Communication device, communication system, display method, and program

Abstract

To solve the following problem: conventionally, kinematic parallax is proposed in which a video image of a partner located in another place is changed in accordance with a point of view (eye positions) in a person on a real space located in one place, to display the partner with a person size in accordance with a distance on a virtual space, on a display in the one place, thereby allowing users to feel as if they are present in the same room with their partners, however, not all the persons on the virtual space hold a conference, or the like, and some virtual users on the virtual space may feel someone's eyes watching and feel discomfort.SOLUTION: A communication apparatus selectively displays an image of a subject located in another place or a character image related to the subject, in accordance with a distance between one place and the other place on a virtual space.SELECTED DRAWING: Figure 34

Description

Technical Field

[0001] The present disclosure relates to a communication device, a communication system, a display method, and a program.

Background Art

[0002] In recent years, telework that utilizes ICT (Information and Communication Technology) and enables a flexible work style that effectively utilizes time and location has been spreading. This makes it possible to achieve work-life balance, improve business efficiency, and enhance customer satisfaction. In addition, in line with the spread of telework, the performance of systems such as video (TV) conferencing via the Internet has been improving (see FIG. 34).

[0003] Also, in communication, the sense of distance between people is very important because it affects ease of conversation and sense of tension (Edward Hall, Hidden Dimension, 2000, Misuzu Shobo). However, in video conferencing, there is a problem that it is difficult to obtain a feeling of being in the same room as the conversation partner (hereinafter referred to as "same-room feeling") like in communication in the real space (or physical space).

[0004] Therefore, in Patent Document 1, a technology is disclosed that realizes a motion parallax that changes the video of the conversation partner at another base according to the viewpoint (eye position) of a person in the real space of one base, and displays the conversation partner on a display on the side of one base with a person size corresponding to the distance in the virtual space, thereby giving each user a sense of being in the same room as the conversation partner.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, not all individuals in the virtual space necessarily participate in meetings or similar activities, and some virtual users in the virtual space may not know who is watching them, leading to a lack of security. [Means for solving the problem]

[0007] The invention according to claim 1 is a communication device for making calls with another communication device at another location, comprising: a receiving means for receiving image data obtained by the other communication device by the other communication device taking a photograph of a subject at the other location and transmitting said image data; and a display control means for switching and displaying an image of a subject at the other location based on the received image data and a character image related to said subject, according to the distance between the local location and the other location in a virtual space. [Effects of the Invention]

[0008] As described above, according to the present invention, the communication device switches and displays the image of the subject at another location and the character image related to that subject depending on the distance between the user's location and the other location in the virtual space. This makes it possible to alleviate as much as possible the problem of virtual users in the virtual space not knowing who is watching them and therefore lacking a sense of security. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a communication system according to an embodiment of the present invention. [Figure 2] This is an electrical hardware configuration diagram of the communication management server. [Figure 3] This is an electrical hardware configuration diagram of a communication device. [Figure 4] This is a functional block diagram of a communication system. [Figure 5] This is a functional block diagram of a communication system. [Figure 6] This is a conceptual diagram of the communication management table. [Figure 7] This is a conceptual diagram of the session management table. [Figure 8] It is a conceptual diagram of a display management table. [Figure 9] It is a conceptual diagram visually showing data managed by a display management table at points on a display. [Figure 10] It is a conceptual diagram of a selection layout information management table. [Figure 11] It is a conceptual diagram of a fixed layout information management table. [Figure 12] It is a conceptual diagram of a position management table showing the latest position of a virtual user. [Figure 13] It is a conceptual diagram showing the arrangement position of virtual participants on world coordinates. [Figure 14] It is a conceptual diagram showing the arrangement position of virtual participants on world coordinates. [Figure 15] It is a sequence diagram showing the participation process in a specific communication session. [Figure 16] It is an example display of a login screen on a communication device. [Figure 17] It is a flowchart showing the position identification process. [Figure 18] It is a diagram showing a selection screen for arrangement position types on a communication device. [Figure 19] It is a diagram showing an example display of a selection screen for selecting a virtual company. [Figure 20] It is a diagram showing an example display of a selection screen for selecting a virtual department. [Figure 21] It is a diagram showing an example display of a selection screen for selecting a virtual room (floor). [Figure 22] It is a sequence diagram showing the transmission process of display information. [Figure 23] It is a sequence diagram showing the process until luminance image data, depth image data, and sound data transmitted from a predetermined communication device (for example, communication device 1a) reach another communication device. [Figure 24] It is a conceptual diagram of luminance image data and depth image data to be transmitted. [Figure 25]It is a sequence diagram showing the process of receiving image data and audio data by the communication device 1d. [Figure 26] It is a flowchart showing the process of image display. [Figure 27] It is a diagram showing an example of the distance from the virtual user D1 and the displayed image. [Figure 28] It is a flowchart showing the calculation process of the position of an object in the world coordinate system. [Figure 29] It is a conceptual diagram of the conversion from the camera coordinate system to the modeling coordinate system and the conversion from the modeling coordinate system to the world coordinate system. [Figure 30] It is a conceptual diagram in which the image of the user D1 and the virtual display are arranged in the modeling coordinate system. [Figure 31] It is an image diagram showing the conversion from the modeling coordinate system to the world coordinate system. [Figure 32] It is a diagram showing the image displayed on the display 118 of the communication device of the user D1. [Figure 33] It is a diagram showing the image displayed on the display 118 of the communication device of the user D1. [Figure 34] It is an example of display on a display in the case of a conventional video conference.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described using the drawings.

[0011] 〔Outline of the Communication System〕 The communication system of the present embodiment is a system for conducting a video call in a virtual space (also referred to as a "virtual area").

[0012] First, the outline of the configuration of the communication system of the present embodiment will be described using FIG. 1. FIG. 1 is a schematic diagram of the configuration of the communication system according to the embodiment of the present invention.

[0013] As shown in Figure 1, the communication system of this embodiment is constructed by a plurality of communication devices 1a, 1b, 1c, 1d and a communication management server 6, and can communicate via a communication network 100 such as the Internet. The connection method of the communication network 100 may be either wireless or wired.

[0014] Communication devices 1a, 1b, 1c, and 1d are located at bases A, B, C, and D, respectively. Hereafter, any of the communication devices 1a, 1b, 1c, and 1d will be referred to as "communication device 1". Communication device 1 can take images and collect sound at its own base, and output images obtained from images taken at other bases and sounds obtained from sound collected at other bases. Communication device 1 will be explained in detail later.

[0015] Figure 1 also shows the case where communication devices 1a, 1b, 1c, and 1d are used by users A1, B1, C1, and D1, respectively. For example, communication device 1a takes pictures and records audio of its own location A and transmits the image data and audio data to communication devices 1b, 1c, and 1d at other locations B, C, and D via the communication network 100 and the communication management server 6. Similarly, communication devices b1, 1c, and 1d take pictures and record audio of their own locations and transmit the image data and audio data to communication devices at other locations via the communication network 100 and the communication management server 6.

[0016] The communication management server 6 is a computer that manages and controls the communication of the communication device 1, and manages the types of image data transmitted and received (general images and special images). Therefore, the communication management server is also a communication control server. The communication management server 6 is composed of one or more computers. If it is composed of multiple computers, the communication management server is also a communication management system (or communication control system).

[0017] Furthermore, communication device 1 also includes digital televisions, smartphones, smartwatches, car navigation systems, medical devices, etc.

[0018] [Hardware configuration] Next, the hardware configuration of the communication device 1 and the communication management server 6 of this embodiment will be described in detail with reference to Figures 2 to 3.

[0019] <Hardware configuration of the communication management server> Next, we will explain the electrical hardware configuration diagram of the communication management server using Figure 2. Figure 2 is the electrical hardware configuration diagram of the communication management server.

[0020] As shown in Figure 2, the communication management server 6, as a computer, is equipped with a CPU 601, ROM 602, RAM 603, HD 604, HDD (Hard Disk Drive) controller 605, display 606, external device connection I / F (Interface) 608, network I / F 609, bus line 610, pointing device 612, and media I / F 614.

[0021] Of these, the CPU 601 controls the operation of the entire communication management server 6. The ROM 602 stores programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as the work area for the CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls the reading or writing of various data to the HD 604 according to the control of the CPU 601. The display 606 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 608 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network I / F 609 is an interface for data communication using the communication network 100. The bus line 610 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 601 shown in Figure 2.

[0022] Furthermore, the pointing device 612 is a type of input means used for selecting and executing various instructions, selecting processing targets, and moving the cursor. The media interface 614 controls the reading or writing (storage) of data to or from the recording medium 613, such as flash memory. The recording medium 613 includes DVDs and Blu-ray Discs.

[0023] <Hardware configuration of communication equipment> Next, we will explain the electrical hardware configuration diagram of the communication device using Figure 3. Figure 3 is an electrical hardware configuration diagram of the communication device.

[0024] Communication device 1 is, for example, a tablet terminal. As shown in Figure 3, communication device 1 includes a CPU 101, ROM 102, RAM 103, EEPROM 104, imaging unit 105, image sensor I / F 106, acceleration / direction sensor 107, media I / F 109, and GPS receiver 111.

[0025] Of these, the CPU 101 controls the operation of the entire communication device 1. The ROM 102 stores programs used to drive the CPU 101, such as the CPU 101 and IPL. The RAM 103 is used as the work area for the CPU 101. The EEPROM 104 reads or writes various data, such as programs for the communication device 1, according to the control of the CPU 101. The imaging unit 105 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) according to the control of the CPU 101 to obtain image data. The image sensor I / F 106 is a circuit that controls the driving of the imaging unit 105. The acceleration / direction sensor 107 is a type of sensor such as an electronic magnetic compass, gyrocompass, or acceleration sensor that detects the Earth's magnetic field. The media I / F 109 controls the reading or writing (storage) of data to or from the recording medium 108, such as flash memory. The GPS receiver 111 receives GPS signals from GPS satellites.

[0026] Furthermore, the communication device 1 includes a long-distance communication circuit 112, an imaging unit 113, an image sensor interface 114, a microphone 115, a speaker 116, an audio input / output interface 117, a display 118, an external device connection interface 119, a short-distance communication circuit 120, an antenna 120a for the short-distance communication circuit 120, and a touch panel 121.

[0027] Of these, the long-distance communication circuit 112 is a circuit that communicates with other devices via the communication network 100. The imaging unit 113 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 101, and its details will be described later. The image sensor interface 114 is a circuit that controls the driving of the imaging unit 113. The microphone 115 is a built-in circuit that converts sound into electrical signals. The speaker 116 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and speech. The sound input / output interface 117 is a circuit that processes the input and output of sound signals between the microphone 115 and the speaker 116 according to the control of the CPU 101. The display 118 is a type of display means such as liquid crystal or organic EL (electroluminescence) that displays images of the subject and various icons. The external device connection interface 119 is an interface for connecting various external devices. The short-distance communication circuit 120 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 121 is a type of input means that allows the user to operate the communication device 1 by pressing the display 118.

[0028] Furthermore, the communication device 1 is equipped with a bus line 110. The bus line 110 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 101 shown in Figure 4.

[0029] The imaging unit 113 will now be described. The imaging unit 105 includes a luminance camera 105a, a depth camera 105b, and a bus line 105e that electrically connects these to the image sensor I / F. Of these, the luminance camera 105a is a device that uses an image sensor such as a CCD or CMOS to convert received light into an electrical signal and obtain image data. For example, an RGB color camera is included.

[0030] The depth camera 105b acquires depth image data by imaging subjects such as people and objects and measuring the distance from the subject to the depth camera 105b using a laser method based on LiDAR (Light Detection And Ranging). The depth camera 105b is a camera that can capture the shape of a subject in three dimensions and detect various movements such as a person's skeleton, and reflect this in real time in subsequent processing. For this reason, the depth camera 105b is composed of a laser transmitter 105b1 and a laser receiver 105b2. The laser transmitter 113b1 emits laser light. The laser light emitted by the laser transmitter 105b1 is reflected by the subject, and the laser receiver 113b2 receives this reflected laser light, converts it into an electrical signal, and extracts the necessary information. Note that the above describes a method of measuring depth using LiDAR, but it is not limited to this. For example, a method of measuring depth using the TOF (Time of Flight) method may also be used. Furthermore, depth may be measured using a stereo measurement method with a standard RGB color camera, and the method of measuring depth is not limited to those described above.

[0031] The imaging unit 113 includes a luminance camera 113a, a depth camera 113b, and a bus line 113e that electrically connects them to the imaging sensor I / F. The depth camera 113b further includes a laser transmitter 113b1 and a laser receiver 113b2. The luminance camera 113a, depth camera 113b, bus line 113e, laser transmitter 113b1, and laser receiver 113b2 have the same configuration as the luminance camera 105a, depth camera 105b, bus line 105e, laser transmitter 105b1, and laser receiver 105b2 in the imaging unit 105, respectively, so their descriptions are omitted.

[0032] Furthermore, recording media such as CD-ROMs on which the above programs are stored, as well as hard drives on which these programs are stored, may be provided domestically or internationally as program products.

[0033] [Functional configuration of the embodiment] Next, the functional configuration of this embodiment will be described using Figures 4 to 14. Figures 4 and 5 are functional block diagrams of the communication system.

[0034] <Functional Configuration of Communication Devices> Here, we will explain the functional configuration of communication device 1a, which is part of communication device 1.

[0035] As shown in Figure 4, the communication device 1a includes a detection unit 30a, a transmitting / receiving unit 31a, a receiving unit 32a, an image / sound processing unit 33a, a display control unit 34a, a judgment unit 35a, a sound collection unit 38a, a storage / reading processing unit 39a, and a detection unit 40. Each of these units is a function or means realized by any of the components shown in Figure 2 operating according to instructions from the CPU 101 that follow a program for the communication device deployed from the EEPROM 104 onto the RAM 103. The communication device 1a also includes a storage unit 3000a constructed from the ROM 102, RAM 103, and EEPROM 104 shown in Figure 2.

[0036] Furthermore, the communication device 1a has a brightness image acquisition unit 36a, a depth image acquisition unit 37a, and a sound collection unit 38a, which are functions realized by operating the imaging unit 105 in accordance with instructions from the CPU 101. The functions realized by the imaging unit 105 are enclosed in dashed lines. (Location management table) The memory unit 3000a contains a location management database 3007a, which is composed of location management tables. Since the location management database 3007a has the same data structure as the location management database 6007 of the communication management server 6, it will be explained later in the section on location management database 6007.

[0037] (Functional configuration of each communication device) Next, we will explain in more detail the functional configuration of the communication device 1a using Figures 3 and 4.

[0038] The detection unit 30a of the communication device 1a is implemented by commands from the CPU 101 to the acceleration / direction sensor 107, and detects the direction of movement and rotation of the communication device 1a.

[0039] The transmitting / receiving unit 31a is realized by the processing of the CPU 101 on the long-distance communication circuit 112, and transmits and receives various data (or information) with other devices (for example, the communication management server 6) via the communication network 100.

[0040] The reception unit 32a is mainly implemented by the processing of the touch panel 121 by the CPU 101 and accepts various selections or inputs from the user.

[0041] The image and sound processing unit 33a is mainly implemented by the CPU 101 and performs various image and sound processing. For example, the image and sound processing unit 33a acts as a generation unit and generates a 3D object of the subject in virtual space based on luminance image and depth image data.

[0042] The display control unit 34a is mainly implemented by the processing of the CPU 101 and displays various images on each of the displays 118.

[0043] The decision unit 35a is mainly implemented by the processing of the CPU 101 and performs various decisions.

[0044] The luminance image acquisition unit 36a is mainly implemented by the processing of the luminance camera 105a by the CPU 101, and acquires luminance image data by photographing the subject. Here, it is referred to as a luminance image, but this term is used for the purpose of contrasting it with depth images, and a luminance image is not an image that contains only luminance information, but also color information.

[0045] The depth image acquisition unit 37a is mainly implemented by the processing of the depth camera 105b by the CPU 101, and acquires depth image data by capturing images of the subject.

[0046] The sound collection unit 38a is mainly implemented by the processing of the microphone 115 by the CPU 101, and collects sound to acquire sound data.

[0047] The memory and read processing unit 39a is mainly implemented by the CPU 101 and stores various data (or information) in the memory unit 3000a and reads various data (or information) from the memory unit 3000a.

[0048] Furthermore, communication devices 1b, 1c, and 1d have the same functions, storage, and database as communication device 1a. The functions of communication devices 1b, 1c, and 1d are essentially implemented by PCs 3b, 3c, and 3d, respectively. In Figures 4 and 5, the only difference is that the transceiver 31a in communication device 1a corresponds to the transceiver 31b in communication device 1b; therefore, a detailed explanation of these is omitted.

[0049] <Functional Configuration of the Communication Management Server> Next, the functional configuration of the communication management server 6 will be explained using Figures 2 and 4.

[0050] As shown in Figure 4, the communication management server 6 has a transmitting / receiving unit 61, a determination unit 65, a creation unit 66, and a storage / reading processing unit 69. Each of these units is a function or means realized by any of the components shown in Figure 2 operating according to instructions from the CPU 601 that follow a communication management (control) program deployed from HD 604 onto RAM 603. The communication management server 6 also has a storage unit 6000 constructed from ROM 602, RAM 603, and HD 604 as shown in Figure 2.

[0051] (Communication management table) Figure 6 is a conceptual diagram showing a communication management table. The storage unit 6000 has a communication management DB 6001 constructed, which is composed of the communication management table shown in Figure 6. In this communication management table, the password, IP address, and user name are associated and managed for each connection ID of all communication devices 1 managed by the communication management server 6. For example, in the communication management table shown in Figure 6, the connection ID of communication device 1a is "01aa", the password is "aaaa", the IP address is "1.2.1.3" as IPv4, and the name is "Hanako R". Note that the IP address is shown in a simplified manner. Also, the IP address may be IPv6 instead of IPv4. Also, instead of an IP address, a domain name (a string of characters that indicates a name used as an email address or homepage address) may be used.

[0052] (Session management table) Figure 7 is a conceptual diagram showing the session management table. The memory unit 6000 has a session management DB 6002 constructed, which is composed of the session management table shown in Figure 7. In this session management table, the session ID and the connection ID of the communication device that participated in the video call of the same communication session are stored and managed in association. Of these, the session ID is an example of session identification information for identifying the communication session that realizes the video call, and is generated for each virtual area such as a virtual conference room. The session ID may be replaced with a meeting ID. The meeting ID is an identifier for managing meetings (conversations between two or more people) on the communication management server.

[0053] (Display management table) Figure 8 is a conceptual diagram showing the display management table. The memory unit 6000 has a display management DB 6004 constructed, which consists of the display management table shown in Figure 8. This display management table stores and manages the vendor ID and product ID of the display's GUID (Globally Unique Identifier), the vertical and horizontal dimensions (position of each corner) and resolution of the display in real space (real coordinate system), and the position of each corner of the display in virtual space (modeling coordinate system), all associated with each other. The real coordinate system refers to a coordinate system based on an arbitrary position in the three-dimensional world, which is the real world.

[0054] Here, using Figure 9, we will visually explain the positions indicated by the data managed in the display management table as points on the display. Figure 9 is a conceptual diagram that visually shows the positions indicated by the data managed in the display management table as points on the display.

[0055] For example, if the GUID of the display is "vid_10ca&pid_0001", it indicates the points representing the positions of each of the four corners of display 118 in physical space (real coordinate system). Similarly, if the GUID of the communication device is "vid_10ca&pid_0001", it indicates the points representing the positions of the four corners of the display in virtual space (modeling coordinate system).

[0056] Furthermore, if the GUID of the display is "vid_11ca&pid_0010", the points indicating the positions of each of the four corners of one of the displays of communication device 1 in physical space (real coordinate system) are shown. Similarly, if the GUID of communication device 1 is "vid_11ca&pid_0010", the points indicating the positions of the four corners of one of the displays of communication device 1 in virtual space (modeling coordinate system) are shown.

[0057] Furthermore, if the display's GUID is "vid_12ca&pid_0100", it indicates the positions of the four corners of a typical laptop's display in physical space (real coordinate system). Similarly, if the display's GUID is "vid_12ca&pid_0100", it indicates the positions of the four corners of a laptop's display in virtual space (modeling coordinate system).

[0058] (Selection and Placement Information Management Table) Figure 10 is a conceptual diagram showing the selection placement information management table. The storage unit 6000 has a selection placement information management DB 6005 constructed, which is composed of the selection placement information management table shown in Figure 10. In this selection placement information management table, selection placement information is configured for each record, and the reference number (NO), company name, department name of this company, and room (floor) of this department are associated and managed. Furthermore, each room is managed in association with a Session ID (Communication ID) and a conversion parameter from Model coordinates to World coordinates. In this case, the conversion parameter indicates the specific location when a particular virtual user enters a virtual room (floor). For example, it is the location where the virtual desk (chair) of a particular virtual user is placed. Note that the Session ID (Communication ID) in the Selection Placement Information Management DB6005 is the same concept as the "Session ID (Communication ID)" item (attribute) in the Session Management DB6002 mentioned above, so its explanation is omitted.

[0059] (Fixed Placement Information Management Table) Figure 11 is a conceptual diagram showing the fixed-location information management table. The storage unit 6000 has a fixed-location information management DB 6006 constructed, which is composed of the fixed-location information management table shown in Figure 11. In this fixed-location information management table, fixed-location information is configured for each record, and the connection ID of the communication device, the session ID (communication ID), and the conversion parameter from Model coordinates to World coordinates are associated and managed. In this case, the conversion parameter indicates a specific location when a designated (arbitrary) virtual user enters a virtual room (floor). For example, the entrance to the virtual room. Note that the session ID (communication ID) in the fixed-location information management DB 6006 is the same concept as the "session ID (communication ID)" item (attribute) in the session management DB 6002 described above, so its explanation is omitted.

[0060] (Location management table) Figure 12 is a conceptual diagram of a location management table showing the latest location of a virtual user. The storage unit 6000 has a location management DB 6007 constructed, which is composed of the location management table shown in Figure 12. While the selected placement information management DB 6005 and fixed placement information management DB 6006 mentioned above are static databases, the location management DB 6007 is a dynamic database. Furthermore, the location management DB 6007 copies the data managed by the selected placement information management DB 6005 and fixed placement information management DB 6006.

[0061] The location management table manages the connection ID of the communication device, the session ID (communication ID), the conversion parameters from the Model coordinate system to the World coordinate system, and the movement vector in association. In this case, the conversion parameters are managed by copying specific conversion parameters from among the conversion parameters managed in the selected placement information management DB6005, or by copying specific conversion parameters from among the conversion parameters managed in the fixed placement information management DB6006. In Figure 19 described later, if the "Select Placement" button b1 is selected, specific conversion parameters managed in the selected placement information management DB6005 are copied to the location management DB6007. Also, in Figure 18 described later, if the "Fixed Placement" button b2 is selected, specific conversion parameters managed in the fixed placement information management DB6006 are copied to the location management DB6007. Furthermore, in these cases, the connection ID and session ID of the logged-in communication device are also copied along with the specific conversion parameters that are copied and managed.

[0062] Furthermore, the location management table also manages a movement vector for each connection ID, i.e., for each virtual user, which shows the distance from the location copied and managed above to the location the virtual user moved to within the virtual space.

[0063] The transformation parameters from the Model coordinate system to the World coordinate system are shown in Figure 32 below, and represent the transformation parameters used when transforming an object from the Model coordinate system in the first virtual space to the World coordinate system in the second virtual space.

[0064] Furthermore, virtual worlds depicted using computer graphics (CG) have a single coordinate system that serves as the basis for the world. This is called the world coordinate system. If the shape of an object is directly given in the world coordinate system, the shape of the object must be redefined every time the object moves within the world coordinate system. Therefore, a coordinate system is defined (called modeling) in which the shape of an object is determined in the 3D image of the computer graphics. This coordinate system is called the modeling coordinate system.

[0065] Here, we will explain world coordinates and modeling coordinates using different terminology. The world coordinate system represents the entire space in which objects are displayed in computer graphics. It is a coordinate system used to indicate the position of objects in space and is used to handle the placement and movement of objects. It can describe the position and orientation of each solid. If all objects were defined from the beginning in one large space called the world coordinate system, it would be difficult to handle the display process. Therefore, modeling coordinates are used to handle the shape and deformation of individual solids placed in space. The origin is taken at one of the vertices or in the vicinity of the solid, and the position and orientation of the points, lines, faces, etc. that make up the solid are described. By placing all display objects, which are freely defined in individual modeling coordinate systems, onto a unified world coordinate system, it is possible to create a graphics space, or a virtual 3D world.

[0066] Here, we will explain the movement of virtual users within the virtual space using Figures 13 and 14. Figures 13 and 14 are conceptual diagrams showing the placement of virtual participants in world coordinates. Figure 13 shows the position of each virtual user when virtual user D1 enters a specific room (floor). Figure 14 shows the state after virtual user D1 has moved in the direction of the arrow after entering the specific room (floor).

[0067] In Figure 13, virtual users A1, B1, C1, and D1 are located at participant positions a11, a12, a13, and a14, respectively. Similarly, virtual display positions d11 to d15 are positioned to correspond to organizer position a11, participant position a12, participant position a13, participant position a14, and bystander position a15, respectively. In this state, the movement vector column of the position management DB 6007 is not managed (or is managed as "0"). In this state, as shown in Figure 14, when virtual user D1 moves to a position directly in front of virtual user A1 at close range, the movement vector column in the position management DB 6007 stores a movement vector indicating the direction and magnitude (distance) of the movement.

[0068] As mentioned above, location management databases 3007a to 3007d have the same data structure as location management database 6007, so their explanation will be omitted.

[0069] (Functional configuration of the communication management server) Next, we will explain in more detail the functional configurations of the communication management server 6 using Figures 2 and 4.

[0070] The transmitting and receiving unit 61 of the communication management server 6 is implemented by the processing of the CPU 601 to the network I / F 609, and transmits and receives various data (or information) with other devices (for example, communication device 1) via the communication network 100.

[0071] The decision unit 65 is mainly implemented by the processing of the CPU 601 and performs various decisions.

[0072] The creation unit 66 is mainly implemented by the CPU 601 and creates image data IDs.

[0073] The memory and read processing unit 69 is mainly implemented by the CPU 601's processing of HD604 and HD605 shown in Figure 2, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0074] [Processing or operation of the embodiment] Next, the processing or operation of this embodiment will be described using Figures 15 to 34.

[0075] <Joining a video call> First, Figures 15 to 23 will be used to explain the process of joining a specific communication session (virtual space). Figure 15 is a sequence diagram showing the process of joining a specific communication session. Figure 16 is an example of the login screen displayed on a communication device.

[0076] First, a user at site A (for example, user A1) enters their connection ID and password on the login screen displayed on the display 118 by the display control unit 34a and presses the "OK" button. The reception unit 32a then receives the connection ID and password, and the transmission / reception unit 31a sends a login request to the communication management server 6 (S21). This login request includes the connection ID, password, and the IP address of the source (communication device 1a). As a result, the transmission / reception unit 61 of the communication management server 6 receives the login request.

[0077] Next, the storage and retrieval processing unit 69 of the communication management server 6 searches the communication management DB 6001 (see Figure 6) using the connection ID and password included in the login request as search keys, and performs authentication by determining whether the same connection ID and password combination is managed (S22). Here, we will continue the explanation assuming that the same connection ID and password combination is managed by the storage and retrieval processing unit 69. In this case, the storage and retrieval processing unit 69 appends the source IP address received in step S21 to the record of the connection ID and password combination used for authentication.

[0078] Next, the storage and retrieval processing unit 69 reads all selected placement information from the selected placement information management DB 6005 and also reads specific fixed placement information, including the connection ID of the communication device 1a, from the fixed placement information management DB 6006 (S23). Then, the transmitting and receiving unit 61 sends a response to the communication device 1a indicating that authentication was successful (S24). This response includes all the selected placement information read above and the fixed placement information related to the communication device 1a. As a result, the transmitting and receiving unit 11a of the communication device 1a receives the response.

[0079] Next, the communication device 1a performs a process to determine the location of the virtual user A1 in the virtual space based on the received selection placement information and fixed placement information (communication device 1a only). Here, the process of step S25 will be explained in detail with reference to Figures 17 to 21. Figure 17 is a flowchart showing the location determination process. Figure 18 is a diagram showing the selection screen for the placement location type in the communication device. Figure 19 is a diagram showing an example of the display of the selection screen for selecting a virtual company. Figure 20 is a diagram showing an example of the display of the selection screen for selecting a virtual department. Figure 21 is a diagram showing an example of the display of the selection screen for selecting a virtual room (floor).

[0080] First, as shown in Figure 18, the display control unit 34a displays a selection screen on the display 118 of the communication device 1a for selecting the type of arrangement based on the selected arrangement information (S180). This selection screen displays a "Selected Arrangement" button b1 and a "Fixed Arrangement" button b2. The "Selected Arrangement" button b1 is for user A1, etc., to press when they select and enter a desired room in one of several virtual spaces. The "Fixed Arrangement" button b2 is for user A1, etc., to press when they enter a location near their predetermined personal seat in the virtual space.

[0081] Next, when user A1 presses either button b1 or b2, the reception unit 32a accepts the selection (S181).

[0082] Next, if the determination unit 35a determines that the "Select Placement" button b1 has been pressed (S182; YES), the display control unit 34a displays a company selection screen on the display 118 for selecting a company to visit, based on the selection placement information, as shown in Figure 19 (S183). This company selection screen displays "Company" buttons b11, b12, and b13, each representing a different company. Note that any number of "Company" buttons may be displayed.

[0083] When user A1 presses the desired button (in this case, the "AA Company" button b11), the reception unit 32a accepts the company selection (S184). Based on the selection arrangement information, the display control unit 34a displays a department selection screen on the display 118 for selecting the department to visit, as shown in Figure 20 (S185). This department selection screen displays "Department" buttons b111, b112, and b113, each representing a different department. Note that any number of "Department" buttons may be displayed.

[0084] When user A1 presses the desired button (in this case, the "AA1 Headquarters" button b111), the reception unit 32a accepts the department selection (S186). Based on this, the display control unit 34a displays a room selection screen on the display 118 for selecting the room to visit, as shown in Figure 21 (S187). This department selection screen displays "room" buttons b1111, b1112, and b1113, each representing a different room (floor). Note that any number of "room" buttons may be displayed.

[0085] When user A1 presses the desired button (in this case, the "AA1 Headquarters" button b111), the reception unit 32a accepts the department selection (S186). As a result, the display control unit 34a displays a room selection screen on the display 118 for selecting the room to visit, as shown in Figure 21 (S187). This department selection screen displays "room" buttons b1111, b1112, and b1113, each representing a different room (floor). Note that any number of "room" buttons may be displayed.

[0086] When user A1 presses the desired button (in this case, the "AA11 room" button b1111), the reception unit 32a accepts the room selection (S188). As a result, location information indicating the selected room is transmitted in step S26, which will be described later.

[0087] On the other hand, if, in step S182, the determination unit 35a determines that the "select placement" button b1 was not pressed (i.e., the "fixed placement" button b2 was pressed) (S182; NO), then, in step S26 described later, fixed position information indicating the position around the user's seat in a specific virtual room is transmitted.

[0088] Next, returning to Figure 15, the transmitting / receiving unit 31a sends a request to join the virtual space to the communication management server 6 (step S26). This request includes location information related to the button selected in step S25. This location information includes a session ID indicating a communication session to enter the selected room, a connection ID of the source (in this case, the communication device 1a), and a conversion parameter from the selection / placement information or fixed placement information received in step S24 that indicates a specific location in the room selected in step S25. As a result, the transmitting / receiving unit 51 of the communication management server 6 receives the request to join.

[0089] Next, the storage / reading processing unit 69 performs the virtual space participation process on the session management DB 6002 (see Figure 7) (S27). Specifically, the storage / reading processing unit 69 manages the session management DB 6002 by associating the connection ID of the communication device 1a received in step S26 with the record of the same session ID as the session ID received in step S26.

[0090] Next, the storage / reading processing unit 69 reads out other connection IDs associated with the same session ID as the connection ID managed in step S27 (S28). Then, the storage / reading processing unit 69 registers the location information including the connection ID of the communication device 1a received in step S26, and the location information including the other connection IDs read out in step S28, as new records in the location management DB 6007 (S29).

[0091] Next, the transmitting / receiving unit 61 transmits a participation request response to the participation request in step S26 to the communication device 1a (S30). This participation request response includes location information, including all connection IDs associated with the same session ID. As a result, the transmitting / receiving unit 31a of the communication device 1a receives the participation request response. Then, the storage / reading processing unit 39a of the communication device 1a updates the contents of the location management DB 3007a by managing all the location information received in step S30 (S31).

[0092] Similarly, the transmitting / receiving unit 61 transmits participation information to the communication device 1b that is participating in the same communication session as communication device 1a (S32). This participation information includes location information, including all connection IDs associated with the same session ID. As a result, the transmitting / receiving unit 31b of communication device 1b receives the participation information. Then, the storage / reading processing unit 39b of communication device 1b updates the contents of the location management DB 3007b by managing all the location information received in step S32 (S33). The same processing as in step S32 is performed for communication devices other than communication device 1b that are participating in the same communication session as communication device 1a, and the same processing as in step S33 is performed for communication devices other than communication device 1b.

[0093] In the above example, after selecting each of the selection buttons b1, b2, etc. shown in Figure 18, the buttons shown in Figures 19 to 21 are selected a total of three times, but this is not limited to this. For example, in Figure 18, the number of selections could be reduced by including buttons that include at least two of the following: company, department, and room.

[0094] <Display information transmission process> Next, we will explain the display information transmission process using Figure 22. Figure 22 is a sequence diagram showing the display information transmission process.

[0095] First, the transmitting / receiving unit 31a of the communication device 1a sends an information addition request to the communication management server 6 (S41). This information addition request includes the connection ID of the source device (communication device 1a) and display information. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the information addition request.

[0096] Next, the storage and read processing unit 69 of the communication management server 6 retrieves the corresponding session ID by searching the session management DB 6002 (see Figure 7) using the connection ID received in step S41 as the search key (S42).

[0097] Furthermore, the memory and read processing unit 69 stores the display information, which is one record of data, in the display management DB 6004 (see Figure 8) (S43). This allows for centralized management of the display information of each communication device 1.

[0098] Furthermore, if there are other communication devices participating in the same communication session, the communication management server 6 needs to share information with these devices by providing them with the image data ID, connection ID, and display information of communication device 1a. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends an information addition notification to other communication devices (for example, communication device 1b) participating in the same communication session as communication device 1a (S48). This information addition notification includes the image data ID, as well as the connection ID and display information of the source communication device 1a, which were received in step S41. As a result, the transmitting / receiving unit 11b of communication device 1b receives the information addition notification.

[0099] Therefore, all communication devices participating in the same communication session's virtual space can share additional information from other communication devices.

[0100] <Transmission of image and sound data> Next, the transmission process of image data and sound data will be explained using Figures 23 and 24. Figure 23 is a sequence diagram showing the process from when luminance image data, depth image data, and sound data transmitted from a predetermined communication device (for example, communication device 1a) reach another communication device. Figure 24 is a conceptual diagram of the transmitted luminance image data and depth image data.

[0101] First, using Figure 23, we will explain the process when communication device 1a transmits image data (in this case, luminance image data and depth image data) and sound data to multiple other communication devices 1b, 1c, and 1d via the communication management server 6.

[0102] First, the communication device 1a starts shooting and sound collection at base A, at which point the luminance image acquisition unit 36a acquires luminance image data, the depth image acquisition unit 37a acquires depth image data, and the sound collection unit 38a acquires sound data (S101). In this case, the communication device 1a performs video recording, but it may also perform still image recording.

[0103] Next, in the communication device 1a, the storage / reading processing unit 39a updates the movement vector of the device (communication device 1a) to the position management DB 3007a (S102). Then, the transmitting / receiving unit 31a of the communication device 1a transmits image data (luminance image data, depth image data) and sound data, as well as the connection ID of the communication device 1a as the source, to the communication management server 6 (S103). In this case, the image data includes an image data ID for identifying the image transmitted from the device, and the movement vector Taa updated in step 102. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data (luminance image data, depth image data, and image data ID, movement vector Taa), sound data, and the connection ID of the communication device 1a as the source.

[0104] Here, the transmitted luminance image data and depth image data will be explained using Figures 3 and 24. In Figure 3, since a shooting unit 105 for taking self-portraits is provided, the image data includes luminance image Lc1 data and depth image Dc1 data. The luminance image Lc1 and depth image Dc1 data are acquired by the shooting unit 105.

[0105] For example, the resolution of the luminance image Lc1 and depth image Dc1 data is 4K each, but as shown in Figure 24, the transmitting communication device 1 compresses the total of 2 data into a single image data and combines them so that the resolution of this single image data becomes 4K. Note that 4K is just an example, and HD, FHD, 8K, etc., may also be used. Next, returning to Figure 23, in the communication management server 6, the storage / reading processing unit 69a updates the movement vector of the communication device 1a to the position management DB 6007 (S104). Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 7) to transmit image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the transmitting communication device 1a to each of the other communication devices 1b to 1d participating in the video call of the same communication session as communication device 1a (S105, S107, S109). As a result, the transmitting and receiving units 31b, 31c, and 31d of each communication device 1b, 1c, and 1d receive image data (luminance image data, depth image data, image data ID, and movement vector Taa), sound data, and the connection ID of the transmitting communication device 1a.

[0106] Then, in each communication device 1b, 1c, and 1d, the memory and read processing units 39b, c, and d update the movement vector of communication device 1a in the position management DB 3007b, c, and d, respectively (S106, S108, S110).

[0107] <Receiving image and audio data> Next, the processing of receiving image data and sound data will be explained using Figures 25 to 33. Figure 25 is a sequence diagram showing the processing of receiving image data and sound data by the communication device 1d.

[0108] As shown in Figure 25, in the communication device 1a, the storage and read processing unit 39a updates the movement vector of the device (communication device 1a) to the position management DB 3007a (S121). The processing in step S121 is the same as step S102 in Figure 23.

[0109] Next, the transmitting / receiving unit 31a transmits image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the communication device 1a as the source to the communication management server 6 (S122). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of the communication device 1a. This step S122 is the same as step S103 in Figure 23. Then, in the communication management server 6, the storage / reading processing unit 69a updates the movement vector of the communication device 1a to the position management DB 6007 (S123). This step S123 is the same as step S104 in Figure 23.

[0110] Next, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 7) to transmit image data (luminance image data, depth image data, image data ID, movement vector Taa), sound data, and the connection ID of the source communication device 1a to other communication devices 1d participating in the video call of the same communication session as communication device 1a (S124). As a result, the transmitting / receiving unit 31d of communication device 1d receives the image data, sound data, and the connection ID of communication device 1a. This process is the same as step S109 in Figure 23.

[0111] Similarly, the transmitting / receiving unit 31d of the communication device 1d receives image data from the communication device 1b (luminance image data, depth image data, image data ID, movement vector Tab), sound data, and the connection ID of the communication device 1b as the source from the communication management server 6 (S125). The transmitting / receiving unit 31d of the communication device 1d also receives image data from the communication device 1c (luminance image data, depth image data, image data ID, movement vector Tac), sound data, and the connection ID of the communication device 1c as the source from the communication management server 6 (S126).

[0112] Next, in the communication device 1d, the image and sound processing unit 33d performs image display and sound output processing based on the data received by the information addition notification received in steps S124 to S125 and step S44 in Figure 22 (S127). In this case, sound output is a general process of converting each sound data (electrical signal) into vibration, so the image display process will be explained in detail using Figure 26.

[0113] <Image display processing> Figure 26 is a flowchart illustrating the image display process.

[0114] First, the memory and read processing unit 39d searches the location management DB 3007d using the connection ID received in steps S124 to S126 of Figure 25 as a search key, and reads all location information (excluding the session ID) including this connection ID (S201). Then, the image and sound processing unit 33d measures the distance between the virtual space location of user D1 of its own device (communication device 1d) read in step S201 and the virtual space locations of users A1, B1, and C1 of the other communication devices 1a, 1b, and 1c, respectively, read in step S201 (S202).

[0115] Here, we will use Figure 27 to explain the relationship between the virtual user D and distance. Figure 27 shows an example of the distance from the virtual user D1 and the displayed image. The displayed image will be explained later. As shown in Figure 27, the distance to a position close to the virtual user D is the first distance, and the distance to a position farther away is the second distance.

[0116] Then, returning to Figure 26, if the distance is less than the first distance (S203), the process proceeds to step S204. Also, if the distance is greater than or equal to the second distance (S203), the process proceeds to step S211. Furthermore, if the distance is greater than or equal to the first distance but less than the second distance (S213), the process proceeds to step S213.

[0117] (Display process when the distance is less than the first distance) Here, we will explain the display process for cases where the distance is less than the first distance.

[0118] ((Process for calculating the position of an object in the world coordinate system)) First, the communication device 1d calculates the position of the object in the world coordinate system (S204). Here, the display process when the image type is a luminance / depth image will be explained in detail using Figures 28 to 31. Figure 28 is a flowchart showing the process of calculating the position of the object in the world coordinate system.

[0119] First, the image and sound processing unit 33d creates a 3D image of an object based on the luminance image data and depth image data of each location, including its own location, in the virtual space (modeling coordinate system) (S221).

[0120] Here, we will use Figure 29 to explain the 3D rendering of an object's image. Figure 29 is a conceptual diagram of the transformation from the camera coordinate system to the modeling coordinate system, and from the modeling coordinate system to the world coordinate system. The camera coordinate system is a coordinate system that defines how the world appears on the projection plane, with the physical camera as the origin, representing the 3D world of the real world. It is necessary to transform the coordinates given in real coordinates into the coordinate system seen in the camera's coordinate system.

[0121] In order to display images of users from multiple real-world locations in a single virtual space such as a virtual conference room, the image and sound processing unit 33d must first define the objects in the real world of each location individually in a Modeling Coordinate System as the first virtual space, thereby creating a 3D model. After that, it is necessary to define the image of each object in a single World Coordinate System.

[0122] Therefore, first, the image and sound processing unit 33d performs 3D modeling by transforming five sets of images, such as the luminance image Lc1 and depth image Dc1, defined in the Camera Coordinate System as shown in Figure 24, into a single first virtual space (modeling coordinate system) using (Equation 1). In Figure 29, user A1 is represented as an object. In this case, the image of user A1 in the modeling coordinate system is represented by a collection of many points, as shown in the enlarged view ev of Figure 29. In this embodiment, since images are captured from five angles by five luminance and depth imaging devices 5a to 5e, the image of user A1 is represented as a collection of many overlapping points. In this way, users A1, B1, C1, and D1 are each defined independently in separate modeling coordinate systems (3D modeled).

[0123]

number

[0124] Vector P is a projection matrix, and vector P-1 is a matrix for transforming from the camera coordinate system to the real coordinate system. Vector M is a matrix for transforming from the real coordinate system to the modeling coordinate system. This output value shows the point (coordinates) and color information in the modeling coordinate system.

[0125] Next, returning to Figure 28, the memory / read processing unit 39d reads the display information of its own device (communication device 1d) stored in the memory unit 3000d (S222). Then, the image / sound processing unit 33d places the virtual display (also called a virtual screen) dd of its own location in the modeling coordinate system based on the "information on the position of each corner of the display in the virtual space (modeling coordinate system)" in the display information (S223). Figure 30 is a conceptual diagram showing the image of user D1 and the virtual display placed in the modeling coordinate system. For example, as shown in Figure 30, the image / sound processing unit 33d further places (defines) the virtual display dd for the image of user D1 at its own location.

[0126] Returning to Figure 28, the memory / reading processing unit 39d reads the "transformation parameters from modeling coordinate system to world coordinate system" and "movement vector" associated with the connection ID of the user (communication device) participating in the video call of the same communication session from the location management DB 3007d (see Figure 12) (S224). Then, the image / sound processing unit 33d performs a modeling transformation of the images of each object and the virtual display of its own location from the modeling coordinate system to the world coordinate system in the virtual space (S225).

[0127] Here, we will again use Figure 29 to explain the transformation from the modeling coordinate system to the world coordinate system.

[0128] As shown in Figure 29, the image and sound processing unit 33d performs a Modeling Transformation on the image of user A1 in the modeling coordinate system, using (Equation 2) to transfer it to a common second virtual space (world coordinate system). Note that the Modeling Transformation is a transformation process for placing display objects defined in the modeling coordinate system onto the world coordinate system, and involves a combination of four transformation processes: scaling, rotation, shearing, and translation of the display objects.

[0129]

number

[0130] Vector W is a matrix used to transform from the modeling coordinate system to the world coordinate system. This output value represents the point (coordinates) and color information in the world coordinate system.

[0131] Figure 31 is an illustrative diagram showing the transformation from the modeling coordinate system to the world coordinate system. As shown in Figure 31, the image and sound processing unit 33d transforms the images A1, B1, C1, and D1 of each modeling coordinate system, as well as the virtual display dd4 of its own location, into a single common world coordinate system. This allows the images of users A1, B1, C1, and D1 at each location to participate in a video call in a common communication session.

[0132] ((Display of images visible to the user)) Next, returning to Figure 26, we will explain the process of displaying an image that user D1 can see on communication device 1.

[0133] As shown in Figure 26, the image and sound processing unit 33a detects skeletal information such as the eyes, nose, and shoulders of user D1 based on the luminance image data acquired by the luminance image acquisition unit 36d and the luminance image data acquired by the depth image acquisition unit 37d (S204).

[0134] Next, the image and sound processing unit 33a uses the center of both eyes in the skeletal information as the viewpoint position and, based on this viewpoint position, creates images of other users such as user A1 that user D1 can see on the virtual display dd in the world coordinate system, as shown in the center of Figure 31 (S205). In other words, the image and sound processing unit 33a converts the displayed objects in the world coordinate system into images that can be displayed on the virtual display dd based on the viewpoint position. This conversion means a conversion from the world coordinate system to the viewpoint coordinate system and is called a field of view conversion. The viewpoint coordinate system is a coordinate system that has the camera position (viewpoint) in the virtual space as its origin and extends in the direction that the user wants to see (the direction in which the subject exists).

[0135] In the world coordinate system, the intersection points of rays (rays) and polygons (virtual triangles, polygons, etc. that make up a 3D model) are calculated using ray tracing, which travels from the viewpoint to pixels on the virtual display dd. These rays are traced backward from the viewpoint, following the light rays traveling from the image direction. This method is called ray tracing and is also called reverse line of sight tracing because it traces the line of sight backward. The intersection point closest to the viewpoint, i.e., the object (called a "polygon" in the field of computer graphics) that first intersects the ray, is found, and the pixels on the virtual display are painted with the color of that object. If no object (polygon) intersects the ray, the pixel is painted with the background color. By performing this process for all pixels on the virtual display dd, the image displayed on the virtual display dd is created.

[0136] In addition to ray tracing, other methods such as the Z-buffer method also exist, and the methods for creating images displayed on the virtual display dd are not limited to those described above.

[0137] Next, the image / sound processing unit 33a converts the image created in the virtual space (world coordinate system) into an image for display on the display 118 of the communication device 1d in real space (S206). Then, the display control unit 34d displays the image converted in step S206 on the display 118 of the communication device 1d in real space (S207). In this case, the space in which the image is displayed on the display 118 is a two-dimensional planar image, and the resolution depends on the performance of the computer, especially on devices such as the display memory installed in the video board (or graphics board), and is therefore called the device coordinate system.

[0138] (Display processing when the distance is greater than or equal to the second distance) Next, we will explain the display process for distances greater than or equal to the second distance.

[0139] The memory / reading processing unit 39d reads pre-stored icon data from the memory unit 3000d. Note that the icon is an example of a character image. Character images also include avatars.

[0140] Next, the image and sound processing unit 33d changes the size of the icon according to the distance between the virtual user D1 and the other virtual user being measured. For example, as shown in Figure 27, when the distance from virtual user D is greater than the second distance, the icon is changed to a large C1 if it is close, then to a medium C2 if it is farther, and then to a small C3 if it is even farther. Although three stages of change are shown here, any number of stages of change may be performed. After the processing in step S212, the process proceeds to the processing in step S205.

[0141] (Display process when the distance is greater than or equal to the first distance but less than the second distance) Next, the display processing for the case where the distance is greater than or equal to the first distance but less than the second distance will be explained. In this case, the image / sound processing unit 33a creates a planar image using only the luminance image Lc that is in front of the virtual user D1 and the other virtual user that was the subject of the measurement, out of a total of 10 images as shown in Figure 24 (S213). For example, as shown in Figure 27, when the distance from virtual user D is greater than or equal to the first distance but less than the second distance, planar image B is created. After the processing in step S213, the process proceeds to the processing in step S205.

[0142] <Example of image display> Next, various display examples from step S207 in Figure 26 will be explained using Figures 32 and 33.

[0143] (Example of display shown to meeting participants (user D1)) ((Initial display)) Figure 32 shows the image displayed on the display 118 of user D1's communication device. Figure 32 is also an example of a display based on the virtual participant placement shown in Figure 13. As shown in Figure 32, in the virtual space (in this case, the world coordinate system), beyond user D1's line of sight, slightly behind and to the right, is a planar image of user A1; further behind and to the left, is a large icon representing the front of user B1; and directly in front, at the very back, is a small icon representing the back of user C1. The names of each user are also displayed. These names are displayed based on name information managed in the communication management DB 6001. This name information, along with the connection ID, is transmitted in advance from the communication management server 6 to each communication device 1. Note that, in principle, only one user exists at each location in the virtual space (world coordinate system).

[0144] In this manner, the display control unit 34d displays a composite image on the display 118 of the communication device 1d, which is a composite image formed by combining various images related to users A1, B1, and C1.

[0145] ((Displayed after moving)) Next, using Figure 33, we will explain the display when user D1 operates the mouse 13 or the like in the real world, causing the virtual user D1 to move closer to the virtual user A1 in the virtual space, as shown in Figure 14. Figure 33 shows the image displayed on the display 118 of user D1's communication device.

[0146] As shown in Figure 33, in the virtual space (in this case, the world coordinate system), in front of user D1's line of sight, a 3D image of user D1 is located near the front, a 2D image of user B1 is located slightly behind and to the left, and an icon representing the back of user C1 is located even further behind and slightly to the left. In this way, the display control unit 34d displays a composite image on the display 118 of the communication device 1d, which is a composite image of the various images related to users A1, B1, and C1.

[0147] [Main effects of the embodiment] As described above, according to this embodiment, the display control unit 34d switches and displays the image of the subject at the other location based on the image data received by the transmitting / receiving unit 31d, and the character image related to this subject, according to the distance between the local location and the other location in the virtual space. This makes it possible to alleviate as much as possible the problem that virtual users C1 etc. in the virtual space do not know who is watching them and therefore do not feel secure.

[0148] Furthermore, the display control unit 34d switches between displaying a planar image of the subject at the other location based on the brightness image data received by the transmitting / receiving unit 31, and a three-dimensional image of the subject at the other location based on the brightness image and depth image data also received, depending on the distance between the user's own location and the other location in the virtual space. This allows user D1 to recognize that a user whose name he has forgotten but whose face he remembers is participating in the virtual space, and to grasp the user's facial expressions. Also, when the distance between user D1 and another user (for example, user A1 in Figure 34) in the virtual space becomes quite close, the image changes from a planar image to a three-dimensional image, allowing user D1 to feel as if he is in the same room as the user with whom he is communicating.

[0149] Furthermore, the display control unit 34d switches and displays multiple sizes of character images according to the distance between the user's own base and other bases in the virtual space. In addition, as the distance between the user's own base and other bases in the virtual space decreases, the display control unit 34d increases the size of the character image. This allows user D1 to know that they are getting closer to the virtual user, even if the user is displayed as an icon. Although user D1 cannot perceive the facial expressions of the virtual user shown as an icon, they can understand who is present in the virtual space from the displayed name.

[0150] 〔supplement〕 The above-mentioned communication management server 6 may simply be called a "server." The communication management server 6 may also be provided in the form of cloud computing. Cloud computing refers to a usage model in which computer resources are provided as a service via a computer network such as the internet, and the provided computer resources are servers. The form of service provided can be SaaS (Software as a Service), PaaS (Platform as a Service), HaaS or IaaS (Hardware / Infrastructure as a Service), but the form of service provision is not limited.

[0151] Furthermore, in the above embodiment, as shown in Figure 29, each communication device 1 first defines the objects in the real space of each location individually in a modeling coordinate system as a first virtual space to create a 3D model, and then converts the image of each object into a single world coordinate system. However, this is not the only method. For example, the communication management server 6 may first define the objects in the real space of each location individually in a modeling coordinate system as a first virtual space to create a 3D model, and then convert the image of each object into a single world coordinate system. In this case, the communication management server 6 transmits the image data converted to the world coordinate system to the communication device 1 at each location. That is, a communication system comprising a first communication device (e.g., communication device 1d), a second communication device (e.g., communication device 1a), and a server (e.g., communication management server 6) includes a providing means that provides a display image generated according to a changeable positional relationship between the position of a subject in the first communication device in a virtual space and the position of a subject in the second communication device in a virtual space, using image data captured by the first communication device and image data captured by the second communication device, and a display control means that displays the display image provided by the providing means.

[0152] Multiple CPUs, as described in the above embodiment, may be provided in a single device.

[0153] Furthermore, each of the functions of the above embodiments can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above.

[0154] Furthermore, in communication between each communication device 1a to 1d and the communication management server 6, other servers, routers, etc., may relay the data. [Explanation of Symbols]

[0155] 1a Communication device (an example of another communication device, an example of the first other communication device) 1b Communication equipment (an example of other communication equipment, a second example of other communication equipment) 1c Communication device (an example of other communication devices) 1d. Communication device (an example of this device) 118 Display (an example of a display means) 6. Communication Management Server 31d Transmitting and receiving unit (Example of transmitting means, example of receiving means, example of a second receiving means) 32d Reception area (an example of reception method) 34d Display control unit (an example of display control means) 61 Transmitting / receiving unit (an example of the first receiving means)

Claims

1. A communication device that makes calls with other communication devices at other locations, The other communication device obtains image data of the other location by photographing the subject at the other location, and the other communication device transmits the image data, and the receiving means receives the image data, A display control means that switches and displays an image of a subject at the other location based on the received image data, and a character image related to that subject, depending on the distance between the local location and the other location in the virtual space. It has, The display control means is If the aforementioned distance is less than the first distance, a stereoscopic image of the subject at the other location is displayed. If the distance is greater than or equal to the first distance and less than the second distance, a planar image of the subject at the other location is displayed. If the distance is greater than or equal to the second distance, the character image is displayed. A communication device characterized by the following features.

2. In the communication device described in claim 1, The image data is the luminance image and depth image data of the other location obtained by the other communication device photographing the subject at the other location in real space. The communication device is characterized in that the display control means switches between displaying a planar image of the subject at the other location based on the received luminance image data and a three-dimensional image of the subject at the other location based on the received luminance image and depth image data, depending on the distance between the local location and the other location in the virtual space.

3. The communication device according to claim 1 or 2, characterized in that the display control means switches and displays multiple sizes of the character images according to the distance between the local site and other sites in the virtual space.

4. The communication device according to claim 3, characterized in that the size of the character image is increased and displayed as the distance between the local base and other bases in the virtual space decreases.

5. The communication device according to any one of claims 1 to 4, characterized in that the character image is an avatar or an icon.

6. A communication system comprising a communication device that communicates with other communication devices at other locations, and a communication management server that can communicate with said communication device, The aforementioned communication management server A first receiving means for receiving image data transmitted by the other communication device, which is image data of the other location obtained by the other communication device taking a photograph of the subject at the other location, A transmission means for transmitting the image data to the communication device, Equipped with, The aforementioned communication device is A second receiving means for receiving the image data from the server, A display control means that switches and displays an image of a subject at the other location based on the received image data, and a character image related to that subject, depending on the distance between the local location and the other location in the virtual space. It has, The display control means is If the aforementioned distance is less than the first distance, a stereoscopic image of the subject at the other location is displayed. If the distance is greater than or equal to the first distance and less than the second distance, a planar image of the subject at the other location is displayed. If the distance is greater than or equal to the second distance, the character image is displayed. A communication system characterized by the following features.

7. A display method performed by a communication device that makes a call with another communication device at another location, The other communication device receives image data obtained by photographing a subject at the other location, and transmits the image data from that other location, which is a receiving step; A display control step that switches between displaying an image of a subject at the other location based on the received image data and a character image related to that subject, depending on the distance between the local location and the other location in the virtual space, Execute, The aforementioned display control step is: If the aforementioned distance is less than the first distance, a stereoscopic image of the subject at the other location is displayed. If the distance is greater than or equal to the first distance and less than the second distance, a planar image of the subject at the other location is displayed. If the distance is greater than or equal to the second distance, the character image is displayed. A method of display characterized by the following features.

8. A program that causes a computer to perform the method described in claim 7.

Images