Video display device, video display system, and method for controlling video display device

Abstract

An image display apparatus includes: a participation detection sensor that detects whether a user is participating in a session in a virtual space received via the image display apparatus; and a communicator that receives, from an external device, video information of a virtual space in which a self-virtual image corresponding to the user is located, video information of another virtual image corresponding to another user, and voice information of the other user, the voice information of the other user being received from the external device on the basis of the video information of the virtual space and the video information of the other virtual image. An image in which another virtual image is disposed in a virtual space is generated and displayed on a display, whether or not a user is in a temporary leaving state in which the own virtual image is still disposed in the virtual space but does not participate in a session is determined, and when it is determined in the temporary leaving state that the other virtual image is speaking to the own virtual image, the user is displayed on the display. And notifying the user.

Description

Technical Field

[0001] The present invention relates to an image display device, an image display system, and a control method for an image display device. Background Art

[0002] In recent years, various products of information terminals led by PCs have been put on the market and sold. Among them, as a portable image display device, a head-mounted display (hereinafter referred to as "HMD") includes a device that superimposes and displays a three-dimensional image of computer-generated augmented reality (AR: Augmented Reality) in a glasses type, and a device that is called immersive and covers the entire eye and displays a three-dimensional image of virtual reality (VR: Virtual Reality) on the display screen of the HMD.

[0003] One of the applications for HMDs is a remote conferencing system. In a remote conferencing system, all participants are located in different places and enter the same virtual conference room via a network, enabling a remote conference with multiple conference users. At this time, one's own avatar (virtual image) and the avatars of other users are configured in the virtual conference room, and each user uses the HMD to view the image of the virtual conference room.

[0004] Regarding the display of avatars, Patent Document 1 describes the following content: "The display control device obtains information from the device used by the user for an online conference. The display control device judges the status of the user based on the obtained information. The display control device controls the display mode of the avatar corresponding to the user presented to the user in the online conference according to the judged status (extracted from the abstract)."

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-95256 Summary of the Invention

[0008] Technical Problem to be Solved by the Invention

[0009] In the technology described in Patent Document 1, participating users can grasp whether they are sitting or whether the user is working, etc. during the online conference through the display control of the avatar. However, even if a user who has left the seat is called, the user who has left the seat cannot perceive that they have been called. Therefore, there is a concern that the conversation will stagnate when a user temporarily leaves the seat during an online conference.

[0010] An object of the present invention is to provide an image display device, an image display system, and a control method for an image display device that can prevent interference with an online conference due to a user leaving the seat.

[0011] Technical means for solving the problem

[0012] To achieve the above object, the present invention has the structure described in the claimed technical solution. For example, the present invention is an image display device, including: a processor; a display; a participation detection sensor that detects whether a user is participating in a session in a virtual space received via the image display device; and a first communicator that receives, from an external device, image information of a virtual space in which a self virtual image corresponding to the user is located, image information of other virtual images corresponding to other users, and voice information of the other users. Wherein, the processor performs the following controls: based on the image information of the virtual space and the image information of the other virtual images, generate an image in which the other virtual images are arranged in the virtual space and display it on the display; based on the sensor output from the participation detection sensor, determine whether the user is in a temporary absence state where the self virtual image is still arranged in the virtual space but not participating in the session, and when it is determined based on the voice information that the other virtual images are speaking to the self virtual image in the temporary absence state, notify the user.

[0013] Advantages of the invention

[0014] According to the present invention, it is possible to provide an image display device, an image display system, and a control method for an image display device that can prevent interference with an online meeting due to a user's absence. Other objects, structures, and effects will be described in the following embodiments. Description of the drawings

[0015] Figure 1 It is a schematic structural diagram of the image display system of this embodiment.

[0016] Figure 2A It is a structural diagram showing an example of a glasses-type (see-through type) HMD.

[0017] Figure 2B It is a structural diagram showing an example of an immersive (non-see-through type) HMD.

[0018] Figure 3 It is a hardware structural diagram of the HMD.

[0019] Figure 4 It is a flowchart showing the processing flow in the processor of the HMD.

[0020] Figure 5 It is a schematic diagram of looking down on a virtual meeting room from above.

[0021] Figure 6A It is a schematic diagram of looking down on a virtual meeting room from above.

[0022] Figure 6B It is a schematic diagram of the display image of the virtual meeting room in the HMD.

[0023] Figure 7 It is a flowchart showing the processing flow during the temporary absence process of step S409 executed by the processor.

[0024] Figure 8A It is a schematic diagram looking down on the virtual meeting room from the top.

[0025] Figure 8B It is a schematic diagram of the display image of the virtual meeting room in the HMD.

[0026] Figure 9 It is a schematic diagram of the display image in the portable information terminal.

[0027] Figure 10 It is a schematic diagram showing the operation of the portable information terminal that has received the notification instruction.

[0028] Figure 11 It is a schematic diagram showing the operation in the remote control mode.

[0029] Figure 12A It is a schematic diagram looking down on the virtual meeting room.

[0030] Figure 12B It is a schematic diagram of the display image of the virtual meeting room in the HMD.

[0031] Figure 13 It is a schematic diagram showing the operation of the HMD for voice notification.

[0032] Figure 14 It is a schematic diagram of the display image of the virtual meeting room in the HMD.

[0033] Figure 15 It is a schematic diagram of the display image of the virtual meeting room in the HMD.

[0034] Figure 16 It is a functional diagram of the video display program executed by the processor.

[0035] Figure 17 It is a flowchart showing the process of the detection process for detecting whether someone is speaking to their own virtual avatar. Detailed implementation mode

[0036] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in each drawing for explaining one embodiment, the same reference numerals are generally assigned to the same components, and their repeated description is omitted.

[0037] The present invention is expected to improve business efficiency in the case of concurrently implementing an online meeting using the metaverse and operations in the actual environment. Therefore, the present invention is expected to improve technologies for labor-intensive industries that require operation assistance and logistics assistance, and thus is expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (achieve higher levels of economic productivity through diversification of operations, technological upgrading, and innovation, including the focus on the development of high-value-added and labor-intensive industries).

[0038] <First Embodiment>

[0039] Figure 1 It is a schematic structural diagram of the image display system of this embodiment. In addition, the present invention is applicable to the case where there are multiple users, but for ease of understanding of the description, in this embodiment, as Figure 1 shown, it is limited to three users (the first user P1, the second user P2, and the third user P3) for explanation. In addition, hereinafter, the image display device is described by taking a head-mounted display (HMD) as an example.

[0040] Figure 1 In the image display system 100 of, the HMD G1 worn by the first user P1 is connected to the communication network 13 via the wireless router R1. The HMD G2 worn by the second user P2 is connected to the communication network 13 via the wireless router R2. The HMD G3 worn by the third user P3 is connected to the communication network 13 via the wireless router R3. Furthermore, the distribution server 14 and the management server 15 are respectively connected to the communication network 13. The distribution server 14 and the management server 15 are examples of external devices.

[0041] The distribution server 14 distributes various image information such as objects (objects) representing a virtual meeting room pre-registered in the image display system 100 and objects in the virtual meeting room described later, avatar images of each user, and live content data to the HMD G1, HMD G2, and HMD G3 respectively. Each of the HMDs G1, G2, and G3 displays an image on the display screen of each HMD, and outputs sound from the speaker of each HMD.

[0042] The management server 15 manages various types of information obtained via the communication network 13. As the information managed by the management server 15, for example, it manages the information about users described later. The information about users includes the motion information of HMDG1 (the motion information of the first user P1) and the voice information of the first user P1, the motion information of HMDG2 (the motion information of the second user P2) and the voice information of the second user P2, and the motion information of HMDG3 (the motion information of the third user P3) and the voice information of the third user P3. The motion information of each user is represented as vector information obtained based on the sensor output, and the sensor output is detected by the sensors mounted in the HMD worn by the user corresponding to each motion of the user such as head shaking, standing, sitting, and moving. The virtual avatars corresponding to each user are displayed to perform actions corresponding to the vector information corresponding to the actions of each user.

[0043] Furthermore, in the information about users, there are user identification information such as a nickname or a screen name including the name, image information of the virtual avatar, management information for managing multiple users who participate in and watch a meeting in a virtual meeting room, and the like.

[0044] With such a system structure, each user can participate in the meeting in the virtual meeting room while watching an image obtained by superimposing the virtual avatars of others different from the user on the image of the virtual meeting room.

[0045] In addition, Figure 1 Here, HMDG1 is paired with a smartphone as the portable information terminal S1 via short-range wireless communication or LAN communication. Voice data, text message data, and image data can be transmitted and received between HMDG1 and the smartphone. The portable information terminal is not limited to a smartphone and can be an electronic device capable of being paired with the HMD, such as a wearable terminal, a tablet, or a smart speaker. The wearable terminal includes a smart watch, wireless earbuds, and wireless headphones.

[0046] Figure 2A It is a structural diagram showing an example of a glasses-type (see-through type) HMD.

[0047] Figure 2AThe HMDG1 shown includes a left display 202L and a right display 202R with display surfaces in a glasses-shaped housing G10. The left display 202L and the right display 202R are, for example, see-through displays. On the display surfaces of the left display 202L and the right display 202R, a real image of the outside world can pass through, and an image generated by a computer is superimposed and displayed on this real image. A control device 11, a camera 71, a communication device 6, a sensor device 5 including various other sensors, etc. are installed in the housing G10. The control device 11 includes a processor 2, a bus 3, and a memory 4 described later. Furthermore, the control device 11 may include a voice recognition unit 82, a decoder 83, and an encoder 84. Since the HMDG2 and G3 have the same structure as the HMDG1, the description thereof is omitted.

[0048] Figure 2B It is a structural diagram showing an example of an immersive (non-see-through type) HMD.

[0049] Figure 2B A major difference between the immersive HMDG1a shown and the glasses-type HMDG1 is that the right display 202R and the left display 202L are non-see-through types. Therefore, the HMDG1a has a control mode called a perspective mode. A user wearing the HMDG1a cannot directly view the scenery outside. When the user wearing the HMDG1a wants to view the scenery outside, for example, switch to the perspective mode, and display the image captured by the camera 71 on the right display 202R and the left display 202L.

[0050] In addition, in the case of the see-through type HMDG1, an operation state in which the superimposed display of the image generated by the computer is not performed, or in the superimposed display state, for example, the display image is displayed in a corner of the field of view so that the real image outside is easy to view, is called the perspective mode.

[0051] In addition, Figure 2B The processor, the communicator, the sensor device 5 including various other sensors, etc. are not shown in the figure, but they are mounted in the same way as in the Figure 2A case.

[0052] The HMD used in the image display system 100 can be a Figure 2A see-through type HMD, or can be a Figure 2B non-see-through type HMD.

[0053] Figure 3 It is a hardware structure diagram of the HMD of this embodiment. Figure 3 The HMDG1 is taken as an example, but the non-see-through type HMD also has the same structure.

[0054] As Figure 3As shown, the HMDG1 is configured to include a processor 2, a bus 3, a memory 4, a sensor device 5, a communication device 6, an image processing device 7, a sound processing device 8, an operation input device 9, and a line-of-sight detection device 10.

[0055] The processor 2 is a microprocessor unit that controls the entire HMDG1 according to a specified operation program. The processor 2 mainly performs input processing for receiving an input from the user P1 of the HMDG1, information transmission and reception with the distribution server 14 and the management server 15 in response to the input processing, system control for processing the image display system 100 based on this information, and generation and display control of the image to be displayed.

[0056] The bus 3 is a data communication line for transmitting and receiving various commands, data, etc. between the processor 2 and each component module within the HMDG1.

[0057] The memory 4 includes a program storage area 41, a data storage area 42, and an erasable work area 43. Among them, the program storage area 41 stores programs for controlling the operation of the HMDG1, etc., the data storage area 42 stores various data including operation setting values, detection values from the sensor device 5 described later, and images, characters, etc. for display, and the work area 43 is a work area (work area) used when various programs are running, etc.

[0058] The memory 4 includes a volatile memory and a non-volatile memory.

[0059] As the volatile memory, a RAM is provided. The work area 43 is formed in the RAM.

[0060] The non-volatile memory includes, for example, semiconductor element memories such as flash memory and SSD (Solid State Drive), etc., rewritable non-volatile storage media, and ROM. In addition, a disk drive such as an HDD (HardDisc Drive) can also be provided as the non-volatile storage medium. Thus, the stored information can be retained even when the HMDG1 is not externally powered.

[0061] The non-volatile storage medium can store operation programs downloaded from the communication network 13 and various data generated by executing the operation programs, contents such as downloaded dynamic images, static images, and sounds, and data such as dynamic images and static images captured by the camera 71. Each operation program stored in the non-volatile storage medium can be updated and functionally expanded by performing a download process from a program server (not shown).

[0062] The sensor device 5 is a general term for various sensors for detecting the state of the HMDG1. The sensor device 5 is configured to include a GPS (Global Positioning System) sensor 51, a geomagnetic sensor 52, an acceleration sensor 54, a gyro sensor 55, and a wearing detection sensor 53.

[0063] Based on the sensor outputs from the GPS sensor 51, the geomagnetic sensor 52, and the acceleration sensor 54, the position, tilt, orientation, and movement of the HMDG1 are measured.

[0064] In addition, based on the sensor output of the wearing detection sensor 53, it is possible to detect whether the first user P1 is in a state of wearing the HMDG1 or has taken it off.

[0065] The wearing detection sensor 53 is a pressure sensor, a touch sensor, a light sensor, etc. arranged inside the HMD, and any sensor that can detect that the HMDG1 is worn on the head of the first user P1 can be used, and its structure can be arbitrary.

[0066] The wearing detection sensor 53 is an example of a participation detection sensor that detects whether the first user P1 is participating in a session in the virtual space received via the HMDG1.

[0067] In addition, the HMDG1 may further include other sensors such as an illuminance sensor, a proximity sensor, and a biometric authentication sensor.

[0068] The communication device 6 includes a LAN (Local Area Network) communicator 61, a mobile wireless communicator 62, and a short-range wireless communicator 63.

[0069] The LAN communicator 61 is connected to an external device via a communication network 13 such as the Internet through an access point, a wireless router, etc. Therefore, the LAN communicator 61 corresponds to the first communicator. The LAN communicator 61 can be a wireless connector such as Wi-Fi (registered trademark). Thus, the HMDG1 is wirelessly connected to an access point, a wireless router, etc. via the LAN communicator 61.

[0070] The mobile wireless communicator 62 performs telephone communication (calling) and data transmission / reception via the communication network 13 through wireless communication with a base station of a mobile wireless communication network (not shown). Communication with the base station etc. can be performed using the W-CDMA (Wideband Code Division Multiple Access) (registered trademark) method, the GSM (registered trademark) (Global System for Mobile communications) method, the LTE (Long Term Evolution) method, or other communication methods such as 4G and 5G. Since the mobile wireless communicator 62 can also communicate with external devices, it corresponds to the first communicator.

[0071] The LAN communicator 61 and the mobile wireless communicator 62 each include an encoding circuit, a decoding circuit, an antenna, etc.

[0072] The short-range wireless communicator 63 has, for example, a communication function in the Bluetooth (registered trademark) method, but is not particularly limited thereto, and may also be other communication methods such as infrared communication. Since the short-range wireless communicator 63 is wirelessly communicatively connected to the portable information terminal that is the notification target, it corresponds to the second communicator.

[0073] The image processing device 7 includes a camera 71, a right display 202R, and a left display 202L. The camera 71 is a camera unit that converts visible light input from a lens into an electrical signal using electronic devices such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor, thereby inputting visible image data of the surroundings and an object.

[0074] Furthermore, as the camera 71, a TOF (Time Of Flight) sensor may also be provided, which can obtain the distance to the object being photographed as a distance image. Thereby, such detection can be accurately performed using the visible image and the distance image. For example, selection etc. can be performed by an action of reaching out a hand and pinching with the thumb and index finger on a plurality of display images displayed on the HMDG1 (hereinafter referred to as "pointing" action). In addition, the distance to the object being gazed at by the user can be measured using the visible image and the distance image. In addition, the surroundings can also be scanned, and a three-dimensional space map can be generated using the outputs of the above-described sensor device 5 and the image processing device 7.

[0075] The right display 202R and the left display 202L respectively irradiate the display surfaces of the right display 202R and the left display 202L with projected light obtained from a display device such as a liquid crystal panel to display images. The right display 202R and the left display 202L are provided with a graphic RAM (not shown). Display is performed on the display screen based on the image data input to the graphic RAM.

[0076] The sound processing device 8 includes a microphone 81, a sound recognition unit 82, a decoder 83, an encoder 84, a right speaker 85R, and a left speaker 85L.

[0077] The microphone 81 converts the voice of the user or the like into sound data and inputs it.

[0078] The right speaker 85R and the left speaker 85L output sound information or the like required by the user.

[0079] The sound recognition unit 82 analyzes the input sound information and extracts instructions, commands, etc. By using the sound recognition unit 82, the operation of the HMDG1 can be performed by voice commands of operation instructions, and it can also be used to analyze the voice conversation content of each user in a meeting.

[0080] The decoder 83 has functions such as decoding processing (sound synthesis processing) of an encoded sound signal and a three-dimensional sound processing function corresponding to the transmission characteristics of sound, and outputs stereo sound to the user of the HMDG1 through the right speaker 85R and the left speaker 85L.

[0081] The encoder 84 performs encoding processing on the input sound information and generates an encoded sound signal.

[0082] The operation input device 9 is a user interface for inputting operation instructions to the HMDG1. The operation input device 9 can be an operation key formed by arranging button switches or the like, or can be configured as a device for inputting and analyzing gesture actions. Furthermore, the operation input device 9 can also be configured as another portable terminal device connected via a wired communication or wireless communication through the communication device 6.

[0083] The line-of-sight detection device 10 detects the line-of-sight direction of the user wearing the HMDG1. As a line-of-sight detection method for the right line-of-sight detection sensor 1001R and the left line-of-sight detection sensor 1001L, there is a method of irradiating the user's eyes with invisible light (such as infrared rays) and obtaining a pupil image or the like from the captured image through image processing, but the detection method is not particularly limited.

[0084] In addition, the HMDG1 is not limited to communication standards and methods, but for example, it can be connected to the following devices (not shown here) via a short-range wireless communication standard such as Bluetooth (registered trademark).

[0085] · A remote controller that can be held by the user's left and right hands respectively and can perform operation inputs such as pressing buttons

[0086] · A strap with sensors that can detect the movements of hands and feet built in

[0087] Thus, the sensors paired with the strap can be used to detect the movements of the user P1's hands, arms and feet. By comprehensively using the sensor device 5, user movement information such as clapping hands, shaking the head, waving hands, swinging hands up and down, standing, sitting, marching in place, striding, jumping, etc. can be detected.

[0088] In addition, angle information representing the horizontal orientation of the HMDG1 worn by the user P1 calculated based on the sensor outputs obtained from the geomagnetic sensor 52, gyro sensor 55, etc., and gaze direction information calculated based on the sensor output of the gaze detection device 10 can be obtained. The above user movement information and gaze direction information are collectively referred to as behavior information.

[0089] In addition, Figure 1 The hardware structure of the portable information terminal S1 shown, except for the fact that it does not have a gaze detection device 10, etc., forms a single display without distinguishing between the right display 202R and the left display 202L, and a touch panel is laminated on the single display so that operation inputs can be performed, is equivalent to the Figure 3 hardware structure shown, so detailed description is omitted.

[0090] Next, Figure 4 ~ FIG. 6 are used to illustrate in detail the first user P1, the second user P2, and the third user P3 using the image display system 100 to enter a virtual conference room to hold a meeting.

[0091] Figure 4 is a flowchart showing the processing flow in the processor 2 of the HMDG1 participating in the meeting in the virtual conference room constructed by the image display system 100. Here, as an example, the processing flow when the first user P1 ( Figure 1 ) enters the virtual conference room is described in sequence below. In addition, the number of people participating in the meeting, etc. have been registered separately in the image display system.

[0092] Step S401: As the login process of the first user P1 in the image display system 100, the processor 2 of the HMDG1 sends authentication information such as user ID and password to the management server 15. After the login process in step S401 is approved by the management server 15, the control is transferred to the next process.

[0093] Step S402: The first user P1 selects one of the displayed virtual image images as his own virtual image, and the processor 2 accepts this selection input. Furthermore, the processor 2 accepts the input of the name or nickname of the first user P1, etc. The input method is not particularly limited. The displayed text can be input by a pointing action (such as a soft keyboard), or it can be input by voice.

[0094] Step S403: As Figure 5 shown, the processor 2 accepts the first user P1 who participates in the meeting to specify the seat position of his own virtual image A1 in the virtual meeting room through a pointing action or the like.

[0095] Figure 5 is a schematic diagram of looking down at the virtual meeting room from the top surface.

[0096] The processor 2 displays the Figure 5 image of the virtual meeting room 501 shown on the right display 202R and the left display 202L. A meeting table 501t is arranged in the virtual meeting room 501. The seat positions S1, S2, and S3 of the meeting table 501t are respectively the seat positions of the virtual images that can be selected in step S403. In this embodiment, it is assumed that in step S403, the first user P1 selects the seat position S1 of his own virtual image A1 through a pointing action.

[0097] Step S404: The processor 2 starts the meeting process.

[0098] When the second user P2 and the third user P3 respectively operate the HMDG2 and HMDG3 worn by each person to perform the processing of the above steps S401 to S403, all users have completed the entry process into the virtual meeting room 501. In this embodiment, it is assumed that in step S403, the second user P2 selects the seat position S2, and the third user P3 selects the seat position S3 and enters the virtual meeting room 501.

[0099] Step S405: The processor 2 sends the participation status information, behavior information of the user P1, and the voice information of the user P1 speaking obtained from the microphone 81 to the management server 15. The participation status information of the user P1 will be described in detail later.

[0100] The management server 15 broadcasts the participation status information and behavior information received from the HMDG1 to the HMDG2 and HMDG3 worn by all the remaining users who have entered the virtual meeting room 501. The same applies to the HMDG2 and G3 as to the HMDG1.

[0101] Step S406: The processor 2 receives the participation status information, angle information, gaze direction information, and voice information of other users broadcasted.

[0102] Step S407: The processor 2 determines the participation status of the first user P1. When the processor 2 determines that the first user P1 is participating in the meeting (S407: Participating), it performs the display process of the virtual meeting room (S408). When the processor 2 determines that the first user P1 has temporarily left the seat (S407: Temporarily Leaving), it performs the process at the time of temporary leave (S409). The details of the process at the time of temporary leave will be described later. Temporarily leaving refers to the situation where the user maintains the state of logging in to the meeting in the virtual space - that is, the user's virtual avatar remains displayed in the virtual meeting room in the HMD of others, but the user is not participating in the meeting in the virtual meeting room.

[0103] The criteria for determining the participation status can be as described below, for example.

[0104] (1) Whether it satisfies: the output of the wearing detection sensor 53 of the HMD G1 indicates wearing and it is used in the perspective mode.

[0105] (2) Whether it satisfies: the output of the wearing detection sensor 53 of the HMD G1 indicates removal.

[0106] (3) Whether it satisfies: the output of the wearing detection sensor 53 of the HMD G1 indicates removal, and it is used in the speaker output mode where the conversation content of other virtual avatars in the meeting is output to the right speaker 85R and the left speaker 85L.

[0107] In the case of the above (1), the participation status information is determined as the "temporary leave 1 mode". The "temporary leave 1 mode" assumes a situation where the user P1 is operating another personal computer, for example, using a keyboard and mouse to perform tasks different from participating in the meeting, but can return to the meeting in the virtual meeting room at any time according to the meeting situation in the virtual meeting room 501.

[0108] In the case of the above (2), the participation status information is determined as the "temporary leave 2 mode". The "temporary leave 2 mode" assumes, for example, a situation where the user P1 places the removed HMD G1 beside and is operating another personal computer, for example, using a keyboard and mouse to perform tasks different from participating in the meeting, or temporarily removes the HMD G1 for other reasons and moves to a place far from the vicinity of the HMD G1.

[0109] The situation assumed in the case of the above (3) is where the user P1 places the removed HMD G1 beside, listens to the sound in the meeting room, and operates another personal computer, for example, using a keyboard and mouse to perform tasks different from participating in the meeting.

[0110] In the case of the above (1), (2), or (3), the process at the time of temporary leave in step S409 is executed. The details of step S409 will be described later.

[0111] In addition, when it is not any one of the above (1), (2), and (3), the user P1 is participating in a remote meeting, sets the participation status to the "normal mode", and executes step S408.

[0112] In addition, the switching process of turning on or off the perspective mode and the speaker mode can be performed, for example, by button operation of the operation input device 9, or by displaying a menu on the right display 202R or the left display 202L and performing a pointing action, without any particular limitation.

[0113] In addition, as the participation detection sensor, instead of using the wearing detection sensor 53, it is also possible to make a determination based on, for example, the change amount over a certain period of time of the sensor outputs of the geomagnetic sensor 52, the acceleration sensor 54, and the gyro sensor 55. In this case, the wearing detection sensor 53 is not required, and there is an effect of being able to reduce the mounted components and lower the cost.

[0114] Step S408: Based on the participation status information and the gaze direction information of other users received in step S406, the processor 2 performs display control of the image of the virtual meeting room as Figure 6B shown, and performs output control of the sound information.

[0115] Figure 6A is a schematic view of the virtual meeting room 501 viewed from above the top surface.

[0116] Figure 6A In, the virtual avatars A1 of the first user P1, the virtual avatar A2 of the second user P2, and the virtual avatar A3 of the third user P3 are respectively seated facing the center of the meeting table 501t.

[0117] Figure 6B is in Figure 6A a schematic view of the display image of the virtual meeting room 501 displayed on the HMDG1 in the state of the schematic view shown.

[0118] Figure 6B The display image 601 shown is displayed on the right display 202R and the left display 202L mounted in the HMDG1. The display image 601 viewed by the first user P1 is the scenery seen from the user's own virtual avatar A1, so the virtual avatar A1 is not displayed.

[0119] Step S410: When the user P1 does not perform an operation such as exiting the virtual meeting room 501 (S410: continue to participate), the processor 2 returns to step S405 and repeats the processing of steps S405 to S409. When the user P1 performs an operation such as exiting the virtual meeting room 501 (S410: exit), the processor 2 stops displaying the virtual avatar A1 in the virtual meeting room 501 and ends the processing.

[0120] Next, use Figures 7 - 11 to illustrate the details of the processing during temporary absence in step S409. Figure 7 It is a flowchart showing the processing flow during the temporary absence processing of step S409 executed by the processor 2.

[0121] Step S701: When the processor 2 determines that it has detected that someone is speaking to its own virtual avatar A1 (S701: detected), it executes the processing from step S702 onwards. On the other hand, when the processor 2 determines that it has not detected that someone is speaking to its own virtual avatar A1 (S701: not detected), it ends the processing during temporary absence.

[0122] As the detection condition for whether someone is speaking to its own virtual avatar, for example, when it is detected that the virtual avatars A2 of other users, i.e., the second user P2, and A3 of the third user P3 in the virtual meeting room 501 call the name of its own virtual avatar A1 and speak or look in the direction of virtual avatar A1 and speak, and in addition, when the following situations are detected, in order to notify the temporarily absent user P1, the processing from step S702 onwards is executed.

[0123] · Voice analysis (natural language and AI processing, etc. when the name is not called)

[0124] · Behaviors of other virtual avatars (turning around, waving, looking back, line of sight, etc.)

[0125] · Gaze status of multiple other virtual avatars, etc. (using majority voting processing)

[0126] · The above combinations

[0127] Figure 8A is a schematic diagram looking down on the virtual meeting room 501 from the top. In addition, Figure 8B is a schematic diagram of the display image of the virtual meeting room 501 in the HMDG1.

[0128] Based on the management information (participation status information and gaze direction information of virtual avatars A2 and A3) received in step S406, when the virtual avatar A3 faces the direction of the virtual avatar A1 as shown in Figure 8A in the HMDG1, the virtual avatar A3 faces its own virtual avatar A1 as shown in Figure 8B The processor 2 determines that when someone says "Please give your opinion" to itself in this state, it detects that someone is speaking to its own virtual avatar A1 in step S701.

[0129] Step S702: Based on the participation status information determined in step S407, the processor 2 proceeds to step S703 when the participation status information is "temporary absence 1 mode", proceeds to step S705 when it is "temporary absence 2 mode", and proceeds to step S711 when it is "temporary absence 3 mode".

[0130] Step S703: As the processing for "temporary absence 1 mode", the processor 2, similar to step S408, based on the participation status information and gaze direction information of other users received in step S406, Figure 6B as shown, cancels the perspective mode and performs display control of the image of the virtual meeting room and output control of sound information.

[0131] Step S704: Since the video information of the virtual meeting room 501 is displayed in step S703, the participation status information is changed to the "normal mode", and the processing at the time of temporary absence ends.

[0132] Thus, in the "temporary absence 1 mode" where the user P1 is using a keyboard and mouse to operate another personal computer, for example, to perform tasks different from participating in the meeting, the first user P1 can return to the meeting in response to being spoken to by other users in the virtual meeting room 501.

[0133] Step S705: As the processing for "temporary absence 2 mode", the processor 2 determines whether the pairing established with the portable information terminal S1 through short-range wireless communication is in a connected state. If it cannot be connected (S705: not connected), it proceeds to step S706. If it can be connected (S705: connected), it proceeds to step S707.

[0134] Step S706: If the distance between the HMDG1 and the portable information terminal S1 is farther than the distance at which short-range wireless communication can be achieved and the pairing established through short-range wireless communication is in an unconnected state, the processor 2 further performs a connection process with the portable information terminal S1 via a mobile wireless communication network with a longer communication distance. Then, it proceeds to step S707.

[0135] Step S707: The processor 2 asks the first user P1 via the portable information terminal S1 whether to connect in the remote control mode. The remote control mode is a mode in which the user can participate in the meeting in the virtual meeting room via the HMDG1 from the portable information terminal S1.

[0136] In the case of not connecting in the remote control mode (step S707: "no"), it proceeds to step S708. In the case of connecting in the remote control mode (step S707: "yes"), it proceeds to step S709.

[0137] Figure 9It is a schematic diagram of a display image in the portable information terminal S1 that receives an inquiry from the HMDG1 and displays a message asking whether to connect in the remote control mode.

[0138] Figure 9 In the screen of, a "Yes" button for making a remote connection and a "No" button for not making a remote connection are displayed. If the first user P1 touches the "Yes" button, the result is sent to the HMDG1, and the processor 2 transfers the process to step S709. If the user P1 touches the "No" button, the result is sent to the HMDG1, and the processor 2 transfers the process to step 708.

[0139] Step S708: The processor 2 sends a notification instruction to the portable information terminal S1.

[0140] Figure 10 It is a schematic diagram showing the operation of the portable information terminal S1 that has received the notification instruction.

[0141] The portable information terminal S1 that has received the notification instruction outputs a notification sound 901 and a message sound 902 from the right speaker 85R and the left speaker 85L. Furthermore, the portable information terminal S1 can also display a message 903. The combination of the display of the notification sound 901, the message sound 902, and the message 903 is not limited.

[0142] In this way, the temporarily absent user P1 can confirm the above notification result with the portable information terminal S1, return to the placement location of the HMDG1, and participate in the remote meeting again.

[0143] Step S709: When the first user P1 touches the "Yes" button in the screen of Figure 9 a remote connection request instruction signal is sent from the portable information terminal S1 to the HMDG1. The processor 2 of the HMDG1 controls in response to the remote connection request instruction signal and forwards the image and sound of the virtual meeting room 501 to the portable information terminal S1.

[0144] Figure 11 It is a schematic diagram showing the operation in the remote control mode.

[0145] As Figure 11 shown, the portable information terminal S1 displays the image of the virtual meeting room 501 forwarded from the HMDG1 and outputs a sound 1100.

[0146] When the first user P1 gives an answer 1101 to the sound 1100 and touches the virtual avatar A3 as a remote control operation, the touch position information is sent to the HMDG1 as a remote control command, and the sound of the answer 1101 is sent to the HMDG1.

[0147] Step S710: The HMDG1 receives the replied voice and remote control command received from the portable information terminal S1.

[0148] The remote control command is, for example, a command to change the orientation of the face of the self-avatar A1. When the first user P1 touches the avatar A3 displayed on the portable information terminal S1, gaze direction information for turning the avatar A1 to face the direction of the avatar A3 is generated based on the information of the touch position. This gaze direction information is a type of remote control command.

[0149] The HMDG1 sends the received gaze direction information and voice information of the reply 1101 to the management server 15.

[0150] Figure 12A is a schematic diagram looking down on the virtual conference room 501. Figure 12B is a schematic diagram of the display image of the virtual conference room 501 in the HMDG2.

[0151] The HMDG2 receives the gaze direction information and voice information of the HMDG1 via the management server 15. The processor 2 of the HMDG2 performs internal processing for display control based on the received gaze direction information and voice information of the HMDG1. As a result, as shown in Figure 12A the avatar A1 faces the direction of the avatar A3 from a top-down perspective, and as shown in Figure 12B images of the avatar A1 facing the avatar A3 are displayed on the right display 202R and the left display 202L of the HMDG2 from the perspective of the avatar A2. Furthermore, in the HMDG2, an image of the avatar A1 speaking the reply 1101 of the first user P1 is displayed.

[0152] Although not shown in the figure, in the HMDG3 of the third user P3, an image of the avatar A1 facing the user himself / herself within the virtual conference room 501 is displayed.

[0153] In this way, even if the first user P1 temporarily leaves the seat, the first user P1 can rejoin the meeting and send voice information without returning to the place where the HMDG1 is placed. Furthermore, through remote control operation ( Figure 11 ), the user P1 can control the orientation of the self-avatar A1 even when the HMDG1 is removed.

[0154] Step S711: As processing in the "temporary leave 3 mode", the processor 2 outputs voice information through the right speaker 85R and the left speaker 85L to notify the first user P1.

[0155] Figure 13 is a schematic diagram showing the operation of the HMDG1 for voice notification.

[0156] The HMDG1 outputs the sound information of the notification sound 1301 and the message sound 1302 through the right speaker 85R and the left speaker 85L to notify the first user P1. The combination of the notification sound 1301 and the message sound 1302 is not limited.

[0157] According to the first embodiment, for a user who removes the HMDG1 and temporarily leaves the virtual meeting room 501, when someone speaks to the user, by sending a notification from the portable information terminal S1, the user can be informed of the necessity to return to the meeting.

[0158] Furthermore, the portable information terminal S1 can be connected to the HMD, and the meeting in the virtual meeting room can be participated in from the portable information terminal S1. At this time, by sending not only the sound information but also the gaze direction information specifying the direction in which the virtual avatar's face is to be oriented, the orientation of the virtual avatar's face can be changed even without wearing the HMD. Thus, in the images of other users participating in the virtual meeting room, the gaze direction of the virtual avatar can be changed, and the unnaturalness of the orientation of the virtual avatar's face caused by the temporary departure can be eliminated.

[0159] Furthermore, in the above embodiment, since the connection method between the HMD and the portable information terminal is switched according to the temporary departure state, even when leaving the HMD and reaching a place beyond the distance where short-range wireless communication pairing is possible, the virtual meeting room can be participated in again.

[0160] In this way, according to this embodiment, the delay in the progress of the meeting and the unnaturalness of the images caused by the temporary departure in the virtual meeting can be eliminated. When virtual meetings are further popularized and people conduct virtual meetings more frequently and for longer periods, users may have a need to remove the HMD. In such a situation, the conversation process in the virtual meeting of the present invention is not likely to stagnate, and a user-friendly virtual meeting system can be provided.

[0161] <Second Embodiment>

[0162] The second embodiment is an embodiment in which, when a participant in a virtual meeting temporarily leaves, other participants are notified of the temporary departure, based on the first embodiment.

[0163] Figure 14 It is a schematic diagram of the display image of the virtual meeting room 501 in the HMDG2.

[0164] Figure 14 It shows the first-person perspective display image seen from the virtual avatar A2 in the virtual meeting room on the HMDG2 of the second user P2 when the first user P1 temporarily leaves the remote meeting.

[0165] Figure 14In this case, the processor 2 of HMDG2 controls to display the text "Temporarily Absent" near the virtual avatar A1 of the first user P1 who is temporarily absent. Although not shown, the same applies to HMDG3 of the third user P3.

[0166] In this way, all other users P2 and P3 participating in the meeting can understand that although the first user P1 is temporarily absent, they can respond at any time.

[0167] In addition, the text displayed during temporary absence is not limited to "Temporarily Absent", as long as it can convey the situation that the first user P1 can immediately return to the meeting. For example, a specific graphic can be used. In addition, the display color and brightness of the virtual avatar A1 itself can also be changed.

[0168] <Third Embodiment>

[0169] The third embodiment is an embodiment in which, based on the first embodiment, other participants are notified that a participant in the virtual meeting is temporarily absent and participates in a remote mode.

[0170] Figure 15 It is a schematic diagram of the display image of the virtual meeting room 501 in HMDG2.

[0171] Figure 15 It shows the first-person perspective display image seen from the virtual avatar A2 in the virtual meeting room on the HMDG2 of the second user P2 when the first user P1 temporarily leaves the remote meeting.

[0172] Figure 15 In this case, when the first user participates in the meeting in a remote control mode, the processor 2 of HMDG2 displays an image of the virtual avatar A1 holding a smartphone 1501.

[0173] In this way, other users can recognize the situation that the absent first user P1 is returning and participating in the meeting in a remote control mode.

[0174] In addition, it is not limited to the virtual avatar image holding a smartphone, as long as other users can understand the situation that the user is participating nearby in a remote control mode, there is no special limitation.

[0175] <Fourth Embodiment>

[0176] The fourth embodiment relates to an example of a detection process (step S701) for detecting when someone is speaking to their own virtual avatar. Figure 16 It is a functional diagram of the video display program executed in the processor 2.

[0177] Figure 16The image display program 410 shown is stored in the program storage area 41 of the memory 4 of the HMDG1, and its functions are realized by being loaded into the work area 43 and executed. The image display program 410 is also installed in the HMDG2 and G3 worn by other users participating in the virtual meeting respectively, and realizes the same functions as the following-described processing.

[0178] The image display program 410 includes a sound output control unit 411, a sound analysis unit 412, a display control unit 413, another virtual avatar field of view calculation unit 414, a notification processing unit 415, a remote mode processing unit 416, an absence determination unit 417, and a communication control unit 418.

[0179] The sound output control unit 411 outputs the sound information of other users received from the distribution server 14 from the right speaker 85R and the left speaker 85L of the HMDG1.

[0180] The sound analysis unit 412 uses an artificial intelligence engine for analyzing natural language to detect the proper nouns of the user included in the sound information and calling phrases such as "What do you think?" and "Hello" that are commonly used when calling others although they do not contain language for identifying specific persons.

[0181] The display control unit 413 generates an image of another virtual avatar corresponding to the image information, gaze direction information, and behavior information received from the distribution server 14, and displays it on the right display 202R and the left display 202L.

[0182] Another virtual avatar field of view calculation unit 414 determines whether its own virtual avatar is included in the line of sight direction of another virtual avatar based on the image information and gaze direction information received from the distribution server 14. Specifically, it calculates a predetermined horizontal direction angle range and a vertical direction angle range with the position of another virtual avatar in the virtual space as the starting point and the vector representing the gaze direction as the center, and uses it as the field of view of another virtual avatar. When its own virtual avatar is included in this field of view, it is determined that another virtual avatar is facing its own virtual avatar directly.

[0183] The notification processing unit 415 notifies the portable information terminal S1 connected to the HMDG1 that someone is speaking to itself when it detects that someone is speaking to its own virtual avatar.

[0184] The remote mode processing unit 416 performs processing regarding the remote mode between the HMDG1 and the portable information terminal S1 when the remote mode is selected.

[0185] The departure determination unit 417 determines whether the first user P1 has worn or removed the HMD G1, determines that the user is participating in the meeting when wearing it, and determines that the user has temporarily left the meeting when removing it. The departure determination unit 417 can only determine whether the user is participating in the meeting or has temporarily left, or can also determine which of the multiple temporary departure modes the user belongs to, as described in the first embodiment.

[0186] The communication control unit 418 performs communication control between the HMD G1 and the distribution server 14, the management server 15, and the portable information terminal S1.

[0187] Figure 17 It is a flowchart showing the process of a detection process for detecting whether someone is speaking to one's own virtual image.

[0188] The voice analysis unit 412 analyzes whether the voice information of other users received from the distribution server 14 contains one's own proper name or common calling phrase (S1701). If one's own proper name exists (S1702: "Yes"), it is determined that someone is speaking to one's own virtual image (S701: "Yes").

[0189] If one's own proper name does not exist (S1702: "No"), the other virtual image view calculation unit 414 calculates the views of all other virtual images (S1703).

[0190] The other virtual image view calculation unit 414 determines whether one's own virtual image is included in the view of any other virtual image. The position information of one's own virtual image can be obtained by accepting operations using the operation input device 9 or received from the distribution server 14. In the negative case (1704: "No"), it is determined that no one is speaking to one's own virtual image (S701: "No").

[0191] When the other virtual image view calculation unit 414 determines that one's own virtual image is included in the view of any other virtual image (S1704: "Yes"), and the voice analysis unit 412 determines that the voice information contains a common expression for calling (S1705: "Yes"), it is determined that someone is speaking to one's own virtual image (S701: "Yes").

[0192] When the Other Virtual Avatar Field of View Calculation Unit 414 determines that the own virtual avatar is included in the field of view of any other virtual avatar (S1704: "Yes"), and when the Voice Analysis Unit 412 determines that the voice information does not include a normal expression for calling (S1706: "No"), the Other Virtual Avatar Field of View Calculation Unit 414 determines whether the own virtual avatar is included in the fields of view of two or more other virtual avatars (S1706). If affirmative, it is determined that someone is speaking to the own virtual avatar (S701: "Yes"). If negative, it is determined that no one is speaking to the own virtual avatar (S701: "No").

[0193] According to the present embodiment, for determining whether someone is calling oneself, it is possible to detect whether someone is speaking to the own virtual avatar by using each condition detection of whether someone says one's own proper name, whether one appears in the field of view of other virtual avatars and someone says a normal expression for calling, and considering the attention of other members.

[0194] The above-described embodiments do not limit the present invention, and the present invention can be implemented in various different embodiments.

[0195] For example, as the image display device, a notebook personal computer, a tablet computer, a smartphone, or a display or projector that is wired or wirelessly connected to a stationary personal computer can be used instead of the HMD.

[0196] In this case, as the participation detection sensor, a front camera that captures the actual space facing the display of each device is used. If the user is not captured in the image of the front camera, it can be determined that the user is in a temporary absence state. The front camera can be integrally formed with the image display device, or an externally installed camera can be used as the front camera.

[0197] In addition, the present invention is not limited to the above-described embodiments and includes various modification examples. For example, the above-described embodiments are described in detail for easy understanding of the present invention and are not limited to having all the structures described.

[0198] In addition, the above has described an example in which the user sets the system when entering the virtual meeting room or in the meeting room, but the present invention is not limited thereto, and the default value of the system can be used, or a preset value preset by the user can be used.

[0199] In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can also be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0200] In addition, part or all of the above-described structures may be configured by hardware or may be configured to be implemented by a processor executing a program. In addition, the control lines and information lines show the parts necessary for explanation and do not necessarily show all the control lines and information lines of the product. Practically, it can be considered that almost all the structures are interconnected.

[0201] The above-described embodiments include the following technical solutions.

[0202] (Supplementary Note 1)

[0203] An image display device, comprising:

[0204] A processor;

[0205] A display;

[0206] A participation detection sensor that detects whether the user is participating in a session in a virtual space received via the image display device; and

[0207] A first communicator that receives, from an external device, image information of a virtual space in which its own virtual avatar corresponding to the user is located, image information of other virtual avatars corresponding to other users, and voice information of the other users, where

[0208] The processor performs the following controls:

[0209] Based on the image information of the virtual space and the image information of the other virtual avatars, generate an image in which the other virtual avatars are arranged in the virtual space and display it on the display,

[0210] Based on the sensor output from the participation detection sensor, determine whether the user is in a temporary absence state in which the user's own virtual avatar is still arranged in the virtual space but not participating in the session,

[0211] When it is determined based on the voice information that the other virtual avatar is speaking to the user's own virtual avatar in the temporary absence state, notify the user.

[0212] (Supplementary Note 2)

[0213] An image display system is constituted by a distribution server communicatively connected to an image display device, where

[0214] In the distribution server,

[0215] Distribute the image information of the virtual space where the self-avatar corresponding to the first user is located to the image display device operated by the first user, and distribute the image information of other avatars corresponding to the second user and located in the virtual space and the voice information of the second user to the image display device.

[0216] The image display device includes:

[0217] A processor;

[0218] A display;

[0219] A participation detection sensor that detects whether the first user is participating in a session in the virtual space; and

[0220] A communicator that receives the image information of the virtual space, the image information of the other avatars, and the voice information, where

[0221] The processor performs the following controls:

[0222] Based on the image information of the virtual space and the image information of the other avatars, generate an image in which the other avatars are configured in the virtual space and display it on the display.

[0223] Based on the sensor output from the participation detection sensor, determine whether the first user is in a temporary absence state where the self-avatar is still configured in the virtual space but not participating in the session.

[0224] When it is determined based on the voice information that the other avatar is speaking to the self-avatar in the temporary absence state, notify the first user.

[0225] (Supplementary Note 3)

[0226] A control method for an image display device, including:

[0227] The step of receiving, from an external device, the image information of other avatars corresponding to other users and the voice information of the other users configured in the virtual space where the self-avatar corresponding to the user is located;

[0228] The step of generating, based on the image information of the virtual space and the image information of the other avatars, an image in which the other avatars are configured in the virtual space and displaying it on a display;

[0229] A step of determining whether the user is in a temporary absence state in which the user's own virtual avatar is still configured in the virtual space but not participating in the session in the virtual space, based on the sensor output of a participation detection sensor that detects whether the user is participating in the session in the virtual space; and

[0230] A step of notifying the user when it is determined based on the voice information that the other virtual avatar is speaking to the user's own virtual avatar in the temporary absence state.

[0231] Description of Reference Numerals

[0232] 2: Processor, 3: Bus, 4: Memory, 5: Sensor device, 6: Communication device, 7: Image processing device, 8: Sound processing device, 9: Operation input device, 10: Line-of-sight detection device, 11: Control device, 13: Communication network, 14: Distribution server, 15: Management server, 41: Program storage area, 42: Data storage area, 43: Workspace, 51: GPS sensor, 52: Geomagnetic sensor, 53: Wear detection sensor, 54: Acceleration sensor, 55: Gyroscope sensor, 61: LAN communicator, 62: Mobile wireless communicator, 63: Near-field wireless communicator, 71: Camera, 81: Microphone, 82: Voice recognition unit, 83: Decoder, 84: Encoder, 85L: Left speaker, 85R: Right speaker, 100: Image display system, 202L: Left display, 202R: Right display, 410: Image display program, 411: Sound output control unit, 412: Sound analysis unit, 413: Display control unit, 414: Other virtual avatar field-of-view calculation unit, 415: Notification processing unit, 416: Remote mode processing unit, 417: Absence determination unit, 418: Communication control unit, 501: Virtual meeting room, 501t: Conference table, 601: Display image, 708: Step, 901: Notification sound, 902: Message sound, 903: Message, 1001L: Left line-of-sight detection sensor, 1001R: Right line-of-sight detection sensor, 1100: Voice, 1101: Reply, 1301: Notification sound, 1302: Message sound, 1501: Smart phone, A1: User's own virtual avatar, G1: HMD, G1a: HMD, G10: Housing, P1: First user, P2: Second user, P3: Third user, R1: Wireless router, R2: Wireless router, R3: Wireless router, S1: Portable information terminal.

Claims

1. An image display device, characterized in that, comprising: a processor; a display; a participation detection sensor that detects whether the user is participating in a session in a virtual space received via the image display device; and a first communicator that receives, from an external device, image information of a virtual space in which the user's own virtual avatar is located, image information of other virtual avatars corresponding to other users, and voice information of the other users, wherein, the processor performs the following controls: Based on the image information of the virtual space and the image information of the other virtual avatars, generate an image in which the other virtual avatars are arranged in the virtual space and display it on the display, Based on the sensor output from the participation detection sensor, determine whether the user is in a temporary absence state where the user's own virtual avatar is still arranged in the virtual space but not participating in the session, When it is determined based on the voice information that the other virtual avatar is speaking to the user's own virtual avatar in the temporary absence state, notify the user.

2. The image display device according to claim 1, characterized in that: The image display device further includes a camera that captures the outside where the user is located, In the processor, As a control mode of the image display device, it includes a perspective mode of displaying an image of the outside captured by the camera on the display, and a normal mode of displaying the image information on the display, When it is determined that the temporary absence state is in effect while the perspective mode is being used, When it is determined that the other virtual avatar is speaking to the user's own virtual avatar in the temporary absence state, cancel the perspective mode and switch to the normal mode.

3. The image display device according to claim 1, characterized in that: The image display device includes a speaker, In the processor, As a control mode of the image display device, it includes a speaker mode of outputting sound based on the voice information from the speaker, and a normal mode of displaying the image information on the display, When the speaker mode is being used, it is determined that the temporary absence state is in effect.

4. The image display device according to claim 1, characterized in that: It further includes a second communicator that is wirelessly communicatively connected to a portable information terminal, In the processor, When it is determined based on the voice information that the other virtual avatar is speaking to the user's own virtual avatar in the temporary absence state, perform a control to send the notification to the portable information terminal to the second communicator.

5. The image display device according to claim 4, characterized in that: The processor executes a remote control mode, In the remote control mode, forward the image information and the voice information to the portable information terminal, and receive the user's voice and a remote control command specifying the viewing direction of the user's own virtual avatar from the portable information terminal and forward them to the external device.

6. The image display device according to claim 4, characterized in that: The image display device is a head-mounted display, The participating detection sensor is a wearing detection sensor of the head-mounted display.

7. The image display device according to claim 1, wherein: the participating detection sensor is a front camera that captures the actual space facing the display, in the processor, if the user is not captured in the image of the front camera, it is determined that the user is in the temporary absence state.

8. An image display system composed of a distribution server communicatively connected to an image display device, wherein: in the distribution server, image information of the virtual space where the self-virtual image corresponding to the first user is located is distributed to the image display device operated by the first user, and image information of other virtual images corresponding to the second user and located in the virtual space and voice information of the second user are distributed to the image display device, the image display device includes: a processor; a display; a participating detection sensor that detects whether the first user is participating in the session in the virtual space; and a communicator that receives the image information of the virtual space, the image information of the other virtual images, and the voice information, wherein, the processor performs the following controls: generating an image in which the other virtual images are arranged in the virtual space based on the image information of the virtual space and the image information of the other virtual images and displaying it on the display, judging whether the first user is in a temporary absence state in which the self-virtual image is still arranged in the virtual space but does not participate in the session based on the sensor output from the participating detection sensor, when it is judged based on the voice information that the other virtual image is speaking to the self-virtual image in the temporary absence state, notifying the first user.

9. A control method for an image display device, wherein, it includes: a step of receiving from an external device image information of an image in which other virtual images corresponding to other users are arranged in a virtual space where a self-virtual image corresponding to a user is located and voice information of the other users; a step of generating an image in which the other virtual images are arranged in the virtual space based on the image information of the virtual space and the image information of the other virtual images and displaying it on a display; a step of judging whether the user is in a temporary absence state in which the self-virtual image is still arranged in the virtual space but does not participate in the session in the virtual space based on the sensor output from a participating detection sensor that detects whether the user is participating in the session in the virtual space; and a step of notifying the user when it is judged based on the voice information that the other virtual image is speaking to the self-virtual image in the temporary absence state.

Images