Video display device, video display system, and method for controlling video display device

Abstract

A video display device comprising a participation detection sensor that detects whether or not a user is participating in a conversation held in a virtual space received via the video display device, and a communication device that receives video information relating to the virtual space in which a self avatar corresponding to the user is present, video information relating to another avatar corresponding to another user, and voice information relating to the other user from an external device. The video display device generates video in which the other avatar is placed in the virtual space is generated on the basis of the video information relating to the virtual space and the video information relating to the other avatar, displays the video on a display, determines whether or not the user is in a temporary absent state in which the user is not participating in the conversation while leaving the self avatar placed in the virtual space, and notifies, when video display device determines that the other avatar is talking to the self avatar during the temporary absent state, the user that the other avatar is talking to the self avatar.

Description

Video display device, video display system, and video display device control method

[0001] The present invention relates to a video display device, a video display system, and a method for controlling a video display device.

[0002] In recent years, various types of information terminals, including PCs, have been available on the market. Among these, head-mounted displays (hereinafter referred to as "HMDs"), which are portable image display devices, include those that display superimposed 3D images of augmented reality (AR) using a computer in the form of glasses, and those that display 3D images of virtual reality (VR) on a display screen of an immersive HMD that covers the entire eye.

[0003] One application for HMDs is a remote conference system. In a remote conference system, multiple conference users can participate in a remote conference through a network, even if they are all in different locations. In this case, each user places their own avatar and the avatars of other users in the virtual conference room, and each user views the image of the virtual conference room on an HMD.

[0004] Regarding the display of avatars, Patent Document 1 states, "The display control device acquires information from the device used by the user for the online conference. The display control device determines the user's situation based on the acquired information. The display control device controls the display mode of the avatar that is presented to the user in the online conference and corresponds to the user, depending on the determined situation (summary excerpt)."

[0005] Japanese Patent Application Laid-Open No. 2022-95256

[0006] The technology described in Patent Literature 1 allows users to grasp the status of participating users, such as whether they are present during an online conference or whether they are working, by controlling the display of their avatars. However, even if a user calls out to a user who is away from their desk, the user will not notice that they have been called. Therefore, there is a concern that if a user temporarily leaves their desk during an online conference, the conversation will be delayed.

[0007] An object of the present invention is to provide a video display device, a video display system, and a method for controlling a video display device that can prevent a user's absence from their seat from interfering with an online conference.

[0008] In order to achieve the above object, the present invention has the configuration described in the claims. For example, the present invention is a video display device comprising: a processor, a display, a participation detection sensor that detects whether a user is participating in a conversation in a virtual space received via the video display device, and a first communication device that receives from an external device video information of the virtual space in which a self-avatar corresponding to the user exists, video information of other avatars corresponding to other users, and audio information of the other users, wherein the processor generates a video of the other avatars placed in the virtual space based on the video information of the virtual space and the video information of the other avatars and displays the video on the display, determines whether the user is in a temporary absence state in which the user places the self-avatar in the virtual space but is not participating in the conversation based on a sensor output from the participation detection sensor, and executes control to notify the user when it is determined based on the audio information that the other avatar is talking to the self-avatar during the temporary absence state.

[0009] According to the present invention, it is possible to provide a video display device, a video display system, and a method for controlling a video display device that can prevent a user's absence from their seat from disrupting an online conference. In addition to the above, the purpose, configuration, and effects will be clarified in the following embodiments.

[0010] 1 is a schematic diagram of a configuration of a video display system according to the present embodiment. FIG. 1 is a schematic diagram showing an example of a glasses-type (transmissive) HMD. FIG. 1 is a schematic diagram showing an example of an immersive (non-transmissive) HMD. FIG. 2 is a hardware configuration diagram of an HMD. FIG. 3 is a flowchart showing a processing procedure in a processor of an HMD. FIG. 4 is a schematic diagram showing a top view of a virtual conference room. FIG. 5 is a schematic diagram showing a top view of a virtual conference room. FIG. 6 is a schematic diagram showing a display image of a virtual conference room in an HMD. FIG. 7 is a flowchart showing a processing procedure in a temporary absence process in step S409 executed by a processor. FIG. 8 is a schematic diagram showing a top view of a virtual conference room. FIG. 9 is a schematic diagram showing a display image of a virtual conference room in an HMD. FIG. 10 is a schematic diagram showing an operation of a mobile information terminal that has received a notification instruction. FIG. 11 is a schematic diagram showing operation in remote control mode. FIG. 12 is a schematic diagram showing a top view of a virtual conference room. FIG. 13 is a schematic diagram showing a display image of a virtual conference room in an HMD. FIG. 14 is a schematic diagram showing an operation in which an HMD is performing audio notification. FIG. 15 is a schematic diagram showing an image displayed of a virtual conference room in an HMD. It is a schematic diagram of a display image of a virtual conference room on an HMD It is a functional diagram of a video display program executed by a processor It is a flowchart showing the flow of a process of detecting speech to a self avatar.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing for explaining an embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0012] The present invention is expected to improve work efficiency when conducting online meetings using the metaverse and work in a real environment simultaneously. Therefore, the present invention is expected to improve technology for labor-intensive industries that require work support and logistics support, and is expected to contribute to Goal 8.2 of the United Nations' Sustainable Development Goals (SDGs) (increasing economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and labor-intensive industries).

[0013] <First Embodiment> Fig. 1 is a schematic diagram of the configuration of a video display system according to this embodiment. Note that although the present invention is applicable to cases where multiple users exist, for ease of understanding, in this embodiment, the explanation will be limited to three users (first user P1, second user P2, and third user P3) as shown in Fig. 1. In addition, the following explanation will be given using a head-mounted display (HMD) as an example of the video display device.

[0014] In the video display system 100 of Fig. 1, the HMDG 1 worn by the first user P1 is connected to the communication network 13 via a wireless router R1. The HMDG 2 worn by the second user P2 is connected to the communication network 13 via a wireless router R2. The HMDG 3 worn by the third user P3 is connected to the communication network 13 via a wireless router R3. Furthermore, a distribution server 14 and a management server 15 are each connected to the communication network 13. The distribution server 14 and the management server 15 are examples of external devices.

[0015] The distribution server 14 distributes various types of video information and live content data, such as virtual conference rooms registered in advance in the video display system 100, objects representing objects in the virtual conference rooms (described later), and avatar images of each user, to HMDG1, HMDG2, and HMDG3. Each of HMDG1, HMDG2, and HMDG3 displays video on the display screen of the HMD and outputs audio from the speaker of the HMD.

[0016] The management server 15 manages multiple pieces of information acquired via the communication network 13. The information managed by the management server 15 includes, for example, information about users, which will be described later. The information about users includes movement information about the HMDG1 (movement information about the first user P1) and voice information about the first user P1, movement information about the HMDG2 (movement information about the second user P2) and voice information about the second user P2, and movement information about the HMDG3 (movement information about the third user P3) and voice information about the third user P3. The movement information for each user is displayed as vector information based on sensor output detected by a sensor mounted on the HMD worn by the user in response to each user's movement, such as shaking their head, standing, sitting, or moving. An avatar corresponding to each user is displayed moving in accordance with the vector information corresponding to each user's movement.

[0017] Further information about users includes user identification information such as nicknames and handle names including names, avatar video information, and management information for managing multiple users who simultaneously participate in and view a conference in a virtual conference room.

[0018] With this system configuration, each user can participate in a conference in a virtual conference room while viewing an image in which the avatar of another person different from the user in question is superimposed on the image of the virtual conference room.

[0019] 1, the HMDG1 is paired with a smartphone serving as a portable information terminal S1 via close proximity wireless communication or LAN communication. Voice data, text message data, and image data can be sent and received between the HMDG1 and the smartphone. The portable information terminal is not limited to a smartphone, and may be any electronic device that can be paired with the HMD, such as a wearable terminal, tablet, or smart speaker. Wearable terminals include smart watches, wireless earphones, and wireless headphones.

[0020] FIG. 2A is a configuration diagram showing an example of an eyeglass-type (transmission type) HMD.

[0021] The HMDG1 shown in FIG. 2A is equipped with a glasses-like housing G10 and a left display 202L and a right display 202R, each including a display surface. The left display 202L and the right display 202R are, for example, transmissive displays. A real image of the outside world is transmitted through the display surfaces of the left display 202L and the right display 202R, and a computer-generated image is superimposed on the real image. The housing G10 is equipped with a control device 11, a camera 71, a communication device 6, a sensor device 5 including various other sensors, and the like. The control device 11 includes a processor 2, a bus 3, and a memory 4, which will be described later. The control device 11 may further include a voice recognition unit 82, a decoder 83, and an encoder 84. HMDGs 2 and 3 have the same configuration as HMDG 1, and therefore will not be described here.

[0022] FIG. 2B is a configuration diagram showing an example of an immersive (non-transmissive) HMD.

[0023] The immersive HMDG 1a shown in FIG. 2B differs significantly from the eyeglass-type HMDG 1 in that the right display 202R and left display 202L are non-transparent. Therefore, the HMDG 1a has a through mode as a control mode. A user wearing the HMDG 1a cannot directly view the outside world. When viewing the outside world while wearing the HMDG 1a, the user switches to the through mode, which displays, for example, an image captured by the camera 71 on the right display 202R and left display 202L.

[0024] In the case of a transparent HMDG1, the operating state in which a computer-generated image is not superimposed, or in which the image is superimposed but the external real image can be easily seen, for example, by displaying the image in the corner of the field of view, is referred to as the through mode.

[0025] Also, in FIG. 2B, a processor, a communication device, a sensor device 5 including various other sensors, and the like are not shown, but are assumed to be installed in the same manner as in FIG. 2A.

[0026] The HMD used in the video display system 100 may be the transmissive HMD shown in FIG. 2A or the non-transmissive HMD shown in FIG. 2B.

[0027] 3 is a diagram showing the hardware configuration of an HMD according to this embodiment. In FIG. 3, an HMD G1 is taken as an example, but a non-transparent HMD also has a similar configuration.

[0028] As shown in FIG. 3, the HMDG 1 includes a processor 2, a bus 3, a memory 4, a sensor device 5, a communication device 6, a video processing device 7, an audio processing device 8, an operation input device 9, and a gaze detection device 10.

[0029] The processor 2 is a microprocessor unit that, in accordance with a predetermined operating program, controls the entire HMDG 1. The processor 2 mainly processes input from the user P1 of the HMDG 1, sends and receives information to and from the distribution server 14 and management server 15 in response to the input, controls the system for processing the video display system 100 based on the information, and generates and controls the display of images to be displayed.

[0030] The bus 3 is a data communication path for transmitting and receiving various commands and data between the processor 2 and each component block within the HMDG 1 .

[0031] The memory 4 is composed of a program storage area 41 that stores programs for controlling the operation of the HMDG 1, a data storage area 42 that stores various data including operation setting values, detection values ​​from the sensor device 5 described below, images for display, characters, etc., and a rewritable work area 43 that includes a work area used for various program operations.

[0032] The memory 4 includes a volatile memory and a non-volatile memory.

[0033] The RAM is provided as a volatile memory, and a work area 43 is formed in the RAM.

[0034] The nonvolatile memory includes, for example, a readable / writable nonvolatile storage medium such as a semiconductor device memory such as a flash memory or an SSD (Solid State Drive), and a ROM. A magnetic disk drive such as an HDD (Hard Disc Drive) may also be provided as a nonvolatile storage medium. This allows the HMDG1 to retain stored information even when power is not being supplied from an external source.

[0035] The nonvolatile storage medium can store operation programs downloaded from the communication network 13, various data created by executing the operation programs, downloaded content such as video, still images, and audio, and data such as video and still images captured using the camera 71. Each operation program stored in the nonvolatile storage medium can be updated and its functions expanded by downloading from a program server (not shown).

[0036] The sensor device 5 is a general term for various sensors used to detect the state of the HMDG 1. The sensor device 5 includes a GPS (Global Positioning System) sensor 51, a geomagnetic sensor 52, an acceleration sensor 54, a gyro sensor 55, and an attachment / detachment detection sensor 53.

[0037] Based on the sensor outputs from the GPS sensor 51, the geomagnetic sensor 52, and the acceleration sensor 54, the position, tilt, direction, and movement of the HMDG 1 are measured.

[0038] Furthermore, based on the sensor output of the attachment / detachment detection sensor 53, it is possible to detect whether the first user P1 is wearing or detaching the HMDG1.

[0039] The attachment / detachment detection sensor 53 may be any type of sensor that can detect that the HMDG1 is attached to the head of the first user P1, such as a pressure sensor, touch sensor, or photo sensor located inside the HMD.

[0040] The attachment / detachment detection sensor 53 is an example of a participation detection sensor that detects whether the first user P1 is participating in a conversation in a virtual space received via the HMDG1.

[0041] The HMDG 1 may further include other sensors such as an illuminance sensor, a proximity sensor, a biometric authentication sensor, etc.

[0042] The communication device 6 includes a LAN (Local Area Network) communication device 61 , a mobile wireless communication device 62 , and a close-proximity wireless communication device 63 .

[0043] The LAN communication device 61 connects to external devices via a communication network 13 such as the Internet via an access point, a wireless router, etc. Therefore, the LAN communication device 61 corresponds to a first communication device. The LAN communication device 61 may be a wireless connection device such as Wi-Fi (registered trademark). As a result, the HMDG 1 wirelessly connects to an access point, a wireless router, etc. via the LAN communication device 61.

[0044] The mobile wireless communication device 62 performs telephone communication (calls) and transmits and receives data via the communication network 13 by wireless communication with a base station of a mobile wireless communication network (not shown). Communication with a base station or the like may be performed by other communication methods such as the W-CDMA (Wideband Code Division Multiple Access) (registered trademark) system, the GSM (Global System for Mobile communications) (registered trademark) system, the LTE (Long Term Evolution) system, or 4G, 5G, etc. The mobile wireless communication device 62 can also communicate with external devices and therefore corresponds to a first communication device.

[0045] The LAN communication device 61 and the mobile radio communication device 62 each include an encoding circuit, a decoding circuit, an antenna, and the like.

[0046] The close proximity wireless communication device 63 has a communication function of the Bluetooth (registered trademark) system, for example, but is not particularly limited to this, and other communication systems such as infrared communication may also be used. The close proximity wireless communication device 63 corresponds to a second communication device for wirelessly connecting to the mobile information terminal that is the notification target.

[0047] The image processing device 7 is composed of a camera 71, a right display 202R, and a left display 202L. The camera 71 is a camera unit that inputs visible image data of the surroundings and an object by converting visible light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0048] The camera 71 may further include a TOF (Time of Flight) sensor capable of capturing the distance to an object being imaged as a distance image. This allows for accurate detection using the visible image and distance image, such as selecting an object by pointing at multiple images displayed on the HMDG 1 and pinching them with the thumb and index finger (hereinafter referred to as "pointing"). Using these visible and distance images, the distance to an object the user is gazing at can also be measured. The surroundings can also be scanned, and a three-dimensional space map can be generated using the outputs of the sensor device 5 and the video processing device 7.

[0049] The right display 202R and the left display 202L each display an image by irradiating the display surface of the right display 202R and the left display 202L with projection light obtained from a display device such as a liquid crystal panel. The right display 202R and the left display 202L each include a video RAM (not shown). Images are displayed on the display screen based on image data input to the video RAM.

[0050] The voice processing device 8 includes a microphone 81, a voice recognition unit 82, a decoder 83, an encoder 84, a right speaker 85R, and a left speaker 85L.

[0051] The microphone 81 converts the user's voice or the like into audio data and inputs it.

[0052] The right speaker 85R and the left speaker 85L output audio information and the like required by the user.

[0053] The voice recognition unit 82 analyzes the input voice information and extracts instruction commands, etc. The voice recognition unit 82 may be used to operate the HMDG 1 using voice commands for operation instructions, or may be used to analyze the content of the voice conversations of each user in a conference.

[0054] The decoder 83 has functions such as decoding the encoded audio signal (audio synthesis processing) and a three-dimensional audio processing function according to the transmission characteristics of the audio, and outputs three-dimensional audio to the user of the HMDG 1 from the right speaker 85R and the left speaker 85L.

[0055] The encoder 84 performs encoding processing on the input audio information to generate an encoded audio signal.

[0056] The operation input device 9 is a user interface for inputting operation instructions to the HMDG 1. The operation input device 9 may be an operation key with an array of button switches or the like, or may be configured as a device for inputting and analyzing gesture movements. Furthermore, the operation input device 9 may be configured as a separate mobile terminal device connected via the communication device 6 by wired or wireless communication.

[0057] The gaze detection device 10 detects the gaze direction of a user wearing the HMDG 1. The right gaze detection sensor 1001R and the left gaze detection sensor 1001L may detect the gaze by irradiating the user's eyes with invisible light (such as infrared light) and obtaining a pupil image from the captured image through image processing, but the detection method is not particularly limited.

[0058] Furthermore, the HMDG1 is not limited by the communication standard or method, but may be connected via the Bluetooth (registered trademark) close-proximity wireless communication standard to the following devices (not shown): a remote control that the user can operate by pressing buttons held in each hand; and a band with built-in sensors that can detect hand and foot movements.

[0059] Therefore, the sensors paired with these bands can detect the movements of the hands, arms, and legs of the user P1. By utilizing these sensor devices 5 in a comprehensive manner, it is possible to detect user motion information such as clapping, shaking head or hands, raising and lowering hands, standing, sitting, stamping, stepping, jumping, etc.

[0060] It is also possible to obtain angle information indicating the horizontal orientation of the HMDG 1 worn by the user P1, calculated based on sensor outputs obtained from the geomagnetic sensor 52, the gyro sensor 55, etc., and gaze direction information calculated from the sensor outputs of the gaze detection device 10. The above user operation information and gaze direction information are collectively referred to as behavior information.

[0061] In addition, the hardware configuration of the mobile information terminal S1 shown in Figure 1 is the same as the hardware configuration shown in Figure 3, except that it does not have an eye gaze detection device 10, etc., that it forms a single display without distinguishing between the right display 202R and the left display 202L, and that a touch panel is stacked on top of the single display to enable operation input, so a detailed explanation will be omitted.

[0062] Next, a detailed description will be given of a conference in which a first user P1, a second user P2, and a third user P3 enter a virtual conference room using the video display system 100, with reference to FIGS.

[0063] 4 is a flowchart showing the processing steps in the processor 2 of the HMDG 1 participating in a conference in a virtual conference room using the video display system 100. As an example, the processing steps when a first user P1 (FIG. 1) enters the virtual conference room will be explained below. Note that the number of participants in the conference and other information are assumed to be registered separately in the video display system.

[0064] Step S401: As a login process for the first user P1 to the video display system 100, the processor 2 of the HMDG 1 transmits authentication information such as a user ID and password to the management server 15. When the login process in step S401 is approved by the management server 15, control is passed to the next process.

[0065] Step S402: The first user P1 selects one of the displayed avatar images as his / her avatar, and the processor 2 accepts the selection input. The processor 2 then accepts input of the first user P1's name, nickname, etc. There are no particular restrictions on the input method, but the first user P1 may input the displayed characters (such as a software keyboard) by pointing, or may input the characters by voice.

[0066] Step S403: As shown in FIG. 5, the processor 2 receives, by a pointing operation or the like, the seat position in the virtual conference room of the self-avatar A1 of the first user P1 who will be participating in the conference.

[0067] FIG. 5 is a schematic diagram of a virtual conference room viewed from above.

[0068] The processor 2 displays an image of the virtual conference room 501 shown in Fig. 5 on the right display 202R and the left display 202L. A conference table 501t is placed in the virtual conference room 501. Seat positions S1, S2, and S3 on the conference table 501t are the seat positions of avatars that can be selected in step S403. In this embodiment, it is assumed that the first user P1 selects the seat position S1 of his / her own avatar A1 by pointing in step S403.

[0069] Step S404: The processor 2 starts the conference process.

[0070] When the second user P2 and the third user P3 perform the above steps S401 to S403 on the HMDG2 and HMDG3 worn by them, the process of entering all users into the virtual conference room 501 is completed. In this embodiment, in step S403, the second user P2 selects seat position S2, and the 33rd user P3 selects seat position S3, and they enter the virtual conference room 501.

[0071] Step S405: The processor 2 transmits the participation status information, behavior information, and speech information of the user P1 obtained from the microphone 81 to the management server 15. The participation status information of the user P1 will be described in detail later.

[0072] The management server 15 broadcasts the participation status information and behavior information received from the HMDG1 to the HMDG2 and HMDG3 worn by all remaining users in the virtual conference room 501. The same applies to the HMDG2 and HMDG3 as to the HMDG1.

[0073] Step S406: The processor 2 receives the broadcasted participation status information, angle information, gaze direction information, and audio information of other users.

[0074] Step S407: Processor 2 determines the participation status of first user P1. Processor 2 determines that first user P1 is participating (S407: Participate) and executes virtual conference room display processing (S408). Processor 2 determines that first user P1 is temporarily absent (S407: Temporary absence) and executes temporary absence processing (S409). Details of temporary absence processing will be described later. Temporary absence refers to a state in which a user remains logged in to the conference in the virtual space, that is, their own avatar remains displayed in the virtual conference room on the HMD of others, but the user is not participating in the conference in the virtual conference room.

[0075] The following may be used as criteria for determining the participation status, for example: (1) Whether the output of the attachment / detachment detection sensor 53 of the HMDG1 indicates that the HMDG1 is attached and is being used in the through mode; (2) Whether the output of the attachment / detachment detection sensor 53 of the HMDG1 indicates that the HMDG1 is detached; (3) Whether the output of the attachment / detachment detection sensor 53 of the HMDG1 indicates that the HMDG1 is detached and is being used in the speaker output mode in which the content of the conversation of other avatars in the conference is output to the right speaker 85R and left speaker 85L.

[0076] In the case of (1) above, the participation status information is determined to be "temporary absence 1 mode." "Temporary absence 1 mode" is assumed to be a situation in which user P1 is using a keyboard and mouse on another personal computer or the like to perform a task other than participating in the conference, and can return to the conference in the virtual conference room at any time depending on the conference status in virtual conference room 501.

[0077] In the case of (2) above, the participation status information is determined to be "temporary absence mode 2." "Temporary absence mode 2" is assumed to occur when user P1 places the detached HMDG 1 nearby and uses the keyboard and mouse of another personal computer or the like to perform work other than participating in the conference, or when user P1 detaches the HMDG 1 temporarily for other business and moves to a location away from the vicinity of the HMDG 1.

[0078] In the case of (3) above, it is assumed that user P1 places the detached HMDG1 next to him, listens to the audio during the conference, and uses the keyboard and mouse on another personal computer or the like to perform work other than participating in the conference.

[0079] In the case of (1), (2), or (3), the temporary absence process is executed in step S409, which will be described in detail later.

[0080] If the above conditions (1), (2), and (3) are not met, the user P1 is participating in a remote conference, and the participation status is set to "normal mode," and step S408 is executed.

[0081] The process of switching the through mode or speaker mode on or off may be performed, for example, by operating a button on the operation input device 9, or by pointing at a menu displayed on the right display 202R or left display 202L; there are no particular restrictions.

[0082] Furthermore, the joining detection sensor may be determined based on the amount of change over a certain period of time in the sensor output of the geomagnetic sensor 52, acceleration sensor 54, or gyro sensor 55, instead of the attachment / detachment detection sensor 53. In this case, the attachment / detachment detection sensor 53 is not required, which has the effect of reducing the number of mounted parts and cutting costs.

[0083] Step S408: The processor 2 controls the display of the image in the virtual conference room as shown in FIG. 6B and controls the output of audio information based on the participation status information and gaze direction information of other users received in step S406.

[0084] FIG. 6A is a schematic diagram of the virtual conference room 501 viewed from above.

[0085] In FIG. 6A, an avatar A1 of a first user P1, an avatar A2 of a second user P2, and an avatar A3 of a third user P3 are seated facing the center of a conference table 501t.

[0086] FIG. 6B is a schematic diagram of the display image of the virtual conference room 501 displayed on the HMDG 1 in the state of the schematic diagram shown in FIG. 6A.

[0087] 6B is displayed on the right display 202R and the left display 202L mounted on the HMDG 1. The display image 601 viewed by the first user P1 is a view from the perspective of the user's own avatar A1, and therefore the avatar A1 is not displayed.

[0088] Step S410: If the user P1 has not performed an operation such as logging out to exit the virtual conference room 501 (S410: Continue participation), the processor 2 returns to step S405 and repeats the processing of steps S405 to S409. If the user P1 has performed an operation such as logging out to exit the virtual conference room 501 (S410: Exit), the processor 2 stops displaying the avatar A1 from the virtual conference room 501 and ends the processing.

[0089] Next, the temporary absence process in step S409 will be described in detail with reference to Figures 7 to 11. Figure 7 is a flowchart showing the processing procedure in the temporary absence process in step S409 executed by the processor 2.

[0090] Step S701: If the processor 2 determines that a conversation with the user's avatar A1 has been detected (S701: YES), the processor 2 executes the processes from step S702 onward. On the other hand, if the processor 2 determines that a conversation with the user's avatar A1 has not been detected (S701: NO), the processor 2 ends the temporary absence process.

[0091] The conditions for detecting whether or not someone is speaking to the user's own avatar include, for example, other users in the virtual conference room 501, i.e., avatar A2 of the second user P2 or avatar A3 of the third user P3, calling the user's own avatar A1 by name and speaking to them, or gazing at the avatar A1 and speaking to him, as well as the above. If the following is detected, the process from step S702 onwards is executed to notify user P1, who is temporarily away from his seat.

[0092] Fig. 8A is a schematic diagram of a top view of the virtual conference room 501. Fig. 8B is a schematic diagram of the display image of the virtual conference room 501 on the HMDG 1.

[0093] Based on the management information (participation status information and gaze direction information of avatar A2 and avatar A3) received in step S406, processor 2 determines that when avatar A3 is facing avatar A1 as shown in Fig. 8A, avatar A3 is facing the user's own avatar A1 as shown in Fig. 8B on the HMDG 1. If processor 2 determines that the user is speaking "Please tell us your opinion" in this state, it detects that the user's own avatar A1 was spoken to in step S701.

[0094] Step S702: Based on the participation status information determined in step S407, if the participation status information is "temporary absence 1 mode", processor 2 proceeds to step S703; if the participation status information is "temporary absence 2 mode", processor 2 proceeds to step S705; and if the participation status information is "temporary absence 3 mode", processor 2 proceeds to step S711.

[0095] Step S703: As part of the processing for the "temporary absence mode 1," processor 2 cancels the through mode and controls the display of the image in the virtual conference room and the output of the audio information, based on the participation status information and gaze direction information of other users received in step S406, as in step S408, as shown in FIG. 6B.

[0096] Step S704: Since the video information of the virtual conference room 501 is displayed in step S703, the participation status information is set to "normal mode" and the temporary absence process is terminated.

[0097] Therefore, in "temporary absence 1 mode" in which user P1 is using another personal computer or the like with a keyboard and mouse to perform work other than participating in the conference, the first user P1 can return to the conference in response to a call from another user in the conference in the virtual conference room 501.

[0098] In step S705, as processing for the "temporary absence mode 2," the processor 2 determines whether the pairing with the mobile information terminal S1 via close proximity wireless communication is in a connected state. If connection is not possible (S705: not connected), the process proceeds to step S706. If connection is possible (S705: connected), the process proceeds to step S707.

[0099] Step S706: If the distance between the HMDG 1 and the portable information terminal S1 is greater than the short-range wireless communication range and pairing by short-range wireless communication is not yet established, the processor 2 performs connection processing with the portable information terminal S1 via a mobile wireless communication network with a longer communication range. Then, the process proceeds to step S707.

[0100] Step S707: The processor 2 sends a message to the first user P1 via the mobile information terminal S1 to inquire whether to connect in remote control mode. The remote control mode is a mode in which the first user P1 can participate in a conference in the virtual conference room from the mobile information terminal S1 via the HMDG1.

[0101] If the connection is not to be made in the remote control mode (step S707: No), the process proceeds to step S708. If the connection is to be made in the remote control mode (step S707: Yes), the process proceeds to step S709.

[0102] FIG. 9 is a schematic diagram of a display image on the portable information terminal S1 that displays a message inquiring whether or not connection in remote control mode is required in response to an inquiry from the HMDG1.

[0103] 9 displays a "Yes" button to establish a remote connection and a "No" button to not establish a remote connection. If the first user P1 taps the "Yes" button, the result is sent to the HMDG1, and the processor 2 proceeds to step S709. If the user P1 taps the "No" button, the result is sent to the HMDG1, and the processor 2 proceeds to step S708.

[0104] Step S708: The processor 2 transmits a notification instruction to the portable information terminal S1.

[0105] FIG. 10 is a schematic diagram showing the operation of the portable information terminal S1 when it receives a notification instruction.

[0106] Upon receiving the notification instruction, the portable information terminal S1 outputs a notification sound 901 and a message voice 902 from the right speaker 85R and the left speaker 85L. The portable information terminal S1 may also display a message 903. The combination of the notification sound 901, the message voice 902, and the message 903 is not limited.

[0107] In this way, user P1 who is temporarily away from his / her seat can check the notification result on his / her mobile information terminal S1, and can return to the location where HMDG1 is located and rejoin the remote conference.

[0108] 9, a remote connection request instruction signal is sent from the portable information terminal S1 to the HMDG 1. In response to the remote connection request instruction signal, the processor 2 of the HMDG 1 controls the transfer of images and audio from the virtual conference room 501 to the portable information terminal S1.

[0109] FIG. 11 is a schematic diagram showing the operation in the remote control mode.

[0110] As shown in FIG. 11, the portable information terminal S1 displays an image of the virtual conference room 501 transferred and controlled by the HMDG1, and outputs a sound 1100.

[0111] When the first user P1 makes a reply 1101 to the voice 1100, the first user P1 taps the avatar A3 as a remote control operation, and the tap position information is transmitted to the HMDG1 as a remote control command, and the voice of the reply 1101 is transmitted to the HMDG1.

[0112] Step S710: The HMDG1 receives the reply voice and remote control command from the portable information terminal S1.

[0113] The remote control command is, for example, a command to change the facial orientation of the first user P1's avatar A1. When the first user P1 taps on the avatar A3 displayed on the mobile information terminal S1, gaze direction information for turning the avatar A1 toward the avatar A3 is generated from information on the tap position. The gaze direction information is a type of remote control command.

[0114] The HMDG 1 transmits the gaze direction information and the voice information of the reply 1101 received from the portable information terminal S 1 to the management server 15 .

[0115] Fig. 12A is a schematic diagram of a bird's-eye view of the virtual conference room 501. Fig. 12B is a schematic diagram of the display image of the virtual conference room 501 on the HMDG 2.

[0116] The HMDG2 receives the gaze direction information and voice information of the HMDG1 via the management server 15. The processor 2 of the HMDG2 performs internal processing for display control based on the received gaze direction information and voice information of the HMDG1, and as a result, from the bird's-eye view, avatar A1 faces the direction of avatar A3 as shown in Fig. 12A, and from the view point of avatar A2, an image of avatar A1 facing avatar A3 is displayed on the right display 202R and left display 202L of the HMDG2 as shown in Fig. 12B. Furthermore, on the HMDG2, an image of avatar A1 speaking the reply 1101 of first user P1 is displayed.

[0117] Although not shown in the figure, the HMDG3 of the third user P3 displays an image of the avatar A1 facing the user in the virtual conference room 501.

[0118] In this way, even if the first user P1 is temporarily away from his / her seat, he / she can rejoin the conference and send audio information without returning to the location of the HMDG 1. Furthermore, by operating the remote control (FIG. 11), the orientation of the first user P1's own avatar A1 can be controlled even if the first user P1 has removed the HMDG 1.

[0119] Step S711: As processing for the "temporary absence mode 3", the processor 2 outputs audio information from the right speaker 85R and the left speaker 85L to notify the first user P1.

[0120] FIG. 13 is a schematic diagram showing the operation of the HMDG 1 performing audio notification.

[0121] The HMDG1 outputs audio information, such as a notification sound 1301 and a message voice 1302, from the right speaker 85R and the left speaker 85L to notify the first user P1. The combination of the notification sound 1301 and the message voice 1302 is not limited.

[0122] According to the first embodiment, if a user removes the HMDG1 and temporarily leaves the virtual conference room 501, the user can be notified of the need to return to the conference by being notified by the mobile information terminal S1 when the user is spoken to.

[0123] Furthermore, by connecting the mobile information terminal S1 to an HMD, a user can participate in a conference in a virtual conference room from the mobile information terminal S1. In this case, by sending not only audio information but also gaze direction information specifying the direction in which the user's avatar wants to face, the user can change the facial direction of the user's avatar without wearing an HMD. This allows the user's avatar's gaze direction to be changed in the video of other users participating in the virtual conference room, eliminating the unnatural facial direction of the user's avatar when the user temporarily leaves the conference room.

[0124] Furthermore, in the above embodiment, the connection method between the HMD and the mobile information terminal is switched depending on whether the user is temporarily away from the seat, so that the user can rejoin the virtual conference room even if the user is located farther away from the HMD than the distance at which pairing via close-proximity wireless communication is possible.

[0125] As described above, according to the present embodiment, it is possible to eliminate delays in the progress of a virtual conference and unnatural images caused by temporary absences during the virtual conference. As virtual conferences become more widespread and take place more frequently and for longer periods of time, it may become necessary for users to remove their HMDs. However, even in such situations, the flow of conversation in the virtual conference is less likely to be disrupted, and a user-friendly virtual conference system can be provided.

[0126] Second Embodiment In addition to the first embodiment, the second embodiment is an embodiment in which, when a participant of a virtual conference temporarily leaves his / her seat, the other participants are notified of the temporary absence.

[0127] FIG. 14 is a schematic diagram of the display image of the virtual conference room 501 on the HMDG 2.

[0128] FIG. 14 shows a first-person display image seen from the perspective of avatar A2 in the virtual conference room, displayed on the HMDG2 of second user P2 when first user P1 is temporarily away from the remote conference.

[0129] 14, the processor 2 of the HMDG 2 controls the display of the text "temporarily away" near the avatar A1 of the first user P1 who is temporarily away from his / her desk. Although not shown, the same is true for the HMDG 3 of the third user P3.

[0130] In this way, all other users P2 and P3 participating in the conference can know that the first user P1 is temporarily away from his / her desk but is available to respond at any time.

[0131] The text displayed during the temporary absence is not limited to "temporarily away," but may be anything that conveys that the first user P1 can return to the conference immediately. For example, a specific graphic may be used. Furthermore, the display color or brightness of the self-avatar A1 may be changed.

[0132] Third Embodiment The third embodiment is an embodiment in which, in addition to the first embodiment, a participant of a virtual conference is notified to other participants that he or she has temporarily left the conference room and is participating in remote mode.

[0133] FIG. 15 is a schematic diagram of the display image of the virtual conference room 501 on the HMDG 2.

[0134] FIG. 15 shows a first-person display image seen from the perspective of avatar A2 in the virtual conference room, displayed on the HMDG2 of second user P2 when first user P1 is temporarily away from the remote conference.

[0135] In FIG. 15, the processor 2 of the HMDG 2 displays an image of an avatar A1 holding a smartphone 1501 when participating in a conference in remote control mode.

[0136] In this way, other users can recognize that the first user P1, who is away from his / her seat, has returned to the conference and is participating in the conference in remote control mode.

[0137] Note that the image is not limited to an avatar holding a smartphone, and is not particularly limited as long as it allows other users to understand the situation of someone nearby participating in remote control mode.

[0138] Fourth Embodiment The fourth embodiment relates to an example of the process of detecting speech to one's own avatar (step S701). FIG. 16 is a functional diagram of a video display program executed by the processor 2.

[0139] 16 is stored in the program storage area 41 of the memory 4 of the HMDG1, and is loaded into the work area 43 and executed to realize its functions. The video display program 410 is also installed in the HMDG2 and HMDG3 worn by each of the other users participating in the virtual conference, and realizes functions similar to the processes described below.

[0140] The video display program 410 includes an audio output control unit 411 , an audio analysis unit 412 , a display control unit 413 , an other avatar field of view calculation unit 414 , a notification processing unit 415 , a remote mode processing unit 416 , an absence determination unit 417 , and a communication control unit 418 .

[0141] The audio output control unit 411 outputs audio information uttered by other users received from the distribution server 14 from the right speaker 85R and left speaker 85L of the HMDG 1.

[0142] The voice analysis unit 412 uses an artificial intelligence engine that analyzes natural language to detect the user's proper nouns contained in the voice information, as well as general terms used to address others, such as "What do you think about...?" and "Hey," which do not include words that identify a specific person.

[0143] The display control unit 413 generates images of other avatars according to the video information, gaze direction information, and behavior information received from the distribution server 14, and displays the images on the right display 202R and the left display 202L.

[0144] The other avatar field of view calculation unit 414 determines whether the user's avatar is included in the line of sight of the other avatar, based on the video information and gaze direction information received from the distribution server 14. Specifically, it calculates the field of view of the other avatar within a predetermined horizontal and vertical angle range centered on a vector indicating the gaze direction, with the position of the other avatar in the virtual space as the starting point. If the user's avatar is included in the field of view, it is determined that the other avatar is facing the user's avatar.

[0145] When a call to the user's own avatar is detected, the notification processing unit 415 notifies the mobile information terminal S1 connected to the HMDG1 that a call has been made.

[0146] When the remote mode is selected, the remote mode processing unit 416 executes processing relating to the remote mode between the HMDG 1 and the portable information terminal S1.

[0147] The absence determination unit 417 determines whether the first user P1 has put on or removed the HMDG1, and determines that the first user P1 is participating in the conference if the HMDG1 is being worn, and that the first user P1 is temporarily away if the HMDG1 is being removed. The absence determination unit 417 may determine only whether the first user P1 is participating in the conference or temporarily away, or may determine which of multiple temporary absence modes the first user P1 is in as described in the first embodiment.

[0148] The communication control unit 418 controls communications between the HMDG 1 and the distribution server 14, management server 15, and mobile information terminal S1.

[0149] FIG. 17 is a flowchart showing the flow of the process of detecting speech to one's own avatar.

[0150] The voice analysis unit 412 analyzes whether the voice information of another user received from the distribution server 14 contains the user's own proper noun or a general term of address (S1701). If the user's own proper noun is present (S1702: Yes), it is determined that the user is speaking to his / her own avatar (S701: Yes).

[0151] If the user's own proper noun is not found (S1702: No), the other avatar field of view calculation unit 414 calculates the field of view of all other avatars (S1703).

[0152] The other avatar field of view calculation unit 414 determines whether the user's avatar is included in the field of view of any other avatar. The position information of the user's avatar may be the seat position information of the user's avatar that the operation input device 9 has accepted, or may be received from the distribution server 14. If the result is negative (1704: No), it is determined that no one has spoken to the user's avatar (S701: No).

[0153] If the other avatar field of view calculation unit 414 determines that the self-avatar is included in the field of view of any other avatar (S1704: Yes) and the audio analysis unit 412 determines that the audio information includes a general expression of a call (S1705: Yes), it determines that someone is speaking to the self-avatar (S701: Yes).

[0154] If the other avatar field of view calculation unit 414 determines that the self avatar is not included in the field of view of any of the other avatars (S1704: Yes) and the audio analysis unit 412 determines that the audio information does not include a general expression of a call (S1706: No), the other avatar field of view calculation unit 414 determines whether the self avatar is included in the field of view of two or more other avatars (S1706). If the result is positive, it is determined that someone is talking to the self avatar (S701: Yes). If the result is negative, it is determined that someone is not talking to the self avatar (S701: No).

[0155] According to this embodiment, whether or not someone is calling out to oneself can be detected by using various conditions that take into account whether or not one's own proper name is used, whether or not other avatars are visible, whether or not the call is made using general expressions, and the degree of attention from other members.

[0156] The above-described embodiments do not limit the present invention, and the present invention can be realized in various different embodiments.

[0157] For example, instead of an HMD, the image display device may be a laptop computer, a tablet computer, a smartphone, or a display or projector connected to a stationary personal computer via a wired or wireless connection.

[0158] In this case, an in-camera that captures an image of the real space facing the display of each device may be used as the participation detection sensor, and if the user is not captured in the image of the in-camera, it may be determined that the user is temporarily away from their seat. The in-camera may be formed integrally with the video display device, or an external camera may be used as the in-camera.

[0159] It should be noted that the present invention is not limited to the above-described embodiment and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0160] Furthermore, the above describes an example of setting the system when a user enters a virtual conference room or while in the room, but the present invention is not limited to this, and it is also possible to use the system's default values ​​or preset values ​​that the user has set in advance.

[0161] It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment.It is also possible to add, delete, or replace part of the configuration of each embodiment with the configuration of another embodiment.

[0162] Furthermore, the above-mentioned components may be partially or entirely configured in hardware, or may be configured to be realized by a processor executing a program. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0163] The above embodiments include the following inventions: (Supplementary Note 1) A video display device comprising: a processor, a display, a participation detection sensor that detects whether a user is participating in a conversation in a virtual space received via the video display device, and a first communication device that receives from an external device video information of the virtual space in which a self-avatar corresponding to the user exists, video information of other avatars corresponding to other users, and audio information of the other users, wherein the processor generates a video in which the other avatar is placed in the virtual space based on the video information of the virtual space and the video information of the other avatars, and displays the video on the display, determines based on sensor output from the participation detection sensor whether the user is in a temporary absence state in which the user has placed the self-avatar in the virtual space but is not participating in the conversation, and executes control to notify the user when it determines based on the audio information that the other avatar is talking to the self-avatar during the temporary absence state.(Supplementary Note 2) In a video display system configured by a distribution server and a video display device communicatively connected, the distribution server distributes video information of a virtual space in which a first user's own avatar corresponding to the first user exists to a video display device operated by the first user, and distributes video information of another avatar corresponding to a second user existing in the virtual space and audio information of the second user to the video display device, the video display device comprising: a processor; a display; a participation detection sensor that detects whether the first user is participating in a conversation in the virtual space; and a communication device that receives the video information of the virtual space, the video information of the other avatar, and the audio information, the processor generates a video in which the other avatar is placed in the virtual space based on the video information of the virtual space and the video information of the other avatar, and displays the video on the display; and determines whether the first user is in a temporary absence state in which the first user has placed their own avatar in the virtual space but is not participating in the conversation, based on a sensor output from the participation detection sensor, a control method for a video display device, the control method comprising: receiving from an external device video information in which an avatar corresponding to another user is placed in a virtual space in which a user's own avatar exists, and audio information of the other user; generating a video in which the other avatar is placed in the virtual space based on the video information of the virtual space and the video information of the other avatar, and displaying the video on a display; determining whether the user is in a temporary absence state in which the user has placed the user's own avatar in the virtual space but is not participating in the conversation in the virtual space, based on a sensor output from a participation detection sensor that detects whether the user is participating in the conversation in the virtual space; and notifying the user when it is determined based on the audio information that the other avatar is talking to the user's own avatar during the temporary absence state.

[0164] 2: Processor 3: Bus 4: Memory 5: Sensor device 6: Communication device 7: Video processing device 8: Audio processing device 9: Operation input device 10: Gaze detection device 11: Control device 13: Communication network 14: Distribution server 15: Management server 41: Program storage area 42: Data storage area 43: Work area 51: GPS sensor 52: Geomagnetic sensor 53: Wear / detach detection sensor 54: Acceleration sensor 55: Gyro sensor 61: LAN communication device 62: Mobile wireless communication device 63: Close proximity wireless communication device 71: Camera 81: Microphone 82: Audio recognition unit 83: Decoder 84: Encoder 85L: Left speaker 85R: Right speaker 100: Video display system 202L: Left display 202R : Right display 410: Video display program 411: Audio output control unit 412: Audio analysis unit 413: Display control unit 414: Other avatar field of view calculation unit 415: Notification processing unit 416: Remote mode processing unit 417: Absence determination unit 418: Communication control unit 501: Virtual conference room 501t: Conference desk 601: Display image 708: Step 901: Notification sound 902: Message sound 903: Message 1001L: Left gaze detection sensor 1001R: Right gaze detection sensor 1100: Sound 1101: Reply 1301: Notification sound 1302: Message sound 1501: Smartphone A1: Self avatar G1: HMD G1a: HMD G10: Housing P1 : First user P2: Second user P3: Third user R1: Wireless router R2: Wireless router R3: Wireless router S1: Portable information terminal

Claims

1. An image display device, A processor; A display and a participation detection sensor that detects whether a user is participating in a conversation in a virtual space received via the video display device; a first communication device that receives, from an external device, image information of a virtual space in which a user's own avatar exists, image information of other avatars corresponding to other users, and audio information of the other users; Equipped with The processor, generating an image in which the other avatar is placed in the virtual space based on the image information of the virtual space and the image information of the other avatar, and displaying the image on the display; determining whether the user is in a temporary absence state in which the user has placed the self-avatar in the virtual space but is not participating in the conversation based on a sensor output from the participation detection sensor; When it is determined that the other avatar is talking to the user's own avatar based on the voice information during the temporary absence state, a control is executed to notify the user. Image display device.

2. 2. The image display device according to claim 1, The image display device further includes a camera that captures an image of an external environment in which the user exists, The processor, a through mode in which the image of the outside world captured by the camera is displayed on the display as a control mode of the image display device, and a normal mode in which the image information is displayed on the display, The state of being away from the seat is determined as being the momentary absence state while the through mode is being used, when it is determined that the other avatar is talking to the self avatar during the temporary absence state, the through mode is cancelled and the self avatar is transitioned to the normal mode. Image display device.

3. 2. The image display device according to claim 1, The image display device includes a speaker, The processor, a speaker mode in which sound based on the audio information is output from the speaker, and a normal mode in which the video information is displayed on the display, as control modes of the video display device; The state in which the user is away from the seat is determined to be the momentary absence state while the speaker mode is being used. Image display device.

4. 2. The image display device according to claim 1, A second communication device that establishes a wireless communication connection with the portable information terminal is further provided, The processor, and when it is determined that the other avatar is talking to the self avatar based on the voice information during the temporary absence state, the second communication device is controlled to transmit the notification to the mobile information terminal. Image display device.

5. 5. The image display device according to claim 4, The processor, executing a remote control mode for transferring the video information and the audio information to the portable information terminal, and receiving from the portable information terminal a remote control command for designating the user's voice and the gaze direction of the user's avatar and transferring the remote control command to the external device; Image display device.

6. 5. The image display device according to claim 4, The image display device is a head-mounted display, The participation detection sensor is a sensor for detecting whether the head mounted display is attached or detached. Image display device.

7. 2. The image display device according to claim 1, The participation detection sensor is an in-camera that captures an image of a real space facing the display, The processor, If the user is not captured in the image captured by the front camera, it is determined that the user is in the temporarily absent state. Image display device.

8. In a video display system configured by communicatively connecting a distribution server and a video display device, The distribution server includes: delivering, to a video display device operated by a first user, video information of a virtual space in which a self-avatar corresponding to the first user exists, and also delivering, to the video display device, video information of another avatar corresponding to a second user and existing in the virtual space, and audio information of the second user; The image display device includes: A processor; A display and a participation detection sensor that detects whether the first user is participating in a conversation in the virtual space; a communication device that receives the video information of the virtual space, the video information of the other avatars, and the audio information; Equipped with The processor, generating an image in which the other avatar is placed in the virtual space based on the image information of the virtual space and the image information of the other avatar, and displaying the image on the display; determining whether the first user is in a temporary absence state in which the first user has placed the self-avatar in the virtual space but is not participating in the conversation based on a sensor output from the participation detection sensor; and when it is determined that the other avatar is talking to the self avatar based on the voice information during the temporary absence state, executing control to notify the first user. Video display system.

9. A method for controlling a video display device, comprising: receiving from an external device video information in which another avatar corresponding to another user is placed in a virtual space in which a self avatar corresponding to the user exists, and audio information of the other user; generating an image in which the other avatar is placed in the virtual space based on the image information of the virtual space and the image information of the other avatar, and displaying the image on a display; A step of determining whether or not the user is in a temporary absence state in which the user has placed the self-avatar in the virtual space but is not participating in the conversation in the virtual space, based on a sensor output from a participation detection sensor that detects whether the user is participating in the conversation in the virtual space; a step of notifying the user when it is determined that the other avatar is talking to the user's own avatar based on the voice information during the temporary absence state; A control method for a video display device comprising: