Display Control Device

Abstract

A display control device according to the present invention comprises: a display control unit that, through a display device, causes a user to recognize a virtual space in which a virtual object is superimposed on a real space, by displaying an image representing a virtual object on a display device; and a transmissivity control unit that controls transmissivity of the virtual object in accordance with an occupancy rate of area of a display region of the display device that the virtual object occupies.

Description

[Technical Field]

[0001] The present invention relates to a display control device. [Background technology]

[0002] When virtual objects overlap each other in a virtual space, a display control device that displays the virtual space on a display device may control the transparency of the virtual objects in order to increase the visibility of both virtual objects for the user.

[0003] For example, Patent Document 1 discloses a program that enables a user to easily and smoothly operate each virtual object in a virtual space without causing the user trouble in operating a terminal. The program causes a computer to function as an object determination unit that determines at least one of the virtual objects as an object to be subjected to transparency processing, and an object control unit that performs transparency processing on the object subjected to the transparency processing in accordance with a transparency rate that is determined in accordance with the positional relationship with a virtual camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-016319 Summary of the Invention [Problem to be solved by the invention]

[0005] When a user uses a smartphone or XR glasses that provide an augmented reality space or mixed reality space in which virtual objects and real space are superimposed, not only do virtual objects overlap with each other, but also virtual objects existing in the virtual space overlap with objects existing in the real space.

[0006] However, the technology disclosed in Patent Document 1 only supports superimposing virtual objects that exist in a virtual space, and does not support superimposing a virtual object on an object that exists in real space.

[0007] Therefore, an object of the present invention is to provide a display control device that can improve the visibility of an object when a virtual object and an object existing in real space are superimposed on each other. [Means for solving the problem]

[0008] A first display control device according to a preferred embodiment of the present invention is a display control device that includes a display control unit that causes a display device to display an image showing a virtual object, thereby allowing a user to recognize a virtual space in which the virtual object is superimposed on real space via the display device, and a transmittance control unit that controls the transmittance of the virtual object in accordance with the occupancy rate of the area occupied by the virtual object in the display area of ​​the display device.

[0009] A second display control device according to a preferred embodiment of the present invention is a display control device including an imaging device that captures an image of a real space in which a real object exists, a recognition unit that recognizes the type of the real object, a display control unit that displays an image of a virtual object on a display device, thereby allowing a user to recognize a virtual space in which the virtual object is superimposed on the real space via the display device, and a transmittance control unit that controls the transmittance of the virtual object depending on the type of real object located behind the virtual object. [Effects of the Invention]

[0010] According to the present invention, when a virtual object and an object existing in real space are superimposed on each other, it is possible to improve the visibility of the object existing in real space. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment. [Figure 2A] FIG. 1 is a diagram showing an example of a real space. [Figure 2B] FIG. 1 is a diagram showing an example of a mixed reality space in which a virtual object is superimposed on a real space. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 10-K. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a server 20. [Figure 5] 10 is a flowchart showing the operation of the terminal device 10-K. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 10A-K. [Figure 7] 10 is a flowchart showing the operation of the terminal device 10A-K. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 10B-K. [Figure 9] FIG. 4 is an explanatory diagram of a first distance and a second distance. [Figure 10] 10 is a flowchart showing the operation of the terminal device 10B-K. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1: First embodiment Hereinafter, with reference to FIGS. 1 to 5, a configuration of an information processing system 1 including a terminal device 10 as a display control device according to a first embodiment of the present invention will be described.

[0013] 1-1: Configuration of the first embodiment 1-1-1: Overall structure Fig. 1 shows the overall configuration of an information processing system 1 according to the first embodiment. As shown in Fig. 1, the information processing system 1 includes terminal devices 10-1, 10-2, ... 10-K, ... 10-J, and a server 20. J is an integer equal to or greater than 1. K is an integer equal to or greater than 1 and equal to or less than J. In this embodiment, the terminal devices 10-1 to 10-J have the same configuration. However, terminal devices that do not have the same configuration as the other terminals may be included.

[0014] In the information processing system 1, the terminal devices 10-1 to 10-J and the server 20 are connected to each other via a communication network NET so as to be able to communicate with each other. K It is assumed that the terminal device 10-K is used.

[0015] The server 20 provides various data and cloud services to the terminal devices 10-1 to 10-J via the communication network NET. In particular, the server 20 provides various contents to be displayed in a virtual space to the terminal devices 10-1 to 10-J.

[0016] The terminal device 10-K is connected to a display 15 of the terminal device 10-K or a display 15 of the terminal device 10-K. K The terminal device 10-K displays virtual objects on the XR glasses worn on the head of the user U. Here, "XR glasses" is a general term for VR (Virtual Reality) glasses, AR (Augmented Reality) glasses, and MR (Mixed Reality) glasses. Also, "the display 15 provided in the terminal device 10-K" and "the display connected to the terminal device 10-K and displayed on the head of the user U" are referred to as "XR glasses." K "XR glasses worn on the head" is an example of a "display device."

[0017] Examples of "virtual objects" include virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, as well as virtual objects representing applications. Examples of text files include memos and source code. Examples of applications include browsers, applications for using SNS, and applications for generating document files.

[0018] The terminal device 10-K is preferably a mobile terminal device such as a smartphone or a tablet, for example.

[0019] 2A and 2B are diagrams showing an example of a mixed reality space in which a virtual object is superimposed on a real space.

[0020] FIG. 2A is an example of a real space RS displayed on a display device. The example of real space RS shown in FIG. 2A is a university classroom. In the university classroom, there are a desk T1, chairs C1 to C4, and a bulletin board NB as some of the objects present in the classroom. Furthermore, as viewed from the user of the display device, the desk T1 and the chairs C1 to C4 are located in front of the bulletin board NB. Furthermore, the chairs C1 to C4 surround the desk T1.

[0021] FIG. 2B illustrates an example of a mixed reality space MS in which a virtual object VO is superimposed on the real space RS illustrated in FIG. 2A. In the mixed reality space MS, the virtual object VO is located in front of the desk T1, the chairs C1-C4, and the bulletin board NB as viewed by the user of the display device. Furthermore, the size of the virtual object VO is such that it covers the desk T1, the chairs C1-C4, and a portion of the bulletin board NB as viewed by the user of the display device. As a result, if the user moves forward in the real space RS while looking at the display device, there is a risk that the user may not notice the desk T1 and the chairs C1-C4 hidden by the virtual object VO and may collide with one of the desk T1 and the chairs C1-C4. Furthermore, because a portion of a notice posted on the bulletin board NB is hidden by the virtual object VO, the user cannot see the content of the notice. Therefore, the terminal device 10 controls the transparency of the virtual object VO to allow the user to view the desk T1, the chairs C1-C4, and the bulletin board NB.

[0022] As another example (not shown), when a user of a display device moves around town while visually viewing a mixed reality space displayed on the display device, a virtual object may block traffic lights and traffic signs, posing a risk to the user. Furthermore, a virtual object may block a signboard installed on a building, preventing the user from obtaining information that should be available to the user. Therefore, the terminal device 10 controls the transparency of the virtual object as described above.

[0023] 1-1-2: Terminal device configuration 3 is a block diagram showing an example of the configuration of the terminal device 10-K. The terminal device 10-K includes a processing device 11, a storage device 12, a communication device 13, a positioning device 14, a display 15, an input device 16, and an inertial sensor 17. The elements of the terminal device 10-K are connected to each other using one or more buses for communicating information.

[0024] The processing device 11 is a processor that controls the entire terminal device 10-K. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 11 may be realized using hardware such as a DSP, ASIC, PLD, or FPGA. The processing device 11 executes various processes in parallel or sequentially.

[0025] The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 also stores a plurality of programs including a control program PR1 executed by the processing device 11. The storage device 12 also stores image information indicating an image to be displayed on the display 15. In particular, the storage device 12 also stores image information indicating an image used by a generation unit 111 (described later) when generating a virtual object VO.

[0026] The communication device 13 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0027] The positioning device 14 acquires location information. The positioning device 14 may be, for example, a GPS (Global Positioning System) device. When the positioning device 14 is a GPS device, the positioning device 14 receives radio waves from multiple satellites. Furthermore, the positioning device 14 generates GPS information as location information from the received radio waves. The location information may be in any format as long as it can identify the location. The location information indicates, for example, the latitude and longitude of the terminal device 10-K. The acquired location information is output to the processing device 11.

[0028] Alternatively, the positioning device 14 may be, for example, a VPS (Visual Positioning System) device. When the positioning device 14 is a VPS device, the positioning device 14 acquires the user U from an imaging device (not shown). K The positioning device 14 acquires image information showing an image obtained by capturing an image of a scene in front of the user U. The positioning device 14 also outputs the image information acquired from the image capturing device to a location information server (not shown) via the communication device 13. The positioning device 14 also acquires VPS information as location information from the location information server via the communication device 13. The location information is used by the user U. K position in the real space RS, and the user U K includes the direction in which the real space RS is viewed.

[0029] The display 15 is a device that displays images and text information. The display 15 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 15. Note that when XR glasses are connected to the terminal device 10-K, the display 15 does not need to be an essential component. Specifically, by using the XR glasses as the display 15, the terminal device 10-K may not be provided with the display 15. The display 15 and the XR glasses are examples of display devices.

[0030] The input device 16 is K For example, the input device 16 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. If the input device 16 includes a touch panel, it may also function as the display 15.

[0031] The inertial sensor 17 is a sensor that detects inertial force. The inertial sensor 17 includes, for example, one or more sensors selected from an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 11 detects the attitude of the terminal device 10-K based on the output information of the inertial sensor 17. Furthermore, the processing device 11 accepts the selection of a virtual object VO, the input of characters, and the input of instructions in the mixed reality space MS based on the attitude of the terminal device 10-K. For example, when a user U K When the user U operates the input device 16 while pointing the central axis of the terminal device 10-K toward a predetermined area in the mixed reality space MS, a virtual object VO to be placed in the predetermined area is selected. K The operation is, for example, a double tap. K By operating the terminal device 10-K, the user can select the virtual object VO without looking at the input device 16 of the terminal device 10-K.

[0032] The processing device 11 reads out the control program PR1 from the storage device 12 and executes the control program PR1. As a result, the processing device 11 functions as a generation unit 111, a display control unit 112, a calculation unit 113, a transmittance control unit 114, and a communication control unit 115.

[0033] The generation unit 111 generates a virtual object VO. The generation unit 111 may generate the virtual object VO using image information stored in the storage device 12. Alternatively, the generation unit 111 may acquire image information from the server 20 via the communication device 13, and generate the virtual object VO using the acquired image information.

[0034] The display control unit 112 displays an image showing the virtual object VO on the display 15, thereby presenting the mixed reality space MS in which the virtual object VO is superimposed on the real space RS to the user U via the display 15. K In particular, as described above, when XR glasses are connected to the terminal device 10-K as the display 15, the display control unit 112 displays an image showing the virtual object VO on the XR glasses (display 15) in a situation where light incident from the outside world passes through the XR glasses, thereby displaying the mixed reality space MS to the user U. K Make them aware of it.

[0035] The calculation unit 113 calculates the occupancy rate of the area of ​​the display area of ​​the display 15 that is occupied by the virtual object VO that the display control unit 112 displays.

[0036] The transmittance control unit 114 controls the transmittance of the virtual object VO in accordance with the occupancy rate of the area occupied by the virtual object VO calculated by the calculation unit 113. Specifically, the transmittance control unit 114 increases the transmittance of the virtual object VO as the occupancy rate calculated by the calculation unit 113 increases. The larger the area of ​​the display area of ​​the display 15 occupied by the virtual object VO, the more the user U KThis is because the area in front of the virtual object VO in the real space RS, more specifically, the area behind the virtual object VO that cannot be seen by the user, becomes wider. Therefore, the higher the occupancy rate, the more transparent the virtual object VO becomes.

[0037] Here, the occupancy of the virtual object VO and the transparency of the virtual object VO may be proportional to each other. For example, when the occupancy is 0%, the transparency control unit 114 may set the transparency of the virtual object VO to 0%, and when the occupancy is 90%, the transparency control unit 114 may set the transparency of the virtual object VO to 90%. Alternatively, the transparency control unit 114 may not set the occupancy of the virtual object VO and the transparency of the virtual object VO to be proportional to each other, but may monotonically increase the transparency of the virtual object VO as the occupancy of the virtual object VO increases.

[0038] As described above, the larger the area of ​​the virtual object VO that blocks the real object RO that exists in the real space RS, the greater the user U K The user U will no longer be able to obtain the information that he or she would normally have obtained from the real object RO. K When a real object RO located in the user's direction is blocked by a virtual object VO, K The risk of this occurring due to the movement of the virtual object VO increases. The terminal device 10-K increases the transparency of the virtual object VO as the occupancy rate of the area of ​​the display area of ​​the display 15 increases. K User U can get the information he / she should have. K This reduces the risk of increased risk due to the movement of people.

[0039] The communication control unit 115 controls the communication device 13 to receive the user U using the terminal device 10-K. K In particular, the communication control unit 115 causes the communication device 13 to transmit operation information indicating the operation of the user U with respect to the virtual object VO. K The server 20 transmits operation information indicating the operation.

[0040] 1-1-3: Server configuration 4 is a block diagram showing an example of the configuration of the server 20. The server 20 includes a processing device 21, a storage device 22, a communication device 23, a display 24, and an input device 25. The elements of the server 20 are connected to each other using one or more buses for communicating information.

[0041] The processing device 21 is a processor that controls the entire server 20. The processing device 21 is configured, for example, using one or more chips. The processing device 21 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 21 may be realized using hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 21 executes various processes in parallel or sequentially.

[0042] The storage device 22 is a recording medium that can be read from and written to by the processing device 21. The storage device 22 also stores a plurality of programs including a control program PR2 executed by the processing device 21. The storage device 22 also stores image information that indicates an image to be displayed on the display 24.

[0043] The communication device 23 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0044] The display 24 is a device that displays images and text information. The display 24 displays various images under the control of the processing device 21. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 24.

[0045] The input device 25 receives operations from an administrator of the information processing system 1. For example, the input device 25 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 25 includes a touch panel, it may also serve as the display 24.

[0046] The processing device 21 reads the control program PR2 from the storage device 22 and executes the control program PR2. As a result, the processing device 21 functions as an acquisition unit 211 and a communication control unit 212.

[0047] The acquisition unit 211 receives the user U from the terminal device 10-K via the communication device 23. K Obtain operation information indicating the operation.

[0048] The communication control unit 212 causes the communication device 23 to transmit various types of data for providing various types of content and cloud services to the terminal device 10-K. In particular, the communication control unit 212 causes the communication device 23 to transmit various types of data necessary for the user of the terminal device 10-K to experience the mixed reality space MS to the terminal device 10-K. Furthermore, when the terminal device 10-K generates a virtual object VO using image information stored in the server 20, the communication control unit 212 reads the image information from the storage device 22 and causes the communication device 23 to transmit the image information to the terminal device 10-K.

[0049] 1-2: Operation of the first embodiment 5 is a flowchart showing the operation of the terminal device 10-K as a display control device according to the first embodiment. Hereinafter, the operation of the terminal device 10-K will be described with reference to FIG.

[0050] In step S1, the processing device 11 functions as a generation unit 111. The processing device 11 generates a virtual object VO.

[0051] In step S2, the processing device 11 functions as the display control unit 112. The processing device 11 causes the display 15 to display an image indicating the virtual object VO generated in step S1, thereby causing the display 15 to display a mixed reality space MS in which the virtual object VO is superimposed on the real space RS.

[0052] In step S3, the processing device 11 functions as the calculation unit 113. The processing device 11 calculates the occupancy rate of the area of ​​the display region of the display 15 that is occupied by the virtual object VO that the display control unit 112 causes to be displayed.

[0053] In step S4, the processing device 11 functions as the transmittance control unit 114. The processing device 11 controls the transmittance of the virtual object VO in accordance with the occupancy calculated in step S3. After that, the terminal device 10-K ends all of the processes shown in the flowchart of FIG.

[0054] 1-3: Effects of the First Embodiment The terminal device 10-K as a display control device according to this embodiment includes a display control unit 112 and a transmittance control unit 114. The display control unit 112 displays an image showing a virtual object VO on a display 15 as a display device, thereby presenting a mixed reality space MS in which the virtual object VO is superimposed on a real space RS to a user U via the display 15. K The transmittance control unit 114 controls the transmittance of the virtual object VO in accordance with the occupancy rate of the area occupied by the virtual object VO in the display area of ​​the display 15.

[0055] Since the terminal device 10-K has the above-mentioned configuration, when a virtual object VO and an object existing in the real space RS are superimposed in an augmented reality space or a mixed reality space, it is possible to improve the visibility of the object existing in the real space RS.

[0056] The larger the area that the virtual object VO occludes from the real object RO in the real space RS, the greater the user U K The user U will no longer be able to obtain the information that he or she would normally have obtained from the real object RO. K When a real object RO located in the user's direction is blocked by a virtual object VO, K The risk of this occurring increases as the virtual object VO moves. The terminal device 10-K increases the transparency of the virtual object VO as the occupancy rate of the area of ​​the screen displayed by the display 15 increases. K In addition, user U can obtain the information that he / she should have obtained. K This reduces the risk of increased risk due to the movement of people.

[0057] 2: Second embodiment Hereinafter, the configuration of an information processing system 1A including a terminal device 10A as a display control device according to a second embodiment of the present invention will be described with reference to FIGS.

[0058] 2-1: Configuration of the second embodiment 2-1-1: Overall structure The information processing system 1A according to the second embodiment differs from the information processing system 1 according to the first embodiment in that it includes terminal devices 10A-1, 10A-2, ... 10A-K, ... 10A-J instead of terminal devices 10-1, 10-2, ... 10-K, ... 10-J. In other respects, the overall configuration of the information processing system 1A is the same as the overall configuration of the information processing system 1 shown in Figure 1, and therefore is not shown in the figures.

[0059] In the following, among the components of the information processing system 1A according to the second embodiment, the same symbols will be used for components that are the same as the components of the information processing system 1 according to the first embodiment, and for the sake of simplicity, explanations of their functions will be omitted.

[0060] 2-1-2: Terminal device configuration 6 is a block diagram showing an example of the configuration of the terminal device 10A-K. Unlike the terminal device 10-K according to the first embodiment, the terminal device 10A-K includes a processing device 11A instead of the processing device 11 and a storage device 12A instead of the storage device 12. Furthermore, the terminal device 10A-K includes an imaging device 18 in addition to the components included in the terminal device 10-K.

[0061] The imaging device 18 captures an image of the external real space RS in which an object exists. The imaging device 18 outputs imaging information indicating an image obtained by capturing an image of the external world. The imaging device 18 further includes, for example, a lens, an imaging element, an amplifier, and an AD converter. The imaging element converts light collected through the lens into an imaging signal, which is an analog signal. The amplifier amplifies the imaging signal and outputs the amplified imaging signal to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The imaging information is output to the processing device 11A. Note that, as described above, if the terminal device 10A-K uses XR glasses connected to the terminal device 10A-K instead of including the display 15, the terminal device 10A-K may use the imaging device included in the XR glasses instead of the imaging device 18. Note that the "object" here is an example of a "real object."

[0062] The storage device 12A stores a control program PR1A instead of the control program PR1 included in the storage device 12 according to the first embodiment. The storage device 12A also stores a learning model LM.

[0063] The learning model LM is a learning model used by the recognition unit 116 (described later) to recognize the type of object included in an image obtained by the imaging device 18 performing imaging.

[0064] The learning model LM is generated by learning training data in the learning phase. The training data used to generate the learning model LM has multiple one-to-one pairs of object photos and object types.

[0065] The learning model LM is generated outside the terminal devices 10A-K. In particular, it is preferable that the learning model LM is generated in a second server (not shown). In this case, the terminal devices 10A-K acquire the learning model LM from the second server (not shown) via the communication network NET.

[0066] The processing device 11A reads out the control program PR1A from the storage device 12A and executes the control program PR1A. As a result, the processing device 11A functions as a transmittance control unit 114A and a recognition unit 116 in addition to the generation unit 111, display control unit 112, and communication control unit 115 similar to those in the first embodiment.

[0067] The recognition unit 116 uses the learning model LM to recognize the type of a real object RO that exists in the real space RS. Specifically, before a virtual object VO is generated, the recognition unit 116 recognizes the type of one or more real objects that exist in the real space RS.

[0068] 2A and 2B, the recognition unit 116 recognizes that the type of the desk T1, which is a real object RO existing in the real space RS, is a "desk." Similarly, the recognition unit 116 recognizes that the types of the chairs C1 to C4, which are real objects RO existing in the real space RS, are "chairs." Furthermore, the recognition unit 116 recognizes that the type of the bulletin board NB, which is a real object RO existing in the real space RS, is a "bulletin board."

[0069] The transmittance control unit 114A controls the transmittance of the virtual object VO in accordance with the type of the real object RO located behind the virtual object VO. In the example shown in Fig. 2B, the transmittance control unit 114A controls the transmittance of the virtual object VO in accordance with the types of the real objects RO located behind the virtual object VO being "desk," "chair," and "bulletin board."

[0070] Here, the transmittance control unit 114A may control the transmittance of the virtual object VO according to the combination of the types of these real objects RO. In the example shown in Fig. 2B, the transmittance control unit 114A may control the transmittance of the virtual object VO based on the fact that the combination of the types of the real objects RO located behind the virtual object VO is a combination of three types: "desk," "chair," and "bulletin board."

[0071] Alternatively, the transmittance control unit 114A may control the transmittance of the virtual object VO based on the type of any one of one or more real objects RO located behind the virtual object VO. In the example shown in FIG. 2B , the transmittance control unit 114A may control the transmittance of the virtual object VO based on the fact that a real object RO whose type is "desk" is located behind the virtual object VO. Alternatively, the transmittance control unit 114A may control the transmittance of the virtual object VO based on the fact that a real object RO whose type is "chair" is located behind the virtual object VO. Alternatively, the transmittance control unit 114A may control the transmittance of the virtual object VO based on the fact that a real object RO whose type is "bulletin board" is located behind the virtual object VO.

[0072] When a real object RO existing in the real space RS is occluded by a virtual object VO, the user U KThe degree of danger that occurs varies depending on the type of the real object RO. The terminal device 10-K recognizes the type of the real object RO and controls the transparency of the virtual object VO according to the recognized type. As a result, the virtual object VO allows the user U to K If the risk of occurrence when the user U's view is obstructed is relatively high, the terminal device 10A-K can suppress the occurrence of a state where the risk is relatively high by relatively increasing the transparency of the virtual object VO. K When the risk of a user U being blocked is relatively low, the terminal device 10A-K relatively reduces the transparency of the virtual object VO. K The visibility of the virtual object VO for the user is ensured.

[0073] 2-2: Operation of the second embodiment 7 is a flowchart showing the operation of the terminal device 10A-K as a display control device according to the second embodiment. Hereinafter, the operation of the terminal device 10A-K will be described with reference to FIG.

[0074] In step S11, the imaging device 18 captures an image of the external real space RS in which the real object RO exists.

[0075] In step S12, the processing device 11A functions as the recognition unit 116. The processing device 11A uses the learning model LM to recognize the type of the real object RO present in the real space RS.

[0076] In step S13, the processing device 11A functions as the generation unit 111. The processing device 11A generates a virtual object VO.

[0077] In step S14, the processing device 11A functions as the display control unit 112. The processing device 11A displays an image showing the virtual object VO generated in step S13 on the display 15, thereby presenting to the user U, via the display 15, a mixed reality space MS in which the virtual object VO is superimposed on the real space RS.K Make them aware of it.

[0078] In step S15, the processing device 11A functions as a transmittance control unit 114A. The processing device 11A controls the transmittance of the virtual object VO according to the type of the real object RO located behind the virtual object VO. Thereafter, the terminal devices 10A-K end all of the processes shown in the flowchart of FIG. 7.

[0079] 2-3: Effects of the second embodiment The terminal devices 10A-K serving as display control devices according to this embodiment include an imaging device 18, a recognition unit 116, a display control unit 112, and a transmittance control unit 114A. The imaging device 18 captures an image of a real space RS in which a real object RO exists. The recognition unit 116 recognizes the type of the real object RO. The display control unit 112 displays an image showing a virtual object VO on a display 15 serving as a display device, thereby presenting a mixed reality space MS in which the virtual object VO is superimposed on the real space RS to the user U via the display 15. K The transparency control unit 114A controls the transparency of the virtual object VO according to the type of the real object RO located behind the virtual object VO.

[0080] Because the terminal device 10A-K has the above-mentioned configuration, when a virtual object VO and an object existing in the real space RS are superimposed in an augmented reality space or a mixed reality space, it is possible to improve the visibility of the object existing in the real space RS.

[0081] When a real object RO that exists in the real space RS is occluded by a virtual object VO, the user U K The degree of danger that occurs varies depending on the type of the real object RO. The terminal device 10-K recognizes the type of the real object RO and controls the transparency of the virtual object VO according to the recognized type. As a result, the virtual object VO allows the user U to KIf the risk of occurrence when the user U's view is obstructed is relatively high, the terminal device 10A-K can suppress the occurrence of a state where the risk is relatively high by relatively increasing the transparency of the virtual object VO. K When the risk of a user U being blocked is relatively low, the terminal device 10A-K relatively reduces the transparency of the virtual object VO. K The visibility of the virtual object VO for the user is ensured.

[0082] 3: Third embodiment Hereinafter, the configuration of an information processing system 1B including a terminal device 10B as a display control device according to a third embodiment of the present invention will be described with reference to FIGS.

[0083] 3-1: Configuration of the third embodiment 3-1-1: Overall structure The information processing system 1B according to the third embodiment differs from the information processing system 1 according to the first embodiment in that it includes terminal devices 10B-1, 10B-2, ... 10B-K, ... 10B-J instead of terminal devices 10-1, 10-2, ... 10-K, ... 10-J. In other respects, the overall configuration of the information processing system 1A is the same as the overall configuration of the information processing system 1 shown in Figure 1, and therefore is not shown in the figures.

[0084] In the following, among the components of the information processing system 1B of the third embodiment, the same symbols will be used for components that are the same as the components of the information processing system 1 of the first embodiment, and for the sake of simplicity, explanations of their functions will be omitted.

[0085] 3-1-2: Terminal device configuration 8 is a block diagram showing an example of the configuration of the terminal device 10B-K. Unlike the terminal device 10-K according to the first embodiment, the terminal device 10B-K includes a processing device 11B instead of the processing device 11 and a storage device 12B instead of the storage device 12.

[0086] The storage device 12B stores a control program PR1B instead of the control program PR1 provided in the storage device 12 according to the first embodiment.

[0087] The processing device 11B reads out the control program PR1B from the storage device 12B and executes the control program PR1B. As a result, the processing device 11B functions as a transmittance control unit 114B, a first acquisition unit 117, and a second acquisition unit 118 in addition to the generation unit 111, the display control unit 112, the calculation unit 113, and the communication control unit 115 similar to those in the first embodiment.

[0088] The first acquisition unit 117 acquires the distance from the virtual object VO to the real object RO located behind the virtual object VO in the mixed reality space MS. This distance is an example of a "first distance."

[0089] FIG. 9 is an explanatory diagram of the first distance and the second distance described later. Note that FIG. 9 is a diagram showing the mixed reality space MS shown in FIG. 2B viewed from a different angle. In the mixed reality space MS shown in FIG. 9, the first distance L1 is the distance between the nearby real object RO and the virtual object VO. The nearby real object RO is located between the user U and the virtual object VO. KFrom the perspective of the virtual object VO, the first distance L1 is the real object RO that is closest to the virtual object VO among the multiple real objects RO located behind the virtual object VO. The multiple real objects RO located behind the virtual object VO are a desk T1, chairs C1 to C4, and a bulletin board NB. More specifically, the first distance L1 is the distance between the proximity point of the nearby real object RO and the proximity point of the virtual object VO. The proximity point of the nearby real object RO is the point on the surface of the nearby real object RO that is closest to the virtual object VO. The proximity point of the virtual object VO is the point on the surface of the virtual object VO that is closest to the nearby real object RO. In the example shown in FIG. 9, the real object RO that is closest to the virtual object VO is the chair C3. Therefore, in the example shown in FIG. 9, the first distance L1 is the distance between the point of the chair C3 that is closest to the virtual object VO and the point of the virtual object VO that is closest to the chair C3. However, the first distance L1 is not limited to this. For example, the first distance L1 may be the distance between the center of gravity of the nearby real object RO and the center of gravity of the virtual object VO.

[0090] 8, the second acquisition unit 118 acquires the distance from the display 15, which serves as a display device, to the virtual object VO in the mixed reality space MS. This distance is an example of a "second distance."

[0091] Referring to FIG. 9, user U K is holding a terminal device 10B-K having a display 15. In the mixed reality space MS shown in Fig. 9, the second distance L2 is the distance between a point on the display 15 that is closest to the virtual object VO and a point on the virtual object VO that is closest to the display 15. However, the second distance L2 is not limited to this. For example, the second distance L2 may be the distance between the center of gravity of the display 15 and the center of gravity of the virtual object VO.

[0092] Returning to the explanation of FIG. 8, the transmittance control unit 114B controls the transmittance of the virtual object VO according to the occupancy rate of the virtual object VO calculated by the calculation unit 113 as well as the first distance L1 acquired by the first acquisition unit 117.

[0093] When the virtual object VO is not transparent, the longer the first distance L1 between the virtual object VO and the real object RO, that is, the further the virtual object VO and the real object RO are from each other, the more the user U K Conversely, the closer the virtual object VO and the real object RO are to each other, the lower the risk of the user U being unable to see the real object RO. K Therefore, the longer the first distance L1, the lower the transparency control unit 114B sets the transparency of the virtual object VO.

[0094] As a result, when the virtual object VO and the real object RO are relatively close to each other, the terminal device 10B-K K However, the increased risk associated with a situation in which the real object RO cannot be seen can be suppressed.

[0095] Alternatively, the transmittance control unit 114B may control the transmittance of the virtual object VO according to the second distance L2 acquired by the second acquisition unit 118 in addition to the occupancy rate of the virtual object VO calculated by the calculation unit 113.

[0096] The longer the second distance L2 between the display 15 and the virtual object VO, i.e., the greater the distance between the user U K The further away the user U is from the virtual object VO, the K Since the user U travels a long distance to reach the virtual object VO, K Therefore, the longer the second distance L2, the lower the transparency of the virtual object VO. As a result, the closer the display 15 and the virtual object VO are to each other, and therefore the closer the user U is to the terminal device 10B-K, the lower the risk of the user U being unable to see the real object RO. KThe closer the virtual object VO is to the user U, the K However, the increased risk associated with a situation in which the real object RO cannot be seen can be suppressed.

[0097] As an example, if the second distance L2 between the display 15 and the virtual object VO is x, the virtual object VO begins to become transparent at a distance of x=a (m), and the transmittance of the virtual object VO is 100 (%) at a distance of x=b (m), the transmittance control unit 114B may calculate the transmittance z (%) of the virtual object VO using the following formula (1).

number

[0098] Alternatively, the transmittance control unit 114B may control the transmittance of the virtual object VO according to both the occupancy rate of the virtual object VO calculated by the calculation unit 113, and the first distance L1 acquired by the first acquisition unit 117 and the second distance L2 acquired by the second acquisition unit 118.

[0099] 3-2: Operation of the third embodiment 10 is a flowchart showing the operation of the terminal device 10B-K as a display control device according to the third embodiment. Hereinafter, the operation of the terminal device 10B-K will be described with reference to FIG.

[0100] In step S21, the processing device 11B functions as the generation unit 111. The processing device 11B generates a virtual object VO.

[0101] In step S22, the processing device 11B functions as the display control unit 112. The processing device 11B displays an image showing the virtual object VO generated in step S21 on the display 15, thereby presenting to the user U, via the display 15, a mixed reality space MS in which the virtual object VO is superimposed on the real space RS. K Make them aware of it.

[0102] In step S23, the processing device 11B functions as the calculation unit 113. The processing device 11B calculates the occupancy rate of the area of ​​the display region of the display 15 that is occupied by the virtual object VO that the display control unit 112 causes to be displayed.

[0103] In step S24, the processing device 11B functions as the first acquisition unit 117. The processing device 11B acquires a first distance L1 from the virtual object VO to the real object RO located behind the virtual object VO in the mixed reality space MS.

[0104] In step S25, the processing device 11B functions as the second acquisition unit 118. The processing device 11B acquires a second distance L2 from the display 15, which serves as a display device, to the virtual object VO in the mixed reality space MS.

[0105] In step S26, the processing device 11B functions as a transmittance control unit 114B. The processing device 11B controls the transmittance of the virtual object VO in accordance with the occupancy rate calculated in step S23, the first distance L1 acquired in step S24, and the second distance L2 acquired in step S25. Thereafter, the terminal device 10B-K ends all of the processing shown in the flowchart of FIG. 10.

[0106] 3-3: Effects of the third embodiment A terminal device 10B-K serving as a display control device according to this embodiment further includes a first acquisition unit 117 in addition to the components included in the terminal device 10-K according to the first embodiment. The first acquisition unit 117 acquires a first distance L1 from a virtual object VO to a real object RO positioned behind the virtual object VO in the mixed reality space MS. The transmittance control unit 114B controls the transmittance of the virtual object VO in accordance with the first distance L1 in addition to the occupancy rate of the virtual object VO.

[0107] Since the terminal device 10B-K has the above-mentioned configuration, when a virtual object VO and an object existing in the real space RS are superimposed in an augmented reality space or a mixed reality space, it is possible to improve the visibility of the object existing in the real space RS.

[0108] In particular, in this embodiment, the terminal device 10B-K controls the transparency of the virtual object VO in accordance with the first distance L1 from the virtual object VO to the real object RO. Specifically, the longer the first distance L1, the lower the transparency of the virtual object VO. As a result, the terminal device 10B-K controls the transparency of the virtual object VO to be lower as the virtual object VO and the real object RO are relatively closer to each other. K However, the increased risk associated with a situation in which the real object RO cannot be seen can be suppressed.

[0109] Furthermore, the terminal device 10B-K as a display control device according to this embodiment further includes a second acquisition unit 118 in addition to the components included in the terminal device 10-K according to the first embodiment. The second acquisition unit 118 acquires a second distance L2 from the display 15 as a display device to the virtual object VO in the mixed reality space MS. The transmittance control unit 114B controls the transmittance of the virtual object VO in accordance with the second distance L2 in addition to the occupancy rate of the virtual object VO.

[0110] Since the terminal device 10B-K has the above-mentioned configuration, when a virtual object VO and an object existing in the real space RS are superimposed in an augmented reality space or a mixed reality space, it is possible to improve the visibility of the object existing in the real space RS.

[0111] In particular, in this embodiment, the terminal device 10B-K controls the transparency of the virtual object VO in accordance with the second distance L2 between the display 15 and the virtual object VO. Specifically, the longer the second distance L2, the lower the transparency of the virtual object VO. As a result, the closer the display 15 and the virtual object VO are to each other, and therefore the closer the user U is to the terminal device 10B-K. KThe closer the virtual object VO is to the user U, the K However, the increased risk associated with a situation in which the real object RO cannot be seen can be suppressed.

[0112] 4: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined.

[0113] 4-1: Variation 1 In the first embodiment, the terminal device 10-K displays a mixed reality space MS in which a virtual object VO is superimposed on a real space RS to a user U via a display 15 or XR glasses connected to the terminal device 10-K. K However, the terminal device 10-K displays a virtual reality space or an augmented reality space to the user U instead of the mixed reality space MS. K Alternatively, the terminal device 10-K may display the virtual reality space, the augmented reality space, or the mixed reality space MS to the user U via a head mounted display (HMD) instead of the XR glasses. K The HMD is an example of a "display device." Furthermore, the virtual reality space, the augmented reality space, and the mixed reality space MS are all examples of a "virtual space."

[0114] As an example, terminal device 10-K may cause a video see-through HMD to display a virtual reality space in which a virtual object VO is superimposed on an image of a real space RS including a real object RO.

[0115] In this case, the display control unit 112 displays, on the HMD, a superimposed image in which an image representing the virtual object VO is superimposed on a captured image obtained by capturing an image of the outside world, thereby presenting the virtual reality space to the user U. K Make them aware of it.

[0116] The terminal device 10A-K according to the second embodiment and the terminal device 10B-K according to the third embodiment may also be modified in the same manner as the first embodiment.

[0117] 4-2: Variation 2 In the information processing system 1 according to the first embodiment, the terminal device 10-K generates a virtual object VO and controls the transparency of the virtual object VO. However, instead of the terminal device 10-K, the server 20 may generate the virtual object VO and then distribute the virtual object VO to the terminal device 10-K. Furthermore, the server 20 may control the transparency of the virtual object VO to be distributed to the terminal device 10-K.

[0118] The information processing system 1A according to the second embodiment and the information processing system 1B according to the third embodiment may also be modified in the same manner as the first embodiment.

[0119] 4-3: Variation 3 The technical features of the information processing system 1 according to the first embodiment to the information processing system 1B according to the third embodiment may be combined with one another.

[0120] For example, by combining the calculation unit 113 provided in the terminal device 10-K according to the first embodiment with the recognition unit 116 provided in the terminal device 10A-K according to the second embodiment, the terminal device according to this modified example may control the transparency of the virtual object VO according to the occupancy rate of the virtual object VO only when a real object RO exists behind the virtual object VO.

[0121] Alternatively, for example, by combining the recognition unit 116 provided in the terminal device 10A-K according to the second embodiment with the first acquisition unit 117 and the second acquisition unit 118 provided in the terminal device 10B-K according to the third embodiment, the terminal device according to this modified example may control the transmittance of the virtual object VO according to the type of the real object RO, the first distance L1, and the second distance L2.

[0122] 5:Other (1) In the above-described embodiment, storage devices 12-12B and storage device 22 are exemplified by ROM and RAM, but may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0123] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0124] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0125] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0126] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0127] (6) Each function illustrated in Figures 1 to 10 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0128] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0129] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0130] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0131] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0132] (10) In the above-described embodiments, the terminal devices 10-1 to 10-J, 10A-1 to 10A-J, and 10B-1 to 10B-J, and the server 20 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. Furthermore, in the present disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0133] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements are considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0134] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0135] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0136] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0137] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the nouns following these articles being plural.

[0138] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0139] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0140] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0141] 1, 1A, 1B... Information processing system, 10, 10A, 10B... Terminal device, 11, 11A, 11B... Processing device, 12, 12A, 12B... Storage device, 13... Communication device, 14... Positioning device, 15... Display, 16... Input device, 17... Inertial sensor, 18... Imaging device, 20... Server, 21... Processing device, 22... Storage device, 23... Communication device, 24... Display, 25... Input Apparatus, 111... Generation section, 112... Display control section, 113... Calculation section, 114, 114A, 114B... Transmittance control section, 115... Output section, 116... Recognition section, 117... First acquisition section, 118... 2 Acquisition section, 211... Acquisition section, 212... Output section, C1, C2, C3, C4... Chair, L1... First distance, L2... Second distance, PR1, PR1A, PR1B, PR2... Control program, T1... Desk

Claims

1. a display control unit that causes a display device to display an image showing a virtual object, thereby allowing a user to recognize, via the display device, a virtual space in which the virtual object is superimposed on a real space; and a transmittance control unit that controls the transmittance of the virtual object in accordance with an occupancy rate of an area of ​​the display area of ​​the display device that the virtual object occupies, A display control device in which the occupancy rate and the transmittance are proportional to each other.

2. 2. The display control device according to claim 1, wherein the display control unit allows the user to recognize the virtual space by displaying on the display device a superimposed image in which an image representing the virtual object is superimposed on an image obtained by capturing an image of the outside world, or by displaying on the display device an image representing the virtual object in a situation in which light incident from the outside world passes through the display device.

3. an imaging device that captures an image of a real space in which a plurality of real objects exist; a recognition unit that recognizes the type of each of the plurality of real objects; a display control unit that causes a display device to display an image showing a virtual object, thereby allowing a user to recognize, via the display device, a virtual space in which the virtual object is superimposed on the real space; and a transmittance control unit that controls the transmittance of the virtual object in accordance with a combination of types of a plurality of real objects positioned behind the virtual object.

Images