Control device and control system

Abstract

To enable a user to grasp a position of an application image displayed in a bird's-eye view by executing an application in virtual space.SOLUTION: A control apparatus 10 that displays, on a display apparatus 15, an image showing virtual space VS according to a position of a user U inside a building, comprises: a generation unit 112 for generating a three-dimensional map TM representing the inside of the building in three dimensions; a management unit 113 for managing a first position and a second position in association with each other, wherein the first position where an application image W obtained by executing an application is placed in the virtual space VS, wherein the second position where a corresponding image I corresponding to the application is placed on the three-dimensional map TM; and a display control unit 115 for displaying, on the display apparatus 15, the application image W at the first position in the virtual space VS and displaying, on the display apparatus 15, the three-dimensional map TM on which the corresponding image I is placed at the second position in accordance with an operation by the user U.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a control device and a control system. [Background technology]

[0002] By using XR technologies such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), various virtual objects may be placed in various locations in the XR space, which can make it difficult for users experiencing the XR space to grasp the location of each virtual object.

[0003] For example, a game device described in Patent Document 1 generates a map object configured by a three-dimensional model corresponding to a three-dimensional virtual space. Player icons representing players are placed on this map object. That is, conventional game devices display a map object on which items appearing in the game and player icons are placed so that a user can visually recognize the game situation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-102066 Summary of the Invention [Problem to be solved by the invention]

[0005] However, because map objects provided by conventional game devices merely represent the virtual space of the game in three dimensions, conventional game devices were unable to display the relationship between the three-dimensional map associated with the real space and the virtual object. In particular, when the virtual object is an application image displayed by executing an application, it was not possible to associate the position of the application image virtually placed in the virtual space with the position on the three-dimensional map and manage it in association with the real space.

[0006] Therefore, an object of the present invention is to provide a control device and a control system that, when a virtual object is an app image displayed by executing an application, can associate the position of the app image virtually placed in a virtual space with the position on a three-dimensional map and manage the position. [Means for solving the problem]

[0007] A preferred embodiment of the present invention provides a control device that causes a display device to display an image showing a virtual space corresponding to a user's position inside a building, and includes: a generation unit that generates a three-dimensional map that represents the interior of the building in three dimensions; a management unit that associates and manages a first position where an app image obtained by executing an application is placed in the virtual space and a second position where a corresponding image corresponding to the application is placed on the three-dimensional map; and a display control unit that causes the display device to display the app image at the first position in the virtual space and the three-dimensional map where the corresponding image is placed at the second position in response to the user's operation. [Effects of the Invention]

[0008] According to the present invention, when a virtual object is an application image displayed by executing an application, it is possible to associate the position where the application image is virtually placed in the virtual space with the real space and manage the position on the three-dimensional map. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of a control system 1. FIG. [Figure 2] FIG. 2 is a diagram showing an example of a three-dimensional map TM. [Figure 3] FIG. 10 is a diagram showing an example of an image showing a virtual space VS visually recognized by a user UJ. [Figure 4] Block diagram showing an example configuration of XR Glasses 20-J. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 10-J. [Figure 6] 10 is a table showing an example of a correspondence relationship table TB1. [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of a server 30. [Figure 8] 4 is a flowchart showing the operation of a terminal device 10-J included in the control system 1. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1: First embodiment Hereinafter, the configuration of a control system 1 including a terminal device 10 as a control device according to a first embodiment of the present invention will be described with reference to FIGS.

[0011] 1-1: Configuration of the first embodiment 1-1-1: Overall structure FIG. 1 is a diagram showing the overall configuration of a control system 1. As shown in FIG. 1, the control system 1 includes terminal devices 10-1, 10-2, ... 10-J, ... 10-K, ... 10-N, XR glasses 20-J, and a server 30. N is an integer equal to or greater than 1. J is an integer equal to or greater than 1 and equal to or less than N. K is an integer equal to or greater than 1 and equal to or less than N, and is different from J. In this embodiment, the terminal devices 10-1 to 10-N have the same configuration. However, terminal devices with different configurations may be included. Note that hereinafter, the terminal devices 10-1 to 10-N may be collectively referred to as "terminal device 10."

[0012] In the control system 1, the terminal devices 10-1, 10-2, ... 10-J, ... 10-K, ... 10-N and the server 30 are connected to each other so as to be able to communicate with each other via a communication network NET. Also, the terminal device 10-J and the XR glasses 20-J are connected to each other so as to be able to communicate with each other. In FIG. 1, the user U J It is assumed that a user U uses a combination of a terminal device 10-J and XR glasses 20-J. K It is assumed that the user U uses the terminal device 10-K. K Instead of the display 15 provided in the terminal device 10-K, which will be described later, the user U J In addition, in FIG. 1, among the terminal devices 10-1 to 10-N, XR glasses similar to the XR glasses 20-J may or may not be connected to the terminal devices 10 other than the terminal device 10-J and the terminal device 10-K. J and user U K General users including the above will be collectively referred to as "user U."

[0013] The server 30 provides various data and cloud services to the terminal devices 10-1 to 10-N via the communication network NET. In particular, the server 30 provides various contents to be displayed in the XR space to the terminal devices 10-1 to 10-N.

[0014] The terminal devices 10-1 to 10-N are connected to a display 15 (to be described later) provided in the terminal devices 10-1 to 10-N, and a terminal device 10-J. JThe system displays a virtual space including virtual objects on the XR glasses 20-J worn on the head of a user. Examples of the virtual objects include virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, as well as virtual objects representing applications. Examples of text files include memos and source code. Examples of applications include browsers, applications for using SNS, and applications for generating document files. Preferably, the terminal devices 10-1 to 10-N are, for example, mobile terminal devices such as smartphones and tablets.

[0015] XR Glasses 20-J is a user J It is a see-through wearable display that is attached to the head of the user U. J The user can view the outside world through the XR glasses 20-J. The XR glasses 20-J are controlled by the terminal device 10-J to display virtual objects on display panels provided in each of the binocular lenses. In addition, the following describes an example in which the XR glasses 20-J are MR glasses. However, the XR glasses 20-J being MR glasses is merely an example, and the XR glasses 20-J may also be VR glasses or AR glasses.

[0016] In particular, in this embodiment, a user U wearing the XR glasses 20-J J is a user U inside the building. J The user views an image showing a virtual space corresponding to the user's position on the XR glasses 20-J. Furthermore, the XR glasses 20-J display a three-dimensional map that shows the interior of the building in three dimensions.

[0017] FIG. 2 is a diagram showing an example of a three-dimensional map TM. FIG. J 10 is a diagram showing an example of an image showing a virtual space VS visually recognized by a user.

[0018] As shown in FIG. 2, the 3D map TM isJ 2 is a three-dimensional map that represents in three dimensions the interior of a building in which a first bedroom B1, a second bedroom B2, a living room L1, and an entrance E1 are located. In the three-dimensional map TM illustrated in FIG. 2, the building includes a first bedroom B1, a second bedroom B2, a living room L1, and an entrance E1. An icon I1 is displayed in the space corresponding to the first bedroom B1. The icon I1 is, for example, an icon corresponding to a video application. The icon I1 indicates that an app image obtained by executing the video application is located at a position corresponding to the first bedroom B1 in a virtual space VS that is superimposed on the real space inside the building. Here, the position in the virtual space VS where the app image obtained by executing the video application is located is an example of a "first position." In addition, the position in the three-dimensional map TM where the icon I1 is located is an example of a "second position."

[0019] Similarly, an icon I2 is displayed in the space corresponding to the living room L1. As an example, the icon I2 is an icon corresponding to a map application. The icon I2 indicates that an app image obtained by executing the map application is placed at a position corresponding to the living room L1 in the virtual space VS superimposed on the real space inside the building. Here, the position in the virtual space VS where the app image obtained by executing the map application is placed is an example of a "first position." Furthermore, the position in the 3D map TM where the icon I2 is placed is an example of a "second position."

[0020] Similarly, an icon I3 is displayed in the space corresponding to the entrance E1. As an example, the icon I3 is an icon corresponding to a ToDo list application. The icon I3 indicates that an app image obtained by executing the ToDo list application is placed at a position corresponding to the entrance E1 in the virtual space VS superimposed on the real space inside the building. Here, the position in the virtual space VS where the app image obtained by executing the ToDo list application is placed is an example of a "first position." Furthermore, the position in the 3D map TM where the icon I3 is placed is an example of a "second position."

[0021] As shown in Figure 3, user U J By operating the user, the user U displays the 3D map TM in the virtual space VS that he / she is viewing. J However, when the user extracts an icon I1 corresponding to a video application from the 3D map TM, an application image W1 obtained by executing the video application corresponding to the icon I1 is displayed in the virtual space VS. J Similarly, user U J However, when the user extracts an icon I2 corresponding to a map application from the 3D map TM, an application image W2 obtained by executing the map application corresponding to the icon I2 is displayed in the virtual space VS. J It is displayed in front of the user U. J However, when the user extracts an icon I3 corresponding to the ToDo list application from the 3D map TM, the application image W3 obtained by executing the ToDo list application corresponding to the icon I3 in the virtual space VS is displayed on the screen of the user U. J It will be displayed right in front of you.

[0022] The icons I1 to I3 are examples of "corresponding images." In the following, general icons including the icons I1 to I3 of this embodiment may be collectively referred to as "icon I." In the following, general application images including the application images W1 to W3 of this embodiment may be collectively referred to as "application image W."

[0023] Here, user U J However, for example, when the icon I2 is extracted from the 3D map TM in the first bedroom B1, that is, when the icon I2 is moved from the 3D map TM to a position corresponding to the first bedroom B1 in the virtual space VS superimposed on the real space inside the building, the app image W is placed at that position. This position is an example of the "third position." Note that the "third position" may be the same position as the above-mentioned "first position."

[0024] In particular, if the first bedroom B1 does not correspond to the second position where the icon I2 was originally placed on the 3D map TM, the icon I2 is relocated to a position corresponding to the first bedroom B1. The position where the icon I2 is relocated is an example of the "fourth position." Note that the "fourth position" may be the same position as the above-mentioned "second position."

[0025] Furthermore, user U J After stopping use of the map application, when the application image W2 of the map application is returned to the 3D map TM, an icon I2 corresponding to the map application is placed on the 3D map TM.

[0026] Here, user U J However, for example, if the app image W2 is moved to a space on the 3D map TM that corresponds to the second bedroom B2, the position to which the app image W2 is moved relative to the 3D map TM, and the position where the icon I2 is placed on the 3D map TM, is an example of the “fifth position.” Note that the “fifth position” may be the same position as at least one of the above-mentioned “first position” and “third position.”

[0027] In this case, the app image W2 is placed in a position corresponding to the second bedroom B2 in the virtual space VS superimposed on the real space inside the building. The position in the virtual space VS where the app image W2 is placed is an example of the "sixth position." Note that the "sixth position" may be the same position as at least one of the "second position" and "fourth position."

[0028] In addition, User U J is user U J and one or more other users U located within the same building. K The 3D map TM can be shared with users U. J is another user U K and user U J Among the application images W1 to W3 visually recognized by the user, one or more application images W can be shared.

[0029] As a result, for example, user U J However, if a user starts a ToDo list application in the first bedroom B1, creates a ToDo list, and places the created ToDo list in the space corresponding to the entrance E1 on the 3D map TM, other users U K is a position corresponding to the entrance E1 in the virtual space VS, J You can check the ToDo list created by

[0030] 1-1-2: XR Glasses Configuration 4 is a block diagram showing an example configuration of the XR glasses 20-J. The XR glasses 20-J include a processing device 21, a storage device 22, a gaze detection device 23, a GPS device 24, a motion detection device 25, a distance measurement device 26, an imaging device 27, a communication device 28, and a display 29. The elements of the XR glasses 20-J are connected to each other by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, or unit.

[0031] The processing device 21 is a processor that controls the entire XR glasses 20-J. The processing device 21 is configured using, for example, one or more chips. The processing device 21 is also configured using, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 21 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 21 executes various processes in parallel or sequentially.

[0032] The storage device 22 is a recording medium that can be read from and written to by the processing device 21. The storage device 22 also stores a plurality of programs including a control program PR2 that the processing device 21 executes.

[0033] The gaze detection device 23 detects the gaze of the user U J The gaze detection device 23 may detect the gaze of the user U and generate gaze information indicating the detection result. Any method may be used to detect the gaze of the user U. The gaze detection device 23 may detect the gaze information based on the position of the inner corner of the eye and the position of the iris, for example. J The gaze detection device 23 outputs the gaze information to a processing device 31, which will be described later. The gaze information output to the processing device 21 is then output to the terminal device 10-J via the communication device 28.

[0034] The GPS device 24 receives radio waves from multiple satellites. The GPS device 24 also generates location information from the received radio waves. The location information indicates the location of the XR glasses 20-J. The location information may be in any format as long as it can identify the location. The location information indicates, for example, the latitude and longitude of the XR glasses 20-J. As an example, the location information is obtained from the GPS device 24. However, the XR glasses 20-J may obtain the location information by any method. The obtained location information is output to the processing device 21. The location information output to the processing device 21 is output to the terminal device 10-J via the communication device 28.

[0035] The motion detection device 25 detects the motion of the XR glasses 20-J. Examples of the motion detection device 25 include inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration. The acceleration sensor detects acceleration on orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as central axes of rotation. The motion detection device 25 can generate attitude information indicating the attitude of the XR glasses 20-J based on the output information of the gyro sensor. The motion detection device 25 can also generate motion information indicating the motion of the XR glasses 20-J based on the output information of the acceleration sensor and the gyro sensor. The motion information includes acceleration data indicating the acceleration on each of the three axes and angular acceleration data indicating the angular acceleration on each of the three axes. The motion detection device 25 outputs the motion information related to the motion of the XR glasses 20-J and the attitude information indicating the attitude of the XR glasses 20-J to the processing device 21. The movement information and posture information output to the processing device 21 is output to the terminal device 10-J via the communication device 28.

[0036] The distance measuring device 26 measures the distance between the XR glasses 20-J and an object around the XR glasses 20-J and outputs the obtained distance information to the processing device 21. The distance measuring device 26 includes, for example, a LIDAR (Light Detection and Ranging) sensor.

[0037] The imaging device 27 outputs imaging information obtained by capturing an image of the outside world. The imaging device 27 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 21. The imaging information output to the processing device 21 is output to the terminal device 10-J via the communication device 28.

[0038] The communication device 28 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 28 is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 28 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 28 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0039] The display 29 is a device that displays images. The display 29 displays various images under the control of the processing device 21. The display 29 includes a lens for the left eye, a display panel for the left eye, and optical members for the left eye, as well as a lens for the right eye, a display panel for the right eye, and optical members for the right eye. As the display panel, various display panels such as a liquid crystal display panel and an organic EL display panel are preferably used. The display 29 is an example of a "display device."

[0040] The processing device 21 functions as an acquisition unit 211, a generation unit 212, an output unit 213, and a display control unit 214, for example, by reading out a control program PR2 from the storage device 22 and executing it.

[0041] The acquisition unit 211 acquires image information indicating an image to be displayed on the XR glasses 20-J from the terminal device 10-J via the communication device 28. The acquisition unit 211 also acquires various control signals from the terminal device 10-J via the communication device 28.

[0042] In addition, the acquisition unit 211 acquires gaze information input from the gaze detection device 23, position information input from the GPS device 24, movement information and posture information input from the movement detection device 25, distance information input from the distance measuring device 26, and imaging information input from the imaging device 27.

[0043] The generation unit 212 generates a user U using at least the movement information, posture information, and distance information acquired by the acquisition unit 211. J The system generates structural information that represents the interior of the building in three dimensions.

[0044] Specifically, user U J The user U wears the XR Glasses 20-J on their head. J The user U moves around the building while keeping all the main points of the building in view. J The generation unit 212 continues to scan the interior of the building as the user U moves. J The structure information is generated by associating the movement information, posture information, and distance information at each point in time during the movement of the object with each other.

[0045] The output unit 213 outputs the gaze information, position information, movement information, posture information, and imaging information acquired by the acquisition unit 211, and the structural information generated by the generation unit 212 to the terminal device 10-J via the communication device 28.

[0046] The display control unit 214 causes the display 29 to display an image indicated by the image information, based on the image information acquired from the terminal device 10-J by the acquisition unit 211. In particular, in this embodiment, the display control unit 214 causes the display 29 to display an image indicating a virtual space VS including a 3D map TM and application images W1 to W3, as exemplified in FIGS.

[0047] 1-1-3: Terminal device configuration 5 is a block diagram showing an example configuration of the terminal device 10-J. The terminal device 10-J includes a processing device 11, a storage device 12, a communication device 13, an imaging device 14, a display 15, an input device 16, and an inertial sensor 17. The elements of the terminal device 10-J are connected to each other by one or more buses for communicating information.

[0048] The processing device 11 is a processor that controls the entire terminal device 10-J. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, ASIC, PLD, FPGA, etc. The processing device 11 executes various processes in parallel or sequentially.

[0049] The storage device 12 is a recording medium that can be read from and written to by the processing device 11. The storage device 12 also stores a plurality of programs including a control program PR1 that the processing device 11 executes.

[0050] Furthermore, the storage device 12 stores a correspondence table TB1. FIG. 6 is a table illustrating an example of the correspondence table TB1. As illustrated in FIG. 6, a correspondence ID, an application image W, position information of the application image W in the virtual space VS, an icon I, and position information of the icon I in the 3D map TM are stored in pairs in the correspondence table TB1. For example, as a data set of correspondence ID=1, an application image W1, coordinate values ​​(x1, y1, z1) in the virtual space VS which are position information of the application image W1 in the virtual space VS, an icon I1, and coordinate values ​​(p1, q1, r1) in the 3D map TM which are position information of the icon I1 in the 3D map TM are stored in pairs in the correspondence table TB1. Note that U sets of data sets with correspondence ID=1 to U are stored in the correspondence table TB1 illustrated in FIG. 5. However, storing U sets of data sets in the correspondence table TB1 is merely an example, and any number of sets of data sets may be stored in the correspondence table TB1.

[0051] Returning to the explanation of FIG. 5, the communication device 13 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0052] The imaging device 14 outputs imaging information obtained by capturing an image of the outside world. The imaging device 14 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 21. The imaging information output to the processing device 11 is output to the server 30 via the communication device 13. Note that when XR glasses 20-J are connected to the terminal device 10-J, the imaging device 14 does not need to be an essential component. In this case, the XR glasses 20-J will further have the same functions as the imaging device 14.

[0053] The display 15 is a device that displays images and text information. The display 15 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 15. Note that when XR glasses 20-J are connected to the terminal device 10-J, the display 15 does not need to be an essential component. In this case, the XR glasses 20-J will further have the same functions as the display 15.

[0054] The input device 16 is a device for inputting information to a user U wearing XR glasses 20-J on his / her head. J For example, the input device 16 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. If the input device 16 includes a touch panel, it may also function as the display 15.

[0055] The inertial sensor 17 is a sensor that detects inertial force. The inertial sensor 17 includes, for example, one or more sensors selected from an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 11 detects the attitude of the terminal device 10-J based on the output information of the inertial sensor 17. Furthermore, the processing device 11 accepts the selection of a virtual object, the input of characters, and the input of instructions in the virtual space VS based on the attitude of the terminal device 10-J. For example, when a user U J By operating the input device 16 while the central axis of the terminal device 10-J is directed to a predetermined area in the virtual space VS, the user U selects a virtual object to be placed in the predetermined area. J The operation is, for example, a double tap. J By operating the terminal device 10-J, the user can select a virtual object without looking at the input device 16 provided on the terminal device 10-J.

[0056] The processing device 11 reads out the control program PR1 from the storage device 12 and executes it, thereby functioning as an acquisition unit 111, a generation unit 112, a management unit 113, a reception unit 114, a display control unit 115, and an output unit .

[0057] The acquisition unit 111 acquires gaze information, position information, movement information, posture information, imaging information, and structural information from the XR glasses 20-J via the communication device 13. The acquisition unit 111 also acquires image information indicating a virtual object to be displayed on the XR glasses 20-J or the display 15 from the server 30 via the communication device 13. The acquisition unit 111 also acquires various control signals from the server 30 via the communication device 13.

[0058] The generation unit 112 generates a user U based on the gaze information, position information, movement information, posture information, imaging information, and image information acquired by the acquisition unit 111. J An image showing a virtual space VS that corresponds to the position of the virtual object and includes the virtual object is generated.

[0059] In particular, the generation unit 112 generates a user U based on the structural information acquired by the acquisition unit 111. J The generation unit 112 generates a 3D map TM that represents in three dimensions the interior of the building in which the user is located. Furthermore, the generation unit 112 links the 3D map TM with position information within the building based on at least the movement information and posture information acquired by the acquisition unit 111. An example of the 3D map TM generated by the generation unit 112 is illustrated in FIG. 2, as described above.

[0060] The management unit 113 manages the user U J The position where an application image W obtained by executing the application is placed in the virtual space VS and the position where an icon I corresponding to the application is placed on the 3D map TM are managed in association with each other. Here, the position where the application image W is placed in the virtual space VS is an example of the above-mentioned "first position." On the other hand, the position where the icon I corresponding to the application is placed on the 3D map TM is an example of the above-mentioned "second position."

[0061] The management unit 113 stores, for example, the position information of the application image W in the local coordinate system set in the virtual space VS and the position information of the icon I in the three-dimensional coordinate system set on the three-dimensional map TM as a pair in a correspondence relationship table TB1 stored in the storage device 12.

[0062] As a result, the first position where the application image W obtained by executing the application is placed in the virtual space VS is associated with the second position where the icon I corresponding to the application is placed on the three-dimensional map TM. J By simply viewing the three-dimensional map TM, the user can grasp the position in the virtual space VS where the application image W is located.

[0063] The reception unit 114 receives the user U J For example, the accepting unit 114 accepts an operation by a user U J The input contents inputted through the input device 16 by the user U JAlternatively, the accepting unit 114 may accept the operation as an operation by the user U. J When the user U moves the terminal device 10-J, the user U moves the terminal device 10-J based on the attitude of the terminal device 10-J detected by the inertial sensor 17. J Here, user U J The operations performed by the touch panel include, for example, selecting a virtual object, inputting characters, and inputting instructions.

[0064] In particular, in this embodiment, the receiving unit 114 receives a command from a user U to move the icon I on the 3D map TM to the above-mentioned "third position" in the virtual space VS while the 3D map TM on which the icon I is placed is displayed on the XR glasses 20-J. J In this case, the management unit 113 places the application image W at a third position in the virtual space VS, and places the icon I at the above-mentioned "fourth position" on the 3D map TM corresponding to the third position.

[0065] As a result, user U J However, it is possible to extract and use the application image W at a third position in the virtual space VS, which does not correspond to the position where the icon I was originally placed on the 3D map TM. J In this case, the user U can rearrange the icon I to a fourth position corresponding to the third position on the three-dimensional map TM. J You can continue to use the application in a location different from where you originally used the application.

[0066] In this embodiment, the receiving unit 114 also receives a request from the user U to move the application image W placed at the third position in the virtual space VS to the above-mentioned "fifth position" on the three-dimensional map TM. J In this case, the management unit 113 places the icon I at the fifth position on the 3D map TM, and places the application image W at the "sixth position" in the virtual space VS corresponding to the fifth position.

[0067] As a result, user UJ The user U can place the icon I at a fifth position on the three-dimensional map TM, which corresponds to a position in the virtual space VS that is different from the position where the application image W is currently being used. J In this case, the user U can rearrange the application image W to a sixth position in the virtual space VS, which corresponds to the fifth position. J You can continue to use the application in a location different from where you are currently using the application.

[0068] The display control unit 115 causes the XR glasses 20-J or the display 15 to display the image generated by the generation unit 112, which shows the virtual space VS including the virtual object.

[0069] In particular, in this embodiment, the display control unit 115 detects the user U J The virtual space VS in which the application image W is arranged is displayed on the XR glasses 20-J or the display 15 according to the position of the application image W. As a result, the user U J can recognize an augmented reality space or mixed reality space in which a virtual space VS is superimposed on the real space inside the building.

[0070] In this embodiment, the display control unit 115 also J In response to the operation by the user U, the display control unit 115 causes the XR glasses 20-J or the display 15 to display the application image W at a first position in the virtual space VS. J In response to the operation by the user, the XR glasses 20-J or the display 15 are caused to display a 3D map TM in which the icon I is placed at the second position.

[0071] Similarly, the display control unit 115 J In response to the operation by the user U, the display control unit 115 causes the XR glasses 20-J or the display 15 to display the application image W at a third position in the virtual space VS. JIn response to the operation by the user, the XR glasses 20-J or the display 15 are caused to display a 3D map TM in which the icon I is placed at the fourth position.

[0072] Similarly, the display control unit 115 J In response to the operation by the user U, the XR glasses 20-J or the display 15 displays the application image W at a fifth position in the virtual space VS. J In response to the operation by the user, the XR glasses 20-J or the display 15 are caused to display a 3D map TM in which the icon I is placed at the sixth position.

[0073] The output unit 116 outputs the gaze information, position information, movement information, posture information, imaging information, and structure information acquired by the acquisition unit 111 to the server 30 via the communication device 13. In addition, the output unit 116 acquires various control signals for the terminal device 10-J via the communication device 13.

[0074] 1-1-4: Server configuration 7 is a block diagram showing an example of the configuration of the server 30. The server 30 includes a processing device 31, a storage device 32, a communication device 33, a display 34, and an input device 35. The elements of the server 30 are connected to each other by one or more buses for communicating information.

[0075] The processing device 31 is a processor that controls the entire server 30. The processing device 31 is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, ASIC, PLD, or FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0076] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a plurality of programs including a control program PR3 that the processing device 31 executes.

[0077] The communication device 33 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0078] The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing device 31. For example, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used as the display 34.

[0079] The input device 35 is a device that accepts operations by an administrator of the server 30. For example, the input device 35 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 35 includes a touch panel, it may also serve as the display 34.

[0080] The processing device 31 functions as an acquisition unit 311 and an output unit 312, for example, by reading out a control program PR3 from the storage device 32 and executing it.

[0081] The acquisition unit 311 acquires, via the communication device 33, gaze information, position information, movement information, posture information, imaging information, and structure information from the terminal device 10-J.

[0082] The output unit 312 outputs image information indicating a virtual object to be displayed on the XR glasses 20-J or the display 15 to the terminal device 10-J via the communication device 33. The output unit 312 also outputs various control signals to the terminal device 10-J via the communication device 33.

[0083] In particular, the output unit 312 outputs image information indicating a virtual object to the terminal device 10-J in accordance with at least one of the gaze information, position information, movement information, posture information, imaging information, and structural information acquired by the acquisition unit 311.

[0084] 1-2: Operation of the first embodiment FIG. 8 is a flowchart showing the operation of the terminal device 10-J included in the control system 1 according to the first embodiment. The operation of the terminal device 10-J will be described below with reference to FIG. 8. Note that the flowchart shown in FIG. 8 is based on the premise that a 3D map TM has already been generated. Also, it is assumed that an application image W is placed at a "first position" in the virtual space VS, and an icon I corresponding to the application image W is placed at a "second position" on the 3D map TM.

[0085] In step S1, the processing device 11 provided in the terminal device 10-J functions as the display control unit 115. The processing device 11 causes the XR glasses 20-J or the display 15 to display a 3D map TM.

[0086] In step S2, the processing device 11 provided in the terminal device 10-J functions as the reception unit 114. If the processing device 11 has received an operation to move the icon I from the 3D map TM, that is, if the answer is YES in step S2, the processing device 11 executes the process of step S3. If the processing device 11 has not received an operation to move the icon I from the 3D map TM, that is, if the answer is NO in step S2, the processing device 11 executes the process of step S5.

[0087] In step S3, the processing device 11 included in the terminal device 10-J functions as the management unit 113. The processing device 11 places the app image W corresponding to the icon I that is the operation target in step S2 at a "third position" in the virtual space VS. Note that the "third position" may be the same position as the above-mentioned "first position."

[0088] In step S4, the processing device 11 included in the terminal device 10-J functions as the management unit 113. The processing device 11 places the icon I at a "fourth position" on the 3D map TM corresponding to the "third position" in step S3. Note that the "fourth position" may be the same position as the above-mentioned "second position."

[0089] In step S5, the processing device 11 included in the terminal device 10-J functions as the reception unit 114. If the processing device 11 has received an operation to move the application image W onto the 3D map TM, that is, if the answer is YES in step S5, the processing device 11 executes the process of step S6. If the processing device 11 has not received an operation to move the application image W onto the 3D map TM, that is, if the answer is NO in step S5, the processing device 11 executes the process of step S1.

[0090] In step S6, the processing device 11 included in the terminal device 10-J functions as the management unit 113. The processing device 11 places the icon I corresponding to the application image W that is the operation target in step S5 at a "fifth position" on the 3D map TM. Note that the "fifth position" may be the same position as the above-mentioned "first position."

[0091] In step S7, the processing device 11 included in the terminal device 10-J functions as the management unit 113. The processing device 11 places the application image W at a "sixth position" in the virtual space VS corresponding to the "fifth position" in step S6. Note that the "sixth position" may be the same position as the above-mentioned "second position."

[0092] 1-3: Effects of the First Embodiment As described above, the terminal device 10 as a control device according to this embodiment is a control device that displays an image showing a virtual space VS corresponding to the position of a user U inside a building on a display 15 or 29 as a display device. The terminal device 10 includes a generation unit 112, a management unit 113, and a display control unit 115. The generation unit 112 generates a 3D map TM that represents the interior of the building in three dimensions. The management unit 113 manages a first position where an application image W obtained by executing an application is placed in the virtual space VS and a second position where an icon I as a corresponding image corresponding to the application is placed on the 3D map TM, in association with each other. The display control unit 115 displays the application image W at the first position in the virtual space VS on the display 15 or 29, and also displays a 3D map TM on the display 15 or 29 in which the icon I is placed at the second position in response to an operation by the user U.

[0093] Since the terminal device 10 has the above configuration, when the virtual object is an application image displayed by executing an application, it is possible to associate the position where the application image W is virtually placed in the virtual space with the real space and manage the position on the 3D map TM.

[0094] In particular, since a first position where an application image W obtained by executing an application is placed in the virtual space VS corresponds to a second position where an icon I corresponding to the application is placed on the three-dimensional map TM, the user U can grasp the position in the virtual space VS where the application image W is placed simply by looking at the three-dimensional map TM.

[0095] Furthermore, according to the above description, in the terminal device 10, the display 15 or 29 serving as a display device is a see-through type that allows the user U to view the outside world. The terminal device 10 further includes an acquisition unit 111. The acquisition unit 111 acquires structural information that represents the interior of the building in three dimensions. The generation unit 112 generates a 3D map TM based on the structural information. The display control unit 115 causes the display 15 or 29 to display a virtual space VS in which an app image W is arranged according to the position of the user U inside the building, thereby allowing the user U to recognize an augmented reality space or mixed reality space in which the virtual space VS is superimposed on the real space inside the building.

[0096] Since the terminal device 10 has the above-described configuration, the user U can grasp where the application image W is located in the real space on which the virtual space VS is superimposed, simply by viewing the three-dimensional map TM.

[0097] According to the above description, the terminal device 10 further includes a reception unit 114. The reception unit 114 receives an operation by the user U to move the icon I on the 3D map TM to a third position in the virtual space VS while the 3D map TM, on which the icon I is placed as a corresponding image, is displayed on the display 15 or 29 as a display device. The management unit 113 places the app image W at the third position in the virtual space VS, and places the icon I as a corresponding image at a fourth position on the 3D map TM that corresponds to the third position.

[0098] Since the terminal device 10 has the above configuration, the user U can extract and use the application image W at a third position in the virtual space VS, which does not correspond to the position where the icon I was originally placed on the 3D map TM. Furthermore, the user U can also rearrange the icon I on the 3D map TM to a fourth position corresponding to the third position. In this case, the user U J You can continue to use the application in a location different from where you originally used the application.

[0099] According to the above description, in the terminal device 10, the reception unit 114 receives an operation by the user U to move the application image W placed at a third position in the virtual space VS to a fifth position on the 3D map TM. The management unit 113 places an icon I as a corresponding image at the fifth position on the 3D map TM, and places the application image W at a sixth position in the virtual space VS corresponding to the fifth position.

[0100] Since the terminal device 10 has the above configuration, the user U can place the icon I at a fifth position on the 3D map TM, which corresponds to a position different from the position where the application image W is currently being used, in the virtual space VS. Furthermore, the user U can re-place the application image W at a sixth position in the virtual space VS, which corresponds to the fifth position. In this case, the user U J You can continue to use the application in a location different from where you are currently using the application.

[0101] Furthermore, according to the above description, the control system 1 according to this embodiment includes a plurality of terminal devices 10. The plurality of terminal devices 10 share the 3D map TM and the application image W in the virtual space VS.

[0102] The control system 1 has the above configuration, so that the 3D map TM and the application image W can be shared among a plurality of users U who each use the terminal device 10.

[0103] 2: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected arbitrarily from the following examples may be combined.

[0104] 2-1: Variation 1 In the above embodiment, icons I1 to I3 are exemplified as corresponding images corresponding to applications, but the corresponding images are not limited to icons I1 to I3. For example, the corresponding images may be thumbnail images of application images W that are displayed when the application is executed.

[0105] 2-2: Variation 2 In the above embodiment, an aspect in which the XR glasses 20-J are connected to the terminal device 10-J has been described, but the device connected to the terminal device 10-J is not limited to the XR glasses 20-J. For example, instead of the XR glasses 20-J, a video see-through head mounted display (HMD) using VR technology, AR technology, or MR technology may be connected to the terminal device 10-J.

[0106] 2-3: Variation 3 In the above embodiment, the terminal device 10-J and the XR glasses 20-J are realized as separate entities, but they may be included in the same housing. That is, the control system 1 according to the above embodiment may include a user device in which the terminal device 10-J and the XR glasses 20-J are included in a single housing, instead of the separate terminal device 10-J and the XR glasses 20-J.

[0107] 2-4: Variation 4 In the above embodiment, the generation unit 212 provided in the XR glasses 20-J generates the structural information, but the generation unit 112 provided in the terminal device 10-J may generate the structural information instead of the generation unit 212. In this case, the acquisition unit 111 provided in the terminal device 10-J acquires the above-mentioned distance information in addition to the line of sight information, position information, movement information, and imaging information from the XR glasses 20-J. The generation unit 112 generates the structural information for the user U. J The structure information is generated by associating the movement information, posture information, and distance information at each point in time during the movement of the object with each other.

[0108] 3:Other (1) In the above-described embodiment, the terminal device 10, the XR glasses 20-J, and the server 30 are exemplified. However, the storage devices provided in the terminal device 10, the XR glasses 20-J, and the server 30 may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) discs), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. Furthermore, the programs executed by the external devices may be transmitted from a network via telecommunications lines. Furthermore, the programs may be transmitted from a communications network NET via telecommunications lines.

[0109] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0110] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0111] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0112] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0113] (6) Each function illustrated in Figures 1 to 8 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically coupled, or may be realized by using two or more devices that are physically or logically separated and connected directly or indirectly (for example, by wire, wirelessly, etc.). A functional block may also be realized by combining software with the one device or the multiple devices.

[0114] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0115] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0116] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0117] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0118] (10) In the above-described embodiments, the terminal device 10 and the server 30 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0119] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0120] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0121] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0122] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0123] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0124] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0125] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0126] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0127] 1...control system, 10...terminal device, 11...processing device, 12...storage device, 13...communication device, 14...imaging device, 15...display, 16...input device, 17...inertial sensor, 20...XR glasses, 21...processing device, 22...storage device, 23...gaze detection device, 24...GPS device, 25...motion detection device, 26...distance measuring device, 27...imaging device, 28...communication device, 29...display, 30...server, 31...processing device, 32...storage device, 33...communication device, 34...display, 3 5...input device, 36...imaging device, 111...acquisition unit, 112...generation unit, 113...management unit, 114...reception unit, 115...display control unit, 116...output unit, 211...acquisition unit, 212...generation unit, 213...output unit, 214...display control unit, 311...acquisition unit, 312...output unit, B1...first bedroom, B2...second bedroom, E1...entrance, I1, I2, I3...icon, L1...living room, PR1, PR2, PR3...control program, TB1...correspondence table, W1, W2, W3...application image

Claims

1. A control device that displays an image showing a virtual space corresponding to a user's position inside a building on a display device, a generation unit that generates a three-dimensional map that represents the interior of the building in three dimensions; a management unit that manages a first position where an application image obtained by executing an application is placed in the virtual space and a second position where a corresponding image corresponding to the application is placed on the three-dimensional map, in association with each other; a display control unit that causes the display device to display the app image at a first position in the virtual space, and causes the display device to display the 3D map in which the corresponding image is arranged at the second position in response to an operation by the user; a receiving unit that receives, in a state in which a three-dimensional map on which the corresponding image is arranged is displayed on the display device, an operation by the user to move the corresponding image on the three-dimensional map to a third position in the virtual space; Equipped with the management unit places the application image at the third position in the virtual space, and places the corresponding image at a fourth position on the three-dimensional map corresponding to the third position; Control device.

2. the display device is a see-through type that allows the user to view the outside world, An acquisition unit that acquires structural information that represents the interior of the building in three dimensions, the generation unit generates the three-dimensional map based on the structural information; the display control unit causes the display device to display the virtual space in which the app image is arranged according to the user's position inside the building, thereby allowing the user to recognize an augmented reality space or a mixed reality space in which the virtual space is superimposed on the real space inside the building; The control device according to claim 1 .

3. the receiving unit receives an operation by the user to move the app image placed at the third position in the virtual space to a fifth position on the three-dimensional map; the management unit places the corresponding image at a fifth position on the three-dimensional map, and places the application image at a sixth position in the virtual space corresponding to the fifth position; The control device according to claim 1 .

4. A control system including a plurality of the control devices according to claim 1, A control system in which the plurality of control devices share the three-dimensional map and the application image in the virtual space.

Images