Interface system

Abstract

To provide an interface system for realizing a real space interface using a wall floor or the like by a space computing device.SOLUTION: The information processing apparatus includes a storage unit 28 that stores metadata including information regarding a shape of a real space and information regarding physical characteristics including a position and a shape of an object in the real space, a position acquisition unit 30 that acquires a position of the spatial computing device 12 in the real space, and a control unit 32 that performs control to display a virtual object indicating predetermined interface information on a surface of the object displayed on the display unit 20 on the basis of the position of the spatial computing device 12 and the physical characteristics of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an interface system between a wall or floor and a user. [Background technology]

[0002] The applicant of this patent has previously proposed technologies for using floors and walls in offices, public spaces, etc. as information interfaces to display guidance information on the floor according to the user's location and display operable applications on the wall (see, for example, Patent Documents 1 to 8). These technologies utilize the physical characteristics of the floor or wall and the positional relationship with the user to present information. In realizing these technologies, the applicant has considered projection mapping, which projects images from a projector to any position, and embedding display modules that display images in facilities (see Non-Patent Documents 1 and 2).

[0003] These technologies are characterized by the fact that they go beyond the size of conventional personal devices and signage, and use the entire walls and floors of large spaces as an interface for displaying information and operating the device.However, there was a problem in that installing devices such as projectors and display modules to meet this scale would be extremely costly.

[0004] Meanwhile, goggle-type headset device products have become known in recent years (see Non-Patent Documents 3 and 4). The device products in Non-Patent Documents 3 and 4 can be understood as realizing XR (Extended Reality) by applying conventional AR (Augmented Reality) technology, but are intended to display virtual objects based on real space through the concept of spatial computing. Spatial computing devices such as these devices are not simply immersive devices; they also enable multiple users to share physical space and virtual objects. Known technologies for placing virtual objects in physical space include those shown in Patent Documents 10 to 12, for example.

[0005] Meanwhile, the applicant of the present patent has already proposed a technology for adding annotations such as metadata to parts of a building that are relevant to its preservation while observing the object on-site (see Patent Document 9). The method described in Patent Document 9 includes the steps of acquiring three-dimensional data of the object, generating an image of the object based on the acquired three-dimensional data, adding annotations (metadata) to predetermined parts of the generated image by a user's operation using a pointing device or the like, and generating an annotated image with the annotations added, and projecting the generated annotated image onto the surface of the corresponding object using a projector or the like. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7374851 [Patent Document 2] Patent No. 7244279 [Patent Document 3] Patent No. 7409782 [Patent Document 4] Japanese Patent Publication No. 2022-67761 [Patent Document 5] Japanese Patent No. 7286302 [Patent Document 6] Japanese Patent No. 7329925 [Patent Document 7] Japanese Unexamined Patent Application Publication No. 2023 - 73301 [Patent Document 8] Japanese Patent No. 7162564 [Patent Document 9] Japanese Patent No. 7390978 [Patent Document 10] Specification of U.S. Patent No. 11348305 [Patent Document 11] Specification of U.S. Patent No. 11709370 [Patent Document 12] Specification of U.S. Patent Application Publication No. 2023 / 0324985 [Non - Patent Document]

[0007] [Non - Patent Document 1] ACM Digital Library, "Smart Elevator Hall: Prototype of In - building Guiding Using Interactive Floor Display", [online], [searched on June 24, Reiwa 6], Internet <URL:https: / / dl.acm.org / doi / abs / 10.1145 / 3380867.<3426207> [Non - Patent Document 2] ACM Digital Library, "Smart bus stop made with interactive surfaces", [online], [searched on June 24, Reiwa 6], Internet <URL:https: / / dl.acm.org / doi / 10.1145 / 3349263.<3349601> [Non - Patent Document 3] Apple homepage, "Apple Vision Pro", [online], [searched on May 20, Reiwa 6], Internet <URL:https: / / www.apple.com / apple - vision - pro> [Non-patent document 4] Apple's website, "Apple Vision Pro is here," [online], [searched May 20, 2024], Internet<URL:https: / / www.apple.com / jp / newsroom / 2023 / 06 / introducing-apple-vision-pro / > Summary of the Invention [Problem to be solved by the invention]

[0008] The inventors have considered using the spatial computing device to display virtual guidance and information on floors and walls. In this way, it is believed that it would be possible to realize a real-space interface on walls and floors, as shown in Patent Documents 1 to 8, without installing a physical display device such as a projector or display module.

[0009] To realize a real-space interface using a spatial computing device that utilizes walls, floors, etc., it is necessary to understand the attributes of the surfaces of the walls, floors, etc. In other words, the information that should be displayed at the feet (floor) based on the spatial position, the information that should be displayed at the hand (wall, tabletop, etc.) where operation is intended, and the information that should be displayed on a large screen in the distance (facade, etc.) should all have different characteristics.

[0010] Therefore, the inventor discovered that by providing components such as wall and floor surfaces with metadata containing physical characteristics and associating these physical characteristics with the information to be displayed, a real-space interface using walls, floors, etc. can be realized using a spatial computing device, leading to the present invention described below.

[0011] The present invention has been made in view of the above, and has as its object to provide an interface system for realizing a real-space interface using walls, floors, etc., by using a spatial computing device. [Means for solving the problem]

[0012] In order to solve the above-mentioned problems and achieve the object, the interface system of the present invention is an interface system for realizing a real-space interface using an object existing in a real space by a spatial computing device, wherein the spatial computing device has a virtual object generation means for generating a virtual object, an imaging means for imaging the real space, a display means for displaying an image captured by the imaging means and a virtual object generated by the virtual object generation means, and a device control means for controlling the virtual object generation means and the display means, and the interface system is characterized by having a storage means for retaining metadata having information regarding the shape of the real space and information regarding physical characteristics including the position and shape of the object in the real space, a position acquisition means for acquiring the position of the spatial computing device in the real space, and a control means for controlling the display of a virtual object showing predetermined interface information on the surface of the object displayed on the display means based on the position of the spatial computing device and the physical characteristics of the object.

[0013] Another interface system according to the present invention is characterized in that, in the above-mentioned invention, the storage means stores data of a user group composed of multiple users, and the control means displays a common virtual object on the display means of the spatial computing devices of multiple users belonging to the same user group, or displays a predetermined virtual object in a common range of the real space.

[0014] Furthermore, another interface system according to the present invention is characterized in that, in the above-described invention, the control means controls the display means of the spatial computing device in a predetermined area set in advance within the real space so as not to display a predetermined virtual object, or controls the display of the predetermined virtual object with restrictions.

[0015] Furthermore, another interface system according to the present invention is characterized in that, in the above-described invention, the spatial computing device has a detection means for detecting the user's line of sight and hands, and the device control means outputs operation information for performing a predetermined operation on a virtual object displayed by the display means based on detection by the detection means. [Effects of the Invention]

[0016] According to the interface system of the present invention, there is provided an interface system for realizing a real-space interface using an object existing in a real space by means of a spatial computing device, wherein the spatial computing device comprises: a virtual object generation means for generating a virtual object; an imaging means for imaging the real space; a display means for displaying an image captured by the imaging means and a virtual object generated by the virtual object generation means; and a device control means for controlling the virtual object generation means and the display means. The interface system comprises: a storage means for retaining metadata having information regarding the shape of the real space and information regarding physical characteristics including the position and shape of the object in the real space; a position acquisition means for acquiring the position of the spatial computing device in the real space; and a control means for controlling the display means to display a virtual object showing predetermined interface information on the surface of the object displayed on the display means, based on the position of the spatial computing device and the physical characteristics of the object. This has the effect of enabling a real-space interface using walls, floors, etc. to be realized at low cost without installing a physical display device such as a projector or display module.

[0017] In addition, according to another interface system of the present invention, the storage means stores data of a user group composed of multiple users, and the control means displays a common virtual object on the display means of the spatial computing devices of multiple users belonging to the same user group, or displays a predetermined virtual object in a common range of the real space, thereby achieving the effect of providing a dedicated real space interface for each user group.

[0018] Furthermore, according to another interface system of the present invention, the control means controls the display means of the spatial computing device in a predetermined area set in advance within the real space so as not to display a predetermined virtual object, or so as to display the predetermined virtual object with restrictions, thereby achieving the effect of being able to control the content of the virtual object to be displayed for each area.

[0019] Furthermore, according to another interface system of the present invention, the spatial computing device has a detection means for detecting the user's line of sight and hands, and the device control means outputs operation information for executing a predetermined operation on a virtual object displayed by the display means based on the detection by the detection means, thereby achieving the effect of being able to operate an interface corresponding to the virtual object based on the output operation information. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of an interface system according to an embodiment of the present invention. [Figure 2] FIG. 2 is an explanatory diagram showing an embodiment of an interface system according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] An embodiment of an interface system according to the present invention will be described in detail below with reference to the drawings. In this embodiment, the real space is assumed to be, for example, a space inside a building used by multiple people or a public space (hereinafter, collectively referred to as a "building space"). Possible examples of such spaces include stations (waiting areas, waiting rooms, concourses, etc.), airports, parks, commercial facilities, building entrances, sidewalks, museums, hospitals, public facilities, event venues, etc. However, the present invention is not limited to these embodiments.

[0022] As shown in FIG. 1, an interface system 10 according to an embodiment of the present invention is for realizing a real-space interface using walls, floors, and the like (objects) present in a space within a building (real space) using a spatial computing device 12. The interface system 10 and the spatial computing device 12 are connected to a metadata determination system 14 via a network N. The metadata determination system 14 determines and records metadata corresponding to the physical characteristics of the object. However, the present invention is not limited to this, and the interface system 10 or the spatial computing device 12 may include the metadata determination system 14, or the spatial computing device 12 or the metadata determination system 14 may also function as the interface system 10.

[0023] (Spatial Computing Device) The spatial computing device 12 is a goggle-type XR headset device worn in front of the eyes of the user's head, and includes a virtual object generation means 16, an imaging means 18, a display means 20, a device control means 22, a detection means 24, and a memory 26. These are interconnected via a bus. The spatial computing device 12 may be, for example, VISION PRO (registered trademark) from Apple Inc.

[0024] The virtual object generation means 16 generates virtual objects (for example, images or videos) that are objects in a virtual space that differs from the real space. For example, the virtual object generation means 16 reads template data of the virtual object stored in the memory 26, generates a virtual object that represents interface information based on a user operation or a control signal from the interface system 10, and stores the generated virtual object in the memory 26.

[0025] The imaging means 18 is installed in the spatial computing device 12 and captures images of the real space around the spatial computing device 12. The captured images and videos are stored in the memory 26 and displayed on the display means 20. The imaging means 18 may be, for example, a 360-degree spherical camera that can capture images of the entire celestial sphere, or may be multiple cameras arranged to capture images of different positions in the surroundings.

[0026] The display means 20 displays, to the user's eyes, an image of the building interior space captured by the imaging means 18, a virtual object generated by the virtual object generation means 16, various notification information and operation states for the user output via the device control means 22, and the like. The display means 20 can be configured, for example, with a display device such as an organic electroluminescence (EL) display. The display means 20 can display virtual objects superimposed on the image of the building interior space captured by the imaging means 18. For example, the display means 20 arranges and displays interface information consisting of virtual objects at positions where walls and floors exist on a celestial sphere image showing the entire surrounding scenery of the spatial computing device 12. The display means 20 may be a semi-transparent display device, in which case virtual objects, etc. may be displayed superimposed on a real space that can be seen through it. If the display means 20 is not semi-transparent, virtual objects, etc. may be displayed superimposed on an image of the real space captured by the imaging means 18.

[0027] The device control means 22 controls the virtual object generation means 16, the imaging means 18, the display means 20, and the detection means 24, and is composed of, for example, a CPU (Central Processing Unit). The device control means 22 controls the operation of each function of the spatial computing device 12 by executing programs such as an OS (Operating System) and software.

[0028] The detection means 24 is composed of an attitude sensor such as an inertial measurement sensor (IMU), a range sensor such as a LIDAR (Laser Detection and Ranging) sensor, and a line of sight / hand detection sensor such as a camera.

[0029] The attitude sensor measures the attitude and orientation of the spatial computing device 12 .

[0030] The range sensor measures the distance to an object (e.g., a wall or floor) by emitting laser light to the surrounding area and measuring the time difference between receiving the reflected light and receiving the laser light. The range sensor can also emit laser light to the surrounding area and obtain point cloud data of the object's three-dimensional coordinates from the reflected light. The device control means 22 can estimate the self-position of the spatial computing device 12 within the building space based on this point cloud data. This self-position estimation method can use, for example, a well-known position estimation method such as SLAM (Simultaneous Localization and Mapping). These methods can measure the self-position, attitude, and direction of the spatial computing device 12, as well as the distance to surrounding objects.

[0031] The gaze and hand detection sensor detects and constantly tracks the user's gaze and hands. Methods for detecting gaze and hands can utilize, for example, well-known techniques commonly used for eye tracking recognition processing using a camera or hand gesture recognition processing. Input operations are performed on the spatial computing device 12 through eye tracking detection and hand gesture detection by the gaze and hand detection sensor. Types of hand gestures include, for example, a tap operation in which the thumb and index finger are brought together, a double tap operation in which the user taps twice, a pinch and hold operation, a pinch and drag operation, a zoom operation, and a rotation operation.

[0032] Specifically, when a user selects a virtual object displayed by the display means 20 with their line of sight and operates it with hand or finger movements, the device control means 22 accepts an input operation for executing a predetermined operation on the virtual object based on detection by a line of sight / hand detection sensor, and transmits the operation information to the interface system 10. The interface system 10 generates control information corresponding to the received operation information and transmits it to the device control means 22. The device control means 22 performs control based on the received control information, allowing the user to operate the interface represented by the virtual object. Note that the spatial computing device 12 may be equipped with a microphone for inputting voice and a speaker for outputting voice. If equipped with a microphone, operation may be possible through voice input.

[0033] The memory 26 stores various programs and data such as virtual objects, images, and unique identification information for the spatial computing device 12 .

[0034] (Interface System) The interface system 10 comprises a storage means 28 , a position acquisition means 30 , and a control means 32 .

[0035] The storage means 28 stores metadata including information about the shape of the space inside the building and information about physical characteristics including the positions and shapes of components such as walls, floors, and desktops in the space inside the building. The metadata can be metadata recorded in the metadata determination system 14. The storage means 28 may also store parameters such as the distance and scale correspondence between these components and a person (such as proximity to a wall or settings within reach).

[0036] As described above, to realize a real-space interface using walls, floors, etc. using the spatial computing device 12, it is necessary to understand the attributes of the surfaces of the walls, floors, etc. In other words, the information to be displayed at the user's feet (floor) based on the spatial position, the information to be displayed at hand (wall, tabletop, etc.) where the user intends to operate, and the information to be displayed on a large screen (facade, etc.) in the distance should each have different characteristics. Therefore, metadata including physical characteristics is assigned to components such as walls and floors, and the information to be displayed is associated with the physical characteristics. This association is performed by the control means 32.

[0037] The idea of ​​assigning metadata to components is an application of the technology described in the above-mentioned Patent Document 9. When assigning metadata, components may be assigned metadata at the design stage of the components as in BIM (Building Information Modeling), or metadata may be assigned to components using a projector and pointing device as in the above-mentioned Patent Document 9, or metadata may be automatically identified and assigned using the imaging means 18 and detection means 24 and software provided in the spatial computing device 12.

[0038] The position acquisition means 30 acquires the position of the spatial computing device 12 in real space. Specifically, the position acquisition means 30 constantly acquires the position of the spatial computing device 12 in real space based on its own position estimated using the range sensor of the detection means 24 of the spatial computing device 12. The acquired position is stored in the storage means 28.

[0039] The control means 32 controls the device control means 22 etc. to display virtual objects showing predetermined interface information on the wall and floor surfaces displayed on the display means 20 based on the position of the spatial computing device 12 and the physical characteristics of the walls, floors, etc. The control means 32 is composed of, for example, a CPU (Central Processing Unit) etc.

[0040] Specifically, the control means 32 identifies the user and his / her position based on the position of the spatial computing device 12 acquired from the position acquisition means 30 and the unique identification information of the spatial computing device 12 stored in the memory 26 of the spatial computing device 12. This associates the position of the spatial computing device 12 with the user. Furthermore, the control means 32 acquires physical characteristics, such as wall surfaces and floor surfaces, corresponding to the position of the spatial computing device 12 from metadata including physical characteristics, such as walls and floors, stored in the storage means 28. This associates the user's position with the wall surfaces and floor surfaces. Furthermore, the control means 32 associates the physical characteristics, such as walls and floors, included in the metadata with information to be displayed there. For example, if the physical characteristics indicate a floor, then the control means 32 associates the physical characteristics with the display of interface information for paths, and if the physical characteristics indicate a wall, then the control means 32 associates the physical characteristics with the display of interface information for operations.

[0041] The control means 32 controls the device control means 22 to display various information, such as interface information specialized for the user, on a wall surface, floor surface, or the like near the user via the display means 20 of the spatial computing device 12. The interface information displayed is specialized for the information to be displayed there. For example, in the case of a floor, interface information for a path is displayed, and in the case of a wall, interface information for operation is displayed.

[0042] FIG. 2(1) is an explanatory diagram of the case where the user wears the spatial computing device 12, and FIG. 2(2) is an explanatory diagram of the case where the user removes the spatial computing device 12.

[0043] As shown in FIG. 2(1), when a user P wearing a spatial computing device 12 views the real space S through the display means 20, a wall interface 42 (interface information) consisting of virtual objects appears to exist on a wall surface 40 of the real space S. The wall interface 42 displays information useful to the user P (e.g., destination information) and operable information, allowing the user P to consider their next action (e.g., a travel route) by looking at the wall interface 42. Input operations can also be performed on the wall interface 42 using eye tracking and hand gestures. While the example shown in the figure illustrates a case where information is displayed only on the wall surface 40, the display according to the present invention is not limited to the wall surface 40 and may be displayed anywhere on the surface of another object present in the real space S. For example, the display may be displayed on a building facility such as a floor surface 44.

[0044] Here, the advantage of associating virtual objects (displays) with the physical surfaces of walls and floors is that physical touches on physical surfaces can be matched with touches on virtual objects. In other words, when operating virtual objects using normal hand gestures, the user must perform operations that are like grabbing air, according to set rules, and there is no tactile feedback unless separate actuators or sounds are prepared, which makes it difficult to know when each operation has been completed.

[0045] In contrast, in this embodiment, interface information is displayed on the physical surface of the wall if it is a wall, or on the physical surface of the tabletop if it is a tabletop. By doing so, you can actually physically touch the surface and operate the interface information with a real sense of touch. If it is on the floor, you can operate the interface information as if you were stepping on it with your feet.

[0046] Note that virtual objects displayed on a physical surface that is out of reach of the hands or body display information corresponding to the physical surface and are not subject to operation by touching the physical surface.

[0047] Furthermore, as shown in FIG. 2(2), when a user P removes the spatial computing device 12, even if the user P looks at the real space S, the user P cannot recognize the wall interface 42.

[0048] According to the above configuration, by connecting the interface system 10, which holds metadata about the characteristics of the real space, to the spatial computing device 12, it is possible to realize a real-space information interface between building facilities such as walls and floors and users at low cost, without installing physical display devices such as projectors or display modules inside the building.

[0049] In the above embodiment, users of the spatial computing device 12 may have their own dedicated accounts and form user groups with multiple users. This allows multiple users to share interface information consisting of virtual objects displayed on the walls, floors, etc. of the virtual space, and the scope of the sharing may be set. In this case, the storage means 28 stores data for user groups composed of multiple users, and the control means 32 collates the user data stored in the storage means 28 and controls the display means 20 of the spatial computing devices 12 of multiple users belonging to the same user group to display common virtual objects, or to display specific virtual objects in a common area of ​​the real space. In this way, a dedicated real-space interface can be provided for each user group.

[0050] In the above embodiment, the building manager may also set prohibitions on the use of interfaces in building facilities such as walls and floors, or in predetermined areas predefined in real space. For example, the manager may restrict the display of wall interfaces on walls in areas where people congregate. In this case, the control means 32 determines whether the position of the spatial computing device 12 acquired by the position acquisition means 30 is within a predetermined area predefined in real space. Data representing the area is stored in the storage means 28 as metadata, and by referencing this metadata, the control means 32 can determine whether the position of the spatial computing device 12 is within the area. If the spatial computing device 12 is determined to be within the area, the control means 32 controls the spatial computing device 12. For example, the control means 32 may control the display means 20 not to display certain interface information (virtual objects), or to display certain interface information with restrictions. In this way, the content of the interface to be displayed can be controlled for each area.

[0051] As described above, the interface system of the present invention is an interface system for realizing a real-space interface using an object existing in a real space by a spatial computing device, wherein the spatial computing device has a virtual object generation means for generating a virtual object, an imaging means for imaging the real space, a display means for displaying an image captured by the imaging means and a virtual object generated by the virtual object generation means, and a device control means for controlling the virtual object generation means and the display means, and the interface system is equipped with a storage means for retaining metadata having information about the shape of the real space and information about physical characteristics including the position and shape of the object in the real space, a position acquisition means for acquiring the position of the spatial computing device in the real space, and a control means for controlling the display means to display a virtual object showing predetermined interface information on the surface of the object displayed on the display means, based on the position of the spatial computing device and the physical characteristics of the object.Therefore, a real-space interface using a wall, floor, etc. can be realized at low cost without installing a physical display device such as a projector or display module.

[0052] In addition, according to another interface system of the present invention, the storage means stores data of a user group composed of multiple users, and the control means displays a common virtual object on the display means of the spatial computing devices of multiple users belonging to the same user group, or displays a predetermined virtual object in a common range of the real space, thereby making it possible to provide a dedicated real space interface for each user group.

[0053] Furthermore, according to another interface system of the present invention, the control means controls the display means of the spatial computing device in a predetermined area set in advance within the real space so as not to display a predetermined virtual object, or so as to display the predetermined virtual object with restrictions, thereby making it possible to control the content of the virtual object to be displayed for each area.

[0054] Furthermore, according to another interface system of the present invention, the spatial computing device has a detection means for detecting the user's line of sight and hands, and the device control means outputs operation information for executing a predetermined operation on a virtual object displayed by the display means based on the detection by the detection means, so that an interface corresponding to the virtual object can be operated based on the output operation information.

[0055] The Sustainable Development Goals (SDGs) are 17 international goals that were adopted at the United Nations Summit in September 2015. The interface system according to this embodiment can contribute to achieving one of the 17 SDGs, for example, goal 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization, and foster innovation." [Industrial Applicability]

[0056] As described above, the interface system according to the present invention is useful for real-space interfaces using walls, floors, etc., and is particularly suitable for realizing real-space interfaces using walls, floors, etc. using a spatial computing device. [Explanation of symbols]

[0057] 10 Interface System 12 Spatial Computing Devices 14 Metadata Identification System 16 Virtual object generation means 18 Imaging means 20 Display means 22 Device Control Means 24 Detection Methods 26 memory 28 Memory means 30 Location acquisition means 32 Control means

Claims

1. An interface system for realizing a real-space interface using an object existing in a real space by a spatial computing device, comprising: the spatial computing device comprises: a virtual object generation means for generating a virtual object; an imaging means for imaging the real space; a display means for displaying an image captured by the imaging means and a virtual object generated by the virtual object generation means; and a device control means for controlling the virtual object generation means and the display means; The interface system is characterized by comprising: a storage means for storing metadata having information regarding the shape of the real space and information regarding physical characteristics of the object in the real space, including the position and shape; a position acquisition means for acquiring the position of the spatial computing device in the real space; and a control means for controlling the display of a virtual object showing interface information on the surface of the object displayed on the display means, based on the position of the spatial computing device and the physical characteristics of the object.

2. the storage means holds data of a user group made up of a plurality of users; The interface system according to claim 1, characterized in that the control means causes a common virtual object to be displayed on the display means of the spatial computing devices of multiple users belonging to the same user group, or causes a predetermined virtual object to be displayed in a common range of the real space.

3. The interface system according to claim 1 or 2, characterized in that the control means controls the display means of the spatial computing device in a predetermined area set in advance in the real space so as not to display a predetermined virtual object, or controls the display means to display the predetermined virtual object with restrictions.

4. The interface system described in claim 1 or 2, characterized in that the spatial computing device has a detection means for detecting the user's line of sight and hands, and the device control means outputs operation information for performing a predetermined operation on a virtual object displayed by the display means based on detection by the detection means.

Images