Information processing device and information processing method

Abstract

The present technology relates to an information processing device and an information processing method capable of setting a projection surface in a more appropriate form. An information processing device is provided, which is provided with: a registration unit that registers display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor provided in a first device, the first device being installed at the projection surface; and a display control unit that projects and displays the display information on the projection surface based on the display control information. The present technology can be applied to a projection system that displays information by projection.

Description

Technical Field

[0001] The present technology relates to an information processing device and an information processing method, and more particularly, to an information processing device and an information processing method capable of setting a projection surface in a more appropriate form. Background Art

[0002] In recent years, research has been conducted on AR technology such as technology for presenting information by projection and performing operations using a projection surface or projection augmented reality (AR) technology for expanding a real space using projection mapping.

[0003] In the case where information is displayed by projection as superimposed on a real object or on a plane on which a real object is mounted, images captured by a fixed camera device are generally used to estimate the posture of the real object or the projection surface based on pre-registered images of the real object, geometric information such as feature points, and three-dimensional information.

[0004] For example, Patent Document 1 discloses a technique in which a video presentation device calculates a positional relationship between video presentation devices based on the position of an infrared light emitting diode (LED) provided in another video presentation device.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: JP 2016-46731 A Summary of the invention

[0008] Technical issues

[0009] However, when estimating the position of a real object or the like using an image captured by a fixed camera, there is a possibility that the accuracy of the estimation may be reduced due to the positional relationship between the camera and the real object or its posture. In addition, there is a possibility that the accuracy of the estimation of the posture of a real object far from the camera may be reduced due to the resolution of the camera.

[0010] Additionally, attaching an infrared light marker, such as an infrared LED, to a real object is complicated.

[0011] Therefore, there is a need for a technology for setting a projection surface by estimating the posture and position of the projection surface in a more appropriate form using an image captured by an imaging device.

[0012] The present technology has been proposed in view of such circumstances, and enables the projection surface to be set in a more appropriate form.

[0013] Solution to the problem

[0014] An information processing device according to an aspect of the present technology is an information processing device including: a registration unit configured to register display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor provided in a first device, the first device being installed at the projection surface; and a display control unit configured to project the display information on the projection surface and display the display information based on the display control information.

[0015] An information processing method according to aspects of the present technology is an information processing method including the following: causing an information processing device to: register display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor set in a first device, the first device being installed at the projection surface; and projecting the display information on the projection surface and displaying the display information based on the display control information.

[0016] In aspects of the present technology, display control information for displaying display information on a projection surface is registered, the display control information being generated based on sensor data obtained from a sensor provided in a first device, the first device being installed at the projection surface; and the display information is projected and displayed on the projection surface based on the display control information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] [ Figure 1 ] Figure 1 is a diagram showing a configuration example of a projection system according to an embodiment of the present technology.

[0018] [ Figure 2 ] Figure 2 It is a diagram showing a method of registering information on a registration target device and a projection surface.

[0019] [ Figure 3 ] Figure 3 It is a diagram showing the flow of projection of display information of an electric device.

[0020] [ Figure 4 ] Figure 4 is a diagram showing an example of a method of estimating the position and posture of a real object.

[0021] [ Figure 5 ] Figure 5 is a diagram showing another example of a method of estimating the position and posture of a real object.

[0022] [ Figure 6 ] Figure 6 is a diagram showing still another example of a method of estimating the position and posture of a real object.

[0023] [ Figure 7 ] Figure 7: is a diagram showing an example of a method of estimating the position and posture of a projection surface in the present technology.

[0024] [ Figure 8 ] Figure 8 is a diagram showing an example of an imaging angle of view of an infrared stereo camera.

[0025] [ Fig. 9 ] Fig. 9 is a block diagram showing a configuration example of a portable terminal.

[0026] [ Fig.10 ] Fig.10 is a diagram showing an example of a posture of a computing portable terminal.

[0027] [ Fig.11 ] Fig.11 is a block diagram showing a configuration example of an information processing device.

[0028] [ Fig.12 ] Fig.12 is a diagram showing a schema example of an installation information database.

[0029] [ Fig.13 ] Fig.13 is a diagram showing an example of imaging of a scintillation pattern of infrared light.

[0030] [ Fig.14 ] Fig.14 is a diagram showing an example of a depth image.

[0031] [ Fig.15 ] Fig.15 is a diagram showing an example of a method of specifying a region by combining a plurality of points.

[0032] [ Fig.16 ] Fig.16 is a diagram showing an example of an image in which a plurality of depth images are superimposed on each other.

[0033] [ Fig.17 ] Fig.17 is a flowchart showing a registration process of a portable terminal.

[0034] [ Fig.18 ] Fig.18 is a flowchart showing the registration process of the information processing device.

[0035] [ Fig.19 ] Fig.19 : is a flowchart showing the projection process of the information processing device.

[0036] [ Fig. 20 ] Fig. 20 is a diagram showing another schema example of the installation information database.

[0037] [ Fig.21 ] Fig.21 is a block diagram showing a configuration example of a projector.

[0038] [ Fig. 22 ] Fig. 22 is a block diagram showing a configuration example of hardware of a computer. DETAILED DESCRIPTION

[0039] Embodiments of the present technology will be described below. The description will be given in the following order.

[0040] 1. Brief description of the projection system

[0041] 2. Detection of the position and posture of the projection surface

[0042] 3. Configuration of each device

[0043] 4. Operation of each device

[0044] 5. Modify the example

[0045] <1. Overview of the projection system>

[0046] Figure 1 is a diagram showing a configuration example of a projection system according to an embodiment of the present technology.

[0047] like Figure 1 As shown, the projection system includes a portable terminal 11, a projector 12, an infrared stereo camera device 13 and an information processing device 14.

[0048] The portable terminal 11 is constituted by a device such as a smartphone or a tablet terminal. The portable terminal 11 is provided with a posture sensor that detects the posture of the portable terminal 11 and an infrared light source that outputs infrared light.

[0049] The portable terminal 11 is placed on a projection surface such as a wall surface, a table surface, or a floor surface, on which the projector 12 projects a display image showing display information. Figure 1 In the example of , the portable terminal 11 is placed on a table in an installation space such as a room, which is provided with components constituting a projection system such as a projector 12. The portable terminal 11 detects its own posture using a posture sensor, and registers information indicating the detected posture in the information processing device 14 as information indicating the posture of the projection surface.

[0050] The projector 12 is a projection device that projects a display image indicating various display information onto a projection surface. The projector 12 is installed at a position where it can project a display image onto a projection surface formed on a wall surface, a table surface, etc. in the installation space. Figure 1In the example of FIG. 1 , the projector 12 is installed on the ceiling of the room. The projector 12 displays the display image by projecting the display image under the control of the information processing device 14.

[0051] The infrared stereo camera device 13 is an imaging device that generates a depth image by detecting infrared light. The infrared stereo camera device 13 is installed at a position where the projection surface of the projector 12 can form an image. Figure 1 In the example shown in FIG. 1 , the infrared stereo camera 13 is installed at a position near the projector 12 .

[0052] For example, the infrared stereo camera 13 detects infrared light output by an infrared light source provided in the portable terminal 11 to generate a depth image. The depth image is used to perform measurement of the distance between the portable terminal 11 and the infrared stereo camera 13 in the information processing device 14, etc.

[0053] The information processing device 14 is composed of, for example, a personal computer (PC) or a dedicated device. The information processing device 14 may also be referred to as a server device, and manages an installation information database in which display control information is registered. The display control information is information including installation information indicating the position and posture of the projection surface. Figure 1 In the example, the information processing device 14 is installed outside the room.

[0054] Portable terminal 11 and information processing device 14 are connected to each other via a network such as the Internet or a wireless local area network (LAN) and can exchange information with each other. Projector 12 and infrared stereo camera 13 are connected to information processing device 14 via wired or wireless communication.

[0055] The user of the projection system having such a configuration can register information about the registration target device and information about any projection surface in association with each other in the installation information database of the information processing apparatus 14 by using the portable terminal 11 .

[0056] A device capable of acquiring display information desired to be projected by the projector 12 is selected as a registration target device from various devices such as devices (electrical devices, etc.) installed in the installation space or devices (electrical devices, etc.) brought into the installation space from the outside. The display information is appropriately projected onto the projection surface registered in the installation information database by the projector 12. The display information is information acquired by the registration target device and projected by the projector 12.

[0057] Figure 2 It is a diagram showing a method of registering information on a registration target device and a projection surface.

[0058] exist Figure 2 In the example of , the electric device 21 located on the table on which the portable terminal 11 is placed is selected as the registration target device. Figure 2 In the example of FIG. 1 , the electric device 21 is a smart speaker, and it is assumed that the electric device 21 is reproducing music or sound of a broadcast program received from a distribution server via the Internet.

[0059] The portable terminal 11 and the electrical device 21 have functions corresponding to a short-range wireless communication standard such as near field communication (NFC) communication. When the user holds the portable terminal 11 on the electrical device 21, the portable terminal 11 performs NFC communication (near-range wireless communication) with the electrical device 21 to acquire device information of the electrical device 21. The device information is information including network information such as a device name, a device type, and an Internet Protocol (IP) address.

[0060] The user holds the portable terminal 11 on the electric device 21, and then moves the portable terminal 11 to a position where it is desired to project the information of the electric device 21. Information is exchanged between the portable terminal 11 and the information processing device 14, and thus the surface on which the portable terminal 11 is placed is registered in the information processing device 14 as a projection surface on which the information of the electric device 21 is projected. The information transmitted from the portable terminal 11 to the information processing device 14 includes the device information of the electric device 21 acquired through the NFC communication.

[0061] Figure 3 2 is a diagram showing the flow of projection of display information of the electric device 21. Figure 3 , the flow of projection of display information is shown in time series in the order of upper, middle, and lower parts.

[0062] exist Figure 3 In the upper part of the table, the electric device 21 is placed substantially at the center of the table surface, and the portable terminal 11 is placed on the right side of the electric device 21. In such a state, for example, the table surface on which the portable terminal 11 is placed is registered in the installation information database of the information processing apparatus 14 as a projection surface of the information of the electric device 21.

[0063] like Figure 3 As shown in the middle part of , after the display control information is registered, the user moves the portable terminal 11 from the table surface to another location.

[0064] In addition, after the display control information is registered, for example, information of music being reproduced is transmitted as display information from the electric device 21 to the information processing apparatus 14. The information processing apparatus 14 and the electric device 21 are also connected to each other via a network.

[0065] In the information processing apparatus 14, the music information transmitted from the electric appliance 21 is received, and a process for projecting a display image for displaying the music information onto the projector 12 is performed based on the display control information.

[0066] like Figure 3 As shown in the lower part of FIG. 1 , in the area D1 (which is the area including the location where the portable terminal 11 is placed), the display information of the electric device 21 is projected by the projector 12. Figure 3 In the lower part, thumbnail images, artist information, music titles, etc. are projected as music information.

[0067] Thus, the user can specify an area used as a projection surface for display information for another piece of equipment by placing the portable terminal 11 at a desired position. In addition, the user can visually confirm the display information of another piece of equipment through the display image projected by the projector 12.

[0068] Although the case where the display information is projected onto the area including the position where the portable terminal 11 is placed has been described above, the display information may be projected onto areas at various positions determined based on the position of the portable terminal 11 as projection surfaces. That is, an area including a position deviated from the area including the position where the portable terminal 11 is placed may be set as a projection surface relative to the position.

[0069] <2. Detection of the position and posture of the projection surface>

[0070] Examples of methods for implementing augmented reality (AR) for displaying information about a real object to be superimposed on the real object or displaying information on a plane on which the real object is mounted include a head-mounted display method, a method of displaying information in a so-called camera-through manner, a method of displaying information by a projector, and the like.

[0071] In the case of implementing AR by each of the methods, the position and posture of the projection surface are estimated. Figures 4 to 8 The method of estimating the position and attitude of each projection surface and the method of estimating the position and attitude of the projection surface in the present technology are described above.

[0072] Figure 4 is a diagram showing an example of a method of estimating the position and posture of a real object.

[0073] Figure 4 The illustrated estimation method is a method of estimating the position and posture of a real object, which is performed in a case where information is displayed in a through-camera manner.

[0074] The smartphone SP that implements AR in a camera-by-camera manner is provided with a camera that images a real object RO installed on a table surface in an installation space, and a display that displays an image captured by the camera.

[0075] For example, Figure 4As shown in the balloon shape of , an image showing the real object RO is captured by the camera of the smartphone SP. Based on such an image, the position and posture of the real object RO are estimated.

[0076] In the smartphone SP used in the estimation method, a camera as an input device and a display as an output device are provided in the same housing, and thus the user can change the direction of the camera and the distance from the real object RO according to the real object RO. For example, based on the estimation result of the position and posture of the real object RO, information about the real object RO is displayed on the display of the smartphone SP so as to be superimposed on the real object RO.

[0077] Figure 5 is a diagram showing another example of a method of estimating the position and posture of a real object.

[0078] Figure 5 The illustrated estimation method is a method of estimating the position and posture of a real object, which is performed in a case where information is displayed by a projector.

[0079] like Figure 5 As shown, in the method of displaying information by a projector, with reference to Figure 4 Unlike the described method, a camera device CAM for imaging the real object RO is installed on the ceiling of a room or the like.

[0080] like Figure 5 As shown in the balloon shape of , an image showing the real object RO and the table is captured by the camera device CAM. Based on such an image, the position and posture of the real object RO are estimated.

[0081] However, since it is difficult to change the position of the camera CAM according to the real object RO, there is a possibility that the estimation accuracy of the position and posture of the real object RO is reduced due to the positional relationship between the camera CAM and the real object RO, etc.

[0082] Furthermore, in the case where the camera CAM images a wide space, there is a possibility that the estimation accuracy of the position and orientation of the real object RO separated from the camera CAM is reduced due to the resolution of the camera CAM.

[0083] Based on the estimation result of the position and posture of the real object RO, the projector 12 installed at the ceiling in the room displays, for example, information on the real object RO to be superimposed on the real object RO.

[0084] Figure 6 is a diagram showing still another example of a method of estimating the position and posture of a real object.

[0085] Figure 6The illustrated estimation method is a method of estimating the position and posture of a real object, which is performed in a case where information is displayed in a through-camera manner.

[0086] Markers M1 and M2 as two markers are attached to the vertices of a rectangular parallelepiped forming a real object RO imaged by the camera of the smartphone SP.

[0087] like Figure 6 As shown in the balloon shape of , an image showing the real object RO and the markers M1 and M2 is captured by the camera device of the smartphone SP. The positional relationship between the smartphone SP and the real object RO is calculated based on the positions of the markers M1 and M2 in such an image and the information indicating the postures of the smartphone SP and the real object RO.

[0088] Based on the positional relationship between the smartphone SP and the real object RO, the smartphone SP can display information even when the real object RO is not included in the angle of view of the camera.

[0089] It is complicated to attach the markers M1 and M2 to the real object RO. For this purpose, the markers can also be displayed on a display provided in the real object RO.

[0090] However, in the case of detecting the position of a marker using an image captured by a camera installed at a ceiling or the like, there is a possibility that the detection accuracy of the marker is reduced due to the installation position of the camera or the distance from the real object RO.

[0091] Figure 7 : is a diagram showing an example of a method of estimating the position and posture of a projection surface in the present technology.

[0092] like Figure 7 As shown, the portable terminal 11 placed adjacent to the real object RO as the registration target device is provided with an infrared light source 31. The portable terminal 11 is also provided with a posture sensor. Note that the real object RO is equivalent to the electrical device 21 such as the above-mentioned smart speaker.

[0093] The portable terminal 11 estimates the posture of the projection surface in the world coordinate system using the posture sensor, and outputs infrared light from the infrared light source 31 .

[0094] The infrared stereo camera device 13 images the installation space and generates a depth image. Figure 8 As shown, the imaging angle of view of the infrared stereo camera 13 includes the real object RO, the portable terminal 11 and the table. In the depth image generated by the infrared stereo camera 13, the real object RO and the portable terminal 11 placed on the table are shown.

[0095] The depth image generated by the infrared stereo camera 13 is supplied to the information processing device 14. Information on the posture of the projection surface estimated using the posture sensor of the portable terminal 11 is also transmitted to the information processing device 14.

[0096] The information processing device 14 calculates a vector pointing from the installation position of the infrared stereo camera 13 to the position of the infrared light source 31 by using the depth image supplied from the infrared stereo camera 13 . Figure 7 The colored arrow A0 in FIG. 1 indicates a vector pointing from the installation position of the infrared stereo camera device 13 to the position of the infrared light source 31 .

[0097] In addition, the information processing device 14 estimates the position of the infrared light source 31 based on the vector. The position of the infrared light source 31 is registered as the position of the projection surface together with the posture of the projection surface estimated by the portable terminal 11.

[0098] That is, it can be said that the portable terminal 11 provided with the infrared light source 31 and the attitude sensor functions as installation information acquisition means for acquiring the position and attitude of the projection surface.

[0099] In this way, in the method of estimating the position and attitude of the projection surface using the installation information acquisition device, the installation information acquisition device is provided as a device of a housing separate from the projector 12 and the infrared stereo camera device 13. Since the projector 12 and the infrared stereo camera device 13 are provided in separate housings, a process for converting the coordinate system of the position and attitude of the projection surface acquired by the installation information acquisition device into the projector coordinate system is performed.

[0100] As described above, in the projection system of the present technology, the position and attitude of the projection surface are estimated by combining position detection using infrared light output by the infrared light source 31 and attitude estimation using the attitude sensor.

[0101] Thus, it is possible to suppress a decrease in the estimation accuracy of the position and posture of the projection surface due to the positional relationship with the projection surface, etc., which may occur when using a camera fixed to a ceiling, etc. In addition, it is possible to suppress a decrease in the estimation accuracy of the position and posture of the projection surface due to the resolution of the camera.

[0102] Furthermore, since there is no high-load processing such as object recognition using an image captured by the camera, it is possible to acquire the position and posture of the projection surface and realize projection according to the information of the projection surface even in a device with low specifications.

[0103] Therefore, the posture and position of the projection surface can be detected in a more appropriate form by using the image captured by the camera device. Therefore, the projection surface can be set in a more appropriate form.

[0104] Furthermore, in the case where the electric device 21 serving as the registration target device is a device such as a home appliance that does not have an information display function, the projection system may provide the information display function based on projection instead of the registration target device.

[0105] <3. Configuration of each device>

[0106] Configuration of the portable terminal 11

[0107] Fig. 9 The portable terminal 11 is a block diagram showing a configuration example. The portable terminal 11 is implemented by executing a predetermined program by a central processing unit (CPU) or the like provided in the portable terminal 11. Fig. 9 At least some of the functional units shown in .

[0108] like Fig. 9 As shown, the portable terminal 11 includes an NFC unit 51 , an object information input unit 52 , an object information storage unit 53 , a posture sensor 54 , a posture estimation unit 55 , a communication unit 56 , an identification code encoding unit 57 , and a proximity sensor 58 .

[0109] The NFC unit 51 is, for example, a reader / writer for NFC communication. The NFC unit 51 performs NFC communication with a registration target device to acquire device information of the registration target device. The device information of the registration target device is provided to the object information input unit 52.

[0110] The object information input unit 52 supplies the device information supplied from the NFC unit 51 to the communication unit 56. In addition, the object information input unit 52 supplies the device information to the object information storage unit 53 and stores the information therein.

[0111] The object information storage unit 53 stores the device information acquired from the registration target device. The object information storage unit 53 is constituted by an auxiliary storage device including an internal or external memory such as a semiconductor memory.

[0112] The attitude sensor 54 is composed of various sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor. Note that the acceleration sensor and the gyro sensor are collectively referred to as an inertial sensor (inertial measurement unit: IMU). The attitude sensor 54 detects the angular velocity, acceleration, magnetic field strength, etc. of the portable terminal 11, and provides sensor data indicating the detection result to the attitude estimation unit 55.

[0113] The posture estimation unit 55 calculates the posture of the portable terminal 11 in the world coordinate system based on the sensor data indicating the detection result supplied from the posture sensor 54. After the device information is acquired from the registration target device by the NFC unit 51, the process of calculating the posture of the portable terminal 11 starts.

[0114] Fig.10 2 is a diagram showing an example of the posture of the computing portable terminal 11 .

[0115] like Fig.10 As shown, after acquiring the device information of the electric device 21 as the registration target device, calculation of the posture of the portable terminal 11 is performed while the portable terminal 11 is placed on the projection surface.

[0116] First, in the attitude estimation unit 55, the attitude of the portable terminal 11 is estimated based on the angular velocity of the portable terminal 11 detected by the gyro sensor. Then, in the attitude estimation unit 55, the attitude of the portable terminal 11 estimated based on the detection result of the gyro sensor is corrected using the attitude of the portable terminal 11 estimated based on the acceleration value of the portable terminal 11 detected by the acceleration sensor and the magnetic field strength detected by the geomagnetic sensor.

[0117] Note that the posture of the portable terminal 11 may be estimated using only the detection result of the acceleration sensor or only the detection result of the geomagnetic sensor.

[0118] like Fig.10 As shown in the balloon shape of , the posture of the portable terminal 11 estimated by the posture estimation unit 55 is indicated by rotation around three axes, namely, the pitch axis (X axis), the roll axis (Y axis), and the yaw axis (Z axis).

[0119] return Fig. 9 , the posture estimation unit 55 fixedly decides information indicating the posture of the portable terminal 11 as posture information when the state in which the amount of change in the posture is equal to or less than a predetermined threshold value has continued for a fixed period of time and provides the posture information to the communication unit 56. The posture information is information indicating the posture of the projection surface in the world coordinate system.

[0120] Note that the posture information includes offset information, which is information from the proximity sensor 58 on which the infrared light source 31 is mounted to the center of the rotation axis of the posture of the portable terminal 11. In addition, the posture information can be generated by the information processing device 14 by sending sensor data indicating the detection result to the information processing device 14.

[0121] The communication unit 56 is composed of a communication module that supports wireless communication such as a wireless LAN (e.g., Wi-Fi (registered trademark)) or a cellular communication (e.g., LTE-Advanced or 5G) or wired communication such as Ethernet (registered trademark). The communication unit 56 transmits the device information provided from the object information input unit 52 and the posture information provided from the posture estimation unit 55 to the information processing device 14.

[0122] In addition, the communication unit 56 receives an identification code (to be described later) transmitted from the information processing device 14 , and supplies the received identification code to the identification code encoding unit 57 .

[0123] The identification code encoding unit 57 encodes the identification code supplied from the communication unit 56 into a blinking pattern. The identification code encoding unit 57 supplies the proximity sensor 58 with information indicating the blinking pattern.

[0124] The proximity sensor 58 is provided with an infrared light source 31 that emits infrared light to the measuring object, and a light receiving element that receives the infrared light reflected by the measuring object. The proximity sensor 58 outputs a blinking pattern of infrared light (which is an encoded identification code) by using the infrared light source 31 such as a light emitting diode (LED).

[0125] Configuration of the information processing device 14

[0126] Fig.11 The information processing device 14 is a block diagram showing a configuration example. The information processing device 14 is implemented by executing a predetermined program by a CPU or the like provided in the information processing device 14. Fig.11 At least some of the functional units shown in .

[0127] like Fig.11 As shown, the information processing device 14 includes a communication unit 71, an installation information storage unit 72, an identification code generation unit 73, a position information acquisition unit 74, a system installation information storage unit 75, a display control unit 76, a position and posture processing unit 77 and a display processing unit 78.

[0128] The communication unit 71 is composed of a communication module that supports wireless communication such as wireless LAN (e.g., Wi-Fi) or cellular communication (e.g., LTE-Advanced or 5G) or wired communication such as Ethernet, etc. The communication unit 71 transmits the identification code provided by the identification code generation unit 73 to the portable terminal 11.

[0129] In addition, the communication unit 71 receives display information transmitted from the registration target device, and provides the received display information together with information of the indicating device as the supply source of the display information to the display control unit 76. For example, the communication unit 71 receives music information transmitted from the smart speaker as the electrical device 21 (which is the registration target device), and provides the received music information to the display control unit 76 together with network information of the smart speaker.

[0130] The communication unit 71 also receives the device information and posture information transmitted from the portable terminal 11. The communication unit 71 provides the device information and posture information to the installation information storage unit 72 and stores the information therein. The installation information storage unit 72 is composed of an auxiliary storage device including an internal or external storage device such as a semiconductor memory or a hard disk drive (HDD).

[0131] In the installation information storage unit 72, the device information and posture information provided from the communication unit 71 and the position information provided from the position information acquisition unit 74 are registered in the installation information database as display control information in association with each other. That is, the communication unit 71 and the position information acquisition unit 74 serve as a registration unit for registering the display control information in the installation information database. The position information is information indicating the position of the projection surface in the world coordinate system.

[0132] Fig.12 is a diagram showing a schema example of an installation information database.

[0133] exist Fig.12 In the "ID", "installation information", "type", "display content" and "network information" are associated with each other. "ID" is an ID given for identifying the registration target device. "Installation information" is information including position information and posture information of the projection surface. "Type" is information indicating the type of the registration target device and is determined based on the device information. "Display content" is information indicating the type of display information, which is information projected onto the projection surface.

[0134] For example, for ID XXX012, information about the projection surface on which the captured image acquired by the camera is projected as display information is registered. The registration target device as the camera, the position information and posture information of the projection surface and the display information as the captured photo information, and the IP address 192.168.0.5 are associated with each other.

[0135] Therefore, for the projection surface associated with ID XXX012, a display image indicating captured photo information transmitted from the camera having the IP address 192.168.0.5 through the network is projected. The captured photo information is information of a captured image acquired by the camera.

[0136] For ID XXX013, information about the projection surface onto which the recording information acquired by the recorder is projected as display information is registered. The registration target device as the recorder, the position information and posture information of the projection surface and the display information as the newly recorded program information, and the IP address 192.168.0.9 are associated with each other.

[0137] Therefore, for the projection surface associated with ID XXX013, a display image indicating newly recorded program information transmitted from the recorder having the IP address 192.168.0.9 via the network is projected. The newly recorded program information is information indicating a new program recorded by the recorder.

[0138] For ID XXX014, information about the projection surface formed in the table is registered. The registration target device of the table and the position information and posture information of the projection surface are associated with each other. For example, in the case where the table does not have a function of sending information to the information processing device 14, the display information and network information are not registered for ID XXX014.

[0139] The table as the registration target device is registered as a projection surface of various information. For example, for the table associated with ID XXX014, a display image indicating the collective information of each piece of equipment is projected.

[0140] Note that the user may input device information of a device that does not have a communication function (eg, a table) by using the portable terminal 11 .

[0141] For ID XXX015, information about the projection surface onto which the music information acquired by the smart speaker is projected as display information is registered. The registration target device as the smart speaker, the position information and posture information of the projection surface, the display information as the music information, and the IP address 192.168.0.1 are associated with each other.

[0142] Therefore, for the projection surface associated with ID XXX015, a display image indicating music information transmitted via the network from the smart speaker which is the electric device 21 having the IP address 192.168.0.1 is projected.

[0143] Return to reference Fig.11 , the identification code generation unit 73 acquires the key information stored in the installation information storage unit 72. The key information is information including a master key such as an ID.

[0144] The identification code generation unit 73 generates an identification code based on the key information. The identification code is information that can identify the registration target device. In the installation information database, a unique ID is set for the registration target device, and thus the identification code is a unique code.

[0145] The identification code generated by the identification code generating unit 73 is provided to the communication unit 71 and transmitted to the portable terminal 11. Note that the identification code is also provided and stored in the installation information storage unit 72. As described above, in the portable terminal 11, the identification code is encoded and output as a blinking pattern of infrared light.

[0146] Fig.13 is a diagram showing an example of imaging of a scintillation pattern of infrared light.

[0147] like Fig.13 As shown, the infrared stereo camera 13 images the flickering pattern of infrared light emitted from the infrared light source 31 of the portable terminal 11 that has received the identification code. For example, a depth image is captured in which an area A1 is shown in the area of ​​the table surface on which the portable terminal 11 is placed, and the area A1 is an area including the position of the infrared light source 31.

[0148] Fig.14 is a diagram showing an example of a depth image.

[0149] For example, Fig.14 As shown, a depth image showing a point P1 is captured by the infrared stereo camera 13. The point P1 is a bright spot showing infrared light output by the infrared light source 31 of the portable terminal 11.

[0150] Such a depth image is provided from the infrared stereo camera device 13 to Fig.11 The position information acquisition unit 74 in.

[0151] The position information acquisition unit 74 acquires the system installation information stored in the system installation information storage unit 75. The system installation information is information including internal parameters and external parameters of the projector 12 and internal parameters and external parameters of the infrared stereo camera 13.

[0152] The position information acquisition unit 74 generates the position information of the projection surface based on the depth image supplied from the infrared stereo camera 13 and the system installation information acquired from the system installation information storage unit 75 .

[0153] like Fig.11 As shown, the position information acquiring unit 74 includes an identification code decoding unit 91 , a position detecting unit 92 and a distance estimating unit 93 .

[0154] The identification code decoding unit 91 detects a flickering pattern of infrared light based on the video as a depth image. The identification code decoding unit 91 decodes the identification code according to the flickering pattern of infrared light.

[0155] The position detection unit 92 detects the position (coordinate value) of the bright spot of infrared light in the depth image by bright spot detection. Specifically, the coordinates of the center of gravity of the bright spot of infrared light are acquired.

[0156] The distance estimation unit 93 calculates the distance (depth value) from the position where the infrared stereo camera 13 is installed to the position of the bright spot detected by the position detection unit 92 based on the brightness value of the depth image.

[0157] Since the coordinates of the center of gravity of the bright spot detected by the position detection unit 92 are the image coordinate system in the infrared stereo camera 13 , the position information acquisition unit 74 calculates a vector toward the bright spot in the camera coordinate system using internal parameters of the infrared stereo camera 13 .

[0158] The position information acquisition unit 74 calculates the position of the infrared light source 31 in the camera coordinate system based on the vector toward the bright spot in the camera coordinate system and the distance between the infrared stereo camera 13 and the bright spot calculated by the distance estimation unit 93 .

[0159] In addition, the position information acquisition unit 74 converts the position of the infrared light source 31 in the camera coordinate system into the position of the infrared light source 31 in the world coordinate system based on the external parameters of the infrared stereo camera 13 .

[0160] The position information acquisition unit 74 acquires the identification code stored in the installation information storage unit 72. The position information acquisition unit 74 specifies the registration target device based on the identification code decoded by the identification code decoding unit 91. The position information acquisition unit 74 provides information indicating the position of the infrared light source 31 in the world coordinate system to the installation information storage unit 72 as the position information of the projection surface, and stores the information therein.

[0161] In the installation information database, the position information is registered in association with the device information of the registration target device specified by the identification code.

[0162] In the system installation information storage unit 75, system installation information including internal parameters, external parameters, etc. of the projector 12 is stored. The system installation information storage unit 75 is constituted by an auxiliary storage device that is the same as or different from the installation information storage unit 72.

[0163] Note that it is assumed that the internal parameters of the projector 12 and its position and posture in the world coordinate system (external parameters), and the internal parameters of the infrared stereo camera 13 and its position and posture in the world coordinate system (external parameters) are known.

[0164] The display control unit 76 controls projection of display information based on the information supplied from the communication unit 71. The display control unit 76 supplies the display information supplied from the communication unit 71 to the display processing unit 78. In addition, the display control unit 76 supplies information of the pointing device as a supply source of the display information to the position and posture processing unit 77.

[0165] The position and posture processing unit 77 acquires display control information associated with a device as a supply source of display information from the installation information storage unit 72 based on information supplied from the display control unit 76. In addition, the position and posture processing unit 77 acquires system installation information stored in the system installation information storage unit 75.

[0166] The position and attitude processing unit 77 calculates a matrix for converting the world coordinate system into the projector coordinate system based on the position information and attitude information in the world coordinate system included in the display control information and the external parameters of the projector 12 included in the system installation information. In addition, the position and attitude processing unit 77 calculates a matrix for converting the projector coordinate system into the projector image coordinate system based on the internal parameters of the projector 12 included in the system installation information.

[0167] The display transformation matrix, which is the two matrices calculated by the position and posture processing unit 77 , is supplied to the display processing unit 78 .

[0168] The display processing unit 78 generates a display image indicating the display information supplied from the display control unit 76. In addition, the display processing unit 78 converts the display image into a display image of the projector image coordinate system by using the display transformation matrix supplied from the position and posture processing unit 77. The display processing unit 78 supplies the display image of the projector image coordinate system to the projector 12 and projects the display image.

[0169] Although the case where the display image is projected onto the area based on the position of one point where the infrared light source 31 exists has been described above, the display image may be projected onto areas where a plurality of points are combined.

[0170] Fig.15 is a diagram showing an example of a method of estimating a region by combining a plurality of points.

[0171] like Fig.15 As shown, the user can move the portable terminal 11 to a plurality of positions in any direction (eg, the direction of the arrow in the figure) within the area A1 , where the area A1 is the imaging viewing angle of the infrared stereo camera device 13 .

[0172] The infrared stereo camera 13 captures a depth image in which infrared light output by the infrared light source 31 is shown at a position to which the portable terminal 11 is moved, and supplies the captured depth image to the information processing device 14 .

[0173] Fig.16 is a diagram showing an example of an image in which a plurality of depth images are superimposed on each other.

[0174] exist Fig.16In the example of , four depth images are captured while the portable terminal 11 is moved, and the four depth images are superimposed on each other.

[0175] For example, Fig.16 As shown, the information processing device 14 detects bright spots of infrared light shown in the four depth images provided from the infrared stereo camera 13 , and detects bright spots of points P11 to P14 .

[0176] In this case, the information processing device 14 registers the area formed by connecting the points P11 to P14 with a straight line as the position information of the projection surface in the installation information database. When projecting the display information, the display image is projected after undergoing a transformation such as enlargement or reduction so that the display image fits inside the area registered as the position information.

[0177] <4. Operation of each device>

[0178] Here, the operation of each device having the above-described configuration will be described.

[0179] Operation of the portable terminal 11

[0180] First, refer to Fig.17 The flowchart of FIG. 1 describes the registration process of the portable terminal 11 .

[0181] Fig.17 The registration process is a series of processes for registering the posture information and the position information in the information processing device 14.

[0182] In step S1, the object information input unit 52 determines whether the device information of the registration target device has been acquired, and waits until it is determined that the device information has been acquired. For example, when the user holds the portable terminal 11 on the registration target device such as a smart speaker as the electrical device 21, the NFC unit 51 performs NFC communication with the registration target device, and in the case where the device information has been acquired, determines that the device information has been acquired.

[0183] In a case where it is determined in step S1 that the device information is acquired, the process proceeds to step S2. In step S2, the posture estimation unit 55 acquires sensor data from the posture sensor 54, and calculates the posture of the portable terminal 11 based on the sensor data.

[0184] In step S3 , the posture estimation unit 55 determines whether the change amount of the posture of the portable terminal 11 is equal to or less than a threshold value, and repeatedly calculates the posture of the portable terminal 11 until the change amount of the posture of the portable terminal 11 is equal to or less than the threshold value.

[0185] In the event that the change amount of the posture of the portable terminal 11 is determined to be equal to or less than the threshold in step S3, the process proceeds to step S4. In step S4, the posture estimation unit 55 determines whether the state where the change amount of the posture is equal to or less than the threshold has continued for a fixed period of time.

[0186] In a case where it is determined in step S4 that the state in which the amount of change in the posture is equal to or smaller than the threshold value has not continued for a fixed period of time, the process returns to step S2 , and the subsequent processes are performed.

[0187] On the other hand, in the event that it is determined in step S4 that the state in which the amount of change in the posture is equal to or less than the threshold has continued for a fixed period of time, the posture estimation unit 55 fixedly decides the posture information, and the process proceeds to step S5 .

[0188] In step S5 , the communication unit 56 transmits the device information and the posture information to the information processing apparatus 14 .

[0189] Thereafter, in step S6 , the communication unit 56 determines whether the identification code transmitted from the information processing device 14 has been received.

[0190] In the event that it is determined in step S6 that the identification code has not been received, the process proceeds to step S7. In step S7, the communication unit 56 determines whether a timeout period has elapsed.

[0191] In the case where it is determined in step S7 that the timeout time has not elapsed, the process returns to step S6 , and the portable terminal 11 waits for transmission of the identification code until the timeout time elapses.

[0192] In the case where it is determined in step S7 that the timeout time has elapsed, the process is terminated.

[0193] On the other hand, in the event that it is determined in step S6 that an identification code has been received, the process proceeds to step S8. In step S8, the identification code encoding unit 57 performs encoding processing on the received identification code, and generates a blinking pattern.

[0194] In step S9 , the proximity sensor 58 outputs a blinking pattern of infrared light from the infrared light source 31 .

[0195] In step S10 , the communication unit 56 determines whether the registration completion notification transmitted from the information processing device 14 has been received.

[0196] In a case where it is determined in step S10 that the registration completion notification has been received, the process is terminated.

[0197] On the other hand, in the event that it is determined in step S10 that the registration completion notification has not been received, the process returns to step S7 and the subsequent process is performed. That is, the process for outputting a blinking pattern of infrared light is continuously performed until the timeout time elapses or until the registration completion notification is received.

[0198] Operation of the information processing device 14

[0199] Next, refer to Fig.18 The flowchart of FIG. 1 describes the registration process of the information processing device 14.

[0200] Fig.18 The registration process in is a series of processes for registering display control information in the installation information database.

[0201] In step S21, the communication unit 71 determines whether the device information and the posture information transmitted from the portable terminal 11 have been received, and waits until it is determined that the device information and the posture information have been received. Fig.17 In the process of step S5 , the device information and the posture information are sent in the portable terminal 11 .

[0202] In the event that it is determined in step S21 that the device information and the posture information have been received, the process proceeds to step S22. In step S22, the communication unit 71 supplies the device information and the posture information to the installation information storage unit 72, and registers the information in the installation information database.

[0203] In step S23 , the identification code generation unit 73 generates an identification code based on the key information associated with the registered device information.

[0204] In step S24, the communication unit 71 transmits the identification code to the portable terminal 11. Fig.17 Prior to the process of step S6, the identification code is received in the portable terminal 11.

[0205] In step S25, the position information acquisition unit 74 acquires a depth image from the infrared stereo camera 13, and waits until infrared light is detected in the depth image. In the case where infrared light is detected, the process proceeds to step S26.

[0206] In step S26 , the identification code decoding unit 91 performs a decoding process on the flicker pattern and decodes the identification code.

[0207] In step S27, the position information acquisition unit 74 determines whether the registration target device is identifiable by using the identification code decoded by the identification code decoding unit 91. For example, in the case where the identification code acquired from the installation information storage unit 72 and the identification code decoded by the identification code decoding unit 91 match each other, it is determined that the registration target device such as a smart speaker as the electrical device 21 is identifiable.

[0208] In a case where it is determined in step S27 that the registration target device is not identifiable, the process proceeds to step S28. In step S28, the communication unit 71 determines whether a timeout time has elapsed.

[0209] In the case where it is determined in step S28 that the timeout time has elapsed, the process is terminated.

[0210] In a case where it is determined in step S28 that the timeout time has not elapsed, the process returns to step S25 , and the subsequent processes are performed.

[0211] On the other hand, in the case where it is determined in step S27 that the registration target device is identifiable, the process proceeds to step S29. In step S29, the position detection unit 92 calculates the coordinates of the center of gravity of the bright spot of infrared light shown in the depth image by bright spot detection. The distance estimation unit 93 calculates the distance from the position of the bright spot to the position where the infrared stereo camera 13 is installed based on the brightness value of the bright spot shown in the depth image.

[0212] In step S30, the position information acquisition unit 74 generates position information of the projection surface based on the calculation result obtained by the position detection unit 92 and the calculation result obtained by the distance estimation unit 93. In addition, the position information acquisition unit 74 provides the position information to the installation information storage unit 72, and registers the position information in the installation information database in association with the device information of the registration target device identified using the identification code.

[0213] In step S31, the communication unit 71 transmits a registration completion notification to the portable terminal 11. Fig.17 Before the process of step S10 , the registration completion notification is received in the portable terminal 11 .

[0214] Then, refer to Fig.19 The flowchart of FIG. 1 describes the projection processing of the information processing device 14 .

[0215] Fig.19 The projection processing in is a series of processing for projecting display information provided from the registration target device based on the registered display control information.

[0216] In step S41 , the communication unit 71 determines whether display information transmitted from a registration target device such as a smart speaker as the electric device 21 has been received, and waits until it is determined that the display information has been received.

[0217] In the case where it is determined in step S41 that display information has been received, the process proceeds to step S42. In step S42, the position and posture processing unit 77 reads the position information and posture information from the installation information storage unit 72. In addition, the position and posture processing unit 77 reads the system installation information from the system installation information storage unit 75.

[0218] In step S43, the position and attitude processing unit 77 calculates a display transformation matrix based on the position information, attitude information, and system installation information.

[0219] In step S44 , the display processing unit 78 generates a display image based on the display information.

[0220] In step S45 , the display processing unit 78 converts the display image into a display image of the projector image coordinate system by using the display transformation matrix.

[0221] In step S46 , display processing unit 78 provides the display image of the projector image coordinate system to projector 12 , and projects the display image onto projector 12 .

[0222] In step S47, the display control unit 76 determines whether the projection has been terminated. For example, in the case where it is detected that the power of the registration target device such as a smart speaker as the electric device 21 has been turned off, it is determined that the projection has been terminated.

[0223] In a case where it is determined in step S47 that the projection has not been terminated, the process returns to step S41 , and the subsequent processes are performed.

[0224] In the case where it is determined in step S47 that the projection has been terminated, the processing is terminated.

[0225] Through the above-described processing, the information processing device 14 can detect the posture information of the projection surface without relying on object identification using the image captured by the camera.

[0226] The information processing device 14 can detect the position information and posture information of the projection surface by suppressing the influence of the resolution of the imaging device, the influence of the installation position of the imaging device, and the like even when the imaging range of the imaging device is wide.

[0227] <5. Modification Example>

[0228] ·Information registered in the installation information database

[0229] The shape such as the irregularity of the projection surface may be registered in the installation information database in association with the installation information.

[0230] Fig. 20 is a diagram showing another schema example of the installation information database.

[0231] exist Fig. 20 In the Fig.12 The described "ID", "installation location", "type", "display information" and "network information" are associated. "Shape" indicates the three-dimensional shape of the projection surface.

[0232] For example, information indicating the shape of the projection surface acquired using the portable terminal 11 is registered in association with the device information, the position information of the projection surface, and the posture information of the projection surface.

[0233] Thus, display information can be displayed according to the shape of irregularities such as the projection surface. For example, a smart speaker as a registration target device is used as a device that provides display information, and its cylindrical shell is also used as a projection surface, thereby displaying a display image projected by the projector 12.

[0234] System Configuration

[0235] In the above description, the information processing device 14 is provided as a device of a housing separate from the portable terminal 11 , the projector 12 , and the infrared stereo camera 13 , but all or some of the functions of the information processing device 14 may be implemented in the projector 12 .

[0236] Fig.21 is a block diagram showing a configuration example of the projector 12 .

[0237] like Fig.21 As shown, the projector 12 includes an information processing unit 111 and a projection unit 112 .

[0238] For example, the information processing unit 111 is provided with a reference Fig.11 All components of the information processing device 14 described above. The information processing unit 111 controls the entire projector 12 and projects a display image based on the registered display control information.

[0239] The projection unit 112 projects a display image under the control of the information processing unit 111 .

[0240] Note that the information processing unit 111 may be provided with some of the components of the information processing device 14 , and some of the remaining components of the information processing device 14 may be provided in another device.

[0241] Infrared stereo camera device 13 and infrared light source 31

[0242] Instead of the infrared stereo camera 13, a receiver including multiple antennas corresponding to short-range wireless communication such as Bluetooth 5.1 may be provided near the projector 12. In this case, instead of the infrared light source 31, a transmitter corresponding to short-range wireless communication such as Bluetooth 5.1 is provided in the portable terminal 11.

[0243] The receiver can detect the position of the transmitter through short-range wireless communication with the transmitter. Information indicating the position of the transmitter detected by the receiver is provided to the information processing device 14 and registered in the installation information database as position information of the projection surface.

[0244] Instead of infrared stereo camera 13, a camera may be provided near projector 12. When the distance between the camera and portable terminal 11 is short, a marker displayed on the display of portable terminal 11 is detected by the camera, and thus position information of the projection surface can be acquired.

[0245] Instead of the infrared stereo camera 13, a wearable device such as AR glasses can be used to obtain the position information of the projection surface. In this case, the information processing device 14 calculates the position of the infrared light source 31 of the portable terminal 11 in the world coordinate system based on the position of the wearable device estimated by the wearable device.

[0246] The detection parameters affecting the exposure time of the infrared stereo camera device 13 and the like may be adjusted according to the detection state of the infrared light source.

[0247] Notification of test status

[0248] In a state where infrared light detection is required (for example, a state where a flashing pattern of infrared light is output from the infrared light source 31 ), if the infrared stereo camera 13 fails to detect infrared light, it may be displayed on the display of the portable terminal 11 that infrared light cannot be detected.

[0249] ·Specify the projection area

[0250] The area of ​​the projection surface can be specified by a plurality of portable terminals 11 placed on the projection surface. In this case, the positions of a plurality of points are detected at a time by infrared light output by the infrared light source 31 provided in the plurality of portable terminals 11. The projected display image is such that the area formed by the detected positions of the plurality of points is registered as the projection surface and is adapted to the projection surface.

[0251] The portable terminal 11 is moved along a surface having a complex shape, and thus an area can be designated as a projection surface.

[0252] · Posture sensor

[0253] Information indicating the posture of the electric device 21 as the registration target device estimated based on sensor data obtained from the posture sensor provided in the electric device 21 may be registered in the installation information database as the posture information of the projection surface.

[0254] Projector 12

[0255] A drive-type projector may be used instead of the projector 12 installed on the ceiling, etc. In this case, the information processing device 14 calculates a display transformation matrix based on the position and posture of the drive-type projector in the world coordinate system.

[0256] Based on the display control information, not only the projection of the projector 12 but also the sound transmission of the speaker can be controlled. In this case, the position information of the output target is registered in the installation information database instead of the position information of the projection surface. In addition, the directivity of the speaker is adjusted based on the position information of the output target, and the sound is transmitted toward the position of the output target.

[0257] ·Getting device information

[0258] The portable terminal 11 and the electric device 21 may be provided with a communication module supporting Bluetooth (registered trademark). The portable terminal 11 may acquire device information from the electric device 21 as the registration target device through Bluetooth communication with the registration target device.

[0259] For example, a device or the like paired with the portable terminal 11 by Bluetooth is selected, and the position information and the posture information of the projection surface are registered.

[0260] In a case where the portable terminal 11 and the electric device 21 are connected to each other through a network such as the Internet or a wireless LAN, the device information can be transmitted from the electric device 21 as a registration target device to the portable terminal 11 through the network.

[0261] ·Communication method

[0262] The communication between the portable terminal 11 and the information processing apparatus 14 can be performed not only by wireless communication via a network such as Wi-Fi but also by short-range wireless communication such as Bluetooth.

[0263] The display information provided from the electric device 21 as the registration target device can be acquired by the information processing apparatus 14 through a device such as the portable terminal 11 which can be connected to the network.

[0264] Register the location of the target device and the location of the projection surface

[0265] Although the case where the display information is projected near the position where the electric device 21 as the registration target device is placed has been described above, the projection surface may be provided at a position irrelevant to the position where the electric device 21 as the registration target device is placed.

[0266] For example, the portable terminal 11 is placed on a wall surface, and thus a part of the area of ​​the wall surface is registered in the installation information database as the position of the projection surface. Information of various devices can be collectively displayed on the wall surface.

[0267] ·computer

[0268] The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, a program for the software is embedded in dedicated hardware to be installed from a program recording medium to a computer or a general-purpose personal computer.

[0269] Fig. 22 : is a block diagram showing an example of the hardware configuration of a computer that executes a program for executing the above-described series of processes.

[0270] A central processing unit (CPU) 201 , a read only memory (ROM) 202 , and a random access memory (RAM) 203 are connected to one another via a bus 204 .

[0271] The input / output interface 205 is also connected to the bus 204. An input unit 206 including a keyboard and a mouse and an output unit 207 including a display and a speaker are connected to the input / output interface 205. In addition, a storage unit 208 including a hard disk, a nonvolatile memory, etc., a communication unit 209 including a network interface, etc., and a drive 210 that drives a removable medium 211 are connected to the input / output interface 205.

[0272] In the computer configured as described above, the CPU 201 performs the above-described series of processing by, for example, loading a program stored in the storage unit 208 onto the RAM 203 via the input / output interface 205 and the bus 204 and executing the program.

[0273] The program executed by the CPU 201 is recorded on, for example, the removable medium 211 , or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting to be installed in the storage unit 208 .

[0274] Note that the program executed by the computer may be a program that executes processing chronologically in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing such as a call time.

[0275] ·other

[0276] In addition, in this specification, a system is a collection of multiple components (devices, modules (components), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a single device housing multiple modules in a single housing are both systems.

[0277] The effects described in this specification are merely examples and are not restrictive, and other effects may be obtained.

[0278] The embodiments of the present technology are not limited to the above-described embodiments, and various changes may be made without departing from the gist of the present technology.

[0279] For example, the present technology may have a configuration of cloud computing in which one function is shared by a plurality of devices via a network and is processed collectively.

[0280] In addition, each step described in the above flowchart may be performed by one device or shared by a plurality of devices.

[0281] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device, or may be shared and executed by a plurality of devices.

[0282] <Example of configuration combination>

[0283] The present technology can be configured as follows. (1)

[0285] An information processing device, comprising:

[0286] a registration unit configured to register display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor provided in a first device installed at the projection surface; and

[0287] A display control unit is configured to project the display information onto the projection surface and display the display information based on the display control information. (2)

[0289] The information processing device according to (1),

[0290] The registration unit registers the display control information in association with a device that is a source of providing the display information. (3)

[0292] The information processing device according to (2),

[0293] The display control information includes installation information, and the installation information includes information indicating at least any one of the position and posture of the projection surface. (4)

[0295] The information processing device according to (3),

[0296] The display control unit displays the display information at a specific position on the projection surface based on the installation information. (5)

[0298] The information processing device according to (3) or (4),

[0299] The installation information includes information indicating a posture of the projection surface, and the posture of the projection surface is estimated based on sensor data obtained from a posture sensor provided in the first device. (6)

[0301] The information processing device according to (5),

[0302] The posture sensor is a sensor including at least any one of an inertial sensor and a geomagnetic sensor. (7)

[0304] The information processing device according to any one of (3) to (6),

[0305] The installation information includes information indicating a position of the projection surface, the position of the projection surface being estimated based on a detection result of light emitted from a light source provided in the first device. (8)

[0307] The information processing device according to (7),

[0308] wherein the light source is included in a proximity sensor, and

[0309] The information processing apparatus further includes a position information acquisition unit that estimates a position of the projection surface based on a captured image obtained by imaging the light emitted from the light source. (9)

[0311] The information processing device according to (8), further comprising:

[0312] an identification information generating unit configured to generate identification information for identifying a second device as a registration target; and

[0313] A communication unit is configured to send the identification information to the first device. (10)

[0315] The information processing device according to (9), further comprising:

[0316] a decoding unit configured to decode a flickering pattern of light emitted from the light source, the flickering pattern being a pattern encoding the identification information,

[0317] The registration unit registers the device information about the second device identified by the identification information and the installation information in association with each other. (11)

[0319] The information processing device according to (9) or (10),

[0320] wherein the communication unit receives the display information from the second device, and

[0321] The display control unit displays the display information received from the second device. (12)

[0323] The information processing device according to (11), further comprising:

[0324] a display processing unit configured to perform processing for converting a display image indicating the display information based on the installation information;

[0325] The display control unit projects and displays the display image converted by the display processing unit. (13)

[0327] The information processing device according to any one of (3) to (12),

[0328] wherein the installation information includes information indicating an area of ​​the projection surface, the area of ​​the projection surface being estimated based on a detection result of light emitted from a light source provided in the first device, and

[0329] The display control unit projects the display information within the range of the area of ​​the projection surface. (14)

[0331] The information processing device according to any one of (3) to (13),

[0332] Wherein, the installation information includes information indicating the shape of the projection surface. (15)

[0334] The information processing device according to (11),

[0335] The display information is obtained from the second device through the first device. (16)

[0337] The information processing device according to (10),

[0338] The device information is acquired by the first device that has performed short-range wireless communication with the second device. (17)

[0340] The information processing device according to (16),

[0341] The short-range wireless communication includes communication using Bluetooth (registered trademark). (18)

[0343] The information processing device according to (10),

[0344] The device information is acquired by the first device that has performed communication with the second device via a network. (19)

[0346] The information processing device according to (12),

[0347] wherein the display processing unit performs processing for converting the display image based on information indicating the position and posture of the drive-type projection device, and

[0348] The display control unit projects and displays the display image converted by the display processing unit through the drive-type projection device. (20)

[0350] An information processing method, comprising:

[0351] The information processing device is caused to perform the following operations:

[0352] registering display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor provided in a first device, the first device being installed at the projection surface, and

[0353] Based on the display control information, the display information is projected onto the projection surface and displayed.

[0354] Reference Mark List

[0355] 11 Portable Terminal

[0356] 12 Projector

[0357] 13 Infrared stereo camera device

[0358] 14 Information processing device

[0359] 21 Electrical equipment

[0360] 31 Infrared light source

[0361] 51 NFC unit

[0362] 52 Object information input unit

[0363] 53 Object information storage unit

[0364] 54 Attitude Sensor

[0365] 55 Attitude Estimation Unit

[0366] 56 Communication Unit

[0367] 57 Identification code encoding unit

[0368] 58 Proximity Sensor

[0369] 71 Communication unit

[0370] 72 Installing the information storage unit

[0371] 73 Identification code generation unit

[0372] 74 Position information acquisition unit

[0373] 75 System installation information storage unit

[0374] 76 Display control unit

[0375] 77 Position and attitude processing unit

[0376] 78 Display control unit

[0377] 91 Identification code decoding unit

[0378] 92 Position detection unit

[0379] 93 Distance Estimation Unit

[0380] 111 Information Processing Unit

[0381] 112 Projection unit

Claims

1. An information processing device, comprising: a registration unit configured to register display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor provided in a first device installed at the projection surface; as well as a display control unit configured to project the display information onto the projection surface and display the display information based on the display control information, The display control information includes installation information, and the installation information includes information indicating a shape of the projection surface.

2. The information processing device according to claim 1, in, The registration unit registers the display control information in association with a device that is a supply source of the display information.

3. The information processing device according to claim 2, in, The installation information includes information indicating at least any one of a position and a posture of the projection surface.

4. The information processing device according to claim 3, in, The display control unit displays the display information at a specific position on the projection surface based on the installation information.

5. The information processing device according to claim 3, in, The installation information includes information indicating a posture of the projection surface, the posture of the projection surface being estimated based on sensor data obtained from a posture sensor provided in the first device.

6. The information processing device according to claim 5, in, The attitude sensor is a sensor including at least any one of an inertial sensor and a geomagnetic sensor.

7. The information processing device according to claim 3, in, The installation information includes information indicating a position of the projection surface, the position of the projection surface being estimated based on a detection result of light emitted from a light source provided in the first device.

8. The information processing device according to claim 7, in, The light source is included in a proximity sensor, and The information processing apparatus further includes a position information acquisition unit that estimates a position of the projection surface based on a captured image obtained by imaging the light emitted from the light source.

9. The information processing device according to claim 8, further comprising: an identification information generating unit configured to generate identification information for identifying a second device as a registration target; as well as A communication unit is configured to send the identification information to the first device.

10. The information processing device according to claim 9, further comprising: a decoding unit configured to decode a flickering pattern of light emitted from the light source, the flickering pattern being a pattern encoding the identification information, The registration unit registers the device information about the second device identified by the identification information and the installation information in association with each other.

11. The information processing device according to claim 9, in, The communication unit receives the display information from the second device, and The display control unit displays the display information received from the second device.

12. The information processing device according to claim 11, further comprising: a display processing unit configured to perform processing for converting a display image indicating the display information based on the installation information; The display control unit projects and displays the display image converted by the display processing unit.

13. The information processing device according to claim 3, in, The installation information includes information indicating an area of ​​the projection surface, the area of ​​the projection surface being estimated based on a detection result of light emitted from a light source provided in the first device, and The display control unit projects the display information within the range of the area of ​​the projection surface.

14. The information processing device according to claim 11, in, The display information is acquired from the second device through the first device.

15. The information processing device according to claim 10, in, The device information is acquired by the first device that has performed short-range wireless communication with the second device.

16. The information processing device according to claim 15, in, The short-range wireless communication includes communication using Bluetooth.

17. The information processing device according to claim 10, in, The device information is acquired by the first device having performed communication with the second device via a network.

18. The information processing device according to claim 12, in, The display processing unit performs processing for converting the display image based on information indicating the position and posture of the drive-type projection device, and The display control unit projects and displays the display image converted by the display processing unit through the drive-type projection device.

19. An information processing method, comprising: The information processing device is caused to perform the following operations: registering display control information for displaying display information on a projection surface, the display control information being generated based on sensor data obtained from a sensor provided in a first device, the first device being installed at the projection surface, and Based on the display control information, projecting the display information onto the projection surface and displaying the display information, The display control information includes installation information, and the installation information includes information indicating a shape of the projection surface.

Images