Information processing device, display, display system, information processing method, program, and display method

Abstract

To easily recognize the positions of objects behind obstacles.SOLUTION: A projection device 500 determines a superimposition position on a surface 30S of a shielding object 30, on which a virtual image of a marker M or marker M3 indicating an object 20 is superimposed, based on first position information indicating the position of the shielding object 30 having a surface and second position information indicating the position of the object 20 located on the opposite side of the surface of the shielding object.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, a display device, a display system, an information processing method, a program, and a display method. [Background technology]

[0002] Patent Document 1 discloses a dynamic target detection system that includes a radar that transmits radar waves into the surrounding area and receives reflected waves from an object to acquire a radar signal, a dynamic signal detection means that detects a dynamic signal from the radar signal output from the radar, a false image determination means that determines whether the detected dynamic signal is an out-of-line-of-line false image, and a real image position estimation means that calculates the distance and direction to the determined false image and estimates the position of a real image outside the line of sight based on the calculated distance and direction to the false image, and that is capable of detecting dynamic targets outside the line of sight with high accuracy with a low-cost configuration. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-135149 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to easily recognize the position of an object located on the opposite side of the surface of an obstruction. [Means for solving the problem]

[0005] The information processing device of the present invention includes a determination means for determining a superposition position on the surface of the obstruction on which a marker indicating an object or a virtual image of the marker is superimposed, based on first position information indicating the position of an obstruction having a surface and second position information indicating the position of an object located on the opposite side of the surface of the obstruction. Effect of the Invention

[0006] According to the present invention, the position of an object located on the opposite side to the surface of an obstruction can be easily recognized. [Brief description of the drawings]

[0007] [Figure 1] 1 is an overall configuration diagram of a projection system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of the measurement device according to the present embodiment. [Diagram 3] FIG. 1 is a diagram showing the hardware configuration of a projection device according to an embodiment of the present invention. [Figure 4] FIG. 1 is a functional block diagram of a projection system according to an embodiment of the present invention. [Diagram 5] 6 is a flowchart showing an example of processing of the projection system according to the present embodiment. [Figure 6] 6 is a flowchart showing an example of a point cloud generation process of the measuring apparatus according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing a marker according to the present embodiment. [Figure 8] 4A to 4C are explanatory diagrams of the position, size, and shape of a marker according to the present embodiment. [Figure 9] FIG. 13 is an explanatory diagram of a modified example according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] FIG. 1 is a diagram showing the overall configuration of a projection (display) system according to an embodiment of the present invention.

[0009] The projection system 10 includes a measurement device 900 and a projection device 500. The measurement device 900 irradiates the surroundings with irradiation light L such as infrared light by a scanning method or a flash method.

[0010] The projection system 10 is an example of a display system, and the projection device 500 is an example of a display device and an information processing device. The projection device 500 is of a movable type that can be moved relative to the measurement device 900.

[0011] The object 20 is located behind an obstruction 30 relative to the projection system 10, i.e., outside the line of sight (NLOS) on the opposite side of the surface 30S of the obstruction 30, and the illumination light L is blocked by the obstruction 30 and is therefore not directly irradiated onto the object 20.

[0012] Of the irradiation light L irradiated to the surrounding area, the irradiation light L irradiated toward the wall 40 is reflected by the wall 40 and then reaches the object 20, and the reflected light reflected by the object 20 is reflected by the wall 40 and received by the measuring device 900 as a multipath component.

[0013] The measuring device 900 detects the position of the object 20 based on the multipath components of the received reflected light, and the projection device 500 projects a marker M indicating the object 20 behind the obstruction 30 so as to be superimposed on the surface 30S of the obstruction.

[0014] Here, the projection device 500 displays a marker M at a superimposed position on the surface 30S of the obstructing object 30, where the marker M intersects with a virtual line VL connecting the projection device 500 and the object 20. This allows the user to recognize the actual position of the object 20 by visually recognizing the superimposed position of the marker M on the surface 30S of the obstructing object 30.

[0015] In the embodiment described in FIG. 1, the position of the object 20 is detected based on the multipath components of the received reflected light. However, in other embodiments, the position of the object 20 may be detected by, for example, a through-the-wall imaging method.

[0016] 2 is a diagram showing the hardware configuration of the measurement device according to this embodiment. The measurement device 900 includes a GNSS sensor 901, a light source 902, a Time Of Flight (TOF) sensor 903, an Inertial Measurement Unit (IMU) 904, and a control device 90.

[0017] The GNSS sensor 901 is a positioning means that measures a position on the earth by receiving signals transmitted from a plurality of GNSS satellites at different times and calculating the distance to the satellite from the difference in the time at which each signal was received. The positioning means may be a device dedicated to positioning, or may be an application dedicated to positioning installed on a PC (Personal Computer), a smartphone, or the like.

[0018] The light source 902 is a semiconductor laser, and emits laser light in a wavelength band other than the visible light region used for distance measurement (infrared light is used here as an example). The light source 902 is an example of a light projecting section.

[0019] The TOF sensor 903 measures the distance from the light source to the object by measuring the reflected light of the laser light emitted from the light source 902 and reflected by the object.

[0020] The IMU 904 is a gyro sensor that detects the angle (attitude) and angular velocity (or angular acceleration) of the measuring device 900. The IMU 904 measures the angles, angular velocities, and accelerations of three axes that govern the motion of the measuring device 900. The IMU 904 is used to calculate the position of the measuring device 900 after it has moved.

[0021] The control device 90 controls the GNSS sensor 901, the light source 902, the TOF sensor 903, and the IMU 904. Hereinafter, this will be referred to as the control device 90. The control device 90 includes a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HDD (Hard Disk Drive) 914, an external device connection I / F (Interface) 915, a network I / F 916, and a bus line 917.

[0022] Of these, the CPU 911 controls the overall operation of the measuring device 900. The CPU 911 reads programs, data, setting information, etc. from the ROM 912 etc. onto the RAM 913 and executes processing. Note that some or all of the control, image data processing, and various functions realized by the CPU 911 may be realized by a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0023] The ROM 912 stores various programs and data, various setting information, etc. The RAM 913 is used as a work area for the CPU 911. The HDD 914 stores sensor data input from the TOF sensor 903 and the IMU 904, etc.

[0024] The external device connection I / F 915 is an interface for connecting various external devices, and is connected to the projection device 500 wirelessly or via a wire to exchange data or signals.

[0025] The network I / F 916 is an interface for performing data communication using a communication network 100 such as the Internet.

[0026] The bus line 917 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 911 shown in FIG.

[0027] FIG. 3 is a diagram showing the hardware configuration of the projection (display, information processing) device according to this embodiment.

[0028] The projection device 500 includes a CPU 501, a ROM 502, a RAM 503, a HDD 504, an operation unit 506, a power switch 507, a network I / F 508, a bus line 510, an LED (Light Emitting Diode) driving circuit 511, an LED light source 512, a projection device 513, a projection lens 514, a fan driving circuit 515, a cooling fan 516, an external device connection I / F 517 and a power supply circuit 518.

[0029] Among these, the CPU 501 controls the operation of the entire projection device 500. The ROM 502 stores a program used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HDD 504 stores image data and the like. The operation unit 506 is provided with various keys, buttons, LEDs, and the like, and is used for a user to perform various operations other than turning the power of the projection device 500 on and off. For example, the operation unit 506 accepts instruction operations such as an operation to adjust the size of a projection image, an operation to adjust a color tone, an operation to adjust a focus, an operation to adjust a keystone, and the like, and outputs the contents of the accepted operation to the CPU 501. The power switch 507 is a switch for switching the power of the projection device 500 on and off. The network I / F 508 is an interface for data communication using the communication network 100 such as the Internet. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 8.

[0030] The LED driving circuit 511 controls turning on and off the LED light source 512 under the control of the CPU 501. When the LED light source 512 is turned on under the control of the LED driving circuit 511, it irradiates the projection light to the projection device 513. The projection device 513 modulates the projection light from the LED light source 512 by a spatial light modulation method based on image data provided via an external device connection I / F 517 or the like, and projects the modulated light as an image onto the projection surface of the screen through a projection lens 514. For example, a liquid crystal panel or a DMD (Digital Micromirror Device) is used as the projection device 513. The LED driving circuit 511, the LED light source 512, the projection device 513, and the projection lens 514 function as a whole as a projection unit (projection means) that projects a projection image onto the projection surface based on image data.

[0031] Fan drive circuit 515 is connected to CPU 501 and cooling fan 516, and drives / stops driving cooling fan 516 based on a control signal from CPU 501. Cooling fan 516 exhausts air from inside projection device 500 by rotating, thereby cooling the inside of projection device 500.

[0032] Furthermore, when power is supplied to the CPU 501, the CPU 501 starts up according to a control program previously stored in the ROM 502, and sends a control signal to the LED drive circuit 511 to turn on the LED light source 512, and sends a control signal to the fan drive circuit 515 to rotate the cooling fan 516 at a predetermined rated speed. Furthermore, when the supply of power from the power supply circuit 518 starts, the projection device 513 of the projection device 500 becomes capable of displaying an image, and power is supplied from the power supply circuit 518 to various other components. When the power switch 507 of the projection device 500 is turned OFF, a power OFF signal is sent from the power switch 507 to the CPU 501, and when the CPU 501 detects the power OFF signal, it sends a control signal to the LED drive circuit 511 to turn off the LED light source 512. After a predetermined time has elapsed, the CPU 501 sends a control signal to the fan drive circuit 515 to stop the cooling fan 516, terminates its own control processing, and finally sends an instruction to the power supply circuit 518 to stop the supply of power from the power source.

[0033] The external device connection I / F 517 is an interface for connecting various external devices. The external device connection I / F 517 is connected to the measuring device 900 wirelessly or via a wire to exchange data or signals.

[0034] The IMU 520 is a gyro sensor that detects the angle (attitude) and angular velocity (or angular acceleration) of the projection device 500. The IMU 520 measures the angles, angular velocity, and acceleration of three axes that govern the motion of the projection device 500. The IMU 520 is used to calculate the position of the projection device 500 after it has moved.

[0035] In the hardware configuration of each of the above devices or terminals, the recording medium such as HD or CD-ROM on which the program is stored can be provided domestically or internationally as a program product.

[0036] FIG. 4 is a functional block diagram of the projection system according to this embodiment. ○Control device functional configuration○ The control device (control board) 90 of the measuring device 900 has a communication unit 91, a control unit 92, a data acquisition unit 93, an acquired data management unit 94, a judgment unit 95, a processing unit 96, a point cloud management unit 97, and a storage / readout unit 99. Each of these units is a function or means realized by any of the components shown in Fig. 2 operating in response to an instruction from a CPU 911 in accordance with a program for the control device loaded from an HDD 914 onto a RAM 913. The control device 90 also has a storage unit 9000 constructed by a ROM 912 and HDD 914 shown in Fig. 2.

[0037] The communication unit 91 is mainly realized by the processing of the CPU 911 for the external device connection I / F 915, and has a function of exchanging various data or information with the projection device 500. The communication unit 91 transmits point cloud data managed by the point cloud management unit 97 to the projection device 500.

[0038] The control unit 92 is mainly realized by the processing of the CPU 911, and controls the light source 902. The data acquisition unit 93 is mainly realized by the processing of the CPU 911, and acquires various detection data obtained by the processing of various sensors such as the GNSS sensor 901, the TOF sensor 903, and the IMU 904.

[0039] The acquired data management unit 94 is mainly realized by the processing of the CPU 911, and stores and manages various data acquired by the data acquisition unit 93 in an acquired data management DB 9001. The judgment unit 95 is mainly realized by the processing of the CPU 911, and makes various judgments.

[0040] The processing unit 96 is mainly realized by the processing of the CPU 911, and calculates distance data and direction data, and generates point cloud data. The point cloud management unit 97 is mainly realized by the processing of the CPU 911, and stores and manages the point cloud data generated by the processing unit 96 in a point cloud data management DB 9002.

[0041] The storage / reading unit 99 is realized mainly by the processing of the CPU 911 , and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000 .

[0042] In the storage unit 9000, an acquired data management DB 9001 that stores and manages acquired data, and a point cloud data management DB 9002 that stores and manages point cloud data are constructed.

[0043] ○Functional configuration of the projection device○ The projection device 500 has a transmission / reception unit 51, a data acquisition unit 52, a projection control unit 53, a determination unit 54, an image generation unit 55, an image management unit 56, a communication unit 57, and a storage / readout unit 59. Each of these units is a function or means realized by any of the components shown in Fig. 4 being loaded from ROM 502 onto RAM 503 and operating in accordance with an instruction from CPU 501 in accordance with a program for the projection device. The projection device 500 also has a storage unit 5000 constructed by ROM 502 and HDD 504 shown in Fig. 4.

[0044] The transmission / reception unit 51 is mainly realized by processing of the CPU 501 for the network I / F 508, and communicates various data or information with other devices via the communication network 100. The transmission / reception unit 51 transmits, for example, image data to other devices.

[0045] The data acquisition unit 52 is mainly realized by the processing of the CPU 501, and acquires various detection data obtained by processing of various sensors such as the IMU 520.

[0046] The projection control unit 53 is mainly realized by the processing of the CPU 501, and projects an image onto the projection surface of the screen by controlling the LED drive circuit 511. The projection control unit 53 is an example of a display unit that displays the marker M.

[0047] The determination unit 54 is realized by the processing of the CPU 501, and performs processing to determine the position, size, and shape of the image. The determination unit 54 is an example of a determination means. The image generation unit 55 is mainly realized by the processing of the CPU 501, and generates the image determined by the determination unit 54.

[0048] The image management unit 56 is mainly realized by the processing of the CPU 501, and stores the image data generated by the image generation unit 55 in the image management DB 5001 and manages it.

[0049] The communication unit 57 is mainly realized by the processing of the CPU 501 for the external device connection I / F 517, and has a function of exchanging various data or information with the control device 90 of the measuring device 900.

[0050] The storage / reading unit 59 is realized mainly by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .

[0051] In the storage unit 5000, an image management DB 5001 that stores and manages image data is constructed.

[0052] In FIG. 4, the measurement device 900 may be configured to communicate with the projection device 500 via the communication network 100, or may be configured integrally with the projection device 500.

[0053] FIG. 5 is a flowchart showing an example of processing of the projection system according to this embodiment.

[0054] The control unit 92 of the control device 90 of the measuring device 900 controls the light source 902 to irradiate the surrounding area with laser light using a scanning method or a flash method (step S1), the data acquisition unit 93 acquires detection data of the reflected light of the laser light irradiated from the light source 902 from the TOF sensor 903, and the acquired data management unit 94 stores the detection data of the reflected light acquired by the data acquisition unit 93 in the acquired data management DB 9001 via the memory / readout unit 99 (step S2).

[0055] Here, the detection data of reflected light acquired by the data acquisition unit 93 includes detection data of reflected light reflected by the obstruction 30 in FIG.

[0056] Also, in step S2, the data acquisition unit 93 acquires location information indicating the position of the measuring device 900 measured by the GNSS sensor 901, and the acquired data management unit 94 stores the location information of the measuring device 900 acquired by the data acquisition unit 93 in the acquired data management DB 9001 via the memory / reading unit 99.

[0057] The processing unit 96 generates point cloud data indicating a three-dimensional point cloud based on the reflected light detection data acquired by the data acquisition unit 93, and the point cloud management unit 97 stores the point cloud data generated by the processing unit 96 in the point cloud data management DB 9002 by the storage / readout unit 99 (step S3). The communication unit 91 transmits the point cloud data generated by the processing unit 96 to the projection device 500, and the communication unit 57 of the projection device 500 receives the point cloud data transmitted from the control device 90.

[0058] Here, the three-dimensional point cloud is a collection of coordinate points in the X, Y, Z directions, etc., corresponding to measurement points on the surface of an object when a space in which the object is located is measured. Each coordinate point is expressed as (1, 3, 5), for example, with the position of the measuring device 900 measured by the GNSS sensor 901 as the origin. In addition, color information may be added to each coordinate point, and RGB values ​​of each coordinate point may be added as color information. The three-dimensional point cloud is also called a point cloud. Point cloud data is data that can be handled by a computer or the like as a collection of coordinate points in a virtual three-dimensional space.

[0059] The point cloud data generated by the processing unit 96 also includes point cloud data indicating a three-dimensional point cloud of the surface 30S of the obstructing object in FIG.

[0060] The point cloud data indicating a three-dimensional point cloud of the surface 30S of the obstruction is calculated based on a first reflected light that is generated when the light emitted from the light projector is reflected by the obstruction 30, and is an example of first positional information indicating the position of the obstruction 30.

[0061] The three-dimensional point cloud on the surface of object 20 is calculated based on the second reflected light that is generated when the irradiated light that has bypassed the obstruction 30 and is irradiated onto object 20 is reflected by object 20 and bypasses the obstruction 30, and is an example of second positional information indicating the position of object 20.

[0062] The determining unit 54 determines a superimposition position on the surface 30S of the obstructing object where the marker M indicating the object 20 is superimposed, based on the point cloud data received by the communication unit 57 (step S4). Step S4 is an example of a determining step.

[0063] Here, if the projection system 10 moves or the projection device 500 moves relative to the measurement device 900 after obtaining the detection data of the reflected light in step S2, the determination unit 54 corrects and determines the superimposition position based on the position of the projection device 500 calculated from the detection data of the IMU 520.

[0064] The position of the projection device 500 calculated based on the detection data of the IMU 520 is an example of third position information that indicates the position of the display means that displays the marker.

[0065] The determination unit 54 determines the size and shape of the marker M at the superimposition position determined in step S4, based on the point cloud data indicating the three-dimensional point cloud on the surface of the object 20 and the superimposition position determined in step S4 (step S5).

[0066] The image generation unit 55 generates an image of the marker M having the size and shape determined in step S5 (step S6), and the projection control unit 53 projects the image of the marker M generated in step S6 onto the superimposition position determined in step S5 (step S7).

[0067] FIG. 6 is a flowchart showing an example of a point cloud generation process of the measuring device according to this embodiment, and shows a process corresponding to step S3 in FIG.

[0068] The processing unit 96 calculates distance data indicating the distance to the measurement point where the irradiated light is reflected, based on the detection data of the reflected light acquired by the data acquisition unit 93 (step S11).

[0069] The judgment unit 95 judges whether the detection data of the reflected light acquired by the data acquisition unit 93 is a multipath component (step S12), and if it is a multipath component, the processing unit 96 calculates directional data indicating the direction of the multipath component of the reflected light based on each reflected light from adjacent measurement points (step S13).

[0070] The processing unit 96 calculates directional data indicating the directions of the multipath components of the reflected light, for example, by a method shown in A Theory of Fermat Paths for Non-Line-of-Sight Shape Reconstruction, cvpr2019, etc.

[0071] The processing unit 96 generates point cloud data of the measurement points based on the distance data and direction data (step S14). Specifically, for the reflected light other than the multipath component, the processing unit 96 generates point cloud data based on the distance data calculated in step S11 and direction data indicating the direction of the irradiated light, and for the multipath component of the reflected light, generates point cloud data based on the distance data calculated in step S11 and the direction data calculated in step S13.

[0072] 7A and 7B are diagrams showing markers according to this embodiment. As shown in Fig. 7A, the projection control unit 53 projects and displays a marker M1 indicating an object 20 behind a shield 30 at a superimposed position on the surface 30S of the shield. Here, the marker M1 is determined by the determination unit 54 to have a shape and size corresponding to the three-dimensional point cloud on the surface of the object 20, and is an image generated by the image generation unit 55.

[0073] 7(b), the projection control unit 53 may project and display a marker M2 indicating the object 20 behind the obstruction 30 at a superimposed position on the surface 30S of the obstruction. Here, the marker M2 is an image generated by the image generation unit 55, the shape and size of which are determined by the determination unit 54 in a planar view of a three-dimensional model obtained by replacing the three-dimensional point cloud on the surface of the object 20.

[0074] With the above configuration, a user can visually recognize the markers M1 and M2 superimposed on the surface 30S of the obstruction 30 and recognize the position, size, and shape of the object 20 that is obscured by the obstruction 30 and cannot be observed with the naked eye.

[0075] This will make it possible to see blind spots, and is expected to help with searching for luggage in places with many shelves, such as warehouses and libraries, as well as prevent the current social problem of children being forgotten to be removed from shuttle buses.

[0076] 7, the projection control unit 53 may constantly project and display markers M1 and M2 indicating the object 20 behind the obstruction 30 at superimposed positions on the surface 30S of the obstruction. This allows the user to quickly recognize the object 20 that is out of the line of sight.

[0077] Furthermore, the projection control unit 53 may project and display markers M1 and M2 indicating the object 20 behind the obstruction 30 at superimposed positions on the surface 30S of the obstruction based on a user operation on the operation unit 506. This allows the user to recognize the object 20 behind a specific obstruction 30.

[0078] Furthermore, the projection control unit 53 may automatically project and display the markers M1 and M2 indicating the object 20 behind the obstruction 30 at a superimposed position on the surface 30S of the obstruction, based on the surrounding environment information.

[0079] As another example of a display system according to the present invention, a display device having a display control unit may be provided instead of the projection device 500 having the projection control unit 53, and the display control unit may display on a display a display screen in which a planar image of a three-dimensional model in which point cloud data indicating the three-dimensional point cloud of the surface of the object 20 has been replaced is superimposed on the surface of the three-dimensional model in which point cloud data indicating the three-dimensional point cloud of the surface 30S of the obstructing object has been replaced.

[0080] FIG. 8 is an explanatory diagram of the position, size, and shape of the marker according to this embodiment.

[0081] Based on the point cloud data 30PC indicating the three-dimensional point cloud on the surface 30S of the obstructing object and the three-dimensional point cloud 20PC of the surface of the object 20, the determination unit 54 determines, by perspective projection transformation, the superimposition position on the surface 30S of the obstructing object where the marker M will be superimposed, as well as the size and shape of the marker M when a projection plane PA is set on the surface 30S of the obstructing object with the viewpoint PV as the projection center.

[0082] As a result, by visually recognizing the marker M superimposed on the surface 30S of the obstruction 30, the user can recognize the position, size and shape of the object 20 that is obscured by the obstruction 30 and cannot be observed with the naked eye, with little cognitive load, as if the obstruction 30 were transparent, unlike when seeing a reflected image in a mirror or the like.

[0083] FIG. 9 is an explanatory diagram of a modified example according to the present embodiment.

[0084] 9, the projection device 500 further includes smart glasses 600, and the projection control unit 53 projects a marker M3 indicating an object 20 behind a shield 30 onto a lens 60 of the smart glasses 600 such that a virtual image of the marker M3 is superimposed on a surface 30S of the shield. The lens 60 is an example of a transmissive reflecting member.

[0085] 3 is provided in the smart glasses 600. As a result, when the smart glasses 600 move, the determination unit 54 corrects and determines the superimposition position of the virtual image of the marker M3 on the surface 30S of the obstructing object based on the position of the smart glasses 600 calculated from the detection data of the IMU 520.

[0086] In this modified example, since the marker M3 cannot be seen by anyone other than the user, it is expected to be used for tracking criminals in criminal investigations.

[0087] ●Summary● [First aspect] As described above, the projection device 500 according to one embodiment of the present invention determines a superimposition position on the surface 30S of the obstruction on which a virtual image of the marker M or marker M3 indicating the object 20 is superimposed, based on first position information indicating the position of the obstruction 30 and second position information indicating the position of the object 20 located behind the obstruction 30, i.e., on the opposite side to the surface 30S of the obstruction 30. The projection device 500 is an example of an information processing device, and the determination unit 54 is an example of a determination means.

[0088] This allows the user to visually recognize the superimposed position of the marker M on the surface 30S of the obstructing object 30, thereby enabling the user to recognize the actual position of the object 20.

[0089] [Second aspect] In the first mode, the determination unit 54 determines the superimposition position based on the first position information, the second position information, and third position information indicating the position of the projection control unit 53 that projects the marker M. The projection control unit 53 is an example of a display unit that displays the marker M.

[0090] As a result, even if the projection device 500 moves, the superimposition position is determined based on the position of the projection device 500, so that the user can correctly visually recognize the superimposition position of the marker M on the surface 30S of the obstructing object 30.

[0091] [Third aspect] In the first or second embodiment, the determination unit 54 further determines the size of the virtual image of the marker M or the marker M3 in accordance with the size of the object 20. This allows the user to recognize the actual size of the object 20.

[0092] [Fourth aspect] In any of the first to third aspects, the determining unit 54 further determines the shape of the virtual image of the marker M or the marker M3 in accordance with the shape of the object 20. This allows the user to recognize the actual shape of the object 20.

[0093] [Fifth aspect] In any of the first to fourth aspects, the second position information indicates the position of the three-dimensional point group 20PC corresponding to the object 20.

[0094] [Sixth aspect] In any of the first to fifth aspects, the first position information is calculated based on a first reflected light that is generated when light emitted from the light source 902 is reflected by the obstruction 30, and the second position information is calculated based on a second reflected light that is generated when light that has been irradiated to the object 20 after bypassing the obstruction 30 is reflected by the object 20 and bypasses the obstruction 30. The light source 902 is an example of a light projecting unit.

[0095] [Seventh aspect] A projection device 500 according to an embodiment of the present invention includes a projection control unit 53 that displays a marker M indicating an object 20 behind a shield 30 on a surface 30S of the shield, or displays the marker M through a lens 60 so that a virtual image of the marker M is superimposed on the surface 30S of the shield. The projection device 500 is an example of a display device, and the lens 60 is an example of a transmissive reflecting member.

[0096] [Eighth aspect] In the seventh embodiment, the marker M or a virtual image of the marker M is superimposed on the surface 30S of the shield at a superimposition position according to the position of the object 20.

[0097] [Ninth aspect] In the seventh or eighth embodiment, the marker M or a virtual image of the marker M is superimposed at a superimposition position according to the position of the projection control unit 53 on the surface 30S of the obstructing object.

[0098] [Tenth aspect] In any of the seventh to ninth embodiments, the virtual image of the marker M or the marker M3 has a size corresponding to the size of the object 20.

[0099] [Eleventh aspect] In any of the first to tenth embodiments, the virtual image of the marker M or the marker M3 has a shape corresponding to the shape of the object 20.

[0100] [12th aspect] In any of the first to eleventh embodiments, the virtual image of the marker M or the marker M3 is an image obtained by viewing the three-dimensional point group 20PC corresponding to the object 20 or a three-dimensional model obtained by replacing the three-dimensional point group 20PC in a plan view.

[0101] [13th aspect] In any of the seventh to twelfth aspects, the marker M or a virtual image of the marker M is superimposed at a superimposition position on the surface 30S of the obstruction, which is determined based on a first reflected light formed when the light emitted from the light source 902 is reflected by the obstruction 30, and a second reflected light formed when the light emitted to the object 20 after bypassing the obstruction 30 is reflected by the object 20 and bypasses the obstruction 30.

[0102] [14th aspect] A projection system 10, which is an example of a display system according to an embodiment of the present invention, includes a measurement device 900 that receives a first reflected light formed when light emitted from a light source 902 is reflected by an obstruction 30, and receives a second reflected light formed when the light emitted to an object 20 after bypassing the obstruction 30 is reflected by the object 20 and bypasses the obstruction 30, and a projection device 500 that displays a marker M indicating the object 20 at a superimposition position on a surface 30S of the obstruction determined based on the first reflected light and the second reflected light, or on a lens 60 so that a virtual image of the marker M is superimposed on the superimposition position.

[0103] [15th aspect] An information processing method performed by a projection device 500 according to one embodiment of the present invention executes a determination step of determining a superimposition position on a surface 30S of the obstruction on which a marker M indicating the object 20 or a virtual image of the marker M is superimposed, based on first position information indicating the position of the obstruction 30 and second position information indicating the position of the object 20 behind the obstruction 30.

[0104] [16th aspect] A program according to one embodiment of the present invention causes a computer to execute a determination step of determining a superposition position on a surface 30S of an obstruction on which a marker M indicating an object 20 or a virtual image of the marker M is superimposed, based on first position information indicating the position of the obstruction 30 and second position information indicating the position of an object 20 behind the obstruction 30.

[0105] [17th aspect] A display method according to one embodiment of the present invention displays a marker M indicating an object 20 behind an obstruction 30 on a surface 30S of the obstruction, or displays the marker M on a lens 60 so that a virtual image of the marker M is superimposed on the surface 30S of the obstruction. [Explanation of symbols]

[0106] 10 Projection system (an example of a display system) 100 Communication Network 900 Measuring Equipment 902 Light source (example of light projector) 500 Projection device (an example of a display device or information processing device) 53 Projection control unit (an example of a display means) 54 Determination unit (an example of a determination means) 20 objects 20PC 3D point cloud of object 30 Shield 30S Cover Surface 30PC 3D point cloud of obstruction 40 Wall 60 Lens (an example of a transmissive reflective material) 600 Smart Glasses L Ranging light IL Projection Light PA Projection Area PV Viewpoint M Marker

Claims

1. An information processing device comprising: a determination means for determining a superposition position on the surface of an obstructing object on which a marker indicating the object or a virtual image of the marker is superimposed, based on first position information indicating the position of the obstructing object having a surface and second position information indicating the position of an object located on the opposite side of the surface of the obstructing object.

2. The information processing apparatus according to claim 1 , wherein the determining means determines the superimposition position based on the first position information, the second position information, and third position information indicating a position of a display means that displays the marker.

3. The information processing apparatus according to claim 1 , wherein the determining means further determines a size of the marker or the virtual image of the marker in accordance with a size of the object.

4. The information processing apparatus according to claim 1 , wherein the determining means further determines a shape of the marker or the virtual image of the marker in accordance with a shape of the object.

5. The information processing apparatus according to claim 1 , wherein the second position information indicates a position of a three-dimensional point cloud corresponding to the object.

6. the first position information is calculated based on a first reflected light that is generated when light emitted from a light projecting unit is reflected by the obstruction, The information processing apparatus according to claim 1 , wherein the second position information is calculated based on second reflected light, which is generated when the irradiated light, which has detouring the obstruction and been irradiated onto the object, is reflected by the object and detouring the obstruction.

7. A display device comprising a display means for displaying, on a surface of an obstruction, a marker indicating an object located on the opposite side of the surface of the obstruction, or for displaying the marker on a transmissive reflective member so that a virtual image of the marker is superimposed on the surface of the obstruction.

8. The display device according to claim 7 , wherein the marker or the virtual image of the marker is superimposed on a surface of the obstruction at a superimposition position according to a position of the object.

9. The display device according to claim 7 , wherein the marker or the virtual image of the marker is superimposed at the superimposition position on the surface of the obstruction, the superimposition position corresponding to the position of the display means.

10. The display device according to claim 7 , wherein the marker or the virtual image of the marker has a size corresponding to a size of the object.

11. The display device according to claim 7 , wherein the marker or the virtual image of the marker has a shape corresponding to a shape of the object.

12. The display device according to claim 7 , wherein the marker or the virtual image of the marker is an image obtained by viewing a three-dimensional point cloud corresponding to the object or a three-dimensional model obtained by replacing the three-dimensional point cloud in a planar view.

13. The marker or the virtual image of the marker is 8. The display device according to claim 7, wherein the illumination light irradiated from a light-projecting unit is superimposed on a superimposition position on the surface of the obstruction that is determined based on a first reflected light that is reflected by the obstruction and a second reflected light that is reflected by the object and bypasses the obstruction after the illumination light is irradiated to the object.

14. a measuring device that receives a first reflected light that is generated when the illumination light emitted from a light projecting unit is reflected by an obstruction, and receives a second reflected light that is generated when the illumination light that has been irradiated onto an object after bypassing the obstruction is reflected by the object and bypasses the obstruction; a display device that displays a marker indicating the object on a transmissive reflective member at a superimposition position on a surface of the obstruction that is determined based on the first reflected light and the second reflected light, or such that a virtual image of the marker is superimposed on the superimposition position; A display system comprising:

15. An information processing method executed by an information processing device, An information processing method that executes a determination step of determining a superposition position on the surface of an obstructing object onto which a marker indicating the object or a virtual image of the marker is superimposed, based on first position information indicating the position of an obstructing object having a surface and second position information indicating the position of an object located on the opposite side of the surface of the obstructing object.

16. A program that causes a computer to execute a determination step of determining a superposition position on the surface of an obstruction on which a marker indicating the object or a virtual image of the marker is superimposed, based on first position information indicating the position of an obstruction having a surface and second position information indicating the position of an object located on the opposite side of the surface of the obstruction.

17. A display method in which a marker indicating an object located on the opposite side of a surface of an obstruction is displayed on the surface of the obstruction, or the marker is displayed on a translucent reflective member so that a virtual image of the marker is superimposed on the surface of the obstruction.

Images