Mixed reality display apparatus, method, and program

Abstract

To provide a mixed reality display apparatus and the like that can contribute to improvement in accuracy of alignment between point cloud data and a photographed object in a real space.SOLUTION: A mixed reality display apparatus includes: a position sensor that generates position data of a three-dimensional sensor; a position data generation unit that generates latest position data on the basis of latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data generated by the position sensor; a correction data generation unit that generates latest correction data on the basis of a positional difference generated by performing alignment between the latest point cloud data and the past point cloud data; and a point cloud data display unit that displays the latest point cloud data in a state in which the latest point cloud data is aligned with a photographed object and corrected on the basis of the latest position data and the latest correction data.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a mixed reality display device, method, and program. [Background technology]

[0002] In industries such as the electric power and construction, the use of 3D sensors is one way to improve the efficiency of facility inspection and monitoring. 3D sensors can output the captured area as 3D data known as point cloud data (3D point cloud data), which represents multiple points with specific 3D coordinates. By processing the point cloud data using dedicated software, it becomes possible to perform inspection and monitoring tasks such as measuring the distance to facilities, measuring the volume of objects, and detecting objects.

[0003] There are two problems with inspection and monitoring work using 3D sensors.

[0004] One problem is the poor visibility of point cloud data. For example, in the case of 3D-LIDAR (Light Detection and Ranging), a common 3D sensor, the 3D-LIDAR emits infrared light to measure the distance to the object being photographed, so the acquired point cloud data does not reproduce the colors of real space. When displaying point cloud data on a display device (such as a personal computer or tablet device), the object must be identified based on the shape of the point cloud, which can be difficult to identify in areas where the point cloud data is sparse.

[0005] Another problem is the poor portability and operability of the display device during inspection work. When operating the display device during inspection work, it is necessary to use one or both hands. This makes it very difficult to perform measurement work when one hand is occupied.

[0006] One solution to these problems is the use of mixed reality (MR) technology. For example, the mixed reality system disclosed in Patent Document 1 includes a portable display device having a see-through display unit and an imaging unit that captures real space, and renders point cloud data of a virtual object in a predetermined real space on the display unit, allowing the user to view the data, thereby solving the problems of visibility and operability. When projecting the point cloud data onto the display unit, alignment is performed so that the projected point cloud data is superimposed on the real space. In Patent Document 1, position information of a three-dimensional sensor and the display unit is acquired by a position measurement means such as a GPS (Global Positioning System) sensor, Bluetooth (registered trademark) beacon, or Wi-Fi, and alignment is performed between the real space and the point cloud data of the virtual object based on the acquired position information. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6171079 Summary of the Invention [Problem to be solved by the invention]

[0008] The following analysis is provided by the present inventors.

[0009] However, since the accuracy of alignment between the real space and the point cloud data of a virtual object depends on the accuracy of the position measurement means, there is a possibility that alignment errors will occur depending on the position measurement means used in the mixed reality system of Patent Document 1. Furthermore, since the accuracy of the point cloud data also depends on the accuracy of the 3D sensor used, there is a possibility that alignment errors will occur depending on the coordinate accuracy of the projected point cloud data.

[0010] A main object of the present invention is to provide a mixed reality display device, method, and program that can contribute to improving the accuracy of alignment between point cloud data and a photographed object in real space. [Means for solving the problem]

[0011] a position data generation unit configured to generate latest position data of the 3D sensor corresponding to the latest point cloud data based on the latest point cloud data generated by the 3D sensor, past point cloud data generated by the 3D sensor, and past position data of the 3D sensor generated by the position sensor and corresponding to the past point cloud data; a correction data generation unit configured to calculate a position difference resulting from aligning the latest point cloud data with the past point cloud data, and to generate latest correction data of the 3D sensor corresponding to the latest point cloud data based on the difference; and a point cloud data display unit configured to display the latest point cloud data in a state in which the latest point cloud data has been aligned and corrected with respect to the object based on the latest position data and the latest correction data.

[0012] The mixed reality display method relating to the second viewpoint includes the steps of: a position sensor detecting the position of a 3D sensor that captures an object in real space, and generating position data of the 3D sensor corresponding to point cloud data generated by the 3D sensor; a computer generating latest position data of the 3D sensor corresponding to the latest point cloud data, based on the latest point cloud data generated by the 3D sensor, past point cloud data generated by the 3D sensor, and past position data of the 3D sensor generated by the position sensor and corresponding to the past point cloud data; the computer calculating a position difference resulting from aligning the latest point cloud data with the past point cloud data, and generating latest correction data of the 3D sensor corresponding to the latest point cloud data, based on the difference; and the computer displaying the latest point cloud data in a state in which the latest point cloud data has been aligned and corrected with respect to the object based on the latest position data and the latest correction data.

[0013] The program relating to the third viewpoint causes a computer to execute the following processes: detecting, via a position sensor, the position of a 3D sensor that captures an image of an object to be captured in real space, and generating position data of the 3D sensor corresponding to point cloud data generated by the 3D sensor; generating latest position data of the 3D sensor corresponding to the latest point cloud data, based on the latest point cloud data generated by the 3D sensor, past point cloud data generated by the 3D sensor, and past position data of the 3D sensor generated by the position sensor that corresponds to the past point cloud data; calculating a position difference resulting from aligning the latest point cloud data with the past point cloud data, and generating latest correction data of the 3D sensor corresponding to the latest point cloud data, based on the difference; and displaying the latest point cloud data in a state in which the latest point cloud data has been aligned and corrected with respect to the object to be captured, based on the latest position data and the latest correction data.

[0014] The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present disclosure can also be embodied as a computer program product. The program is input to a computer device via an input device or a communication interface from the outside, stored in a storage device, and drives a processor according to predetermined steps or processes. The processing results, including intermediate states as needed, can be displayed at each stage on a display device, or the computer device can communicate with the outside via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and a display device as needed, all of which can be connected to each other via a bus. [Effects of the Invention]

[0015] The first to third aspects can contribute to improving the accuracy of alignment between point cloud data and a photographed object in real space. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram schematically illustrating a first example of the configuration of a mixed reality display device according to the present disclosure. [Figure 2] FIG. 10 is an image diagram schematically illustrating an example of the mixed reality display device according to the present disclosure when acquiring point cloud data. [Figure 3] FIG. 1 is an image diagram schematically illustrating an example of point cloud data displayed on a mixed reality display device according to the present disclosure. [Figure 4] 1 is an image diagram showing a schematic example of the positional relationship between the imaging position of a 3D sensor of a mixed reality display device according to the present disclosure and an object to be imaged. [Figure 5] FIG. 10 is an image diagram schematically illustrating an example of the display position of point cloud data when viewed from (0,0,0) on a mixed reality display device according to the present disclosure. [Figure 6]FIG. 10 is a flowchart diagram illustrating an example of the operation of the mixed reality display device according to the present disclosure. [Figure 7] FIG. 10 is a flowchart diagram illustrating an example of detailed operations performed when generating position data of the mixed reality display device according to the present disclosure. [Figure 8] FIG. 10 is a flowchart diagram schematically illustrating an example of detailed operations performed when generating correction data in the mixed reality display device according to the present disclosure. [Figure 9] FIG. 2 is a block diagram schematically illustrating a second example of the configuration of the mixed reality display device according to the present disclosure. [Figure 10] FIG. 2 is a block diagram illustrating a configuration of hardware resources. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following description of the embodiments will be made with reference to the drawings. Note that, where reference numerals are used in this application, they are intended solely to facilitate understanding and are not intended to limit the present invention to the illustrated embodiments. Furthermore, the following embodiments are merely exemplary and do not limit the present invention. Furthermore, connecting lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown, input and output ports exist at the input and output ends of each connecting line in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, and the like shown in this disclosure. The same applies to input and output interfaces. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. The computer device is configured to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless.

[0018] [Form 1] A mixed reality display device according to a first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram schematically illustrating a first example of the configuration of a mixed reality display device according to the present disclosure. FIG. 2 is an image diagram schematically illustrating an example of the mixed reality display device according to the present disclosure when acquiring point cloud data. FIG. 3 is an image diagram schematically illustrating an example of the mixed reality display device according to the present disclosure when displaying point cloud data.

[0019] The mixed reality display device 1 is a device that displays point cloud data (60 in FIG. 3 ) superimposed on a photographed object (50 in FIG. 3 ) in real space (see FIG. 1 ). The mixed reality display device 1 has a function of performing correction processing in the process of displaying the point cloud data 60 captured by the 3D sensor 20 on the display unit 15 so as to improve the accuracy of alignment between the point cloud data 60 and the photographed object 50 in real space. The mixed reality display device 1 can be, for example, smart glasses or a smartphone equipped with a computer function. When the mixed reality display device 1 is smart glasses, the mixed reality display device 1 is worn on the head of the worker 3 as shown in FIG. 2 in order to superimpose the point cloud data (60 in FIG. 3 ) acquired by the 3D sensor 20 on the photographed object (50 in FIG. 3 ) in real space. The mixed reality display device 1 is configured to be able to communicate with an external terminal (not shown) or a server (not shown) via a network (not shown). The mixed reality display device 1 can be used, for example, for measurement work at work sites in the manufacturing and construction industries, and for measurement work on high-altitude facilities in the power and railway industries. The mixed reality display device 1 includes a point cloud analysis unit 10, a three-dimensional sensor 20, and a position sensor 30.

[0020] The point cloud analysis unit 10 is a functional unit that analyzes point cloud data (60 in FIG. 3) (see FIG. 1). For example, a computer can be used as the point cloud analysis unit 10. By executing a predetermined program stored therein, the point cloud analysis unit 10 can be configured to include a point cloud data control unit 11, a position data control unit 12, a correction data control unit 13, an environmental data control unit 14, a display unit 15, and a data storage unit 16.

[0021] The point cloud data control unit 11 is a functional unit that controls the acquisition and storage of point cloud data (60 in FIG. 3) (see FIG. 1). The point cloud data control unit 11 includes a point cloud data acquisition unit 11a and a point cloud data storage unit 11b.

[0022] The point cloud data acquisition unit 11a is a functional unit that acquires point cloud data (60 in FIG. 3) from the three-dimensional sensor 20 (see FIG. 1).

[0023] The point cloud data saving unit 11b is a functional unit that saves the acquired point cloud data (60 in FIG. 3) in the data storage unit 16 (see FIG. 1).

[0024] The position data control unit 12 is a functional unit that controls the acquisition, reference, generation, and storage of position data (see FIG. 1). The position data control unit 12 includes a position data acquisition unit 12a, a data reference unit 12b, a position data generation unit 12c, and a position data storage unit 12d. Here, the position data is data that specifies the position of the 3D sensor 20 (corresponding to the position of the worker 3, the position of the mixed reality display device 1, and the shooting position; including the posture), and may be data including, for example, latitude, longitude, altitude, rotation angle, etc.

[0025] The position data acquisition unit 12a is a functional unit that acquires position data from the position sensor 30 (see FIG. 1).

[0026] The data reference unit 12b is a function that refers to data (point cloud data, position data, correction data, and environmental data) stored in the data storage unit 16 when automatic generation of position data is selected (see FIG. 1).

[0027] The position data generating unit 12c is a functional unit that generates position data (see FIG. 1). If automatic generation of position data is not selected, the position data generating unit 12c does not generate position data, and instead can use the position data acquired from the position sensor 30 by the position data acquiring unit 12a. When automatic generation of position data is selected, the position data generation unit 12c refers to the data stored in the data memory unit 16 via the data reference unit 12b, acquires the latest point cloud data (which may be point cloud data acquired directly from the 3D sensor 20), past point cloud data and its corresponding data (past position data, past correction data, and past environmental data as needed), identifies common points (e.g., feature values ​​such as edges and vertices) between the acquired latest point cloud data and the past point cloud data, registers (aligns) the latest point cloud data with the past point cloud data so that the identified common points overlap, calculates the difference caused by the registration (e.g., the difference between the position of the latest point cloud data before and after alignment with the past point cloud data), calculates the shooting position (position of the 3D sensor 20) corresponding to the latest point cloud data based on the calculated difference, the past position data corresponding to the past point cloud data, and the past correction data, and generates position data corresponding to the latest point cloud data based on the calculated shooting position and, as needed, the past environmental data.

[0028] The position data saving unit 12d is a functional unit that associates the position data acquired by the position data acquiring unit 12a or the position data generated by the position data generating unit 12c with the corresponding point cloud data and saves the data in the data storage unit 16 (see FIG. 1). This makes it clear at which position in real space the point cloud data was captured.

[0029] The correction data control unit 13 is a functional unit that controls the acquisition, reference, generation, and storage of correction data (see FIG. 1). The correction data control unit 13 is composed of a correction data acquisition unit 13a, a data reference unit 13b, a correction data generation unit 13c, and a correction data storage unit 13d. Here, correction refers to aligning the latest point cloud data with the object 50 to be photographed or past point cloud data. The correction data is data that includes differences (amounts of change) that occur due to this alignment, and can be data that includes differences in latitude, longitude, altitude, rotation angle, etc., for example.

[0030] The correction data acquisition unit 13a is a function for acquiring correction data when a user manually performs a correction. For example, as shown in Fig. 3, an example of a manual correction by a user is when an operator 3 visually checks the displayed point cloud data 60 and aligns the point cloud data 60 with the object 50 to be photographed by using gestures, touch panel operations, etc. The purpose of performing such a correction is to eliminate an error between the position detected by the position sensor 30 and the actual position.

[0031] The data reference unit 13b is a functional unit that references past data (point cloud data, position data, environmental data, and correction data) stored in the data storage unit 16 when automatic generation of correction data is selected.

[0032] The correction data generation unit 13c has a function of generating correction data. If automatic generation of correction data is not selected, the correction data generation unit 13c does not generate correction data and instead uses the correction data acquired by the correction data acquisition unit 13a. If automatic generation of correction data is selected, the correction data generation unit 13c refers to the data stored in the data storage unit 16 via the data reference unit 13b, acquires the latest point cloud data (which may be point cloud data acquired directly from the 3D sensor 20), past point cloud data and its corresponding data (past position data, past correction data, and past environmental data as needed), identifies common points (e.g., feature quantities such as edges and vertices) between the acquired latest point cloud data and the past point cloud data, registers (aligns) the latest point cloud data with the past point cloud data so that the identified common points overlap, calculates differences resulting from the registration (e.g., differences in position before and after alignment of the latest point cloud data with the past point cloud data), and generates correction data based on the calculated differences and, as needed, the past environmental data.

[0033] The correction data saving unit 13d is a functional unit that saves the correction data acquired or generated by the correction data generating unit 13c in the data storage unit 16 in association with the corresponding point cloud data (see FIG. 1).

[0034] The environmental data control unit 14 is a functional unit that controls the acquisition and storage of environmental data (see FIG. 1). Environmental data is data containing information about the accuracy of the corresponding point cloud data, and includes, for example, model information of the 3D sensor 20 used, model information of the position sensor 30, weather, shooting location, log data, etc., but may also include other information and is not limited to these. Note that environmental data is also generated when manual correction processing is performed. This is performed for the purpose of measuring the reliability of the point cloud data in the future process of generating position data and correction data based on point cloud data that has been manually corrected. The environmental data control unit 14 includes an environmental data acquisition unit 14a and an environmental data storage unit 14b.

[0035] The environmental data acquisition unit 14a has a function of acquiring environmental data (see FIG. 1). When automatic generation of environmental data is not selected, the environmental data acquisition unit 14a can acquire predetermined information related to environmental data through user input on the mixed reality display device 1, and generate and acquire the environmental data. When automatic generation of environmental data is selected, the environmental data acquisition unit 14a can collect predetermined information related to environmental data from, for example, an internal source (storage unit) or an external source (terminal, server), and generate and acquire the environmental data.

[0036] The environmental data storage unit 14b is a functional unit that stores the environmental data acquired by the environmental data acquisition unit 14a in association with the corresponding point cloud data (see FIG. 1).

[0037] The display unit 15 is a functional unit that displays the point cloud data stored in the data storage unit 16 on the mixed reality display device 1 (for example, a display) (see FIG. 1). The display unit 15 includes a point cloud data display unit 15a and a correction processing unit 15b.

[0038] The point cloud data display unit 15a is a functional unit that displays point cloud data (see FIG. 1). The point cloud data display unit 15a displays the point cloud data in a state where it has been aligned and corrected with respect to the photographed object in real space using the position data and correction data associated with the point cloud data stored in the data storage unit 16. This eliminates the difference in position between the real space and the point cloud data, and improves the accuracy of alignment. In addition to the point cloud data, the point cloud data display unit 15a may also display information such as measurement results and notes.

[0039] The correction processing unit 15b is a function that performs a process of correcting the position of the point cloud data displayed on the mixed reality display device 1 through a manual operation by the user (see FIG. 1). As a method of the correction processing, for example, the user can perform a manual operation on the mixed reality display device 1 (for example, a gesture sensor or a touch panel) to accurately superimpose the captured object 50 in the real space on the point cloud data.

[0040] The data storage unit 16 is a functional unit that stores corresponding point cloud data, position data, environmental data, and correction data in association with each other (see FIG. 1). Note that instead of storing the point cloud data in the data storage unit 16, the point cloud data and the like may be stored in an external storage device (not shown) so that they can be referenced and saved by communicating with the external storage device.

[0041] The three-dimensional sensor 20 is a device that three-dimensionally senses and captures the surface of a photographing object 50 in real space (see FIG. 1). In FIG. 1, the three-dimensional sensor 20 is configured to be mounted on the mixed reality display device 1, but it may also be configured to be independent of the mixed reality display device 1 and externally attached to the mixed reality display device 1 and connected to enable communication. When the mixed reality display device 1 is a pair of smart glasses, the three-dimensional sensor 20 is attached to the mixed reality display device 1. The three-dimensional sensor 20 acquires point cloud data (60 in FIG. 3) of the photographing object 50 in real space. The three-dimensional sensor 20 generates point cloud data by photographing the photographing object 50 and outputs the generated point cloud data to the point cloud analysis unit 10. Note that the point cloud data may be generated by the point cloud analysis unit 10 instead of by the three-dimensional sensor 20. Examples of the 3D sensor 20 that can be used include a time-of-flight (ToF) camera, a stereo camera, a 3D-LIDAR (laser imaging detection and ranging), a depth sensor, a distance sensor, and a distance camera. Here, the point cloud data is data generated by the 3D sensor 20 and is data rendered as a point cloud (a collection of numerous points having XYZ coordinate (3D coordinate) information). The 3D sensor 20 can be configured to have various output formats depending on the environmental conditions required for the solution, such as the shooting distance, angle of view, whether indoors or outdoors, whether there is sunlight, and customer requests. There may be multiple 3D sensors 20 (whether they are the same type of sensor or not), and multiple point cloud data may be combined by the mixed reality display device 1 to monitor a wide area that cannot be monitored with a single 3D sensor 20.

[0042] The position sensor 30 is a sensor that identifies (detects) the position of the 3D sensor 20 (corresponding to the position of the worker 3, the position of the mixed reality display device 1, and the shooting position) (see FIG. 1 ). While the position sensor 30 is configured to be mounted on the mixed reality display device 1 in FIG. 1 , it may also be configured to be externally attached to the mixed reality display device 1 and communicatively connected. The position sensor 30 generates position data by identifying the position of the 3D sensor 20 and outputs the generated position data to the point cloud analysis unit 10. The position data may be output in real time or at predetermined time intervals, but is not limited to these. Note that the position data may not be generated by the position sensor 30, but may be generated by the point cloud analysis unit 10. The position sensor 30 has a function to identify, for example, latitude, longitude, altitude, rotation angle, etc. as the position of the 3D sensor 20. The position of the 3D sensor 20 may also be identified by position identification elements other than latitude, longitude, altitude, and rotation angle (for example, radius vector, declination, etc. of a polar coordinate system). As shown in FIG. 2 , the position sensor 30 may be, for example, a positioning device that uses a camera to read a two-dimensional code 41 affixed to a reference object 40 whose latitude, longitude, and altitude are known, and determines the position of the three-dimensional sensor 20 based on the size and orientation of the read two-dimensional code 41. The reference object 40 and the two-dimensional code 41 are installed for the purpose of calculating the position information of the worker 3. As the position sensor 30, for example, a positioning device that determines the position of the three-dimensional sensor 20 using a GPS positioning device, a Bluetooth (registered trademark) Low Energy (BLE) beacon, or a positioning device that uses radio wave intensity distribution may be used. As the position sensor 30, for example, an inertial measurement unit (IMU) that estimates the position of the three-dimensional sensor 20 using an IMU (Inertial Measurement Unit) sensor such as a gyro sensor or an acceleration sensor may be used.

[0043] Next, a method for calculating the position of the 3D sensor in the mixed reality display device according to the first embodiment will be described with reference to the drawings. Fig. 4 is an image diagram that schematically shows an example of the positional relationship between the shooting position of the 3D sensor in the mixed reality display device according to the present disclosure and the object to be shot. Fig. 5 is an image diagram that schematically shows an example of the display position of point cloud data when viewed from (0,0,0) in the mixed reality display device according to the present disclosure.

[0044] Referring to FIG. 4, a shooting position 51 represents the position of the 3D sensor 20 when the object 50 is photographed by the 3D sensor (20 in FIG. 1) of the mixed reality display device (1 in FIG. 1). The shooting position 51 is the position of the 3D sensor 20 when a certain piece of point cloud data among the point cloud data stored in the data storage unit (16 in FIG. 1) is acquired. The shooting position 51 is derived from position data associated with the point cloud data. The shooting position 52 represents the position of the 3D sensor 20 when the object 50 is photographed by the 3D sensor 20 of the mixed reality display device 1 at a position different from the shooting position 51. At this time, it is assumed that the 3D sensor 20 does not have a position sensor, such as an AR (Augmented Reality) device or GPS, that identifies the position of the 3D sensor 20, and therefore position data cannot be acquired. For simplicity of explanation, it is assumed that the 3D sensor 20 faces the same shooting direction at the shooting positions 51 and 52.

[0045] 5, point cloud data 61 represents point cloud data acquired by photographing the object to be photographed (50 in FIG. 4) at a photographing position (51 in FIG. 4), and point cloud data 62 represents point cloud data acquired by photographing the object to be photographed (50 in FIG. 4) at a photographing position (52 in FIG. 4). Point cloud data 61 is a collection of relative coordinates when photographing position 51 is set as the origin (0,0,0). Similarly, point cloud data 62 is a collection of relative coordinates when photographing position 52 is set as the origin (0,0,0). When point cloud data 61 and point cloud data 62 are arranged in the same space as in FIG. 5, point cloud data 61 from photographing position 51, where 3D sensor 20 was installed close to object to be photographed 50, is displayed in the foreground, and point cloud data 62 from photographing position 52, where 3D sensor 20 was installed far from object to be photographed 50, is displayed in the background. That is, by calculating the difference (amount of change) in position between the point cloud data 61 and the point cloud data 62 in Fig. 5, it is possible to calculate the difference in position between the photographing positions 51 and 52 in Fig. 4. In other words, by performing registration (alignment) so that the point cloud data 62 matches the point cloud data 61, it is possible to calculate the differences in the X-axis, Y-axis, and Z-axis directions between the photographing positions 51 and 52. By using the calculated difference and the position data of the photographing position 51, it is possible to calculate the position data of the photographing position 52.

[0046] 4 and 5, for the sake of simplicity, it is assumed that the imaging direction of the 3D sensor 20 is the same, but even if the imaging direction is different, it is possible to calculate the position data of the imaging position 52 by calculating the difference in the rotation angle. For example, the rotation angle difference (roll, pitch, and raw) when the point cloud data 62 is rotated by registration so as to match the point cloud data 61 is calculated, and the rotation angle of the imaging position 52 can be calculated by using the calculated difference in the rotation angle and the rotation angle of the imaging position 51.

[0047] Calculation of correction data is similar to calculation of position data. By registering point cloud data 61 having position data with point cloud data 62 and calculating the differences in latitude, longitude, altitude, rotation angle, etc., the calculated differences in latitude, longitude, altitude, rotation angle, etc. can be obtained as correction data.

[0048] Next, the operation of the mixed reality display device according to the first embodiment will be described with reference to the drawings. Fig. 6 is a flow chart diagram that schematically shows an example of the operation of the mixed reality display device according to the present disclosure. For the configuration of the mixed reality display device, please refer to Fig. 1.

[0049] First, the point cloud data acquisition unit 11a of the point cloud data control unit 11 of the point cloud analysis unit 10 of the mixed reality display device 1 acquires point cloud data of the object 50 to be photographed in real space acquired by the three-dimensional sensor 20 (step A1).

[0050] Next, the point cloud data saving unit 11b of the point cloud data control unit 11 saves the point cloud data acquired by the point cloud data acquiring unit 11a in the data storage unit 16 (step A2).

[0051] Next, the position data acquisition unit 12a of the position data control unit 12 of the point cloud analysis unit 10 determines whether to automatically generate position data (step A3). Here, the determination of whether to automatically generate position data can be made by displaying a screen that allows the user to select whether to automatically generate position data or acquire it from a position sensor. Alternatively, whether to automatically generate position data may be determined based on a setting that is set in advance, or whether to automatically generate position data may be determined using another method.

[0052] If the position data is not to be automatically generated (NO in step A3), position data acquisition unit 12a acquires the position data of three-dimensional sensor 20 from position sensor 30 (step A4).

[0053] When the position data is automatically generated (YES in step A3), the position data generating unit 12c of the position data control unit 12 generates the position data by referring to the data stored in the data storage unit 16 via the data referring unit 12b (step A5). The detailed operation of generating the position data will be described later (see FIG. 7).

[0054] After step A4 or step A5, the position data storage unit 12d of the position data control unit 12 associates the position data acquired by the position data acquisition unit 12a or the position data generated by the position data generation unit 12c with the corresponding point cloud data and stores it in the data memory unit 16 (step A6).

[0055] Next, the environmental data acquisition unit 14a of the environmental data control unit 14 of the point cloud analysis unit 10 determines whether to automatically generate environmental data (step A7). Here, the determination of whether to automatically generate environmental data can be made by displaying a screen that allows the user to select whether to automatically generate environmental data or to manually generate environmental data, and having the user make the selection. Alternatively, whether to automatically generate environmental data may be determined based on the setting that is set in advance, or whether to automatically generate environmental data may be determined using another method.

[0056] If the environmental data is not automatically generated (NO in step A7), the environmental data acquisition unit 14a generates and acquires the environmental data when the user inputs predetermined information regarding the environmental data (e.g., the model of the 3D sensor, the time of shooting, weather information, log data, etc.) on the mixed reality display device 1 (step A8).

[0057] When the environmental data is automatically generated (YES in step A7), the environmental data acquisition unit 14a collects information on the environmental data from the internal (storage unit) and external (terminal, server) sources, and generates and acquires the environmental data (step A9).

[0058] After step A8 or step A9, the environmental data saving unit 14b of the environmental data control unit 14 associates the environmental data acquired by the environmental data acquiring unit 14a with the corresponding point cloud data and saves it in the data storage unit 16 (step A10).

[0059] Next, the correction data acquisition unit 13a of the correction data control unit 13 of the point cloud analysis unit 10 determines whether to automatically generate correction data (step A11). Here, the determination of whether to automatically generate correction data can be made by displaying a screen that allows the user to select whether to automatically generate correction data or to manually generate correction data, and having the user make the selection. Alternatively, whether to automatically generate correction data may be determined based on the setting that is set in advance, or whether to automatically generate correction data may be determined using another method.

[0060] If the correction data is not to be automatically generated (NO in step A11), the correction data acquisition unit 13a displays the point cloud data on the mixed reality display device 1 (step A12). At this time, the point cloud data is displayed in a state where it has been aligned based on the position data.

[0061] After step A12, the correction data acquisition unit 13a acquires the correction data when the user manually performs the correction (step A13). Here, the correction data can be acquired, for example, by having the operator 3 perform operations (gestures, touch panel operations, etc.) using a GUI (Graphical User Interface) while visually checking the displayed point cloud data 60, thereby aligning the point cloud data 60 with the photographing target 50, and generating and acquiring the correction data based on the difference (amount of change) caused by the alignment.

[0062] When the correction data is automatically generated (YES in step A11), the correction data generating unit 13c generates the correction data by referring to the data stored in the data storage unit 16 via the data referring unit 13b (step A14). Note that the detailed operation of generating the correction data will be described later (see FIG. 8).

[0063] After step A13 or step A14, the correction data storage unit 13d of the correction data control unit 13 associates the correction data acquired by the correction data acquisition unit 13a or the correction data generated by the correction data generation unit 13c with the corresponding point cloud data and stores it in the data memory unit 16 (step A15), and then terminates.

[0064] Next, detailed operations (step A5 in FIG. 6) for generating position data of the mixed reality display device according to form 1 will be described with reference to the drawings. Fig. 7 is a flowchart diagram schematically illustrating an example of detailed operations for generating position data of the mixed reality display device according to the present disclosure.

[0065] When position data is automatically generated (YES in step A3 of FIG. 6), the position data generation unit 12c references the data stored in the data storage unit 16 via the data reference unit 12b and acquires the latest point cloud data (which may be point cloud data directly acquired from the 3D sensor 20) and past point cloud data and its corresponding data (past position data, past correction data, and past environmental data as needed) (step B1). Here, the number of past point cloud data may be one or more and can be determined arbitrarily in advance. Furthermore, if the weather information included in the environmental data is rain or snow, noise may be generated in the point cloud data associated with the environmental data due to the influence of the rain or snow. Therefore, the reliability of the point cloud data may be reduced by the noise due to the rain or snow. In such a case, noise may be removed from the past point cloud data by performing a noise filtering process on the corresponding past point cloud data based on the past environmental data, thereby increasing the reliability of the point cloud data.

[0066] Next, the position data generating unit 12c identifies common points (for example, feature quantities such as edges and vertices) between the acquired latest point cloud data and the past point cloud data (step B2). Here, when identifying common points, a priority may be set for each point cloud. For example, among the point clouds of objects included in both the latest point cloud data and the past point cloud data, a higher priority may be given to objects whose positions and shapes are less likely to change, such as the ground or houses, and a lower priority may be given to objects whose positions and shapes are more likely to change, such as trees or automobiles.

[0067] Next, the position data generating unit 12c performs registration (alignment) between the latest point cloud data and the past point cloud data so that the identified common points overlap (step B3). Here, as a registration method, for example, a method using ICP (Iterative Closest Point) can be mentioned, but other methods may also be used and are not limited to this.

[0068] Next, the position data generator 12c calculates a difference caused by the registration (for example, a difference between the positions of the latest point cloud data before and after alignment with the past point cloud data), and calculates the shooting position (the position of the 3D sensor 20) corresponding to the latest point cloud data based on the calculated difference, the past position data corresponding to the past point cloud data, and the past correction data (step B4). Note that the calculation of the shooting position corresponding to the latest point cloud data may be performed by comparing the shooting positions of the past position data in chronological order based on the shooting times in the past environmental data, and predicting and calculating the shooting position of the latest position data from the trajectory of the shooting positions. Alternatively, the calculation of the shooting position corresponding to the latest point cloud data may be performed by using a specific object existing in real space as a reference (origin) and calculating the shooting position using the relative distance from the object, without using the latitude and longitude information in the past position data.

[0069] Next, the position data generator 12c generates the latest position data corresponding to the latest point cloud data based on the calculated photographing position and, if necessary, past environmental data (step B5), and then proceeds to step A6 in FIG. 6. Here, when a photographing position is calculated using one past point cloud data, the calculated photographing position is generated as is as the position data. When a photographing position is calculated using multiple past point cloud data, the photographing positions calculated for each past point cloud data are integrated to generate one position data. The integration method may involve taking the average of the calculated photographing positions, or weighting the calculated photographing positions for each corresponding past point cloud data before integration. For example, one method may involve extracting information on factors that reduce reliability from the environmental data, quantifying the reliability of the point cloud data from the extracted information, taking into account the accuracy of the 3D sensor 20 used, weather, etc., and assigning a higher weight to a photographing position calculated using point cloud data with a higher reliability value, and a lower weight to a photographing position calculated using point cloud data with a lower reliability value.

[0070] Next, detailed operations (step A14 in FIG. 6) for generating correction data of the mixed reality display device according to form 1 will be described with reference to the drawings. Fig. 8 is a flowchart diagram schematically illustrating an example of detailed operations for generating correction data of the mixed reality display device according to the present disclosure.

[0071] The correction data generation unit 13c refers to the data stored in the data storage unit 16 via the data reference unit 13b, and acquires the latest point cloud data (which may be point cloud data directly acquired from the three-dimensional sensor 20), as well as past point cloud data and its corresponding data (past position data, past correction data, and past environmental data as needed) (step C1). Here, the number of past point cloud data may be one or more, and can be determined arbitrarily in advance.

[0072] Next, the correction data generation unit 13c identifies common points (for example, feature quantities such as edges and vertices) between the acquired latest point cloud data and the past point cloud data (step C2). Here, when identifying the common points, a priority may be set for each point cloud. For example, among the point clouds of objects included in both the latest point cloud data and the past point cloud data, a higher priority may be given to objects whose position or shape is unlikely to change, such as the ground or houses, and a lower priority may be given to objects whose position or shape is likely to change, such as trees or automobiles, and the point cloud of the object with the highest priority may be identified as the common point.

[0073] Next, the correction data generating unit 13c performs registration (alignment) between the latest point cloud data and the past point cloud data so that the identified common points overlap (step C3). Here, as a registration method, for example, a method using ICP can be mentioned, but other methods may also be used and are not limited to this.

[0074] Next, the correction data generating unit 13c calculates the position difference caused by the registration (for example, the position difference between before and after alignment of the latest point cloud data with the past point cloud data) (step C4). Note that the calculation of the difference may be performed by comparing the differences of the past correction data in chronological order based on the shooting times of the past environmental data, and predicting and calculating the difference of the latest correction data from the trajectory of the differences.

[0075] Next, the correction data generator 13c generates correction data based on the calculated difference and, if necessary, past environmental data (step C5), and then proceeds to step A15 in FIG. 6. Here, if the difference is calculated using one past point cloud data, correction data is generated based on the calculated difference. If the difference is calculated using multiple past point cloud data, the differences calculated for each past point cloud data are integrated to generate one correction data. The integration method may involve taking the average of the calculated differences, or weighting the calculated differences for each corresponding past point cloud data before integration. For example, the reliability of the point cloud data may be quantified based on information contained in the environmental data, taking into account the accuracy of the 3D sensor 20 used, weather, etc., and a higher weight may be assigned to a difference calculated using highly reliable point cloud data, and a lower weight may be assigned to a difference calculated using less reliable point cloud data. The reliability may be quantified automatically based on the environmental data and correction data, or may be set manually.

[0076] According to form 1, when referencing point cloud data, in addition to alignment using position data, the data is displayed on the mixed reality display device 1 after correction processing using correction data, which can contribute to improving the accuracy of alignment between the point cloud data and the object 50 to be photographed in real space.

[0077] Furthermore, according to the first aspect, the point cloud data is displayed in synchronization with the real space without performing alignment processing, thereby making it possible to improve the efficiency of the work.

[0078] Furthermore, according to the first aspect, the position data is unchanged, so that measurement accuracy can be ensured when, for example, past point cloud data is compared with the current real space during periodic equipment inspection to check for differences.

[0079] Furthermore, according to the first aspect, by capturing images at multiple locations, point cloud data covering a wide range can be generated without performing registration.

[0080] [Form 2] The mixed reality display device according to the second embodiment will be described with reference to the drawings. Fig. 9 is a block diagram schematically showing a second example of the configuration of the mixed reality display device according to the present disclosure.

[0081] The mixed reality display device 1 is a device that displays point cloud data by superimposing it on a photographed object 50 in real space. The mixed reality display device 1 includes a position sensor 30, a position data generator 12c, a correction data generator 13c, and a point cloud data display unit 15a.

[0082] The position sensor 30 is configured to detect the position of the 3D sensor 20 that captures the object 50 to be captured in real space and generate position data of the 3D sensor 20 that corresponds to the point cloud data generated by the 3D sensor 20.

[0083] The position data generation unit 12c is configured to generate the latest position data of the three-dimensional sensor 20 corresponding to the latest point cloud data based on the latest point cloud data generated by the three-dimensional sensor 20, past point cloud data generated by the three-dimensional sensor 20, and past position data of the three-dimensional sensor 20 corresponding to the past point cloud data generated by the position sensor 30.

[0084] The correction data generation unit 13c is configured to calculate the position difference that occurs when aligning the latest point cloud data with the past point cloud data, and to generate the latest correction data for the 3D sensor 20 that corresponds to the latest point cloud data based on the difference.

[0085] The point cloud data display unit 15a is configured to display the latest point cloud data in a state where the latest point cloud data has been aligned and corrected with respect to the object 50 to be photographed, based on the latest position data and the latest correction data.

[0086] According to form 2, when referencing point cloud data, in addition to alignment using position data, the data is displayed on the mixed reality display device 1 after correction processing using correction data, which contributes to improving the accuracy of alignment between the point cloud data and the object 50 to be photographed in real space.

[0087] The mixed reality display devices according to the first and second aspects can be configured using so-called hardware resources (information processing devices, computers), and may use those having the configuration shown in Fig. 10. For example, the hardware resources 100 include a processor 101, a memory 102, a network interface 103, and the like, which are interconnected by an internal bus 104.

[0088] 10 is not intended to limit the hardware configuration of the hardware resource 100. The hardware resource 100 may include hardware (e.g., an input / output interface) that is not shown. Furthermore, the number of units such as the processor 101 included in the device is not intended to be limited to the example shown in FIG. 10, and for example, multiple processors 101 may be included in the hardware resource 100. The processor 101 may be, for example, a central processing unit (CPU), a microprocessor unit (MPU), a graphics processing unit (GPU), or the like.

[0089] The memory 102 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD).

[0090] The network interface 103 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0091] The functions of the hardware resource 100 are realized by the processing modules described above. The processing modules are realized, for example, by the processor 101 executing a program stored in the memory 102. The programs can be updated by downloading them over a network or by using a storage medium that stores the programs. Furthermore, the processing modules may be realized by semiconductor chips. In other words, it is sufficient that the functions performed by the processing modules be realized by software being executed on some kind of hardware.

[0092] Some or all of the above aspects may be described as, but are not limited to, the following supplementary notes.

[0093] [Appendix 1] a position sensor configured to detect the position of a 3D sensor that captures an image of a target object in real space and generate position data of the 3D sensor corresponding to point cloud data generated by the 3D sensor; a position data generation unit configured to generate latest position data of the three-dimensional sensor corresponding to the latest point cloud data, based on latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data of the three-dimensional sensor generated by the position sensor and corresponding to the past point cloud data; a correction data generation unit configured to calculate a position difference caused by aligning the latest point cloud data with the past point cloud data, and to generate latest correction data of the 3D sensor corresponding to the latest point cloud data based on the difference; a point cloud data display unit configured to display the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed based on the latest position data and the latest correction data; and A mixed reality display device comprising: [Appendix 2] a position data acquisition unit configured to acquire the latest position data from the position sensor when generation of the latest position data is not selected by the position data generation unit; the point cloud data display unit is configured to display the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed, based on the latest position data acquired by the position data acquisition unit and the latest correction data. 2. The mixed reality display device of claim 1. [Appendix 3] a correction data acquisition unit configured, when generation of the latest correction data is not selected by the correction data generation unit, to display the latest point cloud data on the point cloud data display unit, and generate and acquire the latest correction data based on a position difference caused by a user manually aligning the latest point cloud data with the object to be photographed; the point cloud data display unit is configured to display the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed, based on the latest position data and the latest correction data acquired by the correction data acquisition unit. 3. The mixed reality display device according to claim 1 or 2. [Appendix 4] The position data generation unit A process of identifying common points between the latest point cloud data and the past point cloud data; A process of calculating a difference in position caused by aligning the latest point cloud data with the past point cloud data so that the common points overlap; calculating a photographing position corresponding to the latest point cloud data based on the difference, the past position data, and past correction data of the 3D sensor corresponding to the past point cloud data; generating the latest position data based on the photographing position; configured to: A mixed reality display device according to any one of appendices 1 to 3. [Appendix 5] an environment data acquisition unit configured to generate and acquire environment data by collecting information on accuracy of the point cloud data from inside or outside the mixed reality display device, or by acquiring information on accuracy of the latest point cloud data through a user input when automatic generation of the latest environment data is not selected; the position data generation unit is configured to generate the latest position data by using the past environmental data acquired by the environmental data acquisition unit. 5. The mixed reality display device of claim 4. [Appendix 6] the position data generation unit is configured to perform a filtering process to remove noise from the corresponding past point cloud data based on the past environmental data acquired by the environmental data acquisition unit, and to generate the latest position data using the past point cloud data after the filtering process. 6. The mixed reality display device according to claim 5. [Appendix 7] The position data generation unit is configured to extract information on factors that reduce reliability from the past environmental data when using a plurality of the past point cloud data, quantify the reliability of the past point cloud data from the extracted information, assign a large weight to the photographing position calculated using the past point cloud data with a high reliability value, assign a small weight to the photographing position calculated using the past point cloud data with a low reliability value, and integrate the photographing positions to generate one of the latest position data. 6. The mixed reality display device according to claim 5. [Appendix 8] the position data generation unit is configured to generate the single latest position data by averaging the shooting positions calculated for each of the past point cloud data when using the multiple past point cloud data. A mixed reality display device according to any one of appendices 1 to 6. [Appendix 9] The correction data generation unit A process of identifying common points between the latest point cloud data and the past point cloud data; calculating the difference in position caused by aligning the latest point cloud data with the past point cloud data so that the common points overlap; generating latest correction data of the three-dimensional sensor corresponding to the latest point cloud data based on the difference; configured to: When identifying the common location, a higher priority is given to an object whose position and shape do not change among point clouds of objects included in both the latest point cloud data and the past point cloud data, and a lower priority is given to an object whose position and shape change, and the point cloud of the object with the highest priority is identified as the common location. A mixed reality display device according to any one of appendices 1 to 8. [Appendix 10] a correction processing unit that performs a process of correcting the position of the latest point cloud data by a manual operation by a user; 4. The mixed reality display device of claim 3. [Appendix 11] a data storage unit that stores corresponding point cloud data, position data, and correction data in association with each other; A mixed reality display device according to any one of appendices 1 to 10. [Appendix 12] Communicatively connected to an external storage device; A mixed reality display device according to any one of appendices 1 to 11. [Appendix 13] The three-dimensional sensor is provided. A mixed reality display device according to any one of appendices 1 to 12. [Appendix 14] a step in which a position sensor detects a position of a 3D sensor that captures an image of a target object in real space, and generates position data of the 3D sensor corresponding to point cloud data generated by the 3D sensor; A step in which a computer generates latest position data of the three-dimensional sensor corresponding to the latest point cloud data based on latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data of the three-dimensional sensor generated by the position sensor and corresponding to the past point cloud data; a step in which the computer calculates a position difference caused by aligning the latest point cloud data with the past point cloud data, and generates latest correction data for the three-dimensional sensor corresponding to the latest point cloud data based on the difference; a step in which the computer displays the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed based on the latest position data and the latest correction data; A mixed reality display method, including: [Appendix 15] a process of detecting, via a position sensor, the position of a 3D sensor that captures an image of a target object in real space, and generating position data of the 3D sensor corresponding to the point cloud data generated by the 3D sensor; generating latest position data of the three-dimensional sensor corresponding to the latest point cloud data based on latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data of the three-dimensional sensor corresponding to the past point cloud data, generated by the position sensor; a process of calculating a difference in position caused by aligning the latest point cloud data with the past point cloud data, and generating latest correction data of the 3D sensor corresponding to the latest point cloud data based on the difference; a process of displaying the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed based on the latest position data and the latest correction data; A program that causes a computer to execute the following. In addition, Supplementary Notes 14 and 15 relating to the method and program can be expanded as in Supplementary Notes 2-13.

[0094] The disclosures of the above-cited patent documents are incorporated herein by reference and may be used as the basis or part of the present invention, as necessary. Modifications and adjustments of the embodiments are possible within the scope of the entire disclosure of the present invention (including the claims and drawings), and further based on the basic technical concept thereof. Furthermore, various combinations and selections (or non-selections, as necessary) of the various disclosed elements (including each element of each claim, each element of each embodiment or embodiment, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure, including the claims and drawings, and the technical concept thereof. Furthermore, with regard to the numerical values ​​and numerical ranges described in this application, any intermediate values, lower values, and smaller ranges are deemed to be included, even if not explicitly stated. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, in accordance with the spirit of the present invention, are also deemed to be included in (belong to) the disclosures of this application. [Explanation of symbols]

[0095] 1. Mixed reality display device 3. Workers 10 Point cloud analysis section 11 Point cloud data control unit 11a Point cloud data acquisition section 11b Point cloud data storage section 12 Position data control section 12a Position data acquisition unit 12b Data reference section 12c Position data generation unit 12d Location data storage section 13 Correction data control section 13a Correction data acquisition section 13b Data reference section 13c Correction data generation unit 13d Correction data storage section 14 Environmental Data Control Section 14a Environmental data acquisition section 14b Environmental Data Storage Section 15 Display section 15a Point cloud data display section 15b Correction processing section 16 Data storage unit 20 3D Sensor 30 Position Sensor 40 Reference object 41 2D Code 50 Subject of Photography 51, 52 Shooting location 60, 61, 62 Point cloud data 100 Hardware Resources 101 processors 102 memory 103 Network Interface 104 Internal Bus

Claims

1. a position sensor configured to detect a position of a three-dimensional sensor that captures an image of a target object in real space and generate position data of the three-dimensional sensor corresponding to point cloud data generated by the three-dimensional sensor; a position data generation unit configured to generate latest position data of the three-dimensional sensor corresponding to the latest point cloud data, based on latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data of the three-dimensional sensor corresponding to the past point cloud data, generated by the position sensor; a correction data generation unit configured to calculate a position difference resulting from aligning the latest point cloud data with the past point cloud data, and to generate latest correction data of the three-dimensional sensor corresponding to the latest point cloud data based on the difference; a point cloud data display unit configured to display the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed based on the latest position data and the latest correction data; and A mixed reality display device comprising:

2. a position data acquisition unit configured to acquire the latest position data from the position sensor when generation of the latest position data is not selected by the position data generation unit; the point cloud data display unit is configured to display the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed, based on the latest position data acquired by the position data acquisition unit and the latest correction data. The mixed reality display device according to claim 1 .

3. a correction data acquisition unit configured, when generation of the latest correction data is not selected by the correction data generation unit, to display the latest point cloud data on the point cloud data display unit, and generate and acquire the latest correction data based on a position difference caused by a user manually aligning the latest point cloud data with the object to be photographed; the point cloud data display unit is configured to display the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed, based on the latest position data and the latest correction data acquired by the correction data acquisition unit. The mixed reality display device according to claim 1 .

4. The position data generation unit A process of identifying common points between the latest point cloud data and the past point cloud data; A process of calculating a difference in position caused by aligning the latest point cloud data with the past point cloud data so that the common points overlap; calculating a photographing position corresponding to the latest point cloud data based on the difference, the past position data, and past correction data of the three-dimensional sensor corresponding to the past point cloud data; generating the latest position data based on the photographing position; configured to: The mixed reality display device according to claim 1 .

5. an environment data acquisition unit configured to generate and acquire environment data by collecting information on accuracy of the point cloud data from inside or outside the mixed reality display device, or by acquiring information on accuracy of the latest point cloud data through a user input when automatic generation of the latest environment data is not selected; the position data generation unit is configured to generate the latest position data by using the past environmental data acquired by the environmental data acquisition unit. The mixed reality display device according to claim 4 .

6. the position data generation unit is configured to perform a filtering process to remove noise from the corresponding past point cloud data based on the past environmental data acquired by the environmental data acquisition unit, and to generate the latest position data using the past point cloud data after the filtering process. The mixed reality display device according to claim 5 .

7. The position data generation unit is configured to extract information on factors that reduce reliability from the past environmental data when using a plurality of the past point cloud data, quantify the reliability of the past point cloud data from the extracted information, assign a large weight to the photographing position calculated using the past point cloud data with a high reliability value, assign a small weight to the photographing position calculated using the past point cloud data with a low reliability value, and integrate the photographing positions to generate one of the latest position data. The mixed reality display device according to claim 5 .

8. The correction data generation unit A process of identifying common points between the latest point cloud data and the past point cloud data; calculating the difference in position caused by aligning the latest point cloud data with the past point cloud data so that the common points overlap; generating latest correction data of the three-dimensional sensor corresponding to the latest point cloud data based on the difference; configured to: When identifying the common location, a higher priority is given to an object whose position and shape do not change among point clouds of objects included in both the latest point cloud data and the past point cloud data, and a lower priority is given to an object whose position and shape change, and the point cloud of the object with the highest priority is identified as the common location. The mixed reality display device according to any one of claims 1 to 7.

9. a step in which a position sensor detects a position of a 3D sensor that captures an image of a target object in real space, and generates position data of the 3D sensor corresponding to point cloud data generated by the 3D sensor; a step in which a computer generates latest position data of the three-dimensional sensor corresponding to the latest point cloud data based on latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data of the three-dimensional sensor corresponding to the past point cloud data, generated by the position sensor; a step in which the computer calculates a position difference caused by aligning the latest point cloud data with the past point cloud data, and generates latest correction data for the three-dimensional sensor corresponding to the latest point cloud data based on the difference; a step in which the computer displays the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed based on the latest position data and the latest correction data; A mixed reality display method, including:

10. a process of detecting, via a position sensor, the position of a three-dimensional sensor that captures an image of a target object in real space, and generating position data of the three-dimensional sensor corresponding to point cloud data generated by the three-dimensional sensor; generating latest position data of the three-dimensional sensor corresponding to the latest point cloud data based on latest point cloud data generated by the three-dimensional sensor, past point cloud data generated by the three-dimensional sensor, and past position data of the three-dimensional sensor corresponding to the past point cloud data, generated by the position sensor; a process of calculating a difference in position caused by aligning the latest point cloud data with the past point cloud data, and generating latest correction data of the three-dimensional sensor corresponding to the latest point cloud data based on the difference; a process of displaying the latest point cloud data in a state in which the latest point cloud data is aligned and corrected with respect to the object to be photographed based on the latest position data and the latest correction data; A program that causes a computer to execute the following.

Images