Display device, display method, and display program

Abstract

To provide a display device, a display method, and a display program that can easily grasp a displacement amount between a target article and a CAD model in an image in which the target article and the CAD model are superimposed.SOLUTION: A display device 100 is a device that displays a difference between a surface shape of a target article 200 and a surface shape of a CAD model of the target article 200 in an augmented reality. The display device 100 mainly includes an image capturing unit 110, a depth information acquisition unit 120, an inertial information acquisition unit 130, an image generation unit 140, and a display unit 150. The image capturing unit 110 performs image capturing of the target article 200 and the marker 160 to generate a captured image and outputs it to the image generation unit. The depth information acquisition unit 120 acquires depth information of the target article 200, and the inertial information acquisition unit 130 measures inertial information of the display device 100. The image generation unit 140 generates a distance superimposition image based on the depth information, the captured image, and the inertial information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a display program for displaying in augmented reality the deviation between the surface shape of a target object and the surface shape of a CAD model of the target object. [Background technology]

[0002] For example, a superimposed image generating program, a superimposed image generating method, and an information processing device are known which acquire an image of a three-dimensional structure, and based on model information corresponding to the three-dimensional structure, superimpose the three-dimensional structure and a model contained in the acquired image in a determined arrangement, making at least a portion of the model transparent to generate a superimposed image in which the portion of the image that overlaps with the model is visible, and changing the gradation within a predetermined range that includes at least a portion where the deviation between the three-dimensional structure and the model in the generated superimposed image is relatively large depending on the distance from the portion (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-81589 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a configuration in which the gradation within a predetermined range that includes at least the points in the generated superimposed image where the deviation between the three-dimensional structure and the model is relatively large is changed according to the distance from the points where the deviation is relatively large, there is a risk that the exact amount of deviation cannot be intuitively grasped.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a display device, a display method, and a display program that enable easy understanding of the amount of deviation between a target object and a CAD model in an image in which the target object and the CAD model are superimposed. [Means for solving the problem]

[0006] A display device according to a first aspect of the present invention is a display device that displays in augmented reality the difference between the surface shape of a target object and the surface shape of a CAD model of the target object, and the display device includes an imaging unit that images the target object and a marker and outputs the captured image, a depth information acquisition unit that acquires depth information of the target object, an inertial information acquisition unit that acquires inertial information of the display device, and an object image that is an image of the target object and a marker image that is an image of the marker included in the captured image, and uses the depth information and the inertial information to calculate the depth of the CAD model so that the contour of the object image matches the contour of the image of the CAD model in the captured image. and an image generation unit that corrects the image and generates a corrected CAD model, wherein the image generation unit uses the depth information to obtain a first spatial point cloud consisting of position coordinates of a plurality of points included in the surface shape of the target article, and the CAD model has a second spatial point cloud consisting of position coordinates of a plurality of points included in a mesh formed on the surface of the CAD model, and the image generation unit calculates a point distance group indicating the distance between corresponding points in the first spatial point group and the second spatial point group, generates a distance difference image corresponding to the point distance group for each vertex of the corrected CAD model, and generates a distance superimposed image by superimposing the distance difference image on the target article image.

[0007] It is preferable that the image generating section generates a corrected CAD model image based on the corrected CAD model, and generates a distance superimposed image by superimposing the distance difference image and the corrected CAD model image on the target article image.

[0008] It is preferable that the image generating section generates a distance superimposed image in which at least a part of the distance difference image is transparent and a part of the target article image that overlaps with the distance difference image is visible.

[0009] It is preferable that the image generating section generates a distance superimposed image in which at least a part of the corrected CAD model image is made transparent and a part of the target article image that overlaps with the corrected CAD model image is visible.

[0010] It is preferable that the imaging section images two or more markers, and the image generating section generates the corrected CAD model using the images of the two or more markers.

[0011] The depth information acquisition unit is preferably a ToF sensor.

[0012] The inertial information acquisition unit preferably includes an acceleration sensor and / or a gyro sensor and / or a gravity sensor.

[0013] It is preferable that the image generating section generates the distance superimposed image every time a predetermined time elapses.

[0014] The distance overlaid image preferably includes a color that corresponds to the distance between each corresponding point.

[0015] The target article may include a basic part and a secondary part, and the image generation unit may calculate a difference between the position of the secondary part relative to the current basic part and the position of the secondary part relative to the basic part included in the CAD model, and generate a distance difference image based on the difference.

[0016] A display method according to a second aspect of the present invention is a display method for displaying on a display device in augmented reality a difference between a surface shape of a target object and a surface shape of a CAD model of the target object, the display method including a reading step of reading a CAD model having a second spatial point cloud consisting of position coordinates of a plurality of points included in a mesh formed on the surface of the CAD model, an imaging step of imaging the target object and a marker and outputting the captured image, a depth information acquisition step of acquiring depth information of the target object, an inertial information acquisition step of acquiring inertial information of the display device, and a first point cloud acquisition step of acquiring a first spatial point cloud consisting of position coordinates of a plurality of points included in the surface shape of the target object using the depth information. the image of the CAD model is corrected using a CAD model image, which is an image of the target object included in the captured image, a marker image, which is an image of the marker, depth information, and inertial information so that a contour of the target object image matches a contour of the image of the CAD model in the captured image; a point distance group calculation step is calculated to calculate a point distance group indicating the distance between corresponding points in the first spatial point group and the second spatial point group; and an image generation step is generated to generate a distance superimposed image by superimposing the distance difference image on the target object image.

[0017] The target article may comprise a basic part and a sub-part, and the method may further comprise a step of calculating a difference between a position of the sub-part relative to the current basic part and a position of the sub-part relative to the basic part included in the CAD model, and generating a distance difference image based on the difference.

[0018] A display program according to a third aspect of the present invention is a display program executed by a computer for displaying on a display device in augmented reality a difference between a surface shape of a target object and a surface shape of a CAD model of the target object, the display program including a reading step of reading a CAD model having a second spatial point group consisting of position coordinates of a plurality of points included in a mesh formed on the surface of the CAD model, an imaging step in which an imaging unit images the target object and a marker and outputs an imaged image, a depth information acquisition step in which a depth information acquisition unit acquires depth information of the target object, an inertial information acquisition step in which an inertial information acquisition unit acquires inertial information of the display device, and an image generation unit uses the depth information to generate a first spatial point group consisting of position coordinates of a plurality of points included in the surface shape of the target object. the first spatial point cloud acquisition step of acquiring a point cloud corresponding to the target object image included in the captured image, the marker image included in the captured image of the marker, the depth information, and the inertial information, to correct the image of the CAD model so that the contour of the target object image matches the contour of the image of the CAD model in the captured image, thereby generating a corrected CAD model; the image generation unit of the first spatial point cloud and the second spatial point cloud calculating step of calculating a point distance group indicating the distance between corresponding points in the first spatial point cloud and the second spatial point cloud; and the image generation unit of the second spatial point cloud generating a distance difference image corresponding to the point distance group for each vertex of the corrected CAD model, and generating a distance superimposed image by superimposing the distance difference image on the target object image.

[0019] The target article may comprise a basic part and a sub-part, and the method may further comprise a step of calculating a difference between a position of the sub-part relative to the current basic part and a position of the sub-part relative to the basic part included in the CAD model, and generating a distance difference image based on the difference. Effect of the Invention

[0020] According to the present invention, a display device, a display method, and a display program are obtained that enable easy grasping of the amount of deviation between a target article and a CAD model in an image in which the target article and the CAD model are superimposed. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram illustrating a display device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a schematic relationship between two markers and a coordinate system. [Diagram 3] FIG. 2 is a diagram illustrating a screen display. [Figure 4] FIG. 2 is a diagram illustrating a screen display. [Diagram 5] 4 is a flowchart showing a first display method and a display program. [Figure 6] FIG. 2 is a diagram illustrating a screen display. [Figure 7] FIG. 2 is a diagram illustrating a screen display. [Figure 8] 13 is a flowchart showing a second display method and a display program. [Figure 9] FIG. 2 is a diagram illustrating a screen display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] A display device 100 according to a first embodiment of the present invention will now be described with reference to the drawings.

[0023] Referring to FIG. 1, the display device 100 is a device that displays the deviation, i.e., difference, between the surface shape of a target object 200 and the surface shape of a CAD model of the target object 200 in an augmented reality, and is, for example, a tablet computer. The display device 100 mainly includes an imaging unit 110, a depth information acquisition unit 120, an inertial information acquisition unit 130, an image generation unit 140, and a display unit 150. The target object 200 is, for example, a full-scale model (clay model) of a car or a reduced model (clay model), and is installed on a predetermined plane, for example, on a floor surface. Augmented reality is also called "Augmented Reality" and means extending real events by overlaying digital information on things in the real world.

[0024] The imaging unit 110 includes an imaging element such as a CCD or a CMOS, captures the target item 200 and the markers 160 (also represented as 160a and 160b), generates a captured image, and outputs it to the image generation unit. The markers 160 have, for example, a rectangular shape with a pattern drawn on a two-dimensional plane. This pattern is unique for each marker, and the image generation unit 140 can determine the coordinate system of the space in which the target item 200 is placed by analyzing the image of the marker 160 included in the captured image (see FIG. 2). At least two markers 160a and 160b are placed near the target item and on the same plane as the target item 200, for example, on the floor. The image of the target item 200 included in the captured image is called the target item image, and the image of the marker 160 is called the marker image.

[0025] The depth information acquisition unit 120 includes, for example, a ToF sensor, and acquires depth information of the target article 200. The depth information includes distances between the display device 100 and a plurality of points included in the surface shape of the target article 200.

[0026] The inertial information acquisition unit 130 includes, for example, an acceleration sensor and / or a gyro sensor and / or a gravity sensor, and measures inertial information of the display device 100. The inertial information includes at least one of acceleration, angular velocity, and gravitational acceleration. The acceleration sensor measures the acceleration in three axial directions of the display device 100, the gyro sensor measures the angular velocity around three axes of the display device 100, and the gravity sensor measures the gravitational acceleration in three axial directions of the display device 100.

[0027] The image generating unit 140 obtains the position coordinates of a plurality of points included in the surface shape of the target article 200 and the orientations around three axes in the coordinate system of the space in which the target article 200 is placed, based on the depth information, the captured image, and the inertial information. That is, the image generating unit 140 obtains values ​​of six degrees of freedom (DoF). The coordinate system used here is a coordinate system in the coordinate system determined by the image generating unit using the image of the marker 160. A set consisting of the position coordinates and orientations of a plurality of points included in the surface shape of the target article 200 is called a first spatial point group.

[0028] The CAD model is a three-dimensional shape formed by CAD data, which is data that digitizes the design shape of the target object. A set of position coordinates of a plurality of points included in a mesh formed on the surface of the CAD model is called a second spatial point group. These position coordinates are coordinates in a coordinate system determined by the image generating unit 140 using an image of the marker 160. The CAD model has a second spatial point group. The CAD data includes position data of a reference point corresponding to the center point of the marker 160, and is converted by another computer, for example a personal computer or a server, into a file format that can be read by the display device 100 and stored in the display device 100. This file format is, for example, an STL file format. The data obtained by the conversion is called converted CAD data.

[0029] The image generating unit 140 obtains a captured image from the imaging unit 110, and corrects the image of the CAD model so that the contour of the target article image matches the contour of the image of the CAD model in the captured image using the target article image and the marker image included in the captured image, the depth information, and the inertial information, thereby generating a corrected CAD model. Details of this process will be described later.

[0030] The image generating unit 140 calculates a point distance group indicating the distance between corresponding points in the first spatial point group and the second spatial point group, generates a distance difference image corresponding to the point distance group for each vertex of the corrected CAD model, and generates a distance superimposed image by superimposing the distance difference image on the target article image. Each corresponding point means a point that is closest in distance.

[0031] Furthermore, the image generating unit 140 generates a corrected CAD model image based on the corrected CAD model, and overlays the corrected CAD model image on the target article image in addition to the distance difference image, thereby generating a distance superimposed image. Alternatively, the image generating unit 140 may generate an image in which either the distance difference image or the corrected CAD model image is overlaid on the target article image.

[0032] The display unit 150 can display a distance superimposed image, or an image in which a corrected CAD model image is superimposed on a target article image. The image generating unit 140 may display at least a part of the distance difference image at a predetermined transparency level to make it transparent, and generate a distance superimposed image in which a part of the target article image overlapping with the distance difference image is visible. The image generating unit 140 may display at least a part of the corrected CAD model image at a predetermined transparency level to make it transparent, and generate a distance superimposed image in which a part of the target article image overlapping with the corrected CAD model image is visible. This allows the user to view the target article image through the distance difference image and the corrected CAD model image.

[0033] FIG. 3 shows a display unit 150 that displays a distance superimposed image 220 in which a distance difference image 151 and a corrected CAD model image 159 are superimposed on a target article image. The display unit 150 can display the difference between corresponding points in the first spatial point group and the second spatial point group. The distance difference image 151 shown in FIG. 3 shows the difference between corresponding points in the first spatial point group and the second spatial point group as a colored cylinder. The display device 100 displays the difference at each point on the display unit 150 in blue if it is smaller than the first threshold, in green if it is equal to or greater than the first threshold and less than the second threshold, and in red if it is equal to or greater than the second threshold, and in a cylinder 151a with a height according to the size. When the user clicks on the cylinder 151a, a specific value 155 of the difference is displayed. The model operation button 152 is used to fine-tune the position of the corrected CAD model image 159. The model enlargement / reduction button 153 is used to enlarge or reduce the size of the corrected CAD model image 159. The difference display range designation button 154 is used by the user to designate the range in which the difference is to be displayed. The lower and upper limits of the range are designated by a slide bar or by a radio button. The image display designation button 156 is used by the user to designate the image to be displayed. When "overlay CAD data" is selected, the corrected CAD model image 159 is displayed, when "display difference" is selected, the distance difference image 151 is displayed, and when "scan clay model" is selected, the target object 200 is scanned again. The transparency adjustment button 157 is used to adjust the transparency of the corrected CAD model image 159 and the distance difference image 151. The marker fixing button 158 is used to release the position of the marker 160 that has already been recognized, re-recognize the position of the marker 160, and redisplay the model based on the re-recognized position. When the position of the marker is recognized incorrectly, the position of the marker can be recognized again to the correct position by operating the marker fixing button 158. It is also possible to operate the marker fixing button 158 multiple times to recognize the marker 160 multiple times, and use the most accurate position from the recognized position.

[0034] Fig. 4 shows another example of a distance difference image. The distance difference image shown in Fig. 4 shows the difference between corresponding points in the first spatial point group and the second spatial point group using a contour diagram. More specifically, the distance between each corresponding point is shown by color, and if the distance between corresponding points is close, they are shown in blue, and the farther the distance, the closer the color is to red.

[0035] The display device 100 executes the above-mentioned processing in real time. Here, real time includes repeatedly executing the above-mentioned processing every time a predetermined time elapses. When the user moves the display device 100 relative to the target object 200, the imaging unit 110 generates a new captured image at the moved position, the depth information acquisition unit 120 acquires new depth information, the inertia information acquisition unit 130 measures new inertia, the image generation unit 140 generates a new distance superimposed image, and the display unit 150 displays the distance superimposed image or an image in which a corrected CAD model image is superimposed on the target object image.

[0036] Next, a first display method executed by the display device 100 will be described with reference to Fig. 5. The first display method is implemented as a program executable by, for example, a tablet computer, and is executed by, for example, the tablet computer.

[0037] First, when the execution of the first display method is started in step S1001, the user points the imaging unit 110 toward the marker 160 in step S1002. The imaging unit 110 captures the image of the marker 160 and transmits the marker image to the image generation unit 140. Referring to FIG. 2, the image generation unit 140 recognizes the marker images and calculates the center points of each. Then, the image generation unit 140 sets a straight line connecting the two center points to the X-axis, and sets the midpoint between the two center points to the origin. Furthermore, the image generation unit 140 recognizes the plane on which the marker 160 is placed based on the captured image captured by the imaging unit 110, for example, using Apple's ARKit, and sets the plane on which the marker image is placed as the XY plane, and sets right-handed coordinates with the Z-axis direction perpendicular to the XY plane.

[0038] In the next step S1003, the display device 100 reads the CAD model via the converted CAD data stored in advance. As described above, the converted CAD data is obtained by converting the CAD data including the position data of the reference point corresponding to the center point of the marker 160, so the CAD model also includes the position data of the reference point corresponding to the center point of the marker 160.

[0039] In the next step S1004, the image generating unit 140 recognizes planes such as floors and walls included in the captured image and boundaries between planes based on the captured image captured by the imaging unit 110, for example, using Apple's ARKit. By recognizing floors and walls that are larger than the target item 200, the position of the target item 200 can be recognized over a wide range. Therefore, recognition errors can be reduced compared to when only the target item 200 is recognized, and the accuracy of the difference display can be improved.

[0040] In the next step S1005, the image generation unit 140 aligns the position of the CAD model with the target object image included in the captured image based on the center point and set coordinates of the marker 160 calculated in step S1002, and the position information of the floor, walls, etc. recognized in S1004.

[0041] In the next step S1006, the depth information acquisition unit 120 and the inertial information acquisition unit 130 acquire depth information and inertial information of the display device 100. As described above, the depth information is the distance between the display device 100 and a plurality of points included in the surface shape of the target article 200, and the inertial information includes at least one of acceleration, angular velocity, and gravitational acceleration.

[0042] In the next step S1007, the image generating unit 140 recognizes the orientation of the target object based on the distance and inertial information obtained in step S1006. In detail, the image generating unit 140 calculates the position coordinates of multiple points included in the surface shape of the target object 200 and the orientations around three axes in the coordinate system of the space in which the target object 200 is placed based on the depth information, the captured image, and the inertial information, and sets this as a first spatial point group. The position coordinates and orientations of the target object 200 are called orientation data of the target object 200.

[0043] Then, returning to step S1005, the image generating unit 140 uses the orientation data of the recognized target article 200 to align the position of the CAD model with the target article image. That is, the image generating unit 140 aligns the position of the CAD model with the target article image so that the difference between corresponding points in the first spatial point group calculated in step S1007 and the second spatial point group included in the CAD model becomes small. Then, steps S1005 to S1007 are repeated a predetermined number of times, and after the position of the CAD model is sufficiently aligned with the target article image, the process proceeds to step S1008. The predetermined number of times is a preset number of times, for example, 5 to 30 times, or is determined as needed according to the processing capacity of the display device 100. By repeating steps S1005 to S1007 multiple times, the position of the CAD model is gradually brought closer to the target article image, and after sufficient alignment, the process from step S1008 onwards can be executed.

[0044] In the next step S1008, the image generating unit 140 generates a corrected CAD model image (AR image) in accordance with the shape of the aligned CAD model.

[0045] In the next step S1009, an image in which the corrected CAD model image (AR image) is superimposed on the captured image, in other words, an image in which the corrected CAD model image (AR image) is superimposed on the position of the target article image included in the captured image, is displayed on the display unit 150. Here, as shown in FIG. 3, it is possible to display the difference between corresponding points in the first spatial point group and the second spatial point group. On the display unit 150 of the display device 100 that has captured the image of the target article 200, the difference at each point is displayed in blue if it is smaller than the first threshold, in green if it is equal to or greater than the first threshold and less than the second threshold, and in red if it is equal to or greater than the second threshold, and in a cylinder 151a with a height according to the size. The user can operate the operation buttons displayed on the display unit 150 while looking at this, to fine-tune the position, size, shape, etc. of the corrected CAD model image, to enlarge or reduce the display, to change the transparency of the corrected CAD model image, and to perform operations such as displaying or hiding the corrected CAD model image.

[0046] In the next step S1010, the image generating unit 140 calculates a point distance group indicating the distance between corresponding points in the first spatial point group calculated in step S1007 and the second spatial point group included in the CAD model. In other words, for each vertex of the corrected CAD model, the difference between the point on the surface of the target object is calculated. Here, the attitude data of the target object 200 is calculated using the distance between the display device 100 and the target object and the inertial information of the display device 100, the contour of the target object image is obtained using the captured image, and the difference is calculated after minimizing the deviation between the target object image and the corrected CAD model image using the attitude data, the contour of the target object image, and the planes such as floors and walls and the boundaries between the planes recognized using the captured image.

[0047] In the next step S1011, distance difference images 151, 210 are generated for each vertex of the corrected CAD model, which correspond to the point distance group, that is, which represent the difference from the point on the surface of the target article. At this time, the difference, i.e., the amount of deviation, is expressed in blue if it is small, and in red if it is large, with a gradation according to the difference.

[0048] In the next step S1012, an image in which the three distance difference images 151, 210 constituting the AR image are superimposed on the captured image, in other words, a distance superimposed image 220 in which the distance difference images 151, 210 are superimposed on the position of the target item image 241 included in the captured image, is displayed on the display unit 150 (see Figures 3 and 4).

[0049] In the next step S1013, the image generator 140 generates annotations, which include values ​​155, cylinders 151a, and arrows as shown in FIGS.

[0050] In the next step S1014, the image generating unit 140 displays the annotation on the display unit 150. Then, if the marker fixing button 158 has been operated, the process returns to step S1002, and the processes from step S1002 onward are executed again. On the other hand, if the marker fixing button 158 has not been operated, the process returns to step S1004, and the processes from step S1004 onward are executed to create and display a distance superimposed image again.

[0051] According to the present embodiment, a display device, a display method, and a display program are obtained that enable a difference between a target article and a CAD model, that is, a deviation amount, to be easily grasped in an image in which the target article and the CAD model are superimposed.

[0052] It is also possible to display images that allow the differences between the target object and the CAD model to be easily recognized in real time.

[0053] In addition, because the differences are displayed in real time, if the target item is modified on the spot, for example by carving a clay model or piling on clay to change its shape, it is possible to immediately see how much difference there is between the CAD model.

[0054] 3, the difference is displayed in different colors according to the first and second thresholds, but the number of thresholds may be 1 or more. Also, the difference may be displayed in other tubular shapes, such as a square tube, instead of the cylindrical shape 151a.

[0055] A second embodiment of the present invention will be described with reference to Figs. 7 and 8. The display device 100 according to the second embodiment acquires an image of a target object including an image of a basic part in real time, inspects whether an auxiliary part has been newly attached to the basic part correctly, and displays the inspection result together with the basic part and the auxiliary part. Below, configurations different from the first embodiment will be described, and configurations similar to those in the first embodiment will be given the same reference numerals and will not be described again. Below, reference numeral 241 shown in Fig. 7 indicates an image of a basic part, and reference numeral 242d indicates an image of an auxiliary part, but the description will be made assuming that they also indicate the basic part 241 and the auxiliary part 242d, respectively.

[0056] A state in which the auxiliary part 242d is attached to the basic part 241 and a plurality of auxiliary parts 242a to 242c (not shown) are not yet attached will be described with reference to Fig. 7. Here, an image obtained by capturing an image of the basic part 241 and the auxiliary parts 242a to 242d by the imaging unit forms a target article image. The corrected CAD model image is generated based on CAD data of the basic part 241 and the auxiliary parts 242a to 242d correctly attached to the basic part 241, and includes auxiliary part corrected CAD model images 252a to 252d and a basic part corrected CAD model image 251. Fig. 7 shows the display unit 150 on which a distance superimposed image in which a distance difference image and a corrected CAD model image are superimposed on an image of the basic part 241 is displayed.

[0057] The distance difference image displays the difference between the CAD data and the basic parts 241 and the auxiliary parts 242a to 242d as a contour diagram on the auxiliary part corrected CAD model images 252a to 252d and the basic part corrected CAD model image 251. If the difference in a predetermined corresponding area between the CAD data and the basic parts 241 and the auxiliary parts 242a to 242d is within the allowable error range, the contour diagram displays the area in green, if it is outside the allowable error range and the basic parts 241 and the auxiliary parts 242a to 242d are inside the data of the basic parts and the auxiliary parts included in the CAD data (concave difference), the area is displayed closer to blue as the difference increases, and if it is outside the data of the basic parts and the auxiliary parts included in the CAD data (convex difference), the area is displayed closer to red as the difference increases. For example, in FIG. 7, the difference between the basic parts 241 and the auxiliary parts 242d and the CAD data is within the allowable error range, so the area corresponding to the basic parts 241 and the auxiliary parts 242d is displayed in green, for example. On the other hand, since the subsidiary parts 242a to 242e do not exist, the difference from the CAD data is regarded as the maximum value, and the areas corresponding to the subsidiary parts 242a to 242e are displayed in blue.

[0058] The display unit 150 displays an attachment display area 260 in the upper left corner, which indicates whether or not an auxiliary part is attached to the basic part 241. The attachment display area 260 indicates the attachment status indicated by an exclamation mark (!) or a check mark, the part number of the auxiliary part, and the maximum difference between the shapes of the actual basic part and auxiliary part and the CAD data. When a check mark is displayed for the attachment status, it indicates that the auxiliary part is correctly attached to the basic part 241, i.e., the auxiliary part is attached to the basic part 241 with a difference within the allowable error range, and when an exclamation mark is displayed, it indicates that the auxiliary part is not correctly attached to the basic part 241, i.e., it is not yet present in the image or is not attached to the basic part 241 with a difference within the allowable error range. As for the maximum difference, the largest value of the difference between the shapes of the basic part and auxiliary part included in the CAD data and the shapes of the actual basic part 241 and auxiliary part is displayed. When an auxiliary part does not yet exist in the image in reality, the largest value of the differences in a predetermined direction, for example the vertical direction, between the shapes of the basic part and auxiliary part included in the CAD data and the shape of the actual basic part 241 is displayed as the maximum difference. For example, according to the display in Fig. 7, it can be seen that the auxiliary part 242a having the part number "CN-0001" does not yet exist in the image, is placed on the basic part 241 with a difference exceeding the allowable error range, and the maximum difference in the vertical direction from the CAD data is 83 mm (see reference numeral 262a in Fig. 7). Similarly, it can be seen that sub-part 242b with part number "CN-0002" is not yet present in the image and is placed on base part 241 with a difference that exceeds the allowable error range, with the maximum difference in the vertical direction from the CAD data being 81 mm (see reference number 262b in FIG. 7), and sub-part 242c with part number "CN-0003" is also not yet present in the image and is placed on base part 241 with a difference that exceeds the allowable error range, with the maximum difference in the vertical direction from the CAD data being 79 mm (see reference number 262c in FIG. 7). On the other hand, sub-part 242d with part number "CN-0004" is present in the image and is attached to base part 241 with a difference that is within the allowable error range (see reference number 262d in FIG. 7).

[0059] Display unit 150 displays a red downward arrow and part number above a position corresponding to an incorrectly attached secondary part in the corrected CAD model image. For example, in Fig. 7, secondary parts 242a-242c do not exist and have not yet been correctly attached, so red downward arrows and part numbers 253a-253c are displayed above positions corresponding to secondary parts 242a-242c in the corrected CAD model image. In Fig. 7, secondary parts 242a-242c do not exist, so red downward arrows and part numbers 253a-253c are displayed above positions corresponding to secondary parts 242a-242c in the corrected CAD model image.

[0060] A transparency slider 263 for setting the transparency of the corrected CAD model is displayed at the bottom of the display unit 150. The user can operate the transparency slider 263 to change the transparency of the corrected CAD model as desired.

[0061] Next, a second display method executed by the display device 100 will be described with reference to Fig. 8. The second display method is implemented as a program executable by, for example, a tablet computer, and is executed by, for example, the tablet computer. The second display method has steps S1001 to S1009 in common with the first display method, and therefore description thereof will be omitted.

[0062] In step S1201, the display device 100 selects a subsidiary component to be inspected. At this point, the attachment display area 260 displays that the subsidiary components 242a to 242d are not attached to the basic component 241 correctly.

[0063] In the next step S1202, the display device 100 generates an annotation, such as an arrow, indicating the position of the secondary component to be inspected, and displays it on the display unit 150.

[0064] In the next step S1203, the position of the auxiliary part to be inspected is compared with the position of the corresponding auxiliary part in the CAD data, and the difference is calculated.

[0065] In the next step S1011, a distance difference image is generated for each vertex of the corrected CAD model, which corresponds to the point distance group, that is, represents the difference from the point on the surface of the target article. At this time, the difference is expressed by a gradation according to the difference, from blue to green when the difference is small, and red when the difference is large.

[0066] In the next step S1012, an image in which the distance difference image (AR image) is superimposed on the captured image, in other words, a distance superimposed image in which the distance difference image (AR image) is superimposed on the positions of the target article images 241 and 242a to 242d included in the captured image, is displayed on the display unit 150.

[0067] In the next step S1206, the position of the sub-component to be inspected is compared with the position of the corresponding sub-component in the CAD data, and if the position is within the allowable error range, the arrow displayed in step S1201 is erased, whereas if the position is not within the allowable error range, the arrow remains displayed.

[0068] In the next step S1207, the attachment display area 260 updates the display (see FIG. 7). That is, if the auxiliary part is correctly attached to the basic part 241, the attachment status is changed from an exclamation mark to a check mark. If the auxiliary part is not correctly attached to the basic part 241, the exclamation mark displayed in the attachment status is maintained, and the maximum difference value is updated. Then, if the marker fix button 158 (not shown in FIG. 7) is operated, the process returns to step S1002, and the processes from step S1002 onwards are executed again. On the other hand, if the marker fix button 158 is not operated, the process returns to step S1004, and the processes from step S1004 onwards are executed to create and display a distance superimposed image again.

[0069] According to this embodiment, whether the auxiliary part is correctly attached to the basic part can be displayed in real time while the attachment work is being performed, which enables the user to accurately attach the auxiliary part to the basic part and prevents the user from forgetting to attach the auxiliary part.

[0070] A third embodiment of the present invention will be described with reference to Fig. 9. The display device 100 according to the third embodiment differs from the second embodiment in the display of the display unit 150. Below, configurations different from the first and second embodiments will be described, and configurations similar to those of the first and second embodiments will be denoted by the same reference numerals and description thereof will be omitted.

[0071] A state in which a plurality of subsidiary parts 342a to 342d have not yet been attached to a basic part 341 will be described with reference to FIG. 9. For the sake of explanation, the basic part 341 is shown in a see-through state in FIG. 9. Here, an image of the basic part 341 and the subsidiary parts 342a to 342d captured by an imaging unit forms a target article image. The corrected CAD model image is generated based on CAD data of the basic part 341 and the subsidiary parts 342a to 342d correctly attached to the basic part 341. FIG. 9 shows the display unit 150 on which a distance superimposed image in which a distance difference image and a corrected CAD model image are superimposed on an image of the basic part 341 is displayed. The distance difference image is the same as in the second embodiment, and therefore a description thereof will be omitted.

[0072] The display unit 150 has an attachment display area 360 at its upper left that indicates whether or not a sub-part is attached to the basic part 341. The attachment display area 360 indicates the attachment status indicated by using a hyphen (-), an exclamation point (!), or a check mark, the part number of the sub-part, and the maximum difference between the actual position of the sub-part and the CAD data. Regarding the attachment status, when a hyphen (-) is displayed, it indicates that the sub-part has not yet been inspected, when a check mark is displayed, it indicates that the sub-part is correctly attached to the basic part 341, that is, the sub-part is attached to the basic part 341 with a difference within the allowable error range, and when an exclamation point is displayed, it indicates that the sub-part is not correctly attached to the basic part 341, that is, it is not yet present in the image or is not attached to the basic part 341 with a difference within the allowable error range. For example, according to the display in FIG. 9, it is shown that secondary part 342e (not shown) having part number "11C01A00444_0" has not been inspected and therefore the difference has not been measured (see reference numeral 362e in FIG. 9), while secondary part 342a having part number "11C01A00443_0_4" has a maximum difference of 188 mm from the CAD data, indicating that it has been placed on basic part 341 with a difference that exceeds the allowable error range (see reference numeral 362a in FIG. 9).

[0073] The display unit 150 displays a display switching button area 370 on the right side. The display switching button area 370 includes a selection box 371, an overlay button 372, a difference display button 373, a space scan button 374, a shielding effect switching button 375, a marker fixing button 376, and an end button 377. The selection box 371 is used to switch between displaying and not displaying the sub-parts according to the attributes of the sub-parts. The attributes are, for example, a handle, a bolt, a bracket, and / or a pipe. The overlay button 372 is used to switch between displaying and not displaying the corrected CAD model image. The difference display button 373 is used to switch between displaying and not displaying the distance difference image. The space scan button 374 is used to display a point cloud obtained by recognizing the basic part and the sub-parts with a ToF sensor. A space mesh is generated based on the point cloud recognized by the ToF sensor, and the basic part and the sub-parts are displayed using a wire frame. The occlusion effect switching button 375 is a button for switching the effect of hiding the part that should be hidden in reality (occlusion) when the superimposed CAD data is in a positional relationship where it should be displayed partially or entirely behind the basic parts and auxiliary parts in reality. Normally, occlusion is disabled, that is, the part that should be hidden in reality is displayed, and the part of the CAD data that is hidden behind the basic parts and auxiliary parts in reality is also displayed. When the occlusion effect switching button 375 is clicked in this state, the occlusion is enabled and the part that is hidden behind the basic parts and auxiliary parts in reality is not displayed. The marker fixing button 376 is used to release the position of the marker that has already been recognized, re-recognize the marker position, and re-display the model based on the re-recognized position, as in the first embodiment. When the end button 377 is clicked, the processing according to this embodiment is ended.

[0074] According to this embodiment, the same effects as those of the second embodiment can be obtained. In addition, the user can appropriately change the display and appropriately check the progress of the installation work.

[0075] In the above description, the CAD data stored in another computer is converted by a computer provided outside the display device 100. However, the CAD data may be converted by the display device 100.

[0076] The program may be stored in a recording medium, or may be stored and executed in a server connected to the display device 100 via a network. The display device 100 may transmit the depth information and the inertial information to the server, and the server may create a distance superimposed image and transmit it to the display device 100.

[0077] The display device 100 may be executed on a smartphone or a personal computer instead of a tablet computer, and the program may be a program executable by these.

[0078] Although the embodiment using the two markers 160a, 160b has been described, there may be any number of markers 160 as long as there are two or more markers 160. The image generating unit generates a corrected CAD model using images of two or more markers 160.

[0079] The size, shape, and quantity of each member shown in the specification and drawings are examples and are not limited to these. Also, the materials of each member are examples and are not limited to these.

[0080] Although embodiments of the present invention have been described herein with reference to the accompanying drawings, it will be apparent to those skilled in the art that modifications may be made in the structure and relationship of the parts without departing from the scope and spirit of the invention as described. [Explanation of symbols]

[0081] 100 display device 110 Imaging unit 120 Depth information acquisition section 130 Inertial information acquisition section 140 Image Generation Unit 150 Display section 200 Target items

Claims

1. A display device that displays a difference between a surface shape of a target object and a surface shape of a CAD model of the target object in an augmented reality manner, The display device includes: an imaging unit that captures an image of the target object and the marker and outputs a captured image; A depth information acquisition unit that acquires depth information of the target object; an inertial information acquisition unit that acquires inertial information of the display device; an image generating unit that generates a corrected CAD model by correcting the image of the CAD model using a target article image that is an image of the target article and a marker image that is an image of the marker included in the captured image, the depth information, and the inertial information so that a contour of the target article image and a contour of the image of the CAD model match in the captured image; The image generator uses the depth information to obtain a first spatial point group consisting of position coordinates of a plurality of points included in the surface shape of the target article; the CAD model has a second spatial point group consisting of position coordinates of a plurality of points included in a mesh formed on a surface of the CAD model; The image generating unit calculates a point distance group indicating a distance between each corresponding point in the first spatial point group and the second spatial point group, generates a distance difference image corresponding to the point distance group for each vertex of the corrected CAD model, and generates a distance superimposed image by superimposing the distance difference image on the target article image. Display device.

2. The display device according to claim 1 , wherein the image generating section generates a corrected CAD model image based on the corrected CAD model, and generates a distance superimposed image by superimposing the distance difference image and the corrected CAD model image on the target article image.

3. The display device according to claim 1 , wherein the image generating section generates a distance superimposed image in which at least a part of the distance difference image is transparent and a part of the target article image that overlaps with the distance difference image is visible.

4. The display device according to claim 2 , wherein the image generation unit generates a distance superimposed image in which at least a portion of the corrected CAD model image is transparent and a portion of the target article image that overlaps with the corrected CAD model image is visible.

5. The display device according to claim 1 , wherein the imaging unit captures images of two or more of the markers, and the image generating unit generates the corrected CAD model using images of the two or more markers.

6. The display device according to claim 1 , wherein the depth information acquisition unit is a ToF sensor.

7. The display device according to claim 1 , wherein the inertial information acquisition unit includes an acceleration sensor, a gyro sensor, and / or a gravity sensor.

8. The display device according to claim 1 , wherein the image generating section generates the distance superimposed image every time a predetermined time elapses.

9. The display device according to claim 1 , wherein the distance-overlaid image includes a color corresponding to a distance between each of the corresponding points.

10. 2. The display device according to claim 1, wherein the target object comprises a basic part and an auxiliary part, and the image generation unit calculates a difference between a current position of the auxiliary part relative to the basic part and a position of the auxiliary part relative to the basic part included in the CAD model, and generates a distance difference image according to the difference.

11. A display method for displaying a difference between a surface shape of a target object and a surface shape of a CAD model of the target object on a display device in augmented reality, comprising: a reading step of reading the CAD model having a second spatial point group consisting of position coordinates of a plurality of points included in a mesh formed on a surface of the CAD model; an imaging step of imaging the target object and the marker and outputting an image; A depth information acquisition step of acquiring depth information of the target object; an inertial information acquisition step of acquiring inertial information of the display device; a first point cloud acquisition step of acquiring a first spatial point cloud including position coordinates of a plurality of points included in the surface shape of the target article using the depth information; a corrected CAD model generating step of correcting the image of the CAD model using a target article image that is an image of the target article included in the captured image and a marker image that is an image of the marker, the depth information, and the inertial information so that a contour of the target article image and a contour of the image of the CAD model match in the captured image; and a point distance group calculation step of calculating a point distance group indicating distances between corresponding points in the first spatial point group and the second spatial point group; an image generating step of generating a distance difference image corresponding to the group of point distances for each vertex of the corrected CAD model, and generating a distance superimposed image by superimposing the distance difference image on the target article image; A display method comprising:

12. The display method according to claim 11, wherein the target object comprises a basic part and a secondary part, and further comprises a step of calculating a difference between a current position of the secondary part relative to the basic part and a position of the secondary part relative to the basic part included in the CAD model, and generating a distance difference image according to the difference.

13. A display program executed by a computer for displaying a difference between a surface shape of a target object and a surface shape of a CAD model of the target object on a display device in augmented reality, a reading step of reading the CAD model having a second spatial point group consisting of position coordinates of a plurality of points included in a mesh formed on a surface of the CAD model; an imaging step in which an imaging unit captures an image of the target object and the marker and outputs a captured image; a depth information acquisition step in which a depth information acquisition unit acquires depth information of the target object; an inertial information acquisition step in which an inertial information acquisition unit acquires inertial information of the display device; a first point cloud acquisition step in which an image generating unit acquires a first spatial point cloud including position coordinates of a plurality of points included in the surface shape of the target article using the depth information; a corrected CAD model generating step in which the image generating unit corrects the image of the CAD model using a target article image that is an image of the target article included in the captured image and a marker image that is an image of the marker, the depth information, and the inertial information so that a contour of the target article image and a contour of the image of the CAD model match in the captured image; and a point distance group calculation step in which the image generation unit calculates a point distance group indicating distances between corresponding points in the first spatial point group and the second spatial point group; an image generating step in which the image generating unit generates a distance difference image corresponding to the point distance group for each vertex of the corrected CAD model, and generates a distance superimposed image by superimposing the distance difference image on the target article image; A display program comprising:

14. The display program according to claim 13, wherein the target object comprises a basic part and an auxiliary part, and further comprises a step of calculating a difference between a current position of the auxiliary part relative to the basic part and a position of the auxiliary part relative to the basic part included in the CAD model, and generating a distance difference image according to the difference.

Images