Large structure drawing data superimposing device, large structure drawing data superimposing method, and large structure drawing data superimposing program

Abstract

To superpose video data and drawing data with each other with satisfactory accuracy.SOLUTION: An apparatus for superposing a drawing data of a large-sized structure according to the present invention has an imaging unit for capturing a structure in a real space attached with a plurality of markers to generate a video data, a display unit for displaying information, and a control unit for superposing the drawing data of the structure on the vide data to display it in the display unit. The drawing data has a plurality of marker coordinates set at positions corresponding to the plurality of markers on the structure. The control unit matches the plurality of markers included in the vide data and the plurality of marker coordinates included in the drawing data with each other by least square method to superpose the drawing data on the video data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a large-scale structure drawing data superimposing device, a large-scale structure drawing data superimposing method, and a large-scale structure drawing data superimposing program. [Background technology]

[0002] In order to check whether a large-scale structure has been manufactured according to the blueprints, a technique is known in which the manufactured structure is photographed with a camera or the like, and the image data of the design drawings is superimposed on the image (image data) of the photographed structure, thereby checking whether the large-scale structure has been manufactured according to the design drawings (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-201667 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned drawing overlay technology, when drawing data is overlaid on video data of a large structure, the error increases with increasing distance from the starting point, resulting in a larger difference between the video data and the drawing data. Therefore, even if the drawing data is overlaid on the video data, it is difficult to determine which part of the structure the error is, such as whether it is an error in the structure itself or an error in the installation of accessories, etc. Such drawing overlay technology is required to overlay the video data and the drawing data with high accuracy.

[0005] The present disclosure has been made in consideration of the above, and aims to provide a drawing data superposition device for large structures, a drawing data superposition method for large structures, and a drawing data superposition program for large structures that are capable of accurately superimposing video data and drawing data. [Means for solving the problem]

[0006] The drawing data superimposition device for a large structure according to the present disclosure comprises an imaging unit that captures an image of a structure in real space to which multiple markers are attached and generates video data, a display unit that displays information, and a control unit that acquires the video data and drawing data of the large structure and superimposes the acquired video data and drawing data on the display unit, wherein the drawing data has multiple marker coordinates set at positions corresponding to the multiple markers on the large structure, and the control unit superimposes the drawing data on the video data by matching the multiple markers included in the video data with the multiple marker coordinates included in the drawing data using the least squares method.

[0007] The method for overlaying drawing data of a large structure according to the present disclosure includes an acquisition step of acquiring video data of a large structure in real space with a plurality of markers attached and drawing data of the large structure, and a display step of superimposing the drawing data of the large structure on the video data and displaying it on a display unit, wherein the drawing data has a plurality of marker coordinates set at positions corresponding to the plurality of markers on the large structure, and the display step superimposes the drawing data on the video data by matching the plurality of markers included in the video data with the plurality of marker coordinates included in the drawing data using the least squares method.

[0008] The program for overlaying drawing data of a large structure according to the present disclosure causes a computer to execute an acquisition process for acquiring video data of a large structure in real space with multiple markers attached and drawing data of the large structure, and a display process for superimposing the drawing data of the large structure on the video data and displaying it on a display unit, wherein the drawing data has multiple marker coordinates set at positions corresponding to the multiple markers on the large structure, and the display process superimposes the drawing data on the video data by matching the multiple markers included in the video data with the multiple marker coordinates included in the drawing data using the least squares method. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to reduce errors when video data and drawing data are superimposed. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing an example of a drawing data superimposing device according to this embodiment. [Figure 2] FIG. 2 is a functional block diagram showing an example of the drawing data superimposing device according to this embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of the drawing data superimposing method according to this embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the operation of the drawing data superimposing device. [Figure 5] FIG. 5 is a diagram showing an example of the operation of the drawing data superimposing device. [Figure 6] FIG. 6 is a diagram showing an example of the operation of the drawing data superimposing device. [Figure 7] FIG. 7 is a diagram showing an example of the operation of the drawing data superimposing device. [Figure 8] FIG. 8 is a diagram showing an example of the operation of the drawing data superimposing device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of a large-structure drawing data superimposing device, a large-structure drawing data superimposing method, and a large-structure drawing data superimposing program according to the present disclosure will be described with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical.

[0012] Fig. 1 is a schematic diagram showing an example of a drawing data superimposing device 100 according to this embodiment. Fig. 2 is a functional block diagram showing an example of the drawing data superimposing device 100 according to this embodiment.

[0013] 1 and 2, the drawing data superimposing device 100 according to this embodiment includes a photographing unit 10, a display unit 20, and a control unit 30. A portable terminal device such as a tablet or a smartphone is used as the drawing data superimposing device 100. In this embodiment, the drawing data superimposing device 100 is a terminal device in which the photographing unit 10, the display unit 20, and the control unit 30 are integrated.

[0014] The image capturing unit 10 captures an image of a structure (large structure) ST in real space including a plurality of markers M1, and generates video data D1. The video data D1 includes still images, videos, and other images of the structure ST. The image capturing unit 10 outputs the generated video data D1 to the control unit 30. The image capturing unit 10 is configured with an image capturing device such as a visible light camera, a far-infrared camera, or a near-infrared camera. The image capturing unit 10 may also be configured with a combination of a visible light camera, a far-infrared camera, a near-infrared camera, and a LiDAR or IMU device, for example.

[0015] In this embodiment, the structure ST includes, for example, at least one of a civil engineering structure and an architectural structure. Examples of civil engineering structures include bridges, roads, and revetments. Examples of architectural structures include buildings and steel towers. The markers M1 are provided at a plurality of predetermined positions on the structure ST. The markers M1 only need to be identifiable by the imaging unit 10, and can be formed using, for example, an ArUco code, a two-dimensional code, or the like.

[0016] The display unit 20 displays various types of information. Examples of the display unit 20 include a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The display unit 20 displays an image based on a video signal output from the control unit 30.

[0017] The control unit 30 controls the operation of each unit of the drawing data superimposing device 100. The control unit 30 includes a processing unit 31 and a storage unit 32.

[0018] The processing unit 31 performs various types of information processing and includes a processor such as a CPU (Central Processing Unit) and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory).

[0019] The processing unit 31 includes an acquisition unit 33, a superimposition unit 34, and a display control unit 35.

[0020] The acquisition unit 33 acquires the video data D1 generated by the imaging unit 10. The acquisition unit 33 may acquire the video data D1 of a structure imaged by the imaging unit 10 in real time, for example.

[0021] The acquisition unit 33 also acquires drawing data D2 of the structure ST. The drawing data D2 is data of a design drawing of the structure. The drawing data D2 is three-dimensional data. The drawing data D2 can be stored in, for example, the storage unit 32. Marker coordinates M2 corresponding to the marker M1 are set in the drawing data D2. The marker coordinates M2 are set in the drawing data D2 at a position corresponding to the marker M1 of the structure. The drawing data D2 includes an existing portion ST1 (e.g., a girder) to be installed in the structure ST displayed on the display unit 20, and an uninstalled portion S2 (e.g., a drainage pipe) not to be installed in the structure ST displayed on the display unit 20. The uninstalled portion S2 is, for example, a portion corresponding to a configuration to be installed in the structure ST. In FIG. 1, a corresponding portion ST2 of the structure ST corresponding to the uninstalled portion S2 is indicated by a dashed dotted line.

[0022] The superimposing unit 34 superimposes the acquired video data D1 and drawing data D2. The superimposing unit 34 superimposes the drawing data D2 on the video data D1 by matching a plurality of markers M1 of a structure included in the video data D1 with a plurality of marker coordinates M2 included in the drawing data D2 by the least squares method based on the following equation 1. In equation 1, l is the observation vector, v is the residual vector, A is the Jacobian matrix, x is the position of the imaging unit 10, and P is a weighting factor.

number

[0023] The superimposition unit 34 may perform matching using the least squares method for multiple markers M1 and marker coordinates M2, weighting them so that the markers M1 and the marker coordinates M2 corresponding to those markers M1 that are located closer to the imaging unit 10 have higher importance.

[0024] Furthermore, the superimposing unit 34 may perform matching by the least squares method with weights assigned to the plurality of markers M1 and marker coordinates M2 such that the markers M1 and the marker coordinates M2 corresponding to the markers M1 that are located closer to the portion displayed on the display unit 20 are given higher importance. In this case, the superimposing unit 34 can superimpose the existing portion S1 and the uninstalled portion S2 included in the drawing data D2 onto the video data D1.

[0025] When a plurality of markers M1 are simultaneously photographed by the photographing unit 10, the superimposing unit 34 may calculate the position of the photographing unit based on the positions of the plurality of markers M1 photographed simultaneously.

[0026] The display control unit 35 controls the display operation of the display unit 20. For example, the display control unit 35 causes the display unit 20 to display the video data D1 acquired by the acquisition unit 33. Furthermore, when the video data D1 and the drawing data D2 are superimposed by the superimposition unit 34, the display control unit 35 causes the display unit 20 to display the video data D1 and the drawing data D2 in a superimposed state.

[0027] The storage unit 32 stores information such as various programs and data. The storage unit 32 stores drawing data D2 about the structure ST. The storage unit 32 can also store video data D1 of the structure ST. The storage unit 32 includes storage such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0028] The memory unit 32 causes a computer to execute, for example, an acquisition process for acquiring video data D1 of a structure ST in real space including a plurality of markers M1 and drawing data D2 of the structure ST, and a display process for superimposing the drawing data D2 on the captured data and displaying it on the display unit 20, wherein the drawing data D2 has a plurality of marker coordinates M2 set at positions corresponding to the plurality of markers M1 on the structure ST, and the display process stores a drawing data superimposition program for superimposing the drawing data D2 on the captured data by matching the plurality of markers M1 included in the captured data with the plurality of marker coordinates M2 included in the drawing data D2 using the least squares method.

[0029] In the control unit 30, the processor in the processing unit 31 reads out various programs and loads them into memory, thereby executing information processing corresponding to the functions of the above units. Examples of the various programs include programs stored in the storage unit 32 and programs recorded on an external recording medium. The control unit 30 functions as an information processing device (computer) that executes various information processes. Note that the various programs may be executed by an information processing device other than the control unit 30, or the control unit 30 and the other information processing device may cooperate to execute the various programs.

[0030] Next, a description will be given of a drawing data superimposing method using the above-described drawing data superimposing device 100. Fig. 3 is a flowchart showing an example of the drawing data superimposing method according to this embodiment. Figs. 4 to 8 are diagrams showing an example of the operation of the drawing data superimposing device 100.

[0031] 3, the drawing data superimposing method according to this embodiment includes a photographing step S10, an acquiring step S20, and a displaying step S30. Note that the photographing step S10, the acquiring step S20, and the displaying step S30 are processed in real time.

[0032] In the photographing step S10, the worker photographs a structure ST including a plurality of markers M1 using the photographing unit 10 of the drawing data superimposition device 100, as shown in FIG. 4. In the following example, a case where a portion of the structure ST within the photographing range FR is photographed will be described as an example. The photographing unit 10 generates video data D1 of the structure ST and outputs it to the control unit 30. The display control unit 35 outputs the video data D1 from the photographing unit 10 to the display unit 20. As shown in FIG. 5, the video data D1 is displayed on the display unit 20.

[0033] In the acquisition step S20, the acquisition unit 33 of the control unit 30 acquires video data D1 obtained by capturing a structure ST in real space including a plurality of markers M1, and drawing data D2 of the structure ST stored in the memory unit 32.

[0034] In the display step S30, the superimposing unit 34 superimposes the drawing data D2 on the video data D1 by matching the plurality of markers M1 included in the video data D1 with the plurality of marker coordinates M2 included in the drawing data D2 by the least squares method, as shown in Fig. 6. The superimposing unit 34 generates superimposed data D3 by superimposing the drawing data D2 on the video data D1.

[0035] When performing matching by the least squares method, the superimposing unit 34 may weight some of the markers M1 and marker coordinates M2 out of the plurality of markers M1 and marker coordinates M2.

[0036] For example, the superimposing unit 34 can perform matching by the least squares method with weights for a plurality of markers M1 and marker coordinates M2 such that the markers M1 and the marker coordinates M2 corresponding to the markers M1 that are located closer to the imaging unit 10 have higher importance. In this case, the video data D1 and drawing data D2 for the structure ST can be superimposed more accurately the closer they are to the imaging unit 10.

[0037] Furthermore, the superimposing unit 34 can perform matching by the least squares method by weighting the plurality of markers M1 and marker coordinates M2 so that the markers M1 and the marker coordinates M2 corresponding to the markers M1 that are located closer to the portion displayed on the display unit 20 have higher importance. In this case, the video data D1 and drawing data D2 for the portion of the structure ST that is displayed on the display unit 20 can be superimposed with high accuracy.

[0038] When multiple markers M1 are simultaneously photographed by the photographing unit 10, the superimposing unit 34 may calculate the position of the photographing unit 10 based on the multiple markers M1. In this case, the superimposing unit 34 can calculate the position of the photographing unit 10 based on Equation 2. In Equation 2, i is a point index, x i , y i are the projection coordinates on the focal plane FF of the imaging unit 10 (see FIG. 7), x0 and y0 are the focal point, f is the focal length, and X i , Y i , Z i indicates the three-dimensional coordinates of the marker M1, and X0, Y0, and Z0 indicate the position of the imaging unit 10. 11 From 33 Each coefficient of is a component of the rotation matrix R of the imaging unit 10. The rotation matrix R is expressed by the following equation 3. FIG. 7 is a diagram schematically showing the positional relationship between the imaging unit 10, multiple markers M1 that are simultaneously imaged, and the focal plane FF of the imaging unit 10.

number

number

[0039] In this way, by calculating the position of the imaging unit 10 by the superimposing unit 34, the accuracy of the position of the imaging unit 10 is improved.

[0040] As shown in FIG. 8 , the display control unit 35 displays the superimposition data D3 generated by the superimposition unit 34 on the display unit 20. By viewing the video data D1 and the already-constructed portion S1 of the drawing data D2 superimposed on the video data D1 in the superimposition data D3 displayed on the display unit 20, the worker can confirm whether the structure ST has been constructed in accordance with the drawing data D2. Furthermore, by viewing the video data D1 displayed on the display unit 20 and the unconstructed portion S2 of the drawing data D2 superimposed on the video data D1, the worker can confirm whether it is possible to create a configuration of the structure ST corresponding to the unconstructed portion S2. In this case, for example, it is not necessary to accurately superimpose the video data D1 corresponding to the entire structure ST on the entire drawing data D2; it is sufficient to accurately superimpose the video data D1 and the drawing data D2 in the portion displayed on the display unit 20.

[0041] As described above, according to the first aspect of the present disclosure, there is provided a drawing data superposition device 100 including an imaging unit 10 that captures an image of a structure ST in real space to which a plurality of markers M1 are attached and generates video data D1, a display unit 20 that displays information, and a control unit 30 that acquires the video data D1 and drawing data D2 of the structure ST and superimposes the acquired video data D1 and drawing data D2 on the display unit 20, wherein the drawing data D2 has a plurality of marker coordinates M2 set at positions corresponding to the plurality of markers M1 on the structure ST, and the control unit 30 matches the plurality of markers M1 included in the video data D1 with the plurality of marker coordinates M2 included in the drawing data D2 using the least squares method, thereby superimposing the drawing data D2 on the video data D1.

[0042] According to this configuration, when the drawing data D2 is superimposed on the video data D1, the markers M1 included in the video data D1 are matched with the marker coordinates M2 included in the drawing data D2 by the least squares method, thereby reducing errors when superimposing the video data D1 and the drawing data D2. This allows the video data D1 and the drawing data D2 to be superimposed with high accuracy.

[0043] According to the second aspect of the present disclosure, in the first aspect, the control unit 30 performs matching using the least squares method for multiple markers M1 and marker coordinates M2, weighting them so that the markers M1 and the marker coordinates M2 corresponding to those markers M1 that are located closer to the imaging unit 10 have higher importance.

[0044] According to this configuration, the closer to the position of the imaging unit 10, the more accurately the video data D1 and the drawing data D2 can be superimposed.

[0045] According to the third aspect of the present disclosure, in the first aspect, the control unit 30 performs matching using the least squares method for a plurality of markers M1 and marker coordinates M2, weighting them so that the markers M1 and the marker coordinates M2 corresponding to the markers M1 that are located closer to the portion displayed on the display unit 20 have higher importance.

[0046] According to this configuration, the closer the image data D1 and the drawing data D2 are to the portion displayed on the display unit 20, the more accurately the image data D1 and the drawing data D2 can be superimposed.

[0047] According to the fourth aspect of the present disclosure, in the third aspect, the drawing data D2 includes an existing part S1 to be installed in the structure ST displayed on the display unit 20 and an uninstalled part S2 that will not be installed in the structure ST displayed on the display unit 20, and the control unit 30 superimposes the existing part S1 and the uninstalled part S2 on the video data D1.

[0048] According to this configuration, by viewing the video data D1 and the drawing data D2 displayed in a superimposed state on the display unit 20, it is possible to accurately grasp whether the portion of the structure ST corresponding to the existing portion S1 is being constructed in accordance with the drawing data D2. It is also possible to accurately grasp whether the portion of the structure ST corresponding to the unconstructed portion S2 can be formed.

[0049] According to the fifth aspect of the present disclosure, in the first aspect, when multiple markers M1 are simultaneously photographed by the photographing unit 10, the control unit 30 calculates the position of the photographing unit 10 based on the positions of the multiple markers M1 photographed simultaneously.

[0050] According to this configuration, the position of the photographing unit 10 is calculated based on the positions of multiple markers M1 photographed simultaneously, thereby improving the accuracy of the position of the photographing unit 10 when superimposing the video data D1 on the drawing data D2.

[0051] According to a sixth aspect of the present disclosure, in the first aspect, the structure ST includes at least one of a civil engineering structure and an architectural structure. When the structure ST is at least one of a civil engineering structure and an architectural structure, the video data D1 and the drawing data D2 can be superimposed with high accuracy.

[0052] According to a seventh aspect of the present disclosure, there is provided a drawing data superposition program including an acquisition step of acquiring video data D1 of a structure ST in real space with a plurality of markers M1 attached and drawing data D2 of the structure ST, and a display step of superimposing the drawing data D2 of the structure ST on the video data D1 and displaying it on a display unit 20, wherein the drawing data D2 has a plurality of marker coordinates M2 set at positions corresponding to the plurality of markers M1 on the structure ST, and the display step matches the plurality of markers M1 included in the video data D1 with the plurality of marker coordinates M2 included in the drawing data D2 using the least squares method, thereby superimposing the drawing data D2 on the video data D1.

[0053] According to this configuration, when the drawing data D2 is superimposed on the video data D1, the markers M1 included in the video data D1 are matched with the marker coordinates M2 included in the drawing data D2 by the least squares method, thereby reducing errors when superimposing the video data D1 and the drawing data D2. This allows the video data D1 and the drawing data D2 to be superimposed with high accuracy.

[0054] According to an eighth aspect of the present disclosure, a drawing data superposition program is provided that causes a computer to execute an acquisition process for acquiring video data D1 obtained by photographing a structure ST in real space with multiple markers M1 attached and drawing data D2 of the structure ST, and a display process for superimposing the drawing data D2 of the structure ST on the video data D1 and displaying it on a display unit 20, wherein the drawing data D2 has multiple marker coordinates M2 set at positions corresponding to the multiple markers M1 on the structure ST, and the display process superimposes the drawing data D2 on the video data D1 by matching the multiple markers M1 included in the video data D1 with the multiple marker coordinates M2 included in the drawing data D2 using the least squares method.

[0055] According to this configuration, when the drawing data D2 is superimposed on the video data D1, the markers M1 included in the video data D1 are matched with the marker coordinates M2 included in the drawing data D2 by the least squares method, thereby reducing errors when superimposing the video data D1 and the drawing data D2. This allows the video data D1 and the drawing data D2 to be superimposed with high accuracy.

[0056] The technical scope of the present invention is not limited to the above-described embodiments, and appropriate modifications can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0057] D1...video data, D2...data, D3...superimposition data, M1...marker, M2...marker coordinates, R...rotation matrix, S1, ST1...existing part, S2...uninstalled part, FF...focal plane, FR...shooting range, ST...structure, ST2...corresponding part, 10...shooting unit, 20...display unit, 30...control unit, 31...processing unit, 32...storage unit, 33...acquisition unit, 34...superimposition unit, 35...display control unit, 100...data superimposition device

Claims

1. an imaging unit that captures an image of a large structure in real space to which a plurality of markers are attached and generates video data; a display unit that displays information; a control unit that acquires the video data and drawing data of the large structure, and displays the acquired video data and drawing data superimposed on the display unit; Equipped with a plurality of marker coordinates are set in the drawing data at positions corresponding to the plurality of markers on the large structure; The control unit The drawing data is superimposed on the video data by matching the plurality of markers included in the video data with the plurality of marker coordinates included in the drawing data by a least squares method. Drawing data superimposition device for large structures.

2. The control unit performs matching by a least squares method with respect to the plurality of markers and the marker coordinates, weighting the markers and the marker coordinates corresponding to the markers so that the markers and the marker coordinates corresponding to the markers are more important when positioned closer to the imaging unit.

2. The drawing data superimposing device for a large structure according to claim 1.

3. The control unit performs matching by a least squares method with respect to the plurality of markers and the marker coordinates, weighting the markers and the marker coordinates corresponding to the markers so that the markers and the marker coordinates corresponding to the markers are more important when positioned closer to the portion displayed on the display unit.

2. The drawing data superimposing device for a large structure according to claim 1.

4. the drawing data includes an existing portion to be installed in the large structure displayed on the display unit and an uninstalled portion not to be installed in the large structure displayed on the display unit; The control unit superimposes the existing portion and the uninstalled portion on the video data.

4. The drawing data superimposing device for a large structure according to claim 3.

5. When the photographing unit photographs a plurality of the markers simultaneously, the control unit calculates a position of the photographing unit based on positions of the plurality of the markers photographed simultaneously.

2. The drawing data superimposing device for a large structure according to claim 1.

6. The large-scale structure includes at least one of a civil engineering structure and an architectural structure.

2. The drawing data superimposing device for a large structure according to claim 1.

7. an acquisition step of acquiring video data of a large structure in real space to which a plurality of markers are attached and drawing data of the large structure; a display step of superimposing the drawing data of the large structure on the video data and displaying it on a display unit; Including, a plurality of marker coordinates are set in the drawing data at positions corresponding to the plurality of markers on the large structure; The display step superimposes the drawing data on the video data by matching the plurality of markers included in the video data with the plurality of marker coordinates included in the drawing data by a least squares method. A method for overlaying drawing data for large structures.

8. an acquisition process for acquiring video data of a large structure in real space to which a plurality of markers are attached and drawing data of the large structure; a display process in which the drawing data of the large structure is superimposed on the video data and displayed on a display unit; on the computer, a plurality of marker coordinates are set in the drawing data at positions corresponding to the plurality of markers on the large structure; The display process superimposes the drawing data on the video data by matching the plurality of markers included in the video data with the plurality of marker coordinates included in the drawing data by a least squares method. A program for overlaying drawing data for large structures.

Images