Image creation device and support device

Abstract

Even in environments where GPS positioning is difficult, the system easily matches the displayed data, such as measurement data, with the coordinates of the digital twin. [Solution] The input unit 101 receives images acquired from multiple locations on a target structure where markers are provided. The detection unit 102 detects the set markers from among the images from multiple locations. The reference setting unit 103 uses the position of the marker detected by the detection unit 102 as the digital coordinate reference. The creation unit 104 creates a digital twin in virtual space, which reproduces the structure in virtual space based on the images input by the input unit 101, and includes relative positional information between the digital coordinate reference and the image acquisition locations.

Description

Technical Field

[0001] The present invention relates to an image creation device and a support device.

Background Art

[0002] Not limited to the inspection and maintenance management of infrastructure, in various fields, it is important to link inspection / measurement positions and measurement data. Also, in final inspection and quality control, it is important to evaluate the difference between the design value associated with the position and the measured value. In recent years, with the progress of digital technology, digital twin technology that captures information in the real space into the virtual space and associates data such as measurement values, inspection results, and differences from design values with positions has been utilized. According to this technology, it becomes possible to visually confirm.

[0003] For example, a maintenance management support system is disclosed that travels on a road, acquires position information and road surface condition measurement results, and displays the state on a map, enabling visual grasping of repair locations and uninspected locations (Patent Document 1).

[0004] Also, there is a technology for generating a composite image showing the surface of a structure such as a bridge or a tunnel based on a plurality of images and position information. In this technology, by analyzing the composite image or the image before composition to detect cracks on the surface of the structure and superimposing and displaying them on the composite image, it is possible to visually grasp at which positions the cracks have occurred (Patent Document 2).

[0005] Conventionally, the situation of infrastructure etc. has been grasped while referring to data such as measurement results recorded on a medium such as paper and design drawings or map information of structures etc., but by using the above-mentioned technology, it is becoming possible to easily grasp infrastructure information not only for skilled persons.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] However, the aforementioned technology uses a digital twin based on images with a public coordinate system (a coordinate system using latitude and longitude information from GPS, GNSS, etc.) attached, and overlays and displays three-dimensional data of buildings and other structures, measurement information, or information such as the progress of construction, which have location information attached in the public coordinate system. For this reason, in environments where there are shadows or obstructions such as bridge piers, making positioning by GPS etc. difficult, it is difficult to acquire images with location information attached in the public coordinate system necessary for creating the digital twin. Similarly, it is difficult to attach location information in the public coordinate system to the various measurement information that is overlaid and displayed on the digital twin.

[0008] This invention was made to solve the above-mentioned problems, and aims to enable easy matching of displayed content such as measurement data with the coordinates of a digital twin, even in environments where positioning by GPS or the like is difficult. [Means for solving the problem]

[0009] The image creation apparatus according to the present invention comprises an input unit that inputs images acquired from multiple locations on a target structure on which markers are provided; a detection unit that detects set markers from among the acquired images; a reference setting unit that uses the position of the markers detected by the detection unit as a digital coordinate reference; and a creation unit that creates a digital twin based on the images, which includes relative position information between the digital coordinate reference and the acquisition position of the images, and reproduces the structure in a virtual space.

[0010] In one example of the configuration of the image creation device described above, the device further comprises: a first storage unit that stores multiple display contents for each of the multiple divided regions of a structure; a second storage unit that stores relative positional information of the multiple divided regions with respect to the measurement coordinate reference, using the position of a marker as the measurement coordinate reference; and an image generation unit that, along with the digital coordinate reference and the measurement coordinate reference, arranges the multiple display contents stored in the first storage unit to correspond relatively to each of the multiple divided regions and combines them into a digital twin.

[0011] In one example of the configuration of the image creation device described above, the image generation unit aligns the digital coordinate reference with the measurement coordinate reference, thereby making the multiple display contents stored in the first storage unit correspond relatively to each of the multiple divided regions, and combining them into a digital twin.

[0012] The support device according to the present invention includes a position storage unit that stores positional information of multiple locations where images of a target structure are acquired, associating the acquisition order with the starting position based on the location of a marker provided on the structure as the first acquisition position, and a notification unit that notifies the starting position and positional information of the acquisition stored in the position storage unit in the set acquisition order based on the position of the recognized marker.

[0013] In one example configuration of the above-described support device, the notification unit notifies the acquisition location as a trajectory. [Effects of the Invention]

[0014] As explained above, a digital twin of the target structure, which has markers, is created based on images acquired from multiple locations on the structure, with the marker positions as the reference positions. Therefore, even in environments where positioning by GPS or other means is difficult, the coordinates of the digital twin can be easily matched with the displayed content such as measurement data. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 is a configuration diagram showing the configuration of an image creation device according to an embodiment. [Figure 2]FIG. 2 is a perspective view showing the structure 131. [Figure 3] FIG. 3 is a perspective view showing the digital twin 151. [Figure 4] FIG. 4 is a configuration diagram showing the configuration of the support device according to the embodiment. [Figure 5] FIG. 5 is a perspective view showing a notification example of the support device. [Figure 6] FIG. 6 is a configuration diagram showing the hardware configuration of the embodiment. MODE FOR CARRYING OUT THE INVENTION

[0016] Hereinafter, an image creation device according to an embodiment of the present invention will be described with reference to FIG. 1. This image creation device includes an input unit 101, a detection unit 102, a reference setting unit 103, and a creation unit 104.

[0017] The input unit 101 inputs images acquired from a plurality of locations with respect to a target structure provided with markers. For example, each image obtained by imaging (photographing) the structure from a plurality of locations is input. The image to be imaged can be a still image. Also, the image to be imaged can be a moving image. In the case of a moving image, appropriate frames can be cut out and used as a plurality of still images. Also, the image can be an image obtained using a laser scanner. It is preferable to use an image obtained by moving the acquisition point so that the acquisition range of the most recently acquired (imaged) image overlaps with at least half (preferably 70% or more) of the imaging range.

[0018] The input unit 101 can input the position information where the image was acquired (imaged) as supplementary information at the time of creating the digital twin described later. For the captured image (the image may include a marker), a plurality of images acquired at different imaging distances can be targeted. The image can be, for example, an image captured by an operator at a plurality of set locations. For example, the operator can identify and acquire (image) a plurality of set locations using a support device or the like as described later.

[0019] The detection unit 102 detects markers set among images at multiple locations. For example, as shown in FIG. 2, the structure 131 can be a part of a bridge pier made of reinforced concrete. The marker 132 can be, for example, a matrix-type two-dimensional code attached to the structure 131. Also, the shape of a specific location in the target structure can be set as a marker.

[0020] The reference setting unit 103 uses the position of the marker detected by the detection unit 102 as a digital coordinate reference. By performing image processing on the marker detected by the detection unit 102, reference coordinates including the origin can be generated. For example, when imaging markers having at least three specific shapes with relative distances set, the relative positional relationship between the marker and the imaging position can be obtained using the principle of triangulation (detecting the specific shapes by image processing and calculating the positional relationship with the marker using their positional relationship). By performing appropriate coordinate transformation, the imaging (capturing) position with the marker as the origin and reference coordinates can be obtained. By continuously imaging (recording a video), the imaging position with the marker as the origin and reference coordinates can be obtained in real time.

[0021] For example, even if the distances and relative positional relationships of at least three specific shapes provided on the marker are not determined in advance, if the imaging (capturing) device is equipped with a distance measuring sensor such as a Lidar, the relative positional relationship between the marker and the imaging (capturing) position can be derived, and by performing coordinate transformation, the imaging (capturing) position with the marker as the origin and reference coordinates can be obtained.

[0022] Furthermore, sensing information such as inertial sensors can be used to maintain the positional relationship with the marker. When starting image acquisition necessary for creating a digital twin, relative positional relationship information with the marker is acquired and stored. Relative positional relationship information with the marker can include the shooting position coordinates and orientation relative to the marker, and can also include information necessary when converting the normalized distance information described later into actual distance (information related to the scale necessary for deriving the ratio of distance in the image to distance in real space), such as the marker size in the image and the marker size in real space and their ratio, or the size of a specific object in the image (which may be distance, etc.) and the size of an object in real space (which may be distance, etc.) and its ratio. Note that the acquisition of relative positional relationship information with the marker can be done at the end of image acquisition, or it can be done in the middle of image acquisition by storing the relative relationship acquisition time, etc. Next, the structure is photographed (imaged) from multiple locations and input and saved to the input unit 101. It is desirable that the relative positional relationship information with the marker be saved in a form independent of the multiple images (in an independent file or memory space) and that it be possible to retrieve it independently of the images.

[0023] The creation unit 104 uses the images input and saved by the input unit 101 to create a digital twin that reproduces the structure in a virtual space. Rather than creating it in real time when acquiring images for the digital twin, it is generated using dedicated PC equipment after all images have been acquired. If the images are videos, they are converted into images with appropriately reduced frames. The creation unit 104 can, for example, use SfM (Structure from Motion) technology to synthesize multiple images to create a digital twin. In this case, a pose graph is created that represents the relative relationship of the imaging positions for multiple images with different viewpoints. In general pose graph creation, the position information is normalized, and the origin position is arbitrary (generally the center of the entire imaging position space).

[0024] In this embodiment, the image position coordinates of the image in which the marker is captured are transformed using the relative positional relationship information with the marker (transforming the origin and coordinate axes), and the coordinate transformation is also applied to other images while maintaining the relative positional relationship. At this time, the normalized distance information can be used instead of the actual distance information. Next, the 3D space is reconstructed using each image whose relative positional relationship has been determined, and a digital twin is created. Known representations such as 3D point clouds and photogrammetry can be used to represent the digital twin. When normalized coordinates are used, it is desirable to ultimately convert them to the actual distance (size) using the relative positional relationship information with the marker mentioned above.

[0025] Furthermore, digital twins can be created using Nerf (Neural Radiance Fields) technology, which generates images from multiple images using machine learning, and 3DGS (3D Gaussian Splatting) technology, which generates images from free viewpoints more quickly by adding 3D Gaussian data to a coarse point cloud generated by SfM technology.

[0026] Furthermore, if the image input by the input unit 101 includes point cloud data obtained from a laser scanner, infrared camera, or depth camera using a stereo camera, a digital twin can be created using ICP (Iterative Closest Point) technology or the like. The digital twin can be a photograph, a composite image, a point cloud, a mesh, or the like. It can also be a digital twin that has undergone processing such as 3D model conversion. The digital twin of the structure created by the creation unit 104 is displayed on the display unit 108. In all cases, the relative positional relationship information with the aforementioned markers is used at an appropriate timing during the creation process.

[0027] When acquiring (imaging) images, getting closer to the marker makes marker recognition easier, but it becomes difficult to determine which part of the structure is being photographed, which is a disadvantage for creating a pose graph. Conversely, moving further away from the marker makes it easier to grasp the overall structure and create a pose graph, but marker recognition becomes more difficult. For this reason, it is desirable to recognize the marker at close range to set the coordinate system, and then start acquiring images for the digital twin from a slightly further distance.

[0028] Image processing can be used to set coordinate axes for markers, and the camera's position relative to the markers can be obtained in real time. Digital twin creation can be performed once all necessary images are available. Except when obtaining relative relationships with markers, it is not essential to include positional information in the captured images. Relative relationships (generally normalized) can be obtained through the pose graph creation process.

[0029] Furthermore, this image creation device may include a first storage unit 105, a second storage unit 106, and an image generation unit 107. The first storage unit 105 stores multiple display contents for each of the multiple divided regions of the structure. The second storage unit 106 stores relative positional information of the multiple divided regions with respect to the measurement coordinate reference, using the position of the marker as the measurement coordinate reference. The image generation unit 107 combines the multiple display contents stored in the first storage unit, along with the digital coordinate reference and the measurement coordinate reference, with each of the multiple divided regions in relative terms, and combines them into a digital twin. The multiple display contents combined into the digital twin are displayed on the display unit 108 together with the digital twin of the structure.

[0030] The multiple display contents are identification information that identifies locations within multiple divided regions. For example, they are identification information that identifies each location within a matrix-like arrangement of divided regions. For example, a person in charge can perform a predetermined measurement at the location identified by this identification information. The multiple display contents can also be multiple measurement results obtained from measurements taken from the surface side of the structure for each of the multiple divided regions. The measurements may include, for example, the deterioration state of the structure, including salt concentration, delamination, cracks, and voids, and each of the multiple measurement results may indicate the deterioration state of the structure, including salt concentration, delamination, cracks, and voids. The measurements can be performed using a fluorescent X-ray measuring device, a tapping test device, a laser-induced breakdown spectroscopy analyzer, a surface resistance measuring device, an electromagnetic wave radar device, an ultrasonic measuring device, etc.

[0031] For example, the measurement is performed using an X-ray fluorescence analyzer. The X-ray fluorescence analyzer irradiates the target (structure) with X-rays and measures the energy distribution of the resulting fluorescent X-rays to determine the salt concentration of the target. The X-ray fluorescence analyzer is equipped with a terminal that has the function of obtaining the origin coordinates (measurement coordinate reference) defined by the marker through image processing, and the function of estimating its own position. This terminal can estimate its own position using predetermined sensor information, etc., and records the position information at a predetermined timing. The timing of the position information recording is set to synchronize with the end of the X-ray fluorescence measurement, for example.

[0032] The operator captures an image of the marker with a camera on a terminal fixed to the X-ray fluorescence measuring device and begins acquiring relative position coordinates with the marker as the origin. These position coordinates are those of the terminal, not the measurement area (the tip of the device) of the X-ray fluorescence measuring device. Therefore, the tip of the relative position coordinate acquisition device is pressed against the predetermined origin of the marker to reset the coordinates (the tip of the relative position coordinate acquisition device becomes the origin).

[0033] Next, the device is moved to a designated measurement position, and the tip of the X-ray fluorescence measuring device is pressed against the side of the bridge pier to perform the predetermined measurement. This procedure records the salinity and coordinates of the measurement position. The coordinates recorded are the relative coordinates to the marker, the orientation relative to the marker, and generally the rotation angles for each axis, but Euler angles or quaternions can also be used. This procedure is repeated to perform the predetermined measurements, and a list (display content) of all measurement position information and salinity concentrations is stored.

[0034] Since the coordinate systems of the digital twin and the multiple display contents coincide, the display unit 108 superimposes the display contents (measurement data) onto the appropriate position on the digital twin. For example, as shown in Figure 3, the measurement data 152 is superimposed onto the digital twin 151 of the structure and displayed on the display unit 108. The display unit 108 allows observation of measurement data from any position. Known displays can be made, such as color-coding the measurement data and performing appropriate analysis using AI to highlight only abnormal areas and display them on the display unit 108. The display unit 108 can be composed of, for example, VR glasses.

[0035] The digital twin and the displayed content are stored as separate data. By extracting only the displayed content and storing it in appropriate AR glasses or a display terminal, the measurement results can be projected onto the real world at the site where the structure is installed, using the marker as the origin. Furthermore, from a remote location, the displayed content (measurement results) on the digital twin can be used, and at the site, instructions and discussions for detailed inspections and additional measurements can be conducted while viewing the displayed content superimposed on the real world.

[0036] When displaying data on AR glasses, the relative coordinates from the reference marker held by the AR glasses and the relative positional relationship between the measuring part on the measuring instrument used for measurement can be assumed to be obtained by image recognition of the measuring instrument or a mark attached to the measuring instrument. By using AR glasses, location information of the target location can be acquired. Furthermore, if the terminal used for display is not AR glasses, the terminal is fixed to the measuring instrument, and the relative relationship between the terminal's position coordinates and the measuring part is fixed, so it can be corrected (calibrated) in advance.

[0037] Conventional technologies make it difficult to combine and display measurement data acquired on-site with a digital twin. While well-known technologies link both coordinate systems to the public coordinate system of latitude and longitude, acquiring coordinates using GPS positioning, for example, is generally costly, requires expensive equipment and skilled surveying techniques, limiting its application. While the difficulty of determining topography and building outlines is decreasing due to advancements in aerial photography, satellite imagery, and drone technology, adding location information to individual measurements remains extremely difficult. However, according to this embodiment, at least one marker can easily match the displayed content, such as measurement data, with the coordinates of the digital twin.

[0038] Furthermore, when displaying measurement results and other information overlaid on a created digital twin using AR glasses or a tablet screen, the system can receive information on newly stored measurement points and display the additional measurement locations overlaid on the real space on the AR glasses or tablet device at the site.

[0039] Next, the support device will be described with reference to Figure 4. The support device includes a position memory unit 121 and a notification unit 122. The support device can be, for example, a portable device. The support device guides the imaging position.

[0040] The position memory unit 121 stores position information for multiple locations where images of the target structure are acquired, associating the acquisition order with the starting position (digital coordinate reference) based on the location of markers provided on the structure. The notification unit 122 notifies the acquisition start position and position information stored in the position memory unit 121 in the set acquisition order, based on the position of the recognized marker. The notification unit 122 notifies the acquisition location as a trajectory 153, for example, as shown in Figure 5. The notification unit 122 can display and output the notification content, for example, by having a display function such as AR glasses.

[0041] For example, by using a support device and taking images according to the notification, the person in charge can acquire images of a specified number of locations on a target structure where markers are placed. The imaging can be carried out using a mobile robot or an aerial drone.

[0042] Furthermore, the image creation and support devices described above can also be computer devices equipped with a CPU (Central Processing Unit) 301, main memory 302, external storage 303, and network connection device 304, as shown in Figure 6. The CPU 301 operates (executes the program) based on a program stored in the main memory 302, thereby realizing the functions described above. The program described above is a program for the computer to execute the functions of each device. The network connection device 304 connects to the network 305. In addition, each function can be distributed across multiple computer devices.

[0043] As explained above, a digital twin of the target structure, which has markers, is created based on images acquired from multiple locations on the structure, with the marker positions as the reference positions. Therefore, even in environments where positioning by GPS or other means is difficult, the coordinates of the digital twin can be easily matched with the displayed content such as measurement data.

[0044] It should be noted that the present invention is not limited to the embodiments described above, and it is clear that many modifications and combinations can be implemented within the technical concept of the present invention by those with ordinary skill in the art. [Explanation of Symbols]

[0045] 101...Input unit, 102...Detection unit, 103...Reference setting unit, 104...Creation unit, 105...First storage unit, 106...Second storage unit, 107...Image generation unit, 108...Display unit, 121...Position storage unit, 122...Notification unit, 131...Structure, 132...Marker, 151...Digital twin, 152...Measurement data, 153...Trajectory.

Claims

1. An input unit that takes images obtained from multiple locations on a target structure where markers are placed as input, A detection unit for detecting the set marker from the aforementioned image, A reference setting unit that uses the position of the marker detected by the detection unit as a digital coordinate reference, A creation unit creates a digital twin of the structure based on the aforementioned image, which includes relative positional information between the digital coordinate reference and the acquisition position of the aforementioned image. An image creation device equipped with the following features.

2. In the image creation apparatus according to claim 1, A first storage unit that stores multiple display contents for each of the multiple divided regions of the aforementioned structure, A second storage unit stores relative positional information of the plurality of divided regions with respect to the measurement coordinate reference, using the position of the marker as the measurement coordinate reference. An image generation unit combines the digital coordinate reference and the measurement coordinate reference with the image generation unit, which associates the plurality of display contents stored in the first storage unit with each of the plurality of divided regions in relation to the digital twin. An image creation device further equipped with these features.

3. In the image creation apparatus according to claim 2, The image generation unit is an image creation device that matches the digital coordinate reference with the measurement coordinate reference, thereby making the plurality of display contents stored in the first storage unit correspond relatively to each of the plurality of divided regions, and combining them into the digital twin.

4. A position storage unit stores positional information of multiple locations from which images of the target structure are acquired, associating the acquisition order with the starting position based on the location of a marker provided on the structure as the first acquisition position. Based on the recognized marker's position, the notification unit notifies the acquisition start position stored in the position storage unit and the position information in the set acquisition order. A support device equipped with the following features.

5. In the support device according to claim 4, The notification unit is a support device that notifies the acquisition location as a trajectory.

Images