Shape registration system

Abstract

To provide a figure registration system that solves conventional problems, namely, makes good use of respective features of a three-dimensional model and an image to draw an object of interest.SOLUTION: A figure registration system comprises three-dimensional model storage means, image storage means, figure storage means, figure position calculation means, and image extraction means. The figure position calculation means calculates the position of a figure on a three-dimensional space, and the image extraction means extracts a specific image based upon the figure. The figure is stored together with a position calculated by the figure position calculation means, and the image is stored together with a photography position and a photography attitude. Then the image extraction means extracts an image including the figure from a plurality of images stored in the image storage means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention is a technology for managing objects that users should pay attention to, such as cracks, signs, and illegal occupation of roads (hereinafter simply referred to as "objects of interest"). More specifically, it relates to a graphic registration system that uses a 3D model and image to graphically represent objects of interest.

Background Art

[0002] Persons who manage specific facilities such as road administrators and river administrators perform management tasks such as regularly inspecting whether there are any abnormalities in the facilities and recording the inspection results. For example, in the case of a road administrator, they inspect whether there are any cracks or other deformations in concrete structures such as road pavements and retaining walls, conduct facility surveys on road signs and lighting fixtures, and monitor whether there are any illegal occupancies on the managed roads. They leave records each time they conduct inspections or the like.

[0003] When managing roads, rivers, etc., generally the target range is vast (long). Therefore, in addition to the results of inspections or the like (such as the presence or absence of abnormalities), the position of "objects of interest" such as cracks, signs, and illegal occupation of roads that are the targets becomes important. Furthermore, the shape of the objects of interest often becomes useful information along with the position. That is, by recording the attribute information (such as inspection results) of the objects of interest together with information such as their position and shape, more effective management tasks can be performed.

[0004] On the other hand, because roads, rivers, etc. have a vast management target range, it is not realistic to frequently conduct inspections or surveys of the entire area. On the contrary, it is easier to perform high-quality management tasks by conducting inspections or the like more frequently. Therefore, instead of directly visually inspecting on-site, it is conceivable to conduct inspections or the like using photos. That is, first, photos of the management target range are taken, and the necessary information is obtained by visually observing the photos. As a result, inspection or other work can be performed in an office or the like without going to the site, and survey results or the like can be obtained more frequently.

[0005] Furthermore, photogrammetry techniques can be used to determine the position of an object of interest, and by tracing the photograph, the shape of the object can also be extracted. Management methods utilizing photogrammetry techniques have been proposed before; for example, Patent Document 1 proposes a technique for determining the coordinates of an object by photogrammetry and for managing various data related to the photogrammetry process. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 10-246630 [Overview of the project] [Problems that the invention aims to solve]

[0007] By the way, when visually inspecting photographs to identify cracks in, for example, road pavement, it is necessary to review a large number of photographs. This requires considerable effort and time, and human errors such as overlooking cracks are unavoidable. This is mainly because each photograph must be reviewed individually. For example, if there were wide-angle photographs that provide an overview of the area being managed, the effort required for review and human errors could be reduced.

[0008] Therefore, the inventors of this application focused on using a 3D model of the object being managed. By using a 3D model, it is possible to check the road as a continuous flow, and by manipulating the 3D model, it is possible to view it from a desired viewpoint, which significantly reduces the amount of work required and makes it easier to avoid human error.

[0009] However, even when visually inspecting a 3D model that contains color information (e.g., RGB), it can be difficult to clearly extract shapes such as cracks. Generally, this is because 3D models with color information have lower resolution than single photographs. Therefore, it is preferable to use single photographs when extracting shapes such as cracks. However, single photographs alone cannot determine the location of cracks (in this case, their location within the object being managed). In other words, while a 3D model can determine the location of cracks, it is difficult to grasp their shape, while a single photograph makes it easy to extract the shape of cracks, but it is difficult to determine their location.

[0010] The problem addressed by the present invention is to solve the conventional problem, namely, a 3D model and image The objective is to provide a graphic registration system that can visualize objects of interest while utilizing the unique characteristics of each method. [Means for solving the problem]

[0011] The present invention uses a three-dimensional model to determine the position of an object of interest, image This invention focuses on the idea of ​​generating a figure of the object of interest that possesses information about its position and shape by tracing the shape of the object using [a specific method / technology], and is based on a completely new concept.

[0012] The graphic registration system of the present invention comprises a 3D model storage means, an image storage means, a graphic storage means, a graphic position calculation means, and an image extraction means. Of these, the 3D model storage means is a means for storing a 3D model of the target terrain, and the image storage means is a means for storing multiple images of the target terrain. The graphic storage means is a means for storing graphics, the graphic position calculation means is a means for calculating the position of a graphic in 3D space, and the image extraction means is a means for extracting a specific image based on the graphic. The graphics are stored along with the position calculated by the graphic position calculation means, and the images are stored along with the shooting position and shooting orientation. The image extraction means then extracts an image containing a graphic from among the multiple images stored by the image storage means.

[0013] The graphic registration system of the present invention has a server with a 3D model storage means and image Memory means, figure position calculation means, figure memory means, image An extraction means is provided, The system may also be equipped with a display mechanism that displays a 3D model on a terminal device connected to a server via a network. [Effects of the Invention]

[0014] The graphic registration system of the present invention has the following effects: (1) Since a three-dimensional model is used, for example, a road can be confirmed as a series of flows, and it can be viewed from a desired perspective by operating the three-dimensional model. As a result, the labor required for the work is significantly reduced, and it becomes easier to avoid human errors. (2) When a figure is input onto the three-dimensional model, its position information can be obtained, and candidates to be confirmed are presented. Therefore, also in this respect, the labor required for the work is significantly reduced, and it becomes easier to avoid human errors. image (3) Since the shape of the object of interest (such as cracks) can be traced while visually observing a high-resolution image , a more accurate figure of the object of interest can be generated.

Brief Description of the Drawings

[0015] [Figure 1] A block diagram showing the main configuration of the figure registration system of the present invention. [Figure 2] A model diagram schematically showing a shooting parameter frame corresponding to shooting parameters. [Figure 3] A block diagram showing the figure registration system of the present invention configured by a server and a terminal. [Figure 4] A flowchart showing the main processing flow of the figure registration system of the present invention. [Figure 5] A screen diagram showing a state where a 3D model, an input figure, a candidate image, and a shooting parameter frame are displayed on a display means. [Figure 6] A screen diagram showing a state where two selected images are displayed as a stereo image in a separate window. [Figure 7] A screen diagram showing a situation where when the figure of one selected image is adjusted, the figure of the other selected image is adjusted in the same way. [Figure 8] A screen diagram showing a situation where even when the figure of one selected image is adjusted, the figure of the other selected image is not adjusted.

Mode for Carrying Out the Invention

[0016] An example of the graphic registration system of the present invention will be explained with reference to the figures.

[0017] Figure 1 is a block diagram showing the main configuration of the figure registration system 100 of the present invention. As shown in this figure, the figure registration system 100 includes a figure position calculation means 102 and image Extraction means 103, 3D model storage means 109, image It is configured to include a memory means 110 and a graphic memory means 111, and further Graphic input means 101 and Shooting location display means 104, image It can also be configured to include a selection means 105, a graphic adjustment means 106, an attribute information input means 107, and a display means 108.

[0018] The shape registration system 100 consists of a shape input means 101 and a shape position calculation means 102. image Extraction means 103, shooting location display means 104, image The selection means 105, the graphic adjustment means 106, and the attribute information input means 107 can be manufactured as dedicated components, or a general-purpose computer device can be used. This computer device is equipped with a processor such as a CPU, memory such as ROM or RAM, and may also include input means such as a mouse or keyboard and a display. For example, it can be composed of a personal computer (PC) or a server.

[0019] Furthermore, the 3D model storage means 109 and image The memory means 110 and graphic memory means 111 can utilize the memory of a general-purpose computer (e.g., a personal computer) or they can be built on a database server. When built on a database server, they can be located on a local network (LAN: Local Area Network) or they can be a cloud server that stores data via the internet (i.e., wireless communication).

[0020] The following will provide a detailed explanation of each of the main elements that constitute the graphic registration system 100 of the present invention.

[0021] (Means for storing 3D models) The 3D model storage means 109 stores a three-dimensional model (hereinafter referred to as the "3D model") of the terrain targeted by the user (hereinafter simply referred to as the "target terrain"). Here, the 3D model is a representation of the target terrain using three-dimensional coordinates, and refers to terrain models such as DSM (Digital Surface Model) or DEM (Digital Elevation Model).

[0022] 3D models are typically composed of multiple meshes. A mesh is formed by dividing an area into, for example, orthogonal grids, and each mesh has a representative point. Since point clouds obtained by measurement (e.g., laser measurement) are often random data, geometric calculations are often performed to assign heights to the representative points of the mesh. Methods for this calculation include the Triangulated Irregular Network (TIN) method, which determines height using an irregular triangular network formed from random data; the Nearest Neighbor method, which uses the nearest laser measurement point 4; and conventional methods such as the Inverse Distance Weighting (IDW), Kriging method, and averaging method. It should be noted that 3D models can be constructed using various methods, not just laser measurement, including photogrammetry, and it is particularly good to use 3D models that contain color information such as RGB obtained from photographs. Furthermore, the 3D model used here may be constructed specifically for the present invention, or an existing one can be used if available.

[0023] ( image storage means) image The memory means 110 obtains by photographing various parts of the target terrain. image It is used to remember. Typically, the range of the target terrain is vast, therefore image The number of such people also increased, meaning image The memory means 110 has a large number image It stores the following. In addition, in the graphic registration system 100 of the present invention, adjacent imageIt is best to use elements that partially overlap (overlap or side-wrap). image The memory means 110 is, image Along with the shooting position (3D coordinates: X, Y, Z) and shooting orientation (yaw ω, pitch φ, roll κ) when the image was taken, that is, the shooting position and shooting orientation are related (linked). image Remember this. In other words, image The memory means 110 stores image Each of these contains information such as the shooting position and shooting posture.

[0024] (Graphic input means) The shape input means 101 is a means by which an operator inputs shapes onto a 3D model. While visually observing the 3D model displayed on the display means 108, such as an LCD display, the operator can input shapes at desired positions and in desired shapes using a pointing device (such as a mouse, touch panel, pen tablet, touchpad, trackpad, or trackball) or a keyboard. For example, if the operator finds a crack in the asphalt road surface (object of interest) on the displayed 3D model, they can use a pointing device to trace the crack, thus inputting a shape. Here, "shape" includes various shapes such as points, lines, and polygons.

[0025] (Means for calculating the position of a figure) The shape position calculation means 102 is a means for calculating the position (3D coordinates) of shape data (hereinafter simply referred to as "shape data") input by the operator. It performs coordinate calculation processing at the time the operator inputs the shape, or triggered by the operator's instruction. Of course, if the shape is a point, it calculates the coordinates of that point; if the shape is a line, it calculates the coordinates that make up the line segment; and if the shape is a polygon, it calculates the coordinates that make up the surface. The coordinates calculated by the shape position calculation means 102 are coordinates in the 3D coordinate system (3D space) set in the 3D model. Therefore, it reads the necessary information from the 3D model storage means 109 and then performs coordinate calculation processing (Figure 1). The coordinates calculated here (hereinafter referred to as "shape position" for convenience) are associated with (linked to) the input shape data and stored in the shape storage means 111 (Figure 1).

[0026] (Attribute information input means) The attribute information input means 107 is a means for the operator to input attribute information related to the input graphic data, and like the graphic input means 101, it can use a pointing device or keyboard. For example, attribute information such as that the graphic data is a crack, the date and time the crack was discovered, the dimensions of the crack, and the person who input it can be input. The attribute information entered here is associated (linked) with the input graphic data and stored in the graphic storage means 111 (Figure 1).

[0027] ( image extraction means) image The extraction means 103 is, image Numerous items stored by memory means 110 image From among them, including graphic data entered by the operator image This is a means of extracting. As already explained, image Since it has a shooting position and shooting orientation (hereinafter referred to as "shooting specifications"), if you use a 3D model, imageIt is possible to calculate the position (shooting range). In other words, by shooting with those shooting parameters, it is possible to understand which part of the target terrain (i.e., the 3D model) will be the target. Specifically, the 3D model and image Based on the shooting specifications, in the 3D coordinate system of the 3D model image Calculate the shooting range.

[0028] image The extraction means 103 uses the graphic data entered by the operator as a "clue," so to speak, to extract the desired result. image Extract the following. Specifically, the shape position calculated by the shape position calculation means and image The shooting range is compared with (Figure 1), and the graphic data is included within that shooting range. image We will extract the following. Therefore, the extracted here image (Hereinafter, for convenience, "candidate" image " ) There may be one sheet or two or more sheets.

[0029] image Candidates extracted by extraction means 103 image This is displayed on the display means 108 along with the 3D model. At this time, the shape position and image Since the shooting ranges of each are represented in the same 3D coordinate system, the candidates image The position of the graphic data within can also be determined. Therefore, the candidates displayed on the display means 108 image It is preferable to display the graphic data entered by the operator.

[0030] (Means for displaying the shooting location) The shooting location display means 104 is, image Candidates extracted by extraction means 103 imageThis is a means of showing the shooting specifications, and displays the shape (hereinafter referred to as the "shooting specification frame") set according to the shooting position and shooting orientation on the display means 108 together with the 3D model. More specifically, as shown in Figure 2, the shooting specification frame is displayed on the 3D model (i.e., in the 3D coordinate system of the 3D model) at a position corresponding to the shooting position and at an inclination corresponding to the shooting orientation. In Figure 2, at five points (points A to E) image The diagram shows an example of a photograph taken. The photographic parameters for point A are displayed on the display means 108 as photographic parameter frame A, similarly for point B as photographic parameter frame B, for point C as photographic parameter frame C, for point D as photographic parameter frame D, and for point E as photographic parameter frame E. The photographic parameter frame can be configured as a border (i.e., a line) or as a surface (i.e., a polygon).

[0031] ( image selection method) image Selection method 105 is, image Candidates extracted by extraction means 103 image From among the desired candidates, image This is a means for the operator to select, and like the graphic input means 101 and attribute information input means 107, a pointing device or keyboard can be used. image Candidates from among image In selecting the candidate, the candidate displayed on the display means 108 along with the 3D model is selected. image From among the desired candidates image The specification can also specify a desired candidate from among the shooting parameters frames displayed on the display means 108 along with the 3D model. image It is also possible to specify the following: image If you specify that, image The system can also be configured to display the position (shooting range) on the 3D model using coloring or other methods.

[0032] Candidates selected by the operator image (Hereinafter, for convenience, "selected" image The candidate is displayed on the display means 108, and is displayed in a separate window from the 3D model. image Two selections from among them image If you select these two selections image These are displayed side-by-side in a separate window. Therefore, the two selections image When there is some overlap (overlap or side overlap) between them, these selections image The image is displayed as a stereo image, and the operator is selected image It is preferable that the object of interest inside (for example, a crack) can be visualized in tangible form.

[0033] (Means for adjusting shapes) The shape adjustment means 106 selects the 3D model, which is displayed in a separate window from the 3D model. image While visually checking, image This is a means by which the operator adjusts the internal graphic data as needed, and includes a graphic input means 101 and an attribute information input means 107. image Similar to selection method 105, a pointing device or keyboard can be used. Here, adjusting the graphic data refers to operations that change its position, shape, etc.

[0034] The operator is selected by the shape adjustment means 106. image When the internal shape data is adjusted, the shape position calculation means 102 calculates the shape position according to the adjustment (i.e., recalculates). The adjusted shape data, including the recalculated shape position, is stored in the shape storage means 111. At this time, it is possible to specify that the original (pre-adjustment) shape data (including the shape position) is retained as history while the new (adjusted) shape data is stored separately, or it is possible to specify that only the new shape data is stored without retaining the original shape data (so-called overwriting). The adjusted shape data is then used as shape data displayed on the 3D model, and as candidate data. image This is also reflected in the shape data displayed within, i.e., 3D models and candidates. imageThe adjusted shape data will be displayed.

[0035] (System Configuration) The graphic registration system 100 of the present invention can also be configured with a server (e.g., a cloud server) and terminal equipment, as shown in Figure 3. In this case, the server has a graphic position calculation means 102 and image Extraction means 103, shooting location display means 104, 3D model storage means 109, image A memory means 110 and a graphic memory means 111 are provided, and the terminal device has a graphic input means 101 and image It is preferable to provide a selection means 105, a graphic adjustment means 106, an attribute information input means 107, and a display means 108. Here, terminal devices refer to devices such as personal computers (PCs), tablet computers (such as iPad®), and smartphones, which are connected to a server via a network.

[0036] Furthermore, the graphic registration system 100 can also connect multiple terminal devices (terminals A to C in the diagram) to one server, as shown in Figure 3. In addition, from each terminal device image The server sends the following to the receiving server: image of image It is also possible to specify that the data be stored in the memory means 110, or that a large number of data sent from each terminal device are stored. image It is also possible to configure the system so that the server creates the 3D model and stores it in the 3D model storage means 109. In this case, which terminal device image To distinguish between a 3D model and a digital model, the terminal device information and image It's a good idea to associate (link) 3D models with the information before memorizing it.

[0037] (Process flow) The main processes of the graphic registration system 100 will be described in detail below with reference to Figure 4. Figure 4 is a flowchart showing the flow of the main processes of the graphic registration system 100 of the present invention. In this flowchart, the actions to be performed are shown in the center column, the necessary items for those actions are shown in the left column, and the results of those actions are shown in the right column.

[0038] As shown in Figure 4, the 3D model is first read from the 3D model storage means 109 and displayed on the display means 108, such as a liquid crystal display (Step 201). When the operator finds a crack or other object of interest in the 3D model, they input graphic data to trace the crack on the 3D model using the graphic input means 101 (Step 202). Figure 5 shows the state in which the input graphic data is displayed on the 3D model. At this time, the operator can also input attribute information related to the graphic data using the attribute information input means 107. The attribute information entered here is associated (linked) with the graphic data and stored in the graphic storage means 111.

[0039] When graphic data is input, the graphic position calculation means 102 calculates the graphic position (Step 203). The calculated graphic position is then associated (linked) with the graphic data and stored in the graphic storage means 111. Once the graphic position is calculated, candidate images containing the input graphic data are also stored. image but image Extracted by extraction means 103 (Step 204), candidate image The position of the shape data within is determined (Step 205), and candidates containing the shape data are selected. image The data is displayed on the display means 108 (Step 206). Figure 5 shows a candidate including graphic data. image This shows the display along with the 3D model.

[0040] candidate image When extracted, the extracted candidates image The shooting parameters frame is displayed on the display means 108 by the shooting location display means 104 (Step 207). Figure 5 shows the state in which the shooting parameters frame is displayed together with the 3D model. The operator then, image Candidates using the extraction means 103 image Select from among those you wish image Select (Step 208). At this time, the candidates displayed on the display means 108 image Select from among those you wish imageThe specification can also specify this, or in addition to this (or instead), a desired candidate from among the shooting parameters frames displayed on the display means 108. image It is also possible to specify this in the specifications.

[0041] election image Once selected, the selection will be displayed in a separate window from the 3D model. image The following will be displayed (Step 209). At this time, the candidates image Two selections from among them image If you select these two selections image These are displayed side by side in a separate window, meaning these selected image The image is displayed as a stereo image, and the operator is selected image The object of interest within (e.g., cracks) can be visualized in tangible form. Figure 6 shows the selected objects, each containing graphic data. image However, this shows the items displayed side-by-side in separate windows.

[0042] The 3D model is selected in a separate window. image When this is displayed, the operator selects using the shape adjustment means 106. image Selected while visually inspecting image The internal shape data is adjusted as needed (Step 210). The results of these adjustments are then displayed on the 3D model as shape data and candidates. image This is also reflected in the shape data displayed within, i.e., 3D models and candidates. image The adjusted shape data is displayed. Figure 7 shows the selection of one side (for example, the left side) in a separate window. image When the shape data is adjusted, the selection of the other side (for example, the right side) is affected accordingly. image The same applies to the geometric data, which is also adjusted. Note that, as shown in Figure 7, both selections in a separate window... image A mode is provided in which the geometric data is adjusted in conjunction (a so-called linked mode), and one of the selections in a separate window image The result of adjusting the shape data is selected by the other party. imageIt is also possible to provide a mode that is not affected by the graphic data (a so-called independent mode). Figure 8 shows the selection of one side (for example, the left side) in a separate window. image When adjusting the shape data, the selection of the other side (for example, the right side) image The graphic data indicates a state that remains unchanged (i.e., not adjusted).

[0043] election image When the graphic data inside is adjusted, the adjusted graphic data is stored in the graphic storage means 111 by the operator performing a registration operation (Step 211). [Industrial applicability]

[0044] The graphic registration system of the present invention is particularly useful for road and river administrators, including national and local governments. Because the present invention contributes to the efficient maintenance of roads and rivers, it not only allows users to comfortably utilize these facilities, but also contributes to extending the lifespan of infrastructure such as roads and rivers and reducing life cycle costs. Therefore, the present invention is not only industrially applicable but also has the potential to make a significant social contribution. [Explanation of Symbols]

[0045] 100. The present invention's graphic registration system 101 (Shape registration system) Shape input means 102 (Figure registration system) Figure position calculation means 103 (of the shape registration system) image extraction means 104 (Means for displaying the shooting location of the graphic registration system) 105 (of the shape registration system) image Selection method 106 (Shape registration system) Shape adjustment means 107 Attribute information input means (of the graphic registration system) 108 Display means (of the graphic registration system) 109 (3D model storage means of the graphic registration system) 110 (of the shape registration system) image storage means 111 (Figure storage means of the figure registration system)

Claims

1. A 3D model storage means for storing a 3D model of the target terrain, Image storage means for storing multiple images of the aforementioned target terrain, A graphic input means for an operator to input shapes into the three-dimensional model displayed on the display means, A figure storage means for storing the aforementioned figure, A figure position calculation means for calculating the position of the aforementioned figure in three-dimensional space, Image extraction means for extracting a specific image based on the aforementioned figure, The system includes an image selection means for an operator to select a desired image from a plurality of images extracted by the image extraction means, The aforementioned figure is stored along with the position calculated by the figure position calculation means. The aforementioned image is stored along with the shooting position and shooting posture. The image extraction means extracts a plurality of images containing the figure from a plurality of images stored in the image storage means. A graphic registration system characterized by the following features.

2. The system further comprises a shape adjustment means for an operator to adjust the shape, Multiple images selected by the image selection means are displayed together with the graphic in a separate window on the display means. The system has a "linked mode" in which, when the operator adjusts the shape in the image displayed in a separate window, the shape in other images is also adjusted, and an "independent mode" in which, when the operator adjusts the shape in the image displayed in a separate window, the shape in other images remains in its original state. The graphic registration system according to claim 1, characterized by the features described above.

Images