Figure registration system

Abstract

To provide a figure registration system that solves conventional problems, namely, makes good use of respective features of a three-dimensional model and an image to draw an object of interest.SOLUTION: A figure registration system comprises three-dimensional model storage means, image storage means, figure storage means, figure position calculation means, and image extraction means. The figure position calculation means calculates the position of a figure on a three-dimensional space, and the image extraction means extracts a specific image based upon the figure. The figure is stored together with a position calculated by the figure position calculation means, and the image is stored together with a photography position and a photography attitude. Then the image extraction means extracts an image including the figure from a plurality of images stored in the image storage means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention is a technology for managing objects that users should pay attention to, such as cracks, signs, and illegal road occupancies (hereinafter simply referred to as "objects of interest"). More specifically, it is a technology for managing objects of interest, such as cracks, signs, illegal road occupancies, etc., using a 3D model. image This relates to a graphic registration system that utilizes a graphical representation of an object of interest. [Background technology]

[0002] Those who manage specific facilities, such as road and river administrators, regularly inspect those facilities to see if there are any abnormalities and record the results of the inspection. For example, road administrators inspect concrete structures such as road pavement and retaining walls to see if there are any abnormalities such as cracks, conduct facility surveys of road signs and lighting, and monitor whether illegal objects have been installed on roads they manage, and keep records of each inspection.

[0003] When managing roads, rivers, etc., the target area is generally vast (long). For this reason, in addition to the results of inspections (such as the presence or absence of abnormalities), what is important is the location of the "objects of interest" such as cracks, signs, and illegal road occupancies. Furthermore, along with the location, the shape of the object of interest is often useful information. In other words, more effective management work can be carried out by recording the attribute information of the object of interest (such as the inspection results) together with information such as its location and shape.

[0004] On the other hand, because the areas to be managed for roads and rivers are vast, it is not realistic to frequently inspect or investigate the entire area. On the other hand, it is easier to carry out quality management work by inspecting more frequently. For this reason, instead of direct visual inspections on-site, it is possible to carry out inspections using photographs. In other words, first take photographs of the area to be managed, and then visually inspect the photographs to obtain the necessary information. This allows inspections to be carried out in an office, etc., without having to go to the site, and investigation results to be obtained more frequently.

[0005] In addition, photogrammetry technology can be used to determine the position of a target object, and the shape of the target object can be extracted by tracing a photograph. Management methods using photogrammetry technology have been proposed in the past. For example, Patent Document 1 proposes a technology for determining the coordinates of a target object by photogrammetry and managing various types of photogrammetry data in general. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 10-246630 Summary of the Invention [Problem to be solved by the invention]

[0007] By the way, when extracting cracks in road pavement, etc., while visually inspecting photos, it is necessary to check a large number of photos. This requires a considerable amount of effort and time, and human error such as overlooking something is inevitable. This is mainly due to checking each photo one by one. For example, if there were wide-area photos that could be used to get a bird's-eye view of the target area, the effort required for checking and human error could be reduced.

[0008] Therefore, the inventors of the present application focused on using a 3D model of the managed object. By using a 3D model, it is possible to check the road as a series of flows, and by manipulating the 3D model, it is possible to view from a desired viewpoint, which significantly reduces the work effort and makes it easier to avoid human error.

[0009] However, even when visually inspecting a 3D model equipped with color information (e.g., RGB, etc.), it may be difficult to clearly extract the shape of a crack, etc. This is because a 3D model equipped with color information generally has a lower resolution than a single photograph. Therefore, when extracting the shape of a crack, etc., it is desirable to use a single photograph. However, a single photograph alone cannot grasp the position of the crack, etc. (in this case, the position in the managed object). In other words, when a 3D model is used, the position of the crack, etc. can be grasped, but the shape is difficult to grasp, while when a single photograph is used, it is easy to extract the shape of the crack, etc., but the position cannot be grasped.

[0010] The object of the present invention is to solve the conventional problems, namely, to create a 3D model and image The object of the present invention is to provide a graphic registration system that can graphically represent an object of interest while taking advantage of the respective advantages. [Means for solving the problem]

[0011] The present invention utilizes a three-dimensional model to determine the position of an object of interest, image This invention focuses on the fact that by tracing the shape of a target object using a digital camera, a diagram of the target object having information on its position and shape can be generated, and is based on an idea that has not been seen before.

[0012] The graphic registration system of the present invention comprises a 3D model storage means, an image storage means, a graphic storage means, a graphic position calculation means, and an image extraction means. Of these, the 3D model storage means is a means for storing a 3D model of a target terrain, and the image storage means is a means for storing a plurality of images of the target terrain. The graphic storage means is a means for storing a graphic, the graphic position calculation means is a means for calculating the position of the graphic in three-dimensional space, and the image extraction means is a means for extracting a specific image based on the graphic. The graphic is stored together with the position calculated by the graphic position calculation means, and the image is stored together with the shooting position and shooting attitude. The image extraction means extracts an image including a graphic from the plurality of images stored in the image storage means.

[0013] The figure registration system of the present invention has a 3D model storage means and image Storage means, figure position calculation means, figure storage means, image An extraction means is provided, A display means for displaying the three-dimensional model may be provided on a terminal device connected to the server via a network. Effect of the Invention

[0014] The graphic registration system of the present invention has the following advantages. (1) By using a 3D model, for example, it is possible to check a road as a series of steps, and by manipulating the 3D model, it is possible to view it from any desired viewpoint. As a result, the work effort is significantly reduced and human error is easily avoided. (2) When you input a shape onto a 3D model, its position information is obtained and you can confirm it. image In this respect, too, the work effort is significantly reduced and human error becomes easier to avoid, as candidates for the next step are presented. (3) High-resolution image Since it is possible to trace the shape of the target object (such as a crack) while visually inspecting the surface, it is possible to generate a more accurate diagram of the target object. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing the main configuration of a graphic registration system according to the present invention; [Diagram 2] FIG. 13 is a model diagram that illustrates a shooting parameter frame corresponding to the shooting parameters. [Diagram 3] FIG. 1 is a block diagram showing a diagram registration system of the present invention which is composed of a server and a terminal. [Figure 4] FIG. 2 is a flowchart showing a main process flow of the graphic registration system of the present invention. [Diagram 5] 11 is a screen diagram showing a state in which a 3D model, an input figure, candidate images, and a shooting specification frame are displayed on a display means. [Figure 6] A screen diagram showing the two selected images being displayed as a stereo image in a separate window. [Figure 7] 13 is a screen diagram showing a situation in which, when a graphic element in one selected image is adjusted, a graphic element in the other selected image is similarly adjusted. [Figure 8] 13 is a screen diagram showing a situation in which even if a graphic element in one selected image is adjusted, the graphic element in the other selected image is not adjusted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] An example of a graphic registration system according to the present invention will be described with reference to the drawings.

[0017] 1 is a block diagram showing the main components of a graphic registration system 100 according to the present invention. As shown in this figure, the graphic registration system 100 includes a graphic position calculation means 102 and image Extraction means 103, 3D model storage means 109, image The device includes a storage means 110 and a graphic storage means 111, Graphic input means 101 and Shooting location display means 104, image It may also be configured to include a selection means 105, a graphic adjustment means 106, an attribute information input means 107, and a display means 108.

[0018] A figure registration system 100 includes a figure input means 101 and a figure position calculation means 102. image Extraction means 103, shooting location display means 104, image The selection means 105, the graphic adjustment means 106, and the attribute information input means 107 can be manufactured as dedicated means, or a general-purpose computer device can be used. This computer device includes a processor such as a CPU, memories such as a ROM and a RAM, and may further include input means such as a mouse and a keyboard, and a display, and may be configured, for example, as a personal computer (PC) or a server.

[0019] Also, a 3D model storage means 109 image The storage means 110 and the graphic storage means 111 can be a storage device of a general-purpose computer (for example, a personal computer) or can be constructed in a database server. When constructed in a database server, they can be placed in a local network (LAN: Local Area Network) or can be a cloud server that stores data via the Internet (i.e., wireless communication).

[0020] Below, each of the main elements constituting the graphic registration system 100 of the present invention will be described in detail.

[0021] (3D model storage means) The 3D model storage means 109 stores a three-dimensional model (hereinafter referred to as a "3D model") of the terrain targeted by the user (hereinafter simply referred to as the "target terrain"). Here, the 3D model is a representation of the target terrain in three-dimensional coordinates, and is, for example, a terrain model such as a DSM (Digital Surface Model) or a DEM (Digital Elevation Model).

[0022] A 3D model is usually composed of a plurality of meshes. A mesh is formed by dividing the mesh into orthogonal grids, for example, and each mesh has a representative point. Since a point cloud obtained by measurement (for example, laser measurement) is often random data, geometric calculation is often performed to assign height to the representative point of the mesh. This calculation method can be a method using a triangulated irregular network (TIN) that obtains height using an irregular triangulation network formed from random data, a nearest neighbor method that uses the nearest laser measurement point 4, or other conventional methods such as an inverse distance weighting method (IDW), a Kriging method, or an average method. Note that the 3D model is not limited to laser measurement, and can be constructed by various methods including photogrammetry, and it is particularly preferable to use a 3D model that has color information such as RGB obtained from a photograph or the like. The 3D model used here may be constructed for the present invention, or of course, an existing one can be used.

[0023] ( image storage means) image The storage means 110 stores the image data obtained by photographing each part of the target terrain. image Usually, the range of the target terrain is vast, so image The number of image The storage means 110 includes a number of image In the figure registration system 100 of the present invention, imageIt is a good idea to use parts that overlap (overlap or side lap). image The storage means 110 includes: image The shooting position (3D coordinates: X, Y, Z) and shooting attitude (yaw ω, pitch φ, roll κ) when the image was taken are also recorded. In other words, the shooting position and shooting attitude are associated (linked) image In other words, image The storage means 110 stores image Each of these has information on the shooting position and shooting posture.

[0024] (Graphic input means) The graphic input means 101 is a means by which an operator inputs a graphic onto a 3D model, and is a means by which an operator can input a graphic in a desired position and shape using a pointing device (mouse, touch panel, pen tablet, touch pad, track pad, track ball, etc.), a keyboard, etc. while visually viewing the 3D model displayed on a display means 108 such as a liquid crystal display. For example, when an operator finds a crack (target object) in the asphalt road surface on the displayed 3D model, the operator uses a pointing device to trace the crack, that is, input the graphic. Note that the graphic here includes various shapes such as points, lines, and surfaces (polygons).

[0025] (Graphic position calculation means) The figure position calculation means 102 is a means for calculating the position (three-dimensional coordinates) of figure data (hereinafter simply referred to as "figure data") input by an operator, and performs coordinate calculation processing when the operator inputs the figure or when the operator's instruction operation is used as a trigger. Of course, if the figure is a point, the coordinates of the point are calculated, if the figure is a line, the coordinates of the line segments are calculated, and if the figure is a polygon, the coordinates of the faces are calculated. The coordinates calculated by the figure position calculation means 102 are coordinates in a three-dimensional coordinate system (three-dimensional space) set in the 3D model, and therefore, the necessary information is read from the 3D model storage means 109 before the coordinate calculation processing is performed (FIG. 1). The coordinates calculated here (hereinafter for convenience, referred to as "figure position") are associated (linked) with the input figure data and then stored in the figure storage means 111 (FIG. 1).

[0026] (Means for inputting attribute information) The attribute information input means 107 is a means by which an operator inputs attribute information related to the input graphic data, and can utilize a pointing device, keyboard, etc., as with the graphic input means 101. For example, attribute information can be input, such as the fact that the graphic data is a crack, the date and time when the crack was discovered, the crack dimensions, and the person who inputted it. The attribute information input here is associated (linked) with the input graphic data and stored in the graphic storage means 111 (FIG. 1).

[0027] ( image extraction means) image The extraction means 103 is image A large number of image Among them, the shape data entered by the operator is included. image As already explained, image Since the camera has the shooting position and shooting posture (hereinafter referred to as "shooting parameters"), the 3D model can be used to imageIn other words, if you take a picture with that shooting specification, you can determine which range of the target terrain (i.e., the 3D model) is the target. image Based on the shooting specifications, the 3D model's 3D coordinate system image Calculate the shooting range.

[0028] image The extraction means 103 uses the figure data input by the operator as a kind of "clue" to extract the target image Specifically, the figure position calculated by the figure position calculation means and image The range of the image is compared with that of the image (Fig. 1), and the graphic data is included in the range of the image. image Therefore, the extracted image (Hereinafter, for convenience, "candidate" image ") may consist of one sheet or two or more.

[0029] image Candidates extracted by the extraction means 103 image The figure position and the 3D model are displayed on the display means 108. image Since the shooting ranges of each of the candidates are expressed in the same 3D coordinate system, image The position of the graphic data within the candidate displayed on the display means 108 can also be obtained. image It is preferable that the display includes graphic data input by the operator.

[0030] (Means for displaying shooting location) The photographing location display means 104 is image Candidates extracted by the extraction means 103 imageThe imaging parameters are displayed on the display means 108 along with the 3D model in a shape (hereinafter referred to as an "imaging parameter frame") that is set according to the imaging position and imaging posture. More specifically, as shown in FIG. 2, the imaging parameter frame is displayed on the 3D model (that is, in the three-dimensional coordinate system of the 3D model) at a position according to the imaging position and at an inclination according to the imaging posture. In FIG. 2, the imaging parameters are displayed at five points (points A to E). image 1 shows an example of photographing the above-mentioned point A, where the photographing parameters at point A are displayed on the display means 108 as photographing parameter frame A, and similarly, the photographing parameters at point B are displayed as photographing parameter frame B, the photographing parameters at point C are displayed as photographing parameter frame C, the photographing parameters at point D are displayed as photographing parameter frame D, and the photographing parameters at point E are displayed as photographing parameter frame E. The photographing parameter frames can be configured as a frame line (i.e., a line) or as a surface (i.e., a polygon).

[0031] ( image selection method) image The selection means 105 is image Candidates extracted by the extraction means 103 image From among the above, select the desired candidate image The operator can select the candidate by using a pointing device, a keyboard, or the like, in the same manner as the graphic input means 101 and the attribute information input means 107. image Candidates from among image When selecting the candidate, the candidate displayed on the display means 108 together with the 3D model is image The desired candidate from image Alternatively, in addition to (or instead of) this, a desired candidate can be selected from among the shooting specification frames displayed on the display means 108 together with the 3D model. image It is also possible to specify the candidate image If you specify image The position (shooting range) can also be displayed on the 3D model using coloring, etc.

[0032] Candidates selected by the operator image (Hereinafter, for convenience, "selected" image " is displayed on the display means 108, and is displayed in a separate window from the 3D model. image Two selections from image When you select image are displayed side-by-side in separate windows. image If there is some overlap or side lap between these selections, image are displayed as stereo images, and the operator can select image This is advantageous because it allows the user to see the actual object of interest (such as a crack) inside the image.

[0033] (Graphic adjustment means) The graphic adjustment means 106 adjusts the selected image displayed in a separate window from the 3D model. image While visually inspecting the image The figure input means 101 and the attribute information input means 107 are means for an operator to adjust the figure data in the figure as necessary. image A pointing device, a keyboard, etc. can be used as in the selection means 105. Here, adjusting the graphic data means an operation for changing the position, shape, etc. of the graphic data.

[0034] The operator selects the graphic adjustment means 106. image When the graphic data in the area is adjusted, the graphic position calculation means 102 calculates (i.e. recalculates) the graphic position in accordance with the adjustment. The adjusted graphic data, including the recalculated graphic position, is stored in the graphic storage means 111. At this time, the original (pre-adjustment) graphic data (including the graphic position) can be left as history and new (adjusted) graphic data can be stored separately, or the original graphic data can be left alone and only the new graphic data can be stored (so-called overwritten). The adjusted graphic data is then used as the graphic data displayed on the 3D model and as a candidate image This is reflected in the geometric data displayed in the imageThe adjusted shape data is displayed.

[0035] (System Configuration) The figure registration system 100 of the present invention can be configured with a server (for example, a cloud server) and a terminal device as shown in FIG. 3. In this case, the server includes a figure position calculation means 102 and image Extraction means 103, shooting location display means 104, 3D model storage means 109, image The terminal device is provided with a storage means 110 and a graphic storage means 111. image It is preferable to provide a selection means 105, a graphic adjustment means 106, an attribute information input means 107, and a display means 108. Here, the terminal device refers to a device such as a personal computer (PC), a tablet computer (iPad (registered trademark), etc.), or a smartphone, and is connected to the server via a network.

[0036] In addition, the pattern registration system 100 can also connect multiple terminal devices (terminals A to C in the figure) to one server as shown in FIG. 3. Furthermore, each terminal device can image The server that receives the message image of image The storage means 110 may be configured to store the information, or multiple information sent from each terminal device may be stored. image It is also possible to create a 3D model on the server side and store it in the 3D model storage means 109. In this case, the 3D model can be stored in the 3D model storage means 109. image To distinguish between photos and 3D models, image It is a good idea to memorize it by associating (linking) it with a 3D model.

[0037] (Processing flow) The main processing of the graphic registration system 100 will be described in detail below with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the main processing of the graphic registration system 100 of the present invention. In this flow diagram, the action to be performed is shown in the center column, the things necessary for the action are shown in the left column, and the things resulting from the action are shown in the right column.

[0038] As shown in Fig. 4, first, a 3D model is read from the 3D model storage means 109, and the 3D model is displayed on the display means 108 such as a liquid crystal display (Step 201). When an operator finds a crack or the like (target object) in the 3D model, the operator inputs graphic data using the graphic input means 101 so as to trace the crack on the 3D model (Step 202). Fig. 5 shows the state in which the input graphic data is displayed on the 3D model. At this time, the operator can also input attribute information related to the graphic data using the attribute information input means 107. The attribute information input here is associated (linked) with the graphic data and stored in the graphic storage means 111.

[0039] When the graphic data is input, the graphic position calculation means 102 calculates the graphic position (Step 203). The graphic position calculated here is associated (linked) with the graphic data and stored in the graphic storage means 111. When the graphic position is calculated, candidates that include the input graphic data in their images are stored in the graphic storage means 111. image but image The extraction means 103 extracts (Step 204), and the candidates image The position of the graphic data in the region is determined (Step 205), and candidates including the graphic data are selected. image is displayed on the display means 108 (Step 206). image is displayed together with the 3D model.

[0040] candidate image When is extracted, the extracted candidate image The photographing location display means 104 displays the photographing parameter frame on the display means 108 (Step 207). FIG. 5 shows the state in which the photographing parameter frame is displayed together with the 3D model. Then, the operator image Using the extraction means 103, candidates image Select the desired one from image At this time, the candidate displayed on the display means 108 is selected (Step 208). image Select the desired one from imageAlternatively, in addition to (or instead of) this, a desired candidate may be selected from among the shooting specification frames displayed on the display means 108. image It is also possible to specify the following.

[0041] election image When a selection is made, it will be displayed in a separate window from the 3D model. image is displayed (Step 209). image Two selections from image When you select image are displayed side-by-side in a separate window, i.e. image are displayed as stereo images, and the operator can select image The object of interest (e.g., a crack) can be visualized as a solid object. image are displayed side-by-side in separate windows.

[0042] Select in a separate window from the 3D model image When displayed, the operator can select using the graphic adjustment means 106. image Selected while visually inspecting image The graphic data in the 3D model is adjusted as necessary (Step 210). The results of the adjustment are then used to display the graphic data on the 3D model and the candidate image This is reflected in the geometric data displayed in the image The adjusted shape data is displayed in the window. Figure 7 shows the selection of one (for example, the left side) in the other window. image When the shape data of the other (for example, the right side) is adjusted, image The figure data of the two are adjusted in the same way. image A mode in which the shape data of two windows are adjusted in tandem (a so-called linked mode) is provided, and one of the selections in a separate window is adjusted in tandem. image The result of adjusting the shape data of the other selection imageIt is also possible to provide a mode that is not affected by the graphic data of the other window (a so-called independent mode). image When adjusting the shape data of the other (for example, the right) image The graphic data of indicates the situation to remain in that state (i.e., not be adjusted).

[0043] election image When the graphic data within the box is adjusted, the operator performs a registration operation, and the adjusted graphic data is stored in the graphic storage means 111 (Step 211). [Industrial Applicability]

[0044] The graphic registration system of the present invention is particularly useful for road and river administrators, including national and local governments. Since the present invention contributes to the optimal maintenance and management of roads and rivers, it not only allows users to use these facilities comfortably, but also leads to a longer life and reduced life cycle costs for so-called infrastructure such as roads and rivers. Considering this, the present invention can be said to be an invention that can be expected to not only be used industrially, but also to make a great contribution to society. [Explanation of symbols]

[0045] 100 Graphic registration system of the present invention 101 Graphic input means (for graphic registration systems) 102 (Graphic registration system) graphic position calculation means 103 (Graphic Registration System) image extraction means 104 (Graphic registration system) Shooting location display means 105 (Graphic Registration System) image Selection method 106 (Graphic registration system) graphic adjustment means 107 Attribute information input means (for graphic registration system) 108 Display means (of graphic registration system) 109 3D model storage means (of the graphic registration system) 110 (Graphic Registration System) image storage means 111 (Graphic storage means for graphic registration systems)

Claims

1. a three-dimensional model storage means for storing a three-dimensional model of a target terrain; an image storage means for storing a plurality of images of the target terrain; A shape storage means for storing a shape; a figure position calculation means for calculating a position of the figure in a three-dimensional space; and an image extraction means for extracting a specific image based on the graphic. The figure is stored together with the position calculated by the figure position calculation means; The image is stored together with the photographing position and the photographing posture; The image extraction means extracts the image including the graphic from the plurality of images stored in the image storage means. A pattern registration system comprising:

2. the three-dimensional model storage means, the image storage means, the figure position calculation means, the figure storage means, and the image extraction means are provided in a server; a display means for displaying the three-dimensional model is provided in a terminal device connected to the server via a network; 2. The system according to claim 1, wherein the image registration system includes:

Images