Data display system, data display apparatus, data display method, and program

Abstract

To display, on a diagram, a phenomenon captured in an image.SOLUTION: A data display system includes: photographing position pose estimation means for estimating a photographing position pose of image data in a coordinate system of three-dimensional data; phenomenon analysis means for detecting a position of a predetermined phenomenon from the image data; and diagram data display means for displaying the position of the phenomenon on diagram data in a superposed manner, based on the photographing position pose and an analysis result of the phenomenon analysis means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a data display system, a data display device, a data display method, and a program. [Background technology]

[0002] Patent Document 1 discloses a data processing device that calculates distance data from the camera to the structure and tilt data in the three-dimensional direction of the camera from at least three illumination points by the laser pointer in the detailed image data, and combines and integrates multiple pieces of partial image data based on the magnification ratios of the partial image data and the detailed image data in each of two wide and narrow image data sets, to obtain coarse wide-area front image data of the structure with a unified scale. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-135450 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a technique for displaying an event captured in an image on a drawing. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; Including, A data display system is provided.

[0006] According to a second aspect of the present disclosure, an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; Including, A data display device is provided.

[0007] According to a third aspect of the present disclosure, The computer Estimating a photographing position and orientation of the image data in a coordinate system of the three-dimensional data; Detecting a location of a predetermined event from the image data; and displaying the position of the event on the drawing data in a superimposed manner based on the photographing position and orientation and the analysis result of the event analysis means. A method for displaying data is provided.

[0008] According to a fourth aspect of the present disclosure, Computer, an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; A program is provided to function as a Effect of the Invention

[0009] According to the present disclosure, the phenomenon depicted in the image can be displayed on the drawing. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a functional block diagram of the data display system. [Diagram 2] FIG. 2 is a functional block diagram of the data display device. [Diagram 3] 13 is a flow of preparations to be made before using the data display device. [Figure 4] It is a photograph of the object. [Diagram 5] This is three-dimensional data of the object. [Figure 6] This is drawing data for the component. [Figure 7] 1 is a projected image of a component. [Figure 8] This is image data. [Figure 9] This is a mask image. [Figure 10] 13 shows the display state of the three-dimensional data display section. [Figure 11] 13 shows the display state of the three-dimensional data display section. [Figure 12] This is an image of the component. [Figure 13] 13 shows the display state of the drawing data display section. [Figure 14] 13 shows the display state of the drawing data display section. [Figure 15] 3 is a process flow of the data display device. [Figure 16] FIG. 2 is a functional block diagram of the data display device. [Figure 17] 13 is a flow of preparations to be made before using the data display device. [Figure 18] This is a reference image database. [Figure 19] This is the display state of the data display device. [Figure 20] 3 is a process flow of the data display device. [Figure 21] FIG. 13 is a diagram showing a case where a processing circuit provided in a data display device is configured with a processor and a memory. [Figure 22] FIG. 13 is a diagram showing a case where a processing circuit included in a data display device is configured with dedicated hardware. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (Summary of the Disclosure) Hereinafter, an overview of the present disclosure will be described with reference to Fig. 1. As shown in Fig. 1, a data display system 10 includes a photographing position and attitude estimation means 11, an event analysis means 12, and a drawing data display means 13.

[0012] The photographing position and orientation estimation means 11 estimates the photographing position and orientation of the image data in the coordinate system of the three-dimensional data.

[0013] The event analysis means 12 detects a predetermined event from the image data.

[0014] The drawing data display means 13 displays information on the event detected by the event analysis means 12 in a superimposed manner on the drawing data, based on the photographing position and orientation estimated by the photographing position and orientation estimation means 11 .

[0015] According to the above configuration, the phenomenon depicted in the image can be displayed on the drawing.

[0016] (First embodiment) Next, a first embodiment of the present disclosure will be described with reference to the drawings.

[0017] Aging of infrastructure structures, including bridges, is a social issue. Therefore, in the maintenance and management of infrastructure structures, cracks, peeling, exposed rebar, rust, and other abnormalities are investigated and recorded during inspections, and the number and size of the abnormalities are compared with the state at the time of previous inspections to diagnose the safety of the infrastructure structures. As of August 2023, regular inspections every five years are mandatory for bridge maintenance. In bridge inspections, typically, inspection reports are prepared and managed as the results of the inspection, including documents called damage diagrams, which record the location and size of the abnormalities in each component on a drawing, and documents called damage photos, which are photographs that capture each abnormality.

[0018] However, for bridge inspection companies, the task of recording the state of deformation in a damage diagram is not easy. This is because bridges in the real world have a three-dimensional structure, while damage diagrams have a two-dimensional structure, making it difficult to grasp the correspondence between the position of deformation on the bridge and the position of deformation on the diagram. Furthermore, bridge components often have a symmetrical structure, and the diagrams have no geometric features, such as a series of rectangles, which also contributes to the difficulty of grasping the positions.

[0019] Therefore, the inventors have developed a technology to automatically record damage diagrams by taking photos that capture the deformation. However, even if automatic recording of damage diagrams using photos is realized, there may be cases where inspection companies want to check whether the results of the automatic recording are correct. In such cases, the company will have to compare the actual location of the damage on the bridge with the automatically recorded damage diagram and the inspection photos, and once again face the difficulty of determining the location.

[0020] The objective of the present disclosure is to provide a data display system that can present the shooting range of an image in three-dimensional space and display the phenomena captured by the image on a drawing, thereby improving the efficiency of maintenance work by presenting three-dimensional information that is easy for the user to understand that the display on the drawing is working correctly.

[0021] FIG. 2 shows a functional block diagram of the data display device 100. The data display device 100 shown in FIG. 2 is used to realize efficient maintenance of an object, for example, in the maintenance of the object, by presenting the photographed range of the photograph based on the photograph of a predetermined event occurring on the object, and further recording information on the event on drawing data. The object is, for example, a structure that is the subject of maintenance work, such as a bridge, a tunnel, or a road. The predetermined event is an event observed in the maintenance work of the object. The predetermined event is, for example, a deformation occurring on the object, such as cracks, peeling, exposed rebar, or rust.

[0022] 2, the data display device 100 includes a three-dimensional data storage unit 101, an image data input unit 102, an imaging position and attitude estimation unit 103, an event analysis unit 104, a member identification unit 105, a three-dimensional data display unit 106, a drawing data storage unit 107, and a drawing data display unit 108. An LCD 3000 (Liquid Crystal Display) is connected to the data display device 100. The LCD 3000 is a specific example of an output device. The LCD 3000 is a specific example of a display device.

[0023] In this embodiment, the data display device 100 is realized by a single device. However, instead of this, the data display device 100 may be realized by distributed processing using a plurality of devices.

[0024] (Preparation flow) Here, with reference to FIG. 3, a preparation flow to be performed before actually using the data display device 100 will be described.

[0025] As shown in FIG. 3, first, three-dimensional data of the object is generated by three-dimensionally scanning the object and recorded in the three-dimensional data storage unit 101 (S1000). The object is constructed by combining a plurality of members. FIG. 4 shows a photograph of the object. In the example of FIG. 4, the object 1 is a bridge composed of various members such as a member 1a which is a pier, a member 1b which is a deck, and a member 1c which is a girder. FIG. 5 shows three-dimensional data 2 which is three-dimensional data of the object 1. The three-dimensional data 2 belongs to a three-dimensional space defined by a coordinate system 21. In other words, a specific position on the three-dimensional data 2 and a specific position around the three-dimensional data 2 can be expressed by the coordinate system 21. The vertical upward direction is the positive direction of the Z axis. Of the longitudinal directions of the bridge, the direction toward the front of the drawing is the positive direction of the X axis. Of the directions perpendicular to the positive direction of the X axis and the positive direction of the Z axis, the direction toward the right of the drawing is the positive direction of the Y axis. The coordinate system 21 is an orthogonal coordinate system.

[0026] The three-dimensional data 2 is data that represents a three-dimensional structure of an object 1 in the real world. The three-dimensional data 2 includes a collection of multiple three-dimensional points. A three-dimensional point is data that represents a specific position in a three-dimensional space defined by a coordinate system 21. The three-dimensional data 2 is, for example, point cloud data or mesh data.

[0027] The three-dimensional scan is not limited to a specific method as long as it is a method for generating three-dimensional data 2 that represents an object 1 in the real world. The three-dimensional scan is performed, for example, using a LiDAR (Light Detection and Ranging) scanner or a camera. The LiDAR scanner is a sensor that acquires the three-dimensional structure of the surrounding environment as point cloud data with the shooting position as the origin by irradiating laser light and analyzing the reflected light. The method for generating the three-dimensional data 2 by the LiDAR is realized, for example, by performing a three-dimensional scan by the LiDAR from at least one position around the object 1, acquiring at least one point cloud data, and then applying a method for combining multiple point cloud data, such as an ICP (Iterative Closest Point) algorithm. The method for generating the three-dimensional data 2 by the camera is realized, for example, by photographing the object 1 from multiple positions and directions, acquiring multiple photos, and then applying a method for generating point cloud data from multiple photos, such as an SfM (Structure from Motion) algorithm.

[0028] Furthermore, the three-dimensional scan may generate the three-dimensional data 2 as mesh data by a method of converting point cloud data into mesh data or a method of using design information of the object 1. The method of converting point cloud data into mesh data is realized, for example, by generating point cloud data of the object 1 by the above-mentioned method and applying an algorithm for converting the point cloud data into mesh data, such as a Poisson surface reconstruction algorithm. The method of using the design information of the object 1 is realized, for example, by expressing the three-dimensional structure of the object 1 as a combination of simple three-dimensional figures such as a rectangular parallelepiped and a cylinder, and expressing each three-dimensional figure as mesh data.

[0029] Returning to FIG. 3, next, a component label is assigned to each 3D point constituting the 3D data 2 to identify which component of the object 1 the 3D point represents (S1010). In the example of FIG. 5, the 3D point 2a is located at the component 1a, which is a pier of the object 1. Therefore, the 3D point 2a is assigned a component label (hereinafter, component label a) indicating that it is the component 1a. Similarly, the 3D point 2b is assigned a component label indicating that it is the component 1b, and the 3D point 2c is assigned a component label indicating that it is the component 1c. When the object 1 includes a plurality of components of the same type, such as a case where the object 1 includes a plurality of piers, the component labels are assigned to distinguish each of the components of the same type. In other words, the component labels represent the individual components, not the type of the components. The component labels are component identification labels that identify the components.

[0030] The component labels are assigned manually or automatically. For example, a method for automatically assigning component labels uses deep learning for point cloud data, such as Point Net.

[0031] Returning to FIG. 3, next, drawing data for each member of the object 1 is prepared and recorded in the drawing data storage unit 107 (S1020). The drawing data is two-dimensional data that expresses a member that has a three-dimensional structure in the real world as at least one projection drawing. The drawing data is used to create a damage diagram for recording deterioration of a bridge. For example, when inspecting a bridge, an inspector writes down the state of deformation such as cracks and rust that has occurred in the member in the drawing shown by the drawing data. FIG. 6 shows drawing data for member 1a as an example of drawing data. The drawing data 3 expresses a bridge pier, which is a three-dimensional structure, in five projection drawings, projection drawings 31 to 35.

[0032] In step S1020, information associating a position in the projection drawing with a position in the three-dimensional data 2 may be recorded in the drawing data storage unit 107 together with the drawing data. In the example of FIG. 6, the vertex 36 in the projection drawing 32 indicates the same point as the three-dimensional point 2a in the three-dimensional data 2. In this manner, after selecting a plurality of positions from the projection drawing and associating each of them with a position representing the same point in the three-dimensional data 2, the association information can be recorded in the drawing data storage unit 107. As will be described later, the association information may be used in a process in which the drawing data display unit 108 aligns the position of an event occurring in the object 1 with a position in the drawing.

[0033] In step S1020, the drawing data recorded in the drawing data storage unit 107 may be generated from the three-dimensional data 2. For example, the drawing data can be generated by extracting three-dimensional points having a specific component label from the three-dimensional data 2 as three-dimensional data of the component, projecting the three-dimensional data of the component in a plurality of three-dimensional directions to generate projected images, and then extracting the contours of the projected images. FIG. 7 shows a projected image of a component 1a as an example of a projected image. Projected images 31a to 36a are projected images generated by projecting three-dimensional data of a component consisting of three-dimensional points to which a component label a is assigned from the -Y direction, +X direction, +Y direction, -X direction, +Z direction, and -Z direction, respectively. In the example of FIG. 7, the projected image is a black-and-white image in which the area where the three-dimensional points exist is black and the area where they do not exist is white. The drawing data can be generated by extracting the contours from each projected image. However, the projection image 36a is a projection in the -Z direction, that is, in the direction looking up from the ground, and the area represented by the projection image 36a is included in the projection image 31a and the projection image 32a. Therefore, the projection image 36a is not essential for the representation of the member 1a. A projection drawing generated in a direction that is not essential for the representation of the member, such as the -Z direction for the member 1a, may not be recorded in the drawing data generating unit 107. The determination of whether a direction is essential for the representation of the member is, for example, determined by acquiring a distribution of normal directions for the three-dimensional data of the member and determining whether the normal direction is the -Z direction but is less than a predetermined ratio. In the case of the member 1a, since there are almost no points where the normal direction is the -Z direction, it can be determined that the -Z direction is not essential for the representation of the member. Alternatively, when the member has a simple shape such as a rectangular parallelepiped, a direction in which the area in which three-dimensional points exist in the projection image is less than a predetermined ratio can be determined as a direction that is not essential for the representation of the member. In other words, using the example of FIG. 7, a direction in which the number of black pixels in the projected image is equal to or less than a predetermined ratio can be determined as a direction that is not essential for expressing a component.

[0034] Returning to FIG. 3, next, an event occurring in the object 1 is photographed by a camera, and image data is prepared (S1030). FIG. 8 is an example of image data. In the example of FIG. 8, image data 4 photographs an event 5, such as a crack occurring in a bridge pier. The image data 4 is used later as input when using the data display device 100.

[0035] The preparatory flow is not limited to the order shown in Fig. 3. It may be performed in any order, such as step S1010 being performed after step S1000. In particular, step S1030 may be performed before step S1000. In other words, the data display device 100 may be used for image data captured prior to performing a 3D scan of the object 1.

[0036] (Data display device 100) 2, the three-dimensional data storage unit 101 holds the three-dimensional data 2 generated in the advance preparation flow. As described above, each three-dimensional point constituting the three-dimensional data 2 is assigned a member label.

[0037] The image data input unit 102 receives image data 4 and camera information used when capturing the image data 4. The camera information is information that can be acquired from the camera, including the focal length used when capturing the image.

[0038] The photographing position and orientation estimation unit 103 estimates the photographing position and orientation of the image data 4 in the coordinate system 21 as the photographing position and orientation 1031 based on the three-dimensional data 2 and the image data 4. In other words, the photographing position and orientation estimation unit 103 estimates the position and orientation of the camera in the real world when the image data 5 is photographed in the coordinate system 21. The expression of the photographing position and orientation 1031 is not limited to a specific expression as long as it expresses six degrees of freedom of the camera position and orientation. The photographing position and orientation 1031 is expressed, for example, as an external parameter of the camera.

[0039] The method by which the image capturing position and orientation estimation unit 103 estimates the image capturing position and orientation is not limited to a specific one. A technology for estimating the image capturing position and orientation of a camera in a three-dimensional space is typically called Visual Localization. The image capturing position and orientation estimation unit 103 may use a method such as, for example, Render Net, a known technology, in which a rendered image obtained by drawing three-dimensional data using various camera positions and orientations set in a three-dimensional space is compared with image data. In short, first, a plurality of rendered images are generated by drawing three-dimensional data using a plurality of camera positions and orientations set in advance, global feature amounts (global feature amounts) of the image data and each rendered image are calculated, and a rendered image similar to the image data is selected by comparing the global feature amounts. Next, an image matching technology is applied that matches pixels representing the same point between the selected rendered image and the image data. Next, since each pixel of the rendered image represents a specific position of the three-dimensional data, a two-dimensional and three-dimensional correspondence is obtained based on the obtained image matching result, such as matching a specific pixel of the image data with a specific position of the three-dimensional data. Next, the shooting position and orientation of the image data can be estimated by applying a method for calculating the shooting position and orientation of the image data based on the correspondence between two dimensions and three dimensions, such as a PnP (Perspective-n-Points) algorithm.

[0040] The event analysis unit 104 detects the position of the event 5 from the image data 4 shown in Fig. 8. For example, the event analysis unit 104 identifies pixels representing the event 5 from the image data 4 by semantic segmentation of the image, and generates a mask image representing the position of the event 5. For example, the event analysis unit 104 generates the mask image by deep learning such as U-Net. Fig. 9 is an example of a mask image for the event 5 in the image data 4. The mask image 1041 is a black and white image, and the position where the event 5 is detected from the image data 4 is expressed as an area 1042.

[0041] The event analysis unit 104 estimates the size of the event 5 based on the shooting position and orientation 1031. The size of the event 5 is, for example, the length and width of the event 5, and the length of the sides of a rectangle that surrounds the event 5. In the example of FIG. 9, the length of the event 5 is calculated as the length between the coordinates of pixels 1041a and 1041b on the three-dimensional data 2. Moreover, the width of the event 1042 is calculated as the length between the coordinates of pixels 1041c and 1041d on the coordinate system 21.

[0042] Here, for a specific pixel of an image whose shooting position and orientation and focal length are known, the coordinates in the coordinate system 21 can be calculated as follows. First, since the angle of view of the camera is known from the focal length, a half line representing the direction in which the specific pixel is viewed from the shooting position is obtained in the coordinate system 21. Next, the coordinates in the coordinate system 21 for the specific pixel are obtained by obtaining the collision point between the obtained half line and the three-dimensional data 2. When the three-dimensional data 2 is mesh data, the collision point between the half line and the three-dimensional data 2 can be calculated as the intersection point closest to the shooting position among the intersection points between each face constituting the mesh data and the half line. Also, when the three-dimensional data is point cloud data, the collision point between the half line and the three-dimensional data 2 can be calculated as the three-dimensional point closest to the shooting position among the three-dimensional points included in the three-dimensional data 2 whose distance from the half line is equal to or less than a predetermined value. However, in this case, the estimation accuracy of the coordinates is about the same as the density of the point cloud data, and there is a risk that the accuracy will be insufficient in applications such as calculating the width of a crack, which requires an accuracy of 1 millimeter or less. Therefore, from the three-dimensional data 2, three-dimensional points near the obtained collision point are extracted to create a neighborhood point set, and the neighborhood point set is approximated by a plane, and then the intersection of the plane and a half line is newly adopted as the collision point. This allows the coordinates of a specific pixel in the coordinate system 21 to be obtained with a precision finer than the density of the point cloud data.

[0043] Pixels 1041a and 1041b that define the length of the event are calculated, for example, as end points in the longitudinal direction of region 1042. Pixels 1041c and 1041d that define the width of the event are calculated, for example, as end points of region 1042 in a direction perpendicular to the longitudinal direction.

[0044] The member identification unit 105 identifies a member depicted in the image data 4 based on the photographing position and orientation 1031, and outputs the member identification result 1051. For example, the member identification unit 105 calculates member labels for all pixels of the image data 4, and then outputs the member label with the largest number of pixels as the member identification result 1051. Alternatively, the member identification unit 105 calculates a member label for a pixel at the center of the image data 4, and then outputs the member label as the member identification result 1051. The member identification unit 105 calculates a member label for a specific pixel of the image data 4, for example, by first calculating coordinates in the coordinate system 21 for a specific pixel of the image data 4 using the photographing position and orientation 1031 and the focal length by the above-mentioned method. Next, a three-dimensional point closest to the calculated coordinates is searched for among the three-dimensional points of the three-dimensional data 2, and a member label for the found three-dimensional point is referred to, whereby a member label for the specific pixel of the image data 4 can be calculated.

[0045] When the component identification unit 105 is executed after the event analysis unit 104, it may calculate the component identification result 1051 based on the mask image 1041 generated by the event analysis unit 104. For example, the component identification unit 105 calculates component labels over an area 1042 in the mask image 1041, and then outputs the component label with the largest number of pixels as the component identification result 1051.

[0046] The three-dimensional data display unit 106 displays the image data 4 and the three-dimensional data 2 on the LCD 3000 in a superimposed manner, thereby three-dimensionally presenting the shooting range of the image data 4 in the object 1. The three-dimensional data display unit 106 changes the display method from a state in which an outline of the three-dimensional data 2 is presented to a state in which the image data 4 and the three-dimensional data 2 are superimposed and displayed. FIG. 10 shows an example of a state in which an outline of the three-dimensional data 2 is presented. The area 1061 is an area in which the three-dimensional data display unit 106 displays data. The area 1061 is, for example, a display, a window, or a part of a window. The media 1062 is an image, a video, or a three-dimensional data display showing the overall outline of the three-dimensional data 2. The image is, for example, an image that overlooks the entire three-dimensional data 2. The video is, for example, a video in which the three-dimensional data 2 is rotated and displayed in the area 1061. The three-dimensional data display is a media format that accepts input from a mouse, touch panel, etc., and displays the three-dimensional data 2 by rotating and enlarging / reducing it in response to, for example, drag-and-drop with a mouse or sliding on a touch panel.

[0047] FIG. 11 is an example of a state in which the image data 4 and the three-dimensional data 2 are superimposed. The media 1063 includes a display 1064 that superimposes the color of the image data 4 on the three-dimensional data 2, thereby presenting the location of the shooting range of the image data 4 in the target object 1 so that it is easy to visually recognize. The media 1063 may also highlight the three-dimensional data of the component represented by the component identification result 1051 as in the display 1065. Highlighting is, for example, mixing a specific color such as red with the color of the three-dimensional data 2. In the example of FIG. 11, the media 1063 highlights the pier in the display 1065 by mixing gray with the pier. By performing such highlighting, the display 1064 can be made to stand out even more.

[0048] Methods that the three-dimensional data display unit 106 can use to superimpose the image data 4 and the three-dimensional data 2 include, for example, arranging image objects, coloring the three-dimensional data, and superimposing the data on a drawn image from a specified viewpoint.

[0049] The method of arranging an image object is a method of arranging image data 4 as a three-dimensional object on the coordinate system 21, and then displaying the arranged three-dimensional object together with the three-dimensional data 2. The method of arranging an image object is, for example, a method of first arranging an image plane in a perspective projection model of the camera as a three-dimensional planar object on the coordinate system 21 based on the shooting position and attitude 1031 and the focal length. Next, the image data 4 is attached to the image plane. Next, the image plane and the three-dimensional data 2 are displayed together from a position behind the shooting position and attitude 1031. When the method of arranging an image object is used, the media 1063 can take any display form of an image, a video, or three-dimensional data display.

[0050] The method of coloring three-dimensional data is a method of coloring a portion of the three-dimensional data 2 that is included within the shooting range of the image data 4 with the image data 4, and then displaying the colored three-dimensional data 2. As described above, the coordinates in the coordinate system 21 can be calculated for each pixel of the image data 4. Therefore, for example, the three-dimensional data 2 can be colored by overwriting each pixel of the image data 4 with the three-dimensional color of the three-dimensional point of the three-dimensional data 2 that is closest to the pixel of the image data 4 on the coordinate system 21. However, when calculating coordinates using the above-mentioned method, if the three-dimensional data 2 is point cloud data, the coordinates can be calculated with higher accuracy than the density of the point cloud data. In such a case, the three-dimensional data display unit 106 may add a new three-dimensional point to the position of the calculated coordinates when the medium 1063 displays the three-dimensional data 2. In this case, the colored area is drawn with a higher density than the density of the three-dimensional data 2, so that the shooting range of the image data 4 can be made even more noticeable. When using the method of coloring three-dimensional data, the medium 1063 can take any display form of an image, a video, or a three-dimensional data display.

[0051] In the method of superimposing on a drawn image from a predetermined viewpoint, a position and orientation at the time of drawing is set for each component, and the three-dimensional data 2 is drawn from the set position and orientation according to the component identification result 1051 to create a component image, and then the image data 4 is deformed and superimposed on the component image. The position and orientation at the time of drawing is set, for example, so that one surface of the component falls within the angle of view. FIG. 12 shows an example of deforming and superimposing the image data 4 on the component image. In the example of FIG. 12, the photographing position and orientation at the time of drawing is set so that the component 1a in the three-dimensional data 2 is photographed from the front, and a component image 1066 is created. In the example of FIG. 12, only the component 1a in the three-dimensional data 2 is drawn, but the component image may include a portion of the three-dimensional data 2 other than the component 1a as long as the photographing position and orientation at the time of drawing is set with attention paid to the target component. In FIG. 12, an image 1067 obtained by deforming the image data 4 is superimposed on the created component image 1066 as an image.

[0052] As a method for transforming the image data 4, for example, homography transformation may be adopted. In homography transformation, the image data 4 is transformed based on N point correspondences (four or more) between the image data 4 and the component image 1066 so that the distance between the corresponding points becomes closer. The point correspondences are calculated as follows. First, the coordinates in the coordinate system 21 are calculated for the N points of the image data 4 based on the shooting position and orientation 1031 and the focal length. Next, the position in the component image 1066 is calculated for each of the N coordinates. Through the above procedure, the N point correspondences between the image data 4 and the component image 1066 are obtained. The N points of the image data 4 are obtained, for example, by randomly extracting N points from pixels in the image data 4 that represent the same component as the component identification result 1051. Whether or not a specific pixel of the image data 4 represents the same component as the component identification result 1051 can be determined by obtaining the coordinates in the coordinate system 21 for the specific pixel of the image data 4 and checking whether the component label of the closest three-dimensional point represents the same component as the component identification result 1051.

[0053] When changing the display method from a state in which an outline of the three-dimensional data 2 is presented to a state in which the image data 4 and the three-dimensional data 2 are superimposed, the three-dimensional data display unit 106 may complement and display the intermediate states as a moving image. For example, the three-dimensional data display unit 106 may calculate a plurality of positions and orientations between the position and orientation of the drawing in the coordinate system 21 immediately before the display method is changed and the position and orientation of the drawing in the coordinate system 21 immediately after the display method is changed, and create a moving image by drawing the three-dimensional data 2 or a three-dimensional object using each calculated position and orientation.

[0054] The drawing data storage unit 107 holds drawing data for each component of the object.

[0055] The drawing data display unit 108 displays information such as the position and size of the event 5 on the LCD 3000 by superimposing it on the drawing data. The drawing data display unit 106 changes the display method from a state in which only the recorded drawing data is presented to a state in which the position and size of the event detected by the event analysis unit 104 are superimposed on the drawing data linked to the component identification result 1051. FIG. 13 shows an example of a state in which only the recorded drawing data is presented. The area 1081 is an area in which the drawing data display unit 108 displays data. The area 1081 is, for example, a display (LCD 3000), a window, or a part of a window. The drawing data display unit 108 selects one piece of drawing data from the drawing data stored in the drawing data storage unit 107 and displays it in the area 1081. The drawing data selected by the drawing data display unit 107 is, for example, drawing data of a component with the largest area or drawing data of a component inspected first in an inspection operation. In the example of FIG. 13, drawing data 3 of a member 1a is displayed.

[0056] FIG. 14 is an example of a state in which the position and size of an event detected by the event analysis unit 104 are superimposed on the drawing data associated with the component identification result 1051. The drawing data display unit 108 displays the drawing data 1082 associated with the component identification result 1051 in an area 1081, and displays position information 1083 regarding the position of the event detected by the event analysis unit 104 and detailed information 1084 regarding the size of the event detected by the business analysis unit 104. In the example of FIG. 14, the drawing data 1082 is drawing data of the component 1a in which the event 5 occurs, the position information 1083 is the shape of the event 5 on the drawing data 1082, and the detailed information 1084 is the actual width and actual length of the event 6, and a classification according to the size and degree of the event 6. The drawing data display unit 108 may also display an image number 1085 for identifying the image data 4 on the drawing. By displaying image number 1085, the user can efficiently search for the photograph that is the basis for the damage displayed on the drawing.

[0057] The shape of event 5 on drawing data 1082 can be calculated by the following procedure. First, when drawing data 1082 is composed of a plurality of projections, drawing data display unit 108 identifies which projection the surface on which event 5 occurred belongs to. For example, for a pixel in area 1042 representing event 5 in mask image 1041 generated by event analysis unit 104, coordinates in coordinate system 21 are calculated, and the surface on which event 5 occurred is identified by calculating which surface of the component the calculated coordinates belong to. Alternatively, the surface on which event 5 occurred is identified by calculating which surface of the component the photographing position and attitude 1031 is closest to.

[0058] Next, the drawing data display unit 108 identifies the shape of the event 5 in the identified projection drawing. For example, the drawing data display unit 108 generates a projection image by projecting three-dimensional data of the component in the projection direction of the projection drawing, and converts the shape of the event 5 on the projection image into the shape on the projection drawing by aligning the projection image with the projection drawing. The shape of the event 5 on the projection image is found, for example, by calculating a homography transformation from the mask image 1041 generated by the event analysis unit 104 to the projection image, and converting an area 1042 of the mask image 1041 into the coordinate system of the pixels on the projection image by the calculated homography transformation.

[0059] The method of aligning the projection image with the projection drawing may involve, for example, extracting a contour from the projection image, approximating the extracted contour with a polygon to calculate vertices, and then aligning them with the vertices of the projection drawing. Alternatively, if information that associates positions in the projection drawing with positions in the three-dimensional data 2 is generated in the advance preparation flow, the information may be used for alignment. Since the projection image is generated from the three-dimensional data 2, information that associates positions in the projection drawing with positions in the projection image can be calculated. The projection image can be aligned with the projection drawing so that the distance between the associated positions is reduced.

[0060] The drawing data display unit 108 may change the display method of the position information 1083 and the detailed information 1084 depending on whether the event is new or, if it is new, whether the event has changed. For example, when the shape of the event has already been recorded at a position similar to the position information 1083, the event is determined to be not new. For example, when the shape of the event already recorded is compared with the position information 1083, or the size of the event already recorded is compared with the detailed information 1084, the amount of change in the shape or size of the event is calculated, and when the amount of change exceeds a predetermined threshold, the event is determined to have changed. For example, the drawing data display unit 108 may display the position information 1083 and the detailed information 1084 in red when the event is new or has changed, and in black when the event has not changed. Furthermore, when the event is not new, the drawing data display unit 108 may display the size of the already recorded event and the size of the event 5 side by side in the display of the detailed information 1084.

[0061] Next, the control flow of the data display device 100 will be described with reference to FIG.

[0062] First, the image data input unit 102 receives the image data 4 (S1100).

[0063] Next, the image capturing position and orientation estimation unit 103 estimates the image capturing position and orientation 1031 (S1110).

[0064] Next, the member identifying unit 105 identifies the member depicted in the image data 4 based on the photographing position and orientation 1031 (S1120).

[0065] Next, the three-dimensional data display unit 106 changes the display method from a state in which an outline of the three-dimensional data 2 is presented to a state in which the image data 4 and the three-dimensional data 2 are superimposed and displayed (S1130).

[0066] Next, the event analysis unit 104 detects an event 5 from the image data 4, and calculates the size of the event 5 based on the image capturing position and orientation 1031 (S1140).

[0067] Next, the drawing data display unit 106 changes the display method from presenting the recorded drawing data to superimposing the position and size of the event detected by the event analysis unit 104 on the drawing data linked to the component identification result 1051 (S1150).

[0068] The control flow is not limited to the order shown in Figure 15. It may be performed in any order, such as step S1110 being performed after step S1100, step S1120 being performed after step S1110, step S1130 being performed after step S1110, step S1140 being performed after S1100, and step S1150 being performed after steps S1120 and S1140.

[0069] The first embodiment has been described above, and the above embodiment has the following features.

[0070] As shown in FIG. 2, the data display device 100 includes a photographing position and orientation estimation unit 103, an event analysis unit 104, a member identification unit 105, a three-dimensional data display unit 106, and a drawing data display unit 108. The photographing position and orientation unit 103 estimates the photographing position and orientation of the image data on the coordinate system of the three-dimensional data. The event analysis unit 104 detects the position and size of the event from the image data. The member identification unit 105 identifies the member shown in the image data based on the photographing position and orientation. The three-dimensional data display unit 106 displays the image data and the three-dimensional data in a superimposed manner. The drawing data display unit 108 displays the event in a superimposed manner on the drawing data of the member based on the photographing position and orientation. According to the above configuration, when the data display device 100 receives image data, it presents the image data superimposed on the three-dimensional data of the object, and further detects the event shown in the image data, and displays the detection result in a superimposed manner on the drawing data of the member shown in the image. In other words, according to the above configuration, the shooting range of an image can be presented in three-dimensional space, and the phenomenon captured by the image can be displayed on a drawing.

[0071] The data display device 100 includes an imaging position and orientation estimation unit 103 that estimates the imaging position and orientation of image data in the coordinate system of three-dimensional data, an event analysis unit 104 that detects the position of a predetermined event from the image data, and a drawing data display unit 108 that superimposes and displays the event position on the drawing data based on the imaging position and orientation and the analysis result of the event analysis unit 104. With the above configuration, the event depicted in the image can be displayed on the drawing.

[0072] Second embodiment The second embodiment will be described below, focusing on the differences between the second embodiment and the first embodiment, and overlapping descriptions will be omitted.

[0073] In the first embodiment described above, a user can photograph an event occurring on an object with a camera, intuitively grasp the location where the photograph was taken in three-dimensional space, and can further confirm that information about the detected event is automatically superimposed on the drawing data.

[0074] However, for users who wish to carry out detailed analysis of changes in events over time, it may be necessary not only to check updates to the drawing data but also to directly compare it with previous photographs.

[0075] Therefore, in this embodiment, previous photographs necessary for direct comparison are automatically presented.

[0076] 16 shows a functional block diagram of the data display device 200. The data display device 200 includes a reference image data DB (database) 209 and an image data display unit 210 in addition to the functions of the data display device 100.

[0077] (Preparation flow) 17, a preparation flow to be performed before actually using the data display device 200 will be described. The preparation flow of this embodiment is obtained by adding step S2040 to the preparation flow of the first embodiment.

[0078] Step S2040 is a step of generating a reference image data DB and storing the generated reference image data DB in the reference data DB 209. The reference image data DB is a DB that manages reference image data showing various events occurring in the object 1 by linking them to the shooting date and time, the focal length, the shooting position and orientation in the coordinate system 21, the event analysis result that records the mask image of the event, the size of the event, and the position of the event on the drawing data, etc. The reference image data is, for example, an inspection photo taken in a past inspection. In the reference image data DB, each reference image data is managed using a uniquely assigned image ID as a key. FIG. 18 is an example of the reference image data DB. The shooting position and orientation are calculated, for example, by applying the shooting position and orientation estimation unit 103 to the reference image data. The event analysis result is calculated, for example, by applying the event analysis unit 104 to the reference image data.

[0079] The order of the advance preparation flow is not limited to that shown in Fig. 17. S2040 can be performed at any time after S1000.

[0080] (Data display device 200) Returning to FIG. 16, the reference image data DB 209 holds the reference image data DB generated in the advance preparation flow.

[0081] The image data display unit 210 searches for similar reference image data from the reference image data DB based on the shooting position and orientation estimated by the shooting position and orientation estimation unit 103 on the LCD 3000, and displays the found reference image data together with the image data 4. A method of searching for similar reference image data, for example, searches for reference image data with high similarity in shooting position and orientation. For example, when the shooting position and orientation are expressed by the external parameters of the camera, the high similarity in the shooting position and orientation is measured by a low norm of the difference in the rotation matrix included in the external parameters, or a low norm of the difference in the translation vector included in the external parameters. Alternatively, a method of searching for similar reference image data, for example, searches for reference image data with a high overlap rate of the shooting range. In this case, based on the shooting position and orientation 1031 and focal length of the image data 4, a set of three-dimensional points included in the angle of view of the image data 4 out of the three-dimensional data 2 is calculated as the shooting range of the image data 4. Also, based on the shooting position and orientation of the reference image data and the focal length of the reference image data, a set of three-dimensional points included in the angle of view of the reference image data out of the three-dimensional data 2 is calculated as the shooting range of the reference image data. Then, the overlapping rate between the shooting range of the image data 4 and the shooting range of the reference image data is calculated. Alternatively, the method of searching for similar reference image data may be to search for reference image data in which the positions of events on the drawing data are similar, for example.

[0082] The data display device 200 may display at least two of the display results of the three-dimensional data display unit 106, the display result of the drawing data display unit 108, and the display result of the image data display unit 210 side by side. FIG. 19 shows an example in which the display results of the three-dimensional data display unit 106, the display result of the drawing data display unit 108, and the display result of the image data display unit 210 are simultaneously displayed side by side on the LCD 3000. In the example of FIG. 19, the data display device 200 displays the area 1061, the area 1081, and the area 2101 side by side on the area 2001. The area 2001 is, for example, a display (LCD 3000), a window, or a part of the window. The area 2101 is the display area of ​​the image data display unit 210. The area 2101 includes a display 2102 related to the reference image data and a display 2103 related to the image data 4. Area 2101 is, for example, a display (LCD 3000), a window, or a part of a window. In the example of Fig. 19, display 2102 displays the detection result of an event for reference image data superimposed on the reference image data. Display 2103 displays the detection result of an event for image data 4 superimposed on image data 4. A method of superimposing the detection result of an event on image data may be, for example, mixing red at the position of the event on the image data. In the example of Fig. 19, the detection result of the event is superimposed on the image data by drawing a hatched line at the position of the event.

[0083] Next, the control flow of the data display device 200 will be described with reference to Fig. 20. The control flow of the data display device 200 is the control flow of the data display device 100 to which an image data display step S2160 for operating the image data display unit 210 has been added.

[0084] The control flow of the data display device 200 is not limited to the order shown in Figure 20. Step S2160 can be performed at any time after step S1110.

[0085] The second embodiment of the present disclosure has been described above. The above embodiment has the following features.

[0086] The data display device 200 includes a reference image data DB 209 and an image data display unit 210. The reference image DB 209 manages a plurality of reference images by linking the shooting date, the estimation result of the shooting position and orientation, and the detection result of the event detection. The image data display unit 210 searches for reference image data whose shooting position and orientation are similar to that of the image data, and displays the image data and the searched reference image data side by side. The above configuration enables a user who wishes to perform a detailed analysis of the aging of an event to not only check the updates in the drawing data, but also to directly compare it with previous photographs. [Industrial Applicability]

[0087] The present disclosure can be applied to applications in which defects captured in inspection photographs taken during bridge inspection work are recorded on drawing data.

[0088] Next, a hardware configuration of the data display device 100 will be described. In the data display device 100, the three-dimensional data storage unit 101 and the drawing data storage unit 107 are memories. The image data input unit 102, the imaging position and orientation estimation unit 103, the event analysis unit 104, the member identification unit 105, the three-dimensional data display unit 106, and the drawing data display unit 108 are realized by processing circuits. The processing circuit may be a processor and memory that executes a program stored in the memory, or may be dedicated hardware.

[0089] FIG. 21 is a diagram showing an example of a case where the processing circuit of the data display device 100 is configured with a processor and a memory. When the processing circuit is configured with a processor 4000 and a memory 4001, each function of the processing circuit of the data display device 100 is realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 4001. In the processing circuit, each function is realized by the processor 4000 reading and executing the program stored in the memory 4001. That is, the processing circuit includes a memory 4001 for storing a program that will result in the processing of the data display device 100 being executed. It can also be said that these programs cause a computer to execute the procedures and methods of the data display device 100.

[0090] Here, the processor 4000 may be a CPU (Central Processing Unit), a processing device, an arithmetic device, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor), etc. Also, the memory 4001 may be, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), an EEPROM (Electrically EPROM), or other non-volatile or volatile semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc), etc.

[0091] Fig. 22 is a diagram showing an example in which the processing circuit of the data display device 100 is configured with dedicated hardware. When the processing circuit is configured with dedicated hardware, the processing circuit 4002 shown in Fig. 22 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. Each function of the data display device 100 may be realized by the processing circuit 4002 on a function-by-function basis, or each function may be realized collectively by the processing circuit 4002.

[0092] It should be noted that the functions of the data display device 100 may be partially realized by dedicated hardware and partially realized by software or firmware. In this manner, the processing circuit can realize each of the above-mentioned functions by dedicated hardware, software, firmware, or a combination of these.

[0093] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be appropriately combined with other embodiments.

[0094] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but to one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create an embodiment not explicitly shown or described, for example. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0095] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; Including, Data display system. (Appendix 2) a three-dimensional data display unit that displays the image data and the three-dimensional data in a superimposed manner based on the photographing position and orientation, 2. A data display system as described in appended claim 1. (Appendix 3) the three-dimensional data display means displays the image data and the three-dimensional data in a superimposed manner to present a shooting range of the image data in the object represented by the three-dimensional data; 3. A data display system as described in appended claim 2. (Appendix 4) the three-dimensional data display means changes the display method from a state in which only the three-dimensional data is displayed to a state in which the image data and the three-dimensional data are superimposed and displayed; 3. A data display system as described in appended claim 2. (Appendix 5) the three-dimensional data display means arranges the image data as a three-dimensional object on the coordinate system, and simultaneously displays the arranged three-dimensional object and the three-dimensional data, thereby superimposing the image data and the three-dimensional data. 3. A data display system as described in appended claim 2. (Appendix 6) the three-dimensional data display means displays the image data and the three-dimensional data in a superimposed manner by coloring a portion of the three-dimensional data included in an imaging range of the image data based on the image data. 3. A data display system as described in appended claim 2. (Appendix 7) the three-dimensional data display means creates a member image depicting the three-dimensional data from a predetermined viewpoint, and transforms and superimposes the image data on the member image, thereby superimposing and displaying the image data and the three-dimensional data. 3. A data display system as described in appended claim 2. (Appendix 8) an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; Including, Data display device. (Appendix 9) The computer Estimating a photographing position and orientation of the image data in a coordinate system of the three-dimensional data; Detecting a location of a predetermined event from the image data; and displaying the position of the event on the drawing data in a superimposed manner based on the photographing position and orientation and the analysis result of the event analysis means. How data is displayed. (Appendix 10) Computer, an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; A program that functions as a

[0096] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 7 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 8 to 10 in the same dependent relationship as Supplementary Notes 2 to 7. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0097] 1. Object 1a, 1b, 1c Parts 2. Three-dimensional data 2a, 2b, 2c 3D points 21 Coordinate Systems 3. Drawing data 31, 32, 33, 34, 35 Projection 36 Vertex 31a, 32a, 33a, 34a, 35a, 36a Projected image 4. Image data 5. Events 10 Data Display System 11. Shooting position and orientation estimation means 12 Event Analysis Methods 13. Component Identification Means 14 Three-dimensional data display means 15 Drawing data display means 100 Data display device 101 Three-dimensional data storage unit 102 Image data input section 103 Shooting position and orientation estimation unit 104 Event Analysis Department 1041 Mask Image 1041a, 1041b, 1041c, 1041d pixels 1042 area 105 Component Identification Unit 106 Three-dimensional data display section 1061 area 1062, 1063 Media 1064, 1065 display 1066 Parts Images 1067 images 107 Drawing data storage unit 108 Drawing data display section 1081 area 1082 Drawing data 1083 Location information 1084 More information 1085 Image number 200 Data display device 2001 Window 210 Image data display unit 2101 area 2102, 2103 display

Claims

1. an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; Including, Data display system.

2. a three-dimensional data display unit that displays the image data and the three-dimensional data in a superimposed manner based on the photographing position and orientation, 2. The data display system of claim 1.

3. the three-dimensional data display means displays the image data and the three-dimensional data in a superimposed manner to present a shooting range of the image data in the object represented by the three-dimensional data; 3. The data display system according to claim 2.

4. the three-dimensional data display means changes the display method from a state in which only the three-dimensional data is displayed to a state in which the image data and the three-dimensional data are superimposed and displayed; 3. The data display system according to claim 2.

5. the three-dimensional data display means arranges the image data as a three-dimensional object on the coordinate system, and simultaneously displays the arranged three-dimensional object and the three-dimensional data, thereby superimposing the image data and the three-dimensional data.

3. The data display system according to claim 2.

6. the three-dimensional data display means displays the image data and the three-dimensional data in a superimposed manner by coloring a portion of the three-dimensional data included in an imaging range of the image data based on the image data.

3. The data display system according to claim 2.

7. the three-dimensional data display means creates a member image depicting the three-dimensional data from a predetermined viewpoint, and transforms and superimposes the image data on the member image, thereby superimposing and displaying the image data and the three-dimensional data.

3. The data display system according to claim 2.

8. an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; Including, Data display device.

9. The computer Estimating a photographing position and orientation of the image data in a coordinate system of the three-dimensional data; Detecting a location of a predetermined event from the image data; and displaying the position of the event on the drawing data in a superimposed manner based on the photographing position and orientation and the analysis result of the event analysis means. How data is displayed.

10. Computer, an image capture position and orientation estimation means for estimating an image capture position and orientation of image data in a coordinate system of three-dimensional data; an event analysis means for detecting a position of a predetermined event from the image data; a drawing data display means for displaying the position of the event superimposed on the drawing data based on the photographing position and attitude and the analysis result of the event analysis means; A program that functions as a

Images