Training data generation device, training data generation method, training data generation program, point cloud interpolation device, point cloud interpolation method, and point cloud interpolation program

Abstract

To provide a training data generation device that easily and quickly complements point cloud data of objects. [Solution] The training data generation device 1a includes a registration means for registering the maximum distance of the straight section of the model object, the number of connections of the straight section, the angle between the straight sections at the connections, and the diameter of the object; an end generation means for generating positional information of the ends of the straight section of the model object; a three-dimensional data generation means for generating three-dimensional data of the model object; a point cloud without defects for generating point cloud data of the model object; a missing point cloud generation means for generating missing parts in the point cloud where the point cloud is more missing than other parts, thereby generating missing point cloud data in the point cloud data of the model object; a correspondence means for grouping the point cloud without defects generated for the model object and the missing point cloud data generated for the same model object in which the point cloud without defects was created; and a storage means for storing the point cloud without defects and the missing point cloud data as training data for each group.

Description

Technical Field

[0001] The present invention relates to a teacher data generation device, a teacher data generation method, a teacher data generation program, a point cloud completion device, a point cloud completion method, and a point cloud completion program.

Background Art

[0002] In demolition work or renovation work of a plant facility, etc., in order to create a construction plan, it is necessary to grasp the lengths and positions of pipes, supports, and gantries that exist in large numbers as the amount of objects within the plant facility. Therefore, it is important to acquire these three-dimensional images. As for the generation of three-dimensional images, there is a method of acquiring point cloud data and generating it based on the acquired point cloud data.

[0003] When part of the point cloud data is discontinuous or the density of the point cloud is low, etc., if part of the point cloud data is missing, an accurate three-dimensional image cannot be generated. For this reason, as in Patent Document 1, a method of complementing the missing point cloud data has been proposed. The one in Patent Document 1 detects the missing part of the point cloud data in the region of an object and complements the point cloud data of the detected missing part. As a complementing method, for example, linear complementation is performed using the point cloud data in the vicinity of the missing part, or in the case of an object whose outer surface is cylindrical like a pipe, the point cloud data within the missing part is complemented based on the curvature of the point cloud data in the vicinity of the missing part and the base vectors of the cylinder.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Numerous pieces of equipment and piping exist within plant facilities. Therefore, when acquiring point cloud data of piping, many missing sections may be present due to being obscured by other objects. Furthermore, because piping has many bends and complex shapes, it can be difficult to determine whether a section is missing or not. Consequently, interpolation methods like those described in Patent Document 1 have the problem of requiring significant time and effort to interpolate point cloud data of long, sizable objects (such as piping) within plant facilities.

[0006] In view of the above problems, the present invention provides a training data generation device, a training data generation method, a training data generation program, a point cloud interpolation device, a point cloud interpolation method, and a point cloud interpolation program that can easily and quickly interpolate point cloud data of an object. [Means for solving the problem]

[0007] The present invention is a training data generation device that generates training data used to fill in missing parts of three-dimensional point cloud data representing three-dimensional information of a long, rectangular object having a single straight section or multiple straight sections connected at angles to each other, wherein the training data generation device is a model of an object in three-dimensional space on data generated by a user, and includes a registration means for registering the maximum distance of a straight section of a long, rectangular model object having a single straight section or multiple straight sections connected at angles to each other, the number of connections between the straight sections, the angle between the straight sections at the connections when the number of connections is one or more, and the diameter of the model object; an end generation means for generating positional information of the ends of each straight section of the model object based on the maximum distance and angle information registered in the registration means; and based on the diameter information registered in the registration means and the positional information of the ends of each straight section of the model object generated by the end generation means, The system comprises: a three-dimensional data generation means for generating three-dimensional data of a model object; a lossless point cloud generation means for generating lossless point cloud data, which is point cloud data without missing parts of the model object, from the three-dimensional data; a missing point cloud generation means for generating missing parts in the lossless point cloud data at arbitrary positions and ranges, such that the point cloud is missing more than other parts, thereby generating a missing point cloud data in the point cloud data of the model object; a correspondence means for associating the lossless point cloud data generated by the lossless point cloud generation means for a model object with the missing point cloud data generated by the lossless point cloud generation means for the same model object for which the lossless point cloud data was created, and grouping the lossless point cloud data and missing point cloud data of a single model object into the same group; and a storage means for storing the lossless point cloud data and missing point cloud data as training data for each group.

[0008] According to the training data generation device of the present invention, for long objects present in plant facilities such as pipes, a model object is first generated that serves as a model of the object in three-dimensional space in the data. For each model object, both missing point cloud data and missing point cloud data are generated and grouped. In other words, missing point cloud data of various forms (position, size, shape) of missing parts are prepared for a single model object, and the corresponding missing point cloud data is grouped, allowing the device to learn how to impute each form of the missing part. In this way, point cloud data for imputing missing parts can be acquired accurately and quickly.

[0009] In the above configuration, the positional information of the ends of each straight section of the model object is generated in a three-dimensional space in the data by moving a point starting from the origin, based on the movement trajectory of a point consisting of a straight section and a curved section. The end generation means includes a direction determination unit that arbitrarily determines the direction in which the point moves in six directions of mutually orthogonal first, second, and third axes, a distance determination unit that arbitrarily determines the distance to the next curved section, which is less than or equal to the maximum distance registered in the registration means, and the direction determination unit The system may also include a movement unit that moves a point based on the direction and the distance moved by the distance determination unit, an end unit that terminates the movement of the point, a start point extraction unit that extracts the position coordinates of the start point from the movement trajectory and sets the extracted start point as one end of the model object, an end point extraction unit that extracts the position coordinates of the end point from the movement trajectory and sets the extracted end point as the other end of the model object, and a curved section extraction unit that extracts the position coordinates of the curved section from the movement trajectory and sets the extracted curved section as the connection section of the model object.

[0010] The aforementioned elongated object is preferably installed within a plant facility.

[0011] The present invention provides a method for generating training data used to fill in missing parts of three-dimensional point cloud data representing the three-dimensional information of a long, rectangular object having a single straight section or multiple straight sections connected at angles to each other, wherein the method is a model of an object in three-dimensional space on data generated by the user, and registers the maximum distance of a straight section of a long, rectangular model object having a single straight section or multiple straight sections connected at angles to each other, the number of connections between the straight sections, the angle between the straight sections at the connections when the number of connections is one or more, and the diameter of the model object, generates positional information of each end of a straight section of the model object based on the registered maximum distance and angle information, and combines the registered diameter information with the generated end of the model object. Based on the positional information of the ends of each straight section, three-dimensional data of the model object is generated. From the three-dimensional data, missing point cloud data, which is point cloud data without missing parts of the model object, is generated. Missing parts are generated at arbitrary positions and ranges of the missing point cloud data, where the point cloud is missing more than other parts, thereby generating missing point cloud data in which the point cloud data of the model object has missing parts. The missing point cloud data generated for the model object and the missing point cloud data generated for the same model object from which the missing point cloud data was created are associated, and the missing point cloud data and missing point cloud data of a single model object are grouped together as the same group. For each group, the missing point cloud data and missing point cloud data are stored as training data.

[0012] The present invention is a training data generation program that generates training data used to fill in missing parts of three-dimensional point cloud data representing three-dimensional information of a long, rectangular object having a single straight section or multiple straight sections connected at angles to each other, and is a model of an object in three-dimensional space on data generated by the user, and includes a process to register the maximum distance of a straight section of a long, rectangular model object having a single straight section or multiple straight sections connected at angles to each other, the number of connections between the straight sections, the angle between the straight sections at the connections when the number of connections is one or more, and the diameter of the model object; a process to generate position information for each end of a straight section of the model object based on the registered maximum distance and angle information; and a process to combine the registered diameter information and the generated position information for each end of a straight section of the model object. Based on this, the system includes a process for generating three-dimensional data of a model object, a process for generating missing point cloud data from the three-dimensional data, which is point cloud data without missing parts of the model object, a process for generating missing parts in the point cloud data at arbitrary positions and ranges of the missing point cloud data, thereby generating missing point cloud data in the point cloud data of the model object that has missing parts, a process for associating the missing point cloud data generated for a model object with the missing point cloud data generated for the same model object from which the missing point cloud data was created, and grouping the missing point cloud data and missing point cloud data of a single model object into the same group, and a process for storing the missing point cloud data and missing point cloud data as training data for each group, and the system causes a computer to execute each of the above processes.

[0013] The point cloud interpolation device of the present invention interpolates missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at an angle to each other. The device comprises: a learning means that uses missing point cloud data and missing point cloud data stored in the storage means of the training data generation device as training data to machine learn the overall shape of the object and the interpolation method for missing parts of the missing point cloud data for each group; a group selection means that selects a group of point cloud data that approximates the object shape of the three-dimensional point cloud data of the object from among the groups learned by the learning means; a missing point cloud data selection means that selects missing point cloud data that approximates the position coordinates and point cloud density of the missing parts of the three-dimensional point cloud data from among the groups selected by the group selection means; and an interpolation means that interpolates missing parts of the three-dimensional point cloud data of the object based on the interpolation method learned by the learning means for the missing point cloud data selected by the missing point cloud data selection means.

[0014] According to the point cloud interpolation device of the present invention, the missing point cloud data and missing point cloud data stored in the storage means of the training data generation device are used as training data, and the overall shape of the object and the interpolation method for missing parts of the missing point cloud data are learned for each group. As a result, a group of point cloud data that approximates the object shape of the actual three-dimensional point cloud data is selected from the learned groups, and then missing point cloud data that approximates the position coordinates and point cloud density of the missing parts of the three-dimensional point cloud data is selected from that group. As a result, the missing parts of the three-dimensional point cloud data of the actual piping can be interpolated based on the interpolation method learned by the learning means.

[0015] In the above configuration, it is preferable that the learning means and the complementarizing means be AI.

[0016] The present invention provides a point cloud interpolation method for interpolating missing parts of three-dimensional point cloud data representing the three-dimensional information of a long object having a single straight section or multiple straight sections connected at angles to each other. The method involves using intact point cloud data and missing point cloud data stored in the storage means of the training data generation device as training data, learning the overall shape of the object and the interpolation method for missing parts of the missing point cloud data for each group, selecting a group of point cloud data that approximates the object shape of the object's three-dimensional point cloud data from the learned groups, selecting missing point cloud data that approximates the position coordinates and point cloud density of the missing parts of the three-dimensional point cloud data from the selected groups, and interpolating the missing parts of the object's three-dimensional point cloud data based on the interpolation method learned by the learning means for the selected missing point cloud data.

[0017] The point cloud interpolation program of the present invention interpolates missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at angles to each other. The program comprises: a process of machine learning the overall shape of the object and a method for interpolating missing parts of the missing point cloud data for each group, using the intact point cloud data and missing point cloud data stored in the storage means of the training data generation device as training data; a process of selecting a group of point cloud data that approximates the object shape of the object's three-dimensional point cloud data from the learned groups; a process of selecting missing point cloud data that approximates the position coordinates and point cloud density of the missing parts of the three-dimensional point cloud data from the selected groups; and a process of interpolating missing parts of the object's three-dimensional point cloud data for the selected missing point cloud data based on the interpolation method learned by the learning means, with each of the above processes being executed by a computer. [Effects of the Invention]

[0018] The training data generation device, training data generation method, and training data generation program of the present invention can generate training data that is effective for point cloud interpolation of elongated objects, and the point cloud interpolation device, point cloud interpolation method, and point cloud interpolation program that use the generated training data can interpolate the point cloud data of an object easily and quickly.

Brief Description of Drawings

[0019] [Figure 1] (a) is a configuration diagram of a teacher data generation device, and (b) is a configuration diagram of a point cloud completion system including a point cloud completion device. [Figure 2] It is a front view showing an example of a pipe. (a) is a pipe consisting of a single straight part, and (b) is a pipe in which a plurality of straight parts are connected via a connection part. [Figure 3] It is a configuration diagram showing an example of the internal hardware configuration of the teacher data generation device and the point cloud completion device shown in FIG. 1. [Figure 4] It is a block diagram showing the configuration of the teacher data generation device. [Figure 5] It is an explanatory diagram showing an example of moving a point in a virtual three-dimensional space. (a) shows the movement trajectory of the point, and (b) shows the points extracted from the movement trajectory. [Figure 6] It is a diagram showing the process of generating a model pipe by three-dimensional data generation means. (a) shows a state where the outline of the model pipe is drawn by connecting the endpoints of the straight parts, (b) shows a state where mesh processing is performed on the outline of (a), (c) shows a state where the three-dimensional shape of (b) is converted into a point cloud, and (d) shows a state where the point cloud data of (c) is normalized. [Figure 7] It is a diagram showing an example of data in which a missing part is generated in the point cloud data of FIG. 6(d). [Figure 8] It is a schematic diagram showing the form of the point cloud of the missing part. [Figure 9] It is an explanatory diagram showing the concept of generating a group of point cloud data for each model pipe. [Figure 10] It is a flowchart showing the teacher data generation method. [Figure 11] It is a block diagram showing the configuration of the point cloud completion device. [Figure 12] It is a diagram showing a part of the point cloud of the pipe and the state after completing the point cloud in the missing part. [Figure 13] It is a flowchart showing the point cloud completion method.

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0021] As shown in Figures 1(a) and 1(b), the training data generation device 1a and the point cloud completion device 1b are information processing devices of a computer, which will be described later. As shown in Figure 1(b), the point cloud completion system is configured with a computer including the point cloud completion device 1b and the point cloud acquisition means 2. The training data generation device 1a generates training data, which is point cloud data of an object, for use in completing missing parts of three-dimensional point cloud data representing the three-dimensional information of the object. The point cloud completion device 1b uses the training data generated by the training data generation device 1a to complete missing parts of the point cloud data obtained by the point cloud acquisition means 2. For this reason, the point cloud completion device 1b includes all the components of the training data generation device 1a.

[0022] In this embodiment, the "object" refers to a long, rectangular object consisting of a single straight section 3 as shown in Figure 2(a), or a plurality of straight sections 3 as shown in Figure 2(b), with connecting sections 4 where the ends of the straight sections 3 are connected at an angle. Specifically, this includes, for example, piping, supports, and frames within a plant facility. In this embodiment, piping 12 is described as an example of an object. As shown in Figure 2(b), the connecting section 4 is the part where the straight sections 3 are connected, and is the bent portion of the piping 12. The angle between the connecting sections (bending angle) at the connecting section 4 ranges from 0° to 180°, and in Figure 2(b), the bending angle at the connecting section 4 is 90°. Furthermore, the straight section 3 is a straight portion of the object that does not have an angle. In the case of Figure 2(a), both ends of the straight section 3 are the ends 13, 13 of the piping 12. In the case of Figure 2(b), the straight section 3 refers to the section from the end 13 of the piping 12 to the nearest connecting section 4, or between adjacent connecting sections. In this case, the ends of the straight section 3 refer to both ends 13 and the connecting section 4 of the pipe 12.

[0023] Point cloud acquisition means 2 is a three-dimensional laser scanner that performs 3D digital measurement and outputs point cloud data. A three-dimensional laser scanner is a measuring instrument that can acquire the three-dimensional coordinates of the surface shape by irradiating a laser onto the object to be measured. It measures non-contact at a speed of tens of thousands of points or more per second (depending on the model), and obtains high-density, planar point cloud data. Furthermore, even on the back side of the object to be measured or in a wide area that cannot be measured from a single location, it is possible to combine and coordinate multiple scan data by making the target a common point, enabling safe measurement without entering dangerous areas.

[0024] Three-dimensional coordinates are calculated from the distance to the object being measured and the irradiation angle, which are determined from the laser reflection time. Furthermore, point cloud data can be colored according to the color of the photograph taken by the built-in digital camera and the laser reflection intensity (the degree of laser reflection changes depending on the material and color of the object being measured). Thus, point cloud data is a collection of data from multiple points that represent the three-dimensional shape of an object, and includes measurement data indicating the position of each point (relative or absolute coordinate information, generally three-dimensional Cartesian coordinates (x, y, z)) and color information (R, G, B) for each point.

[0025] The point cloud data acquired by the point cloud acquisition means 2 is processed by a computer, specifically the point cloud interpolation device 1b.

[0026] Figure 3 shows an example of the hardware configuration of a computer including a training data generation device and a three-dimensional image generation device 1. The computer, which is the three-dimensional image generation device 1, includes a CPU (Central Processing Unit) 100, memory 101, storage 102 for storing programs, I / O (Input / Output) 103, and an interface (I / F) 104 for communication network connection. The CPU 100, memory 101, storage 102, I / O 103, and I / F 104 are connected to each other via a system bus 105.

[0027] The CPU 100 reads the program stored in the storage 102 into the memory 101 and controls each device connected to the system bus 105.

[0028] Memory 101 is a type of memory such as RAM (Random Access Memory) or ROM (Read Only Memory). The training data generation program or point cloud interpolation program of the present invention is loaded into this memory 101, thereby realizing the training data generation method or point cloud interpolation method of the present invention.

[0029] Storage 102 is a device for storing data for a long period of time, and is a storage device such as a hard disk, SSD (Solid State Drive), or memory card. Storage 102 may be installed inside the computer or may be connected to the computer by wire or wireless connection.

[0030] I / O 103 controls the input and output of data and control signals between the computer and other input / output means. Other input / output means include input means 106 such as a keyboard, touch panel, mouse, and microphone, output means 107 such as a display, printer, and speaker, and interfaces between these input / output devices and the computer.

[0031] I / F104 is a network interface for communication between a computer and external devices. I / F104 may be a network interface for connecting to a wired line or a network interface for connecting to a wireless line.

[0032] The teacher data generation method of this embodiment can be realized by executing the teacher data generation program of this embodiment using the teacher data generation device 1a of this embodiment described above.

[0033] Furthermore, by executing the point cloud interpolation program of this embodiment using the point cloud interpolation device 1b of this embodiment described above, the point cloud interpolation method of this embodiment can be realized.

[0034] Next, the various means within the training data generation device 1a that executes the aforementioned training data generation program will be explained with reference to Figure 4. The training data generation device 1a includes a registration means 5, an end generation means 6, a three-dimensional data generation means 7, a point cloud without missing data means 8, a point cloud with missing data means 9, a corresponding means 10, and a storage means 11.

[0035] The registration means 5 will now be explained. The registration means 5 registers the information necessary to generate the model object described later in Figure 6. The model object (model pipe 50 in this embodiment) is not an actual object, but something generated in data by the user. It is a model of a virtual object (pipe) created in three-dimensional space in the data by the training data generation device 1a to complement the missing parts of the point cloud data of the actually acquired object (pipe). The model pipe 50 is either a long pipe with a single straight section 52 or a long pipe with multiple straight sections 52 whose ends are connected at an angle. When the user specifies and inputs the maximum distance of the straight sections 52 of the model pipe 50, the number of connection points 51 between the straight sections, the angle between the straight sections at the connection points 51 when the number of connection points 51 is 1 or more, and the diameter of the model pipe 50, the registration means 5 registers and stores this information.

[0036] The maximum distance of a straight section 52 refers to the maximum distance that the user can conceive of in one straight section 52 of the model pipe 50 to be generated. Therefore, each straight section 52 of the generated model pipe 50 will be the maximum distance or less. The number of connection points 51 between straight sections is the number of connection points 51 when the model pipe 50 has multiple straight sections 52 (for example, as shown in Figure 6), and is the number of straight sections 52 minus 1. For example, if you want to generate a model pipe 50 with the shape shown in Figure 6, the number of straight sections 52 is 5, so the number of connection points 51 is entered as 4. If you want to generate a model pipe 50 formed from a single straight section 52, the number of connection points 51 is entered as 0. When the number of connection points 51 is 1 or more, the angle between the straight sections at the connection point 51 is the smaller of the angles made by the two straight sections in the plane containing the two connected straight sections (from 0° to 180°). For example, if you want to generate a model pipe with the shape shown in Figure 6, 90° is entered. If you want to generate a model pipe 50 formed from a single straight section 52, no connection section 51 is entered because there is no connection section 51. The diameter of the model pipe 50 is the outer diameter in a cross-section perpendicular to the axial direction of the model pipe 50. The maximum distance, number of connections, angle of the connections, and diameter can be arbitrarily set by the user based on the model pipe to be generated, and values ​​that are likely to exist can be entered.

[0037] The end generation means 6 generates position information for the ends of each straight section 52 of the model pipe 50 (i.e., the connection part 51, one end 51a of the model pipe 50, and the other end 51b of the model pipe) (see Figures 6(a) and (b)) based on information obtained from the registration means 5, including the maximum distance of the straight section 52 of the model pipe 50, the number of connection parts 51 between the straight sections, and the angle between the straight sections at the connection parts 51 when the number of connection parts 51 is one or more. As an example of the generation method, as shown in Figure 5, the end generation means 6 moves a point starting from the origin O in a three-dimensional space of data, and generates position information for the ends 51, 51a, and 51b based on the movement trajectory 45 of the point consisting of a straight section 46 and a curved section 47. The end generation means 6 includes a direction determination unit 20, a distance determination unit 21, a movement unit 22, an end unit 23, a start point extraction unit 24, an end point extraction unit 25, and a curved section extraction unit 26.

[0038] The direction determination unit 20 randomly determines the direction in which the point moves in the six positive and negative directions of the mutually orthogonal first axis (x-axis), second axis (y-axis), and third axis (z-axis), using angles registered in the registration means 5. The distance determination unit 21 randomly determines the distance to the next bend 47, which is less than or equal to the maximum distance registered in the registration means 5.

[0039] The movement unit 22 moves the point based on the direction determined by the direction determination unit 20 and the distance determined by the distance determination unit 21. The termination unit 23 terminates the movement of the point. Various settings can be made for the timing of the termination of the point's movement. For example, if the model piping 50 is formed from a single straight section 52, the point's movement can be terminated when it has moved a set distance from the origin O. If the model piping 50 is formed from a straight section 52 and a connection section 51, the point's movement can be terminated when the point overlaps with the movement trajectory 45.

[0040] The starting point extraction unit 24 extracts the position coordinates of the starting point of the movement trajectory 45 (origin O in this embodiment) and sets the extracted starting point as one end 51a of the model pipe 50. The ending point extraction unit 25 extracts the position coordinates of the ending point 48 of the movement trajectory 45 and sets the extracted ending point 48 as the other end 51b of the model pipe 50. The bend extraction unit 23 extracts the position coordinates of the bend 47 in the movement trajectory 45 and sets the extracted bend 47 as the connection 51 of the model pipe 50.

[0041] As shown in Figure 5(a), in a three-dimensional space on data having x, y, and z axes, a method for randomly generating bends 47 and endpoints 48 when generating a model pipe 50 having straight sections 52 and connecting sections 51 by moving a point starting from the origin O will be described. In this case, it is assumed that an angle of 90° and a maximum distance of 2 are registered in the registration means 5. First, the direction determination unit 20 randomly determines the direction of movement of the point from the origin O (in the +x direction from the origin O in the drawing), and the distance determination unit 21 randomly determines the distance of movement of the point to be less than or equal to the maximum distance registered in the registration means 5 (2 in the drawing), so the point moves 2 in the +x direction. This position becomes the first bend 47a of the movement trajectory 45.

[0042] Furthermore, when the direction determination unit 20 randomly determines the direction of movement of the point (in the diagram, from the first bend 47a in the +z direction) and the distance determination unit 21 randomly determines the distance the point moves within a range less than or equal to the maximum distance registered in the registration means 5 (2 in the diagram), the point moves 2 units in the +z direction. This position becomes the second bend 47b of the movement trajectory 45. Moreover, when the direction determination unit 20 randomly determines the direction of movement of the point (in the diagram, from the second bend 47b in the -x direction) and the distance determination unit 21 randomly determines the distance the point moves within a range less than or equal to the maximum distance registered in the registration means 5 (2 in the diagram), the point moves 2 units in the -x direction.

[0043] At position 48, the direction determination unit 20 randomly determines the direction of movement of the point (in the diagram, the -z direction from position 48), and the distance determination unit 21 randomly determines the distance of movement of the point to be less than or equal to the maximum distance registered in the registration means 5 (2 in the diagram). As a result, the point moves 2 units in the -z direction. In this case, the point coincides with the movement trajectory 45 (origin O), so the termination unit 22 terminates the movement of the point at the position immediately before the point coincides with the movement trajectory 45 (endpoint 48).

[0044] As shown in Figure 5(b), the starting point extraction unit 24 extracts the position coordinates of the starting point (origin O in this embodiment) from the movement trajectory 45, and sets the extracted starting point as one end 51a of the model pipe 50. The ending point extraction unit 25 extracts the position coordinates of the ending point 48 from the movement trajectory 45, and sets the extracted ending point 48 as the other end 51b of the model pipe 50. The bend extraction unit 26 extracts the position coordinates of the bend 47 from the movement trajectory 45, and sets the extracted bend 47 as the connection 51 of the model pipe 50. In this way, the ends (51, 51a, 51b) of all the straight sections 52 of the model pipe 50 are generated.

[0045] By performing the same method while varying the distance and direction of movement to the bend of the point from those described above, it is possible to generate the ends of straight sections of model pipes of various shapes. This makes it possible to generate model pipes of various shapes.

[0046] Even when generating a model pipe consisting of a single straight section 52 without a connecting section 51, the system generates positional information for both ends 51a and 51b of the straight section 52 of the model pipe (in this case, these coincide with both ends of the model pipe 50 itself) based on the movement trajectory of the point, which consists only of the straight section, by moving a point starting from the origin O in a three-dimensional space on the data having x, y, and z axes. That is, the direction determination unit 20 randomly determines the direction of movement of the point from the origin O, and the distance determination unit 21 randomly determines the distance the point moves, which is less than or equal to the maximum distance registered in the registration means 5. The movement unit 22 then moves the point in the determined direction and distance. When the point has moved by the determined distance, the termination unit 22 terminates the movement of the point.

[0047] The starting point extraction unit 24 extracts the position coordinates of the starting point (origin O in this embodiment) from the movement trajectory, and sets the extracted starting point as one end 51a of the model pipe 50. The ending point extraction unit 25 extracts the position coordinates of the ending point from the movement trajectory, and sets the extracted ending point as the other end 51b of the model pipe 50.

[0048] By performing the same method while varying the distance and direction of point movement from those described above, it is possible to generate the ends of straight sections of model pipes of various lengths. This makes it possible to generate model pipes with straight shapes of various lengths within the maximum distance.

[0049] The three-dimensional data generation means 7 generates three-dimensional shape data of the model object 50, which is mesh data. Specifically, the three-dimensional data generation means 7 first connects the ends with straight lines based on the diameter information registered by the registration means 5 and the position information of the ends of each straight section 52 of the model object 50 generated by the end generation means 6 (in the case of a model pipe with a connection, the connection 51 and both ends 51a and 51b of the model pipe 50; in the case of a model pipe without a connection, both ends 51a and 51b of the model pipe 50). That is, if the model pipe has a connection 51, as shown in Figure 6(a), one end 51a and the connection 51, the connection 51s themselves, and the connection 51 and the other end 51b are connected with straight lines to generate the contour of the model pipe 50 consisting of the straight section 52 and the connection 51. If the model pipe 50 does not have a connection 51, both ends 51a and 51b are connected with straight lines to generate the contour of the model pipe 50.

[0050] Next, as shown in Figure 6(b), the diameter is used as the length of the outer diameter in the cross-section of the pipe, and the contour of the model pipe 50 is meshed using a known and commonly used method to generate the three-dimensional shape of the hollow pipe. Meshing is the computational process of creating a polytope network that linearly approximates and represents geometric shapes and regions. Here, a polytope is a generalized figure; for example, polytopes from zero to three dimensions are points, line segments, polygons, and polyhedra, respectively. The mesh is given as the union of polytopes.

[0051] The point cloud generation means 8 generates point cloud data without missing values ​​for the model object 50. Specifically, the point cloud generation means 8 samples the created three-dimensional data (surface of the mesh data) and generates point cloud data of the model piping 50 as shown in Figure 6(c). Specifically, first, the number of point clouds to be sampled is determined. Then, a shape (triangle) is selected with a probability proportional to the area of ​​each shape (e.g., triangle) of the mesh, and within the selected shape (triangle), coordinates are randomly selected using varicentric coordinates with u and v such that 0≦u≦1 and 0≦v≦1 are selected, and these points are added to the point cloud. Point cloud data is generated in this way, and the point cloud data is normalized using a publicly known method. This makes it possible to generate point cloud data without missing values ​​for the model object 50 (point cloud data 53) as shown in Figure 6(d).

[0052] The missing point cloud generation means 9 generates missing points in the intact point cloud data 53 at arbitrary positions and ranges where the point cloud is missing compared to other parts, thereby generating missing point cloud data 54 with missing points 55 in the point cloud data of the model object 50, as shown in Figure 7. That is, point cloud data at a certain position and size (range) is deleted from the intact point cloud data 53 shown in Figure 6(d) as a virtual missing point 55. Here, examples of missing points 55 include cases where there are no points in the entire radial direction as shown in Figure 8(a), resulting in discontinuity in the model pipe 50; cases where there are no points in a part of the radial direction as shown in Figure 8(b), resulting in a narrower part of the model pipe 50; and cases where the point cloud density is lower than in other parts as shown in Figure 8(c). The missing points 55 may be generated by specifying the position, range, and shape of the missing points 55 by the user, or they may be generated by randomly deleting points at arbitrary positions and ranges. Furthermore, by generating data with different shapes for missing portions 55 from the intact point cloud data 53 of a single model pipe 50, it is possible to generate multiple missing point cloud data 54, each with different locations, ranges, and shapes of missing portions 55.

[0053] The corresponding means 10 associates the intact point cloud data 53 generated by the intact point cloud generation means 8 with the missing point cloud data 54 generated by the missing point cloud generation means 9 for the same model pipe 50 for which the intact point cloud data 53 was created, and groups the intact point cloud data 53 and the missing point cloud data 54 for a single model pipe 50 as the same group. That is, as shown in Figure 9, for a single model pipe 50A, intact point cloud data 53A and missing point cloud data 54A with a virtual missing portion 55 are generated. As described above, the missing point cloud creation means 9 generates multiple missing point cloud data 54A1, 54A2, etc., for a single model pipe 50A intact point cloud data 53A, each with different locations, ranges, and shapes of the missing portion 55. The corresponding means 10 associates all the point cloud data related to the model pipe 50A. "Matching" means linking all the missing point cloud data 53A and missing point cloud data 54A1, 54A2, etc., generated for the same model object (for example, model pipe 50A) into a single group 56A. In the same way, a group 56B is generated for model pipes of other shapes 50B, and although not shown in the illustration, groups 56C, 56D, etc. are generated for each of the other model pipes 50C, 50D, etc.

[0054] The storage means 11 stores the intact point cloud data 53 and the missing point cloud data 54 as training data for each group 56. In this way, training data used to fill in missing parts of the point cloud data can be created and stored.

[0055] Next, we will explain how to generate training data using this training data generation device, with reference to Figure 10. As an example of an object, we will describe the case of generating training data to be used to fill in missing points in the point cloud of a plant facility, using piping within a plant facility as an example.

[0056] First, the user inputs the maximum distance of the straight section 52 of the model pipe 50, the number of connection points 51 of the straight section 52, the angle between the straight sections at the connection points 51 when the number of connection points 51 is one or more, and the diameter of the model pipe 50 via the input means 106. The registration means 5 then registers and stores these values ​​(step S1). The maximum distance, number of connection points, angle, and diameter can be arbitrarily set by the user, and optimal values ​​that are assumed to be for an object within a plant facility can be input.

[0057] The end generation means 6 generates positional information for the ends 51, 51a, and 51b of each straight section of the model pipe 50 (ends 51a and 51b if the model pipe consists of a single straight section 52) based on the maximum distance of the straight section 52, the number of connection points 51, and the angle between the straight sections at the connection points 51 when the number of connection points 51 is 1 or more, as registered in the registration means 5 (step S2).

[0058] As shown in Figure 6(b), the three-dimensional data generation means 7 generates three-dimensional data of the model object, which is mesh data (step S3).

[0059] As shown in Figure 6(d), the lossless point cloud generation means 8 generates lossless point cloud data 53, which is point cloud data without any missing parts of the model pipe 50 (step S4). The lossy point cloud generation means 9 generates lossy parts 55 at arbitrary positions and ranges of the lossless point cloud data 53, where the point cloud is missing more than other parts, and generates lossy point cloud data 54 having lossy parts 55 in the point cloud data of the model pipe 50, as shown in Figure 7 (step S5).

[0060] The response means 10 associates the lossless point cloud data 53 generated by the lossless point cloud generation means 8 with the lossy point cloud data 54 generated by the lossy point cloud generation means 9 for the same model pipe 50 for which the lossless point cloud data 53 was created, and groups the lossless point cloud data 53 and lossy point cloud data 54 for one model pipe 50 as the same group 56 (step S6).

[0061] The storage means 11 stores the point cloud data 53 with no missing data and the point cloud data 54 generated from a single model pipe 50 as training data for each group 56 (step S7).

[0062] If there is no training data to be used for machine learning for the other model pipes 50, the process ends. If further training data is needed for machine learning for the other model pipes 50, steps S1 to S7 are repeated (step S8). In this way, training data used to fill in missing parts of the point cloud data can be created and stored.

[0063] The training data generation device and training data generation method of this embodiment first generate a model pipe 50, which serves as a model of the object in three-dimensional space in the data, for long-length objects present in plant facilities such as pipes. For each model pipe 50, complete point cloud data 53 and missing point cloud data 54 are generated and grouped. In other words, missing point cloud data 54 of missing parts 55 of various forms (position, size, shape) are prepared for each model pipe 50, and the corresponding complete point cloud data 53 is grouped, so that a method for imputing missing parts 55 can be learned for each form of missing part 55. In this way, point cloud data for imputing missing parts 55 can be obtained accurately and quickly. The specific method for imputing missing parts of the point cloud based on this training data will be described later.

[0064] Next, the various means within the point cloud interpolation device 1b that executes the point cloud interpolation program will be explained using Figure 11. The point cloud interpolation device 1b consists of a learning unit 30 that performs machine learning using intact point cloud data 53 and missing point cloud data 54 as training data for each group 56 of model pipes 50, and an interpolation unit 31 that interpolates missing parts of the three-dimensional point cloud data representing the three-dimensional information of the actual pipes based on machine learning.

[0065] The learning unit 30 comprises all the components of the training data generation device 1a shown in Figure 4, and a learning means 32. The learning means 32 performs machine learning based on the training data generated by the training data generation device 1a and stored in the storage means 11. Specifically, the overall shape of the piping is learned for each group 56 from the intact point cloud data 53 and the missing point cloud data 54. For each missing point cloud data 54, the learning means learns how to interpolate each missing part 55 to obtain intact point cloud data, based on information such as the position coordinates, state of absence, and point cloud density of the missing part 55, and the corresponding intact point cloud data 53.

[0066] For example, as shown in Figure 8(a), if the missing portion 55 is in the form of a break in the piping, the point cloud is interpolated so that the density of that portion becomes the same as the other portions, connecting the broken portion. Also, as shown in Figure 8(b), if the missing portion 55 is in the form of a narrowing of the piping, the point cloud is interpolated so that the density of that portion becomes the same as the other portions, making the other portions the same width. Furthermore, as shown in Figure 8(c), if the point cloud density of the missing portion 55 is lower than that of the other portions, the point cloud is interpolated so that the density of the point cloud in that portion becomes the same as that of the other portions. In this way, a interpolation method is learned through machine learning so that the missing portion 55 has the same point cloud density as the other portions.

[0067] The interpolation unit 31 includes a group selection means 33, a missing point cloud data selection means 34, and an interpolation means 35.

[0068] The group selection means 33 selects from the groups learned by the learning means 32 a group 56 of point cloud data whose object shape approximates the overall shape of the pipe in the three-dimensional point cloud data of the actual pipe acquired by the point cloud acquisition means 2.

[0069] The missing point cloud data selection means 34 selects missing point cloud data from the group 56 selected by the group selection means 33 that approximates the position coordinates and point cloud density of the missing portion of the actual three-dimensional point cloud data. Since point cloud data has position information for each point, if there is a missing portion in the actual three-dimensional point cloud data, the location of that missing portion can be determined. Therefore, among the missing point cloud data 54 in the selected group 56, if there is a missing point cloud data 54 that has a missing portion at the same position coordinates as the missing portion of the actual three-dimensional point cloud data, that missing point cloud data 54 is selected.

[0070] The interpolation means 35 interpolates the missing parts of the actual three-dimensional point cloud data of the piping based on the interpolation method learned by the learning means 32 for the missing point cloud data 54 selected by the missing point cloud data selection means 34. As a result, as shown in Figure 12, the interpolated portion 57 (the portion that was missing before interpolation) of the point cloud data of the piping has the same point cloud density as the other point cloud portions. The learning means 32 and the interpolation means 35 can use publicly available AI for interpolating missing parts of images.

[0071] Next, we will explain, using Figure 13, how to use this point cloud interpolation device to interpolate missing parts of three-dimensional point cloud data representing the three-dimensional information of an object (pipe).

[0072] First, using the intact point cloud data 53 and missing point cloud data 54 stored in the storage means 11 of the training data generation device 1a as training data, the learning means 32 learns the overall shape of the piping for each group, and learns how to interpolate each missing part 55 to obtain intact point cloud data from information such as the position coordinates, state of absence, and point cloud density of the missing part 55 and the corresponding intact point cloud data 53 (step S11).

[0073] When actual three-dimensional point cloud data of the piping is input, the group selection means 33 selects a group 56 of point cloud data that approximates the object shape of the three-dimensional point cloud data from among the groups learned by the learning means 32 (step S12). In addition, the missing point cloud data selection means 34 selects missing point cloud data 54 from the group 56 selected by the group selection means 33 that approximates the position coordinates and point cloud density of the missing parts of the three-dimensional point cloud data (step S13).

[0074] The interpolation means 35 interpolates the missing parts of the actual three-dimensional point cloud data of the piping based on the interpolation method learned by the learning means 32 for the missing point cloud data 54 selected by the missing point cloud data selection means 34 (step S14).

[0075] As shown in Figure 12, if the missing parts of the actual piping's 3D point cloud data can be filled in (step S15), the process ends. If the missing parts cannot be filled in (step S15), the process returns to step S11 to machine-learn the interpolation method with other training data, or returns to step S12 to select a different group from those that have already been machine-learned and repeats up to step S15. In this way, the missing parts of the actual piping's 3D point cloud data can be filled in.

[0076] According to the point cloud interpolation device and point cloud interpolation method of this embodiment, the intact point cloud data 53 and missing point cloud data 54 stored in the storage means 11 of the training data generation device 1a are used as training data, and the overall shape of the piping and the interpolation method for missing parts 55 of the missing point cloud data 54 are learned for each group 56. As a result, a group 56 of point cloud data that approximates the object shape of the actual three-dimensional point cloud data is selected from the learned groups, and then missing point cloud data 54 that approximates the position coordinates of the missing parts of the three-dimensional point cloud data is selected from that group 56. As a result, the missing parts of the three-dimensional point cloud data of the actual piping can be interpolated based on the interpolation method learned by the learning means.

[0077] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and can be modified in various ways. For example, the object is not limited to piping, but can be other elongated objects within a plant facility, such as electrical conduits, cable trays, etc. Also, it is not limited to a plant facility, but may be an object within another building. The shape of the model object can be varied, and the number of straight sections, the maximum distance of the straight sections, the angle between the straight sections, and the diameter of the straight sections can be arbitrarily set by the user based on the shape of the model object they wish to generate. The method for randomly generating the ends of the straight sections is not limited to that of the embodiments, and may be generated, for example, by specifying coordinates by the user.

[0078] Model objects can be generated using, for example, Open3D, but publicly available software may also be used. While AI is preferable for learning and completion, publicly available software may be used instead. [Explanation of Symbols]

[0079] 1a Training data generation device 1b Point cloud completion device 3. Straight section 4 Connection part 5. Registration Methods 6 End generation means 7. Three-dimensional data generation means 8. Means for generating a point cloud without defects 9 Missing point cloud generation means 10. Countermeasures 11 Memory means 12 Model Objects 13 End of pipe 20 Direction determination unit 21 Distance determination unit 22 Mobile section 23 Ending section 24 Starting point extraction unit 25. End point extraction section 26. Bent section extraction section 32 Learning Methods 33 Group Selection Methods 34. Means for selecting data with missing points 35. Complementary measures 45 Movement trajectory 46 Straight section 47. Curved section 48 Final stop 50 Model Piping 51 Connection part 51a, 51b Ends of model piping 52 Straight section 53. Point cloud data without missing data 54. Missing point cloud data 55 Defective part

Claims

1. A training data generation device that generates training data used to fill in missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at an angle to each other, A user-generated model of a three-dimensional object in data, comprising a registration means for which the maximum distance of a straight section, the number of connections between straight sections, the angle between straight sections at connections where there is one or more connections, and the diameter of the model object are registered, for a long model object having a single straight section or multiple straight sections connected at angles to each other. An end generation means generates positional information for the ends of each straight section of a model object based on the maximum distance and angle information registered in the registration means, A three-dimensional data generation means generates three-dimensional data of a model object based on the diameter information registered by the registration means and the position information of the ends of each straight section of the model object generated by the end generation means. A lossless point cloud generation means generates lossless point cloud data, which is point cloud data without missing parts of the model object, from the aforementioned three-dimensional data. A missing point cloud generation means generates missing points in the point cloud data at arbitrary locations and ranges, where the point cloud is missing more than other parts, thereby generating missing point cloud data for a model object that has missing points. Correspondence means for associating the lossless point cloud data generated by the lossless point cloud generation means for a model object with the lossy point cloud data generated by the lossy point cloud generation means for the same model object for which the lossless point cloud data was created, and for grouping the lossless point cloud data and lossy point cloud data of a single model object as the same group, A training data generation device characterized by comprising storage means for storing both missing point cloud data and missing point cloud data as training data for each group.

2. The positional information of the ends of each straight section of the model object is generated in the three-dimensional space of the data by moving a point starting from the origin, based on the movement trajectory of the point consisting of the straight section and the curved section. The end generation means is A direction determination unit that arbitrarily determines the direction in which a point moves in six directions, positive and negative, of mutually orthogonal first, second, and third axes, A distance determination unit that arbitrarily determines the distance to the next bend, provided that the distance is less than or equal to the maximum distance registered in the registration means, A moving unit that moves a point based on the direction determined by the direction determination unit and the distance moved by the distance determination unit, An ending section that terminates the movement of the point, A starting point extraction unit extracts the position coordinates of the starting point from the movement trajectory and sets the extracted starting point as one end of the model object, An endpoint extraction unit extracts the position coordinates of the endpoint of the movement trajectory and sets the extracted endpoint as the other end of the model object, The method for generating training data according to claim 1, further comprising a curve extraction unit that extracts the position coordinates of curved parts from the movement trajectory and uses the extracted curved parts as connection points of the model object.

3. The method for generating training data according to claim 1, characterized in that the elongated object is installed within a plant facility.

4. A method for generating training data to generate training data used to fill in missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at an angle to each other, This is a model of a three-dimensional object in data generated by the user, and it is a long model object having a single straight section or multiple straight sections connected at angles to each other. The maximum distance of the straight sections, the number of connections between the straight sections, the angle between the straight sections at the connections when there is one or more connections, and the diameter of the model object are registered. Based on the registered maximum distance and angle information, position information for the ends of each straight section of the model object is generated. Based on the registered diameter information and the positional information of the ends of each straight section of the generated model object, three-dimensional data of the model object is generated. From the aforementioned three-dimensional data, a lossless point cloud data is generated, which is point cloud data without missing parts of the model object. By generating missing points in the point cloud data at arbitrary locations and ranges, the system generates missing point cloud data for the model object, where the point cloud is missing more than other parts of the data. By associating the lossless point cloud data generated for a model object with the lossy point cloud data generated for the same model object from which the lossless point cloud data was created, the lossless point cloud data and lossy point cloud data for a single model object are grouped together as the same group. A method for generating training data, characterized by storing both missing point cloud data and missing point cloud data as training data for each group.

5. A training data generation program that generates training data used to fill in missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at angles to each other, A user-generated model of a three-dimensional object in data, comprising a long model object having a single straight section or multiple straight sections connected at angles, the maximum distance of the straight sections, the number of connections between the straight sections, the angle between the straight sections at the connections when the number of connections is one or more, and the diameter of the model object, Based on the registered maximum distance and angle information, a process is performed to generate positional information for the ends of each straight section of the model object. A process to generate three-dimensional data of a model object based on registered diameter information and the position information of the ends of each straight section of the generated model object, The process of generating lossless point cloud data, which is point cloud data without missing parts of the model object, from the aforementioned three-dimensional data, A process to generate missing points in the point cloud data of a model object, where the point cloud is missing in any location and range compared to other parts of the data, thereby generating missing points in the point cloud data of the model object. A process to associate the missing point cloud data generated for a model object with the missing point cloud data generated for the same model object from which the missing point cloud data was created, and to group the missing point cloud data and missing point cloud data of a single model object into the same group. For each group, the system includes a process for storing both missing point cloud data and missing point cloud data as training data. A training data generation program characterized by causing a computer to perform each of the aforementioned processes.

6. A point cloud interpolation device for interpolating missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at an angle to each other, A learning means that uses the data without missing points and the data with missing points stored in the storage means of the training data generation device of claim 1 as training data to machine learn the overall shape of an object and a method for imputing missing parts of the data with missing points for each group, A group selection means selects a group of point cloud data that approximates the object shape of the object's three-dimensional point cloud data from among the groups learned by the aforementioned learning means, A missing point cloud data selection means selects missing point cloud data from the group selected by the group selection means that approximates the position coordinates and point cloud density of the missing portion of the three-dimensional point cloud data. A point cloud interpolation device characterized by comprising: interpolation means for interpolating missing parts of three-dimensional point cloud data of an object based on an interpolation method learned by the learning means, with respect to the missing point cloud data selected by the missing point cloud data selection means.

7. The point cloud interpolation device according to claim 6, characterized in that the learning means and interpolation means are AI.

8. A point cloud interpolation method for filling in missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at an angle to each other, Using the data without missing points and the data with missing points stored in the storage means of the training data generation device of claim 1 as training data, the overall shape of the object and the method for imputing missing parts of the data with missing points are machine-learned for each group. From the trained groups, select the group of point cloud data that approximates the object's shape in three dimensions. From the selected group, select the missing point cloud data that approximates the position coordinates and point cloud density of the missing portion of the three-dimensional point cloud data. A point cloud interpolation method characterized by interpolating the missing parts of the three-dimensional point cloud data of an object based on an interpolation method learned by the learning means for selected missing point cloud data.

9. A point cloud interpolation program for filling in missing parts of three-dimensional point cloud data representing three-dimensional information of a long object having a single straight section or multiple straight sections connected at angles to each other, The process involves using the data without missing points and the data with missing points stored in the storage means of the training data generation device of claim 1 as training data to machine learn the overall shape of an object and a method for imputing missing parts of the data with missing points for each group, The process involves selecting a group of point cloud data that approximates the object's three-dimensional point cloud data shape from among the learned groups, and A process of selecting missing point cloud data from the selected group that approximates the position coordinates and point cloud density of the missing portion of the three-dimensional point cloud data, The system includes a process for filling in missing parts of the object's three-dimensional point cloud data based on the interpolation method learned by the learning means, for selected missing point cloud data. A point cloud interpolation program characterized by causing a computer to perform each of the aforementioned processes.

Images