Three-dimensional image generating device, three-dimensional image generating method, and three-dimensional image generating program

Abstract

To provide a three-dimensional image generating device, method, and program for simply and precisely generating a three-dimensional image.SOLUTION: A three-dimensional image generating device includes: point group image generating means for generating point group images of a designated object viewed from respective view points from point group data which is three-dimensional information; object identifying means for performing machine learning using, as teacher data, shapes of the designated object viewed from the view points from teacher images including the designated object and identifying the type of an object on the basis of the shapes of the designated object learned by the machine learning from the generated point group images; object computation means for computing the coordinates and dimensions of the designated object from the point group data of the designated object; and three-dimensional image generating means for selecting a previously recorded model image of the designated object matching the type of the identified designated object, adjusting the dimensions of the selected model image based on the computed dimensions information of the designated object, and generating a three-dimensional image in which the model image having the adjusted dimensions is put in a three-dimensional coordinate position based on the coordinate information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional image generating device, a three-dimensional image generating method, and a three-dimensional image generating program. [Background technology]

[0002] In the dismantling work of plant facilities, it is necessary to understand the type, size, and location of objects within the facility in order to create a dismantling plan. Therefore, it is important to acquire 3D images of the site. Patent Document 1 discloses a method for generating 3D images. The method in Patent Document 1 registers model shape information, compares a labeled point cloud with each of multiple sets of 3D model shape information, and generates 3D information by replacing a specific area with the 3D model shape using the 3D model shape that is closest in shape. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-103167 Summary of the Invention [Problem to be solved by the invention]

[0004] In the method of Patent Document 1, if an optimal model shape is not found among the pre-prepared model shapes for the acquired point cloud data (if no model shape with matching dimensions or shape exists), the dimensions and shape of the model shape must be readjusted. It is difficult to register a model whose dimensions and shape perfectly match those of the object in the point cloud data, and the model must be adjusted until they match, which is time-consuming and labor-intensive. Furthermore, to achieve a perfect match, multiple models with different dimensions and shapes must be registered, making the registration process cumbersome. Furthermore, many objects appear differently depending on the viewpoint, so models viewed from every angle of the object must be registered, which requires even more models and complicates the registration process. In particular, individual objects within plant facilities, such as valves and pumps, have complex shapes and appear differently from different viewing angles. Therefore, models viewed from every viewpoint must be registered for each object. This makes the model registration process particularly difficult and makes it difficult to generate a 3D image.

[0005] In view of the above problems, the present invention provides a three-dimensional image generating device, a three-dimensional image generating method, and a three-dimensional image generating program that can generate three-dimensional images simply and accurately. [Means for solving the problem]

[0006] The three-dimensional image generating device of the present invention is a three-dimensional image generating device that generates a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, and includes: a point cloud image generating means that generates point cloud images at each of a plurality of different viewpoints of the designated object from point cloud data, which is three-dimensional information; an object identification means that performs machine learning on the shapes of the designated object as viewed from the plurality of viewpoints from a teacher image including the designated object, using these shapes as teacher data, and identifies the type of object based on the machine-learned shape of the designated object from the point cloud image generated by the point cloud image generating means; an object calculation means that calculates the coordinates and dimensions of the designated object from the point cloud data of the designated object; and a three-dimensional image generating means that registers model images of the designated object in advance, selects a model image that matches the type of the designated object specified by the object identification means, adjusts the dimensions of the selected model image based on dimensional information of the designated object calculated by the object position calculation means, and places the dimension-adjusted model image at a three-dimensional coordinate position based on the coordinate information, thereby generating a three-dimensional image.

[0007] According to the 3D image generating device of the present invention, the type of a designated object is identified from point cloud images obtained by viewing the designated object from multiple different viewpoints. This allows the designated object to be recognized from multiple viewpoints, and the type of the designated object can be determined even if the object has a complex shape. Furthermore, the object calculation means acquires coordinate information and dimensional information for the designated object separately from identifying the object's type. This allows the coordinate information and dimensional information to be added to the information on the type of the designated object, enabling the generation of a 3D image. This eliminates the need for model readjustment and the registration of numerous models with different dimensions and shapes, which were previously required.

[0008] In the above configuration, the point cloud image generating means may include a three-dimensional point cloud generating unit that generates a three-dimensional point cloud by three-dimensionally transforming the point cloud of a specified object from point cloud data, a virtual camera setting unit that sets a virtual camera to be placed at a virtual viewpoint position with respect to the three-dimensional point cloud, a rotation unit that rotates the virtual camera relatively around the three-dimensional point cloud, and a virtual viewpoint image creating unit that generates virtual viewpoint images for each of a plurality of virtual viewpoints when the virtual camera views the three-dimensional point cloud from different virtual viewpoints. That is, the point cloud image can be the virtual viewpoint image. In this case, it is preferable that the virtual viewpoint image is a two-dimensional RGB image.

[0009] Another three-dimensional image generating device of the present invention is a three-dimensional image generating device that generates a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, and includes: an object identification means that performs machine learning of the shapes of the designated object as viewed from a plurality of viewpoints from a teacher image including the designated object, as teacher data, and identifies the type of object based on the machine-learned shape of the designated object from point cloud data for each of the designated object viewed from a plurality of different viewpoints; an object calculation means that calculates the coordinates and dimensions of the designated object from the point cloud data of the designated object; and a three-dimensional image generating means that registers model images of the designated object in advance, selects a model image that matches the type of the designated object identified by the object identification means, adjusts the dimensions of the selected model image based on dimensional information of the designated object calculated by the object position calculation means, and places the dimension-adjusted model image at a three-dimensional coordinate position based on the coordinate information to generate a three-dimensional image.

[0010] According to another three-dimensional image generating device of the present invention, a three-dimensional image can be generated from acquired point cloud data without generating a point cloud image as a two-dimensional image.

[0011] In the above configuration, the object calculation means may include a point cloud division unit that divides the point cloud data for each designated object identified by the object identification means when the point cloud data includes a plurality of designated objects.

[0012] In this case, a point cloud enlargement unit may be provided that, when the point cloud data is displayed, enlarges and displays the size of each of the points that make up the point cloud data.

[0013] In the above configuration, the dimension may be the length from the center coordinate of the designated object to the coordinate position farthest from the center coordinate of the designated object.

[0014] The specified object may be equipment within a plant facility. While many of the equipment within a plant facility have complex shapes and dangerous locations, the need for generating 3D images is particularly high during demolition work, etc. By using the 3D image generating device of the present invention, 3D images can be generated accurately and safely even in such cases.

[0015] The three-dimensional image generation method of the present invention is a three-dimensional image generation method that generates a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, and generates point cloud images at each of a plurality of different viewpoints when the designated object is viewed from point cloud data, which is the three-dimensional information; machine learning is performed on each of the shapes of the designated object viewed from the plurality of viewpoints from a teacher image including the designated object, using the shapes of the designated object as teacher data; identifying the type of object from the point cloud image based on the machine-learned shape of the designated object; calculating the coordinates and dimensions of the designated object from the point cloud data of the designated object; selecting a model image that matches the identified type of the designated object from pre-registered model images of the designated object; adjusting the dimensions of the selected model image based on the calculated dimensional information of the designated object; and placing the dimension-adjusted model image at a three-dimensional coordinate position based on the coordinate information to generate a three-dimensional image.

[0016] The three-dimensional image generation program of the present invention is a three-dimensional image generation program that generates a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, and includes the following processes: generating point cloud images at each of a plurality of different viewpoints when the designated object is viewed from point cloud data, which is three-dimensional information; machine learning the shapes of the designated object as viewed from the plurality of viewpoints from a teacher image including the designated object, using the shapes of the designated object as teacher data, and identifying the type of object from the point cloud image based on the machine-learned shapes of the designated object; calculating the coordinates and dimensions of the designated object from the point cloud data of the designated object; and selecting a model image that matches the identified type of the designated object from a pre-registered model image of the designated object, adjusting the dimensions of the selected model image based on the calculated dimensional information of the designated object, and placing the dimension-adjusted model image at a three-dimensional coordinate position based on the coordinate information, thereby generating a three-dimensional image, and causes a computer to execute each of the above processes. [Effects of the Invention]

[0017] The three-dimensional image generating device, the three-dimensional image generating method, and the three-dimensional image generating program of the present invention can generate three-dimensional images simply and accurately. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a configuration diagram of a three-dimensional image generation system including a three-dimensional image generation device. [Figure 2] FIG. 2 is a configuration diagram showing an example of the internal hardware configuration of the three-dimensional image generating device shown in FIG. 1. [Figure 3] 1 is a block diagram showing the configuration of a 3D image generation device according to a first embodiment. [Figure 4] FIG. 10 is a diagram illustrating the relationship between a three-dimensional point cloud of a designated object and a virtual camera in a virtual space. [Figure 5] 1A and 1B are diagrams illustrating point cloud data and virtual viewpoint images at different virtual viewpoints. [Figure 6]10A and 10B are diagrams showing an example of displayed point cloud data, where (a) shows a state in which a specified range has been specified in the point cloud data, and (b) shows a state in which point cloud data outside the specified range has been deleted. [Figure 7] 10 is an explanatory diagram showing the concept of dividing point cloud data for each designated object by a point cloud dividing means. FIG. [Figure 8] FIG. 2 is a flowchart illustrating a three-dimensional image generating method according to the first embodiment. [Figure 9] FIG. 10 is an explanatory diagram showing a method for identifying an object. [Figure 10] FIG. 10 is a block diagram showing the configuration of a 3D image generation device according to a second embodiment. [Figure 11] FIG. 10 is a block diagram showing the configuration of a 3D image generation device according to a third embodiment.

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0020] As shown in FIG. 1, a three-dimensional image generating device 1 of the first embodiment is an information processing device of a computer, which will be described later. A three-dimensional image generating system is configured by a computer including the three-dimensional image generating device 1 and point cloud acquisition means 2. The three-dimensional image generating device 1 generates a three-dimensional image of a specified object using point cloud data obtained by the point cloud acquisition means 2. Here, the "specified object" refers to an object type designated in advance by a user, and refers to the type of object to be displayed in the three-dimensional image. Furthermore, the "type of object" refers to the type of object (e.g., valve, pump, etc.). In this embodiment, a pump and a valve, which are equipment within a plant facility, will be described as examples of the specified object.

[0021] The point cloud acquisition means 2 is a three-dimensional laser scanner that performs 3D digital measurements and outputs point cloud data. A three-dimensional laser scanner is a measuring device that can acquire the three-dimensional coordinates of the surface shape of a measurement target by radiating a laser beam onto the target. It performs non-contact measurements at a speed of tens of thousands of points per second or more (depending on the model), obtaining high-density, planar point cloud data. Furthermore, even for the back side of a measurement target or a wide area that cannot be measured from a single location, it is possible to combine multiple scan data by using a target as a common point and assign coordinates, enabling safe measurements without entering dangerous areas.

[0022] The three-dimensional coordinates are calculated from the distance to the measurement target, which is determined from the laser reflection time, and the irradiation angle. Point cloud data can also be colored according to the color of the photograph taken by the built-in digital camera or the laser reflection intensity (the degree of laser reflection varies depending on the material and color of the measurement target). Point cloud data is thus a collection of data for multiple points that indicate the three-dimensional shape of an object, and includes measurement data (relative or absolute coordinate information, typically three-dimensional Cartesian coordinates (x, y, z)) that indicates the position of each point, and color information (R, G, B) for each point.

[0023] The point cloud data acquired by the point cloud acquisition means 2 is processed by a computer, specifically, a three-dimensional image generation device 1.

[0024] 2 is a diagram showing an example of the hardware configuration of a computer including the 3D image generation device 1. The computer that is the 3D image generation device 1 includes a CPU (Central Processing Unit) 100, memory 101, storage 102 that stores programs, I / O (Input / Output) 103, and an interface (I / F) 104 for connecting to a communication network. The CPU 100, memory 101, storage 102, I / O 103, and I / F 104 are connected to one another via a system bus 105.

[0025] The CPU 100 reads a program stored in the storage 102 into the memory 101 and controls each device connected to the system bus 105 .

[0026] The memory 101 is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The three-dimensional image generation program of the present invention is loaded into this memory 101, thereby realizing the three-dimensional image generation method of the present invention.

[0027] The storage 102 is a device that stores data for a long period of time, and is a storage device such as a hard disk, a solid state drive (SSD), a memory card, etc. The storage 102 may be provided inside the computer, or may be connected to the computer via a wired or wireless connection.

[0028] The I / O 103 controls the input and output of data and control signals between the computer and other input and output means. The other input and output means include input means 106 such as a keyboard, touch panel, mouse, and microphone, output means 107 such as a display, printer, and speaker, and interfaces between these input and output devices and the computer.

[0029] The I / F 104 is a network connection interface for communication between the computer and an external device. The I / F 104 may be a network interface for connection to a wired line or a network interface for connection to a wireless line.

[0030] The 3D image generating method of the present embodiment can be realized by executing the 3D image generating program of the present embodiment by the 3D image generating device 1 of the present embodiment. The 3D image generating program of the present embodiment is a 3D image generating program for generating a 3D image from a 3D point cloud of a space having a designated object of a predetermined type, and includes the following processes: machine learning a shape of the designated object as viewed from a plurality of viewpoints from a teacher image including the designated object as teacher data; identifying the type of the object based on the machine-learned shape of the designated object from a plurality of point cloud data acquired for the designated object from the plurality of viewpoints; calculating coordinates and dimensions of the designated object from the acquired point cloud data of the designated object; and selecting a model image that matches the identified type of the designated object from pre-registered model images of the designated object, adjusting the dimensions of the selected model image based on the calculated dimensional information of the designated object, and arranging the dimension-adjusted model image at a three-dimensional coordinate position based on the coordinate information to generate a 3D image. The program causes a computer to execute each of the above processes.

[0031] Next, the various means included in the 3D image generation device 1 of the first embodiment that executes the above-mentioned 3D image generation program will be described with reference to Fig. 3. The 3D image generation device 1 includes a point cloud image generation means 3, an object identification means 11, an object calculation means 22, and a 3D image generation means 31.

[0032] The point cloud image generating means 3 will now be described. When a designated object 28 (see FIG. 5) in point cloud data 27 (see FIG. 5), which is three-dimensional information, is viewed from a plurality of different viewpoints, the point cloud image generating means 3 generates point cloud images (virtual viewpoint images 29A to 29E) (see FIG. 5) from each viewpoint, and Open3D software or the like can be used for this. The point cloud image generating means 3 includes a point cloud input unit 4, a three-dimensional point cloud generating unit 5, a virtual camera setting unit 6, a rotation unit 8, and a virtual viewpoint image creating unit 9.

[0033] The point cloud input unit 4 recognizes the point cloud of a specified object from the acquired point cloud data. The three-dimensional point cloud generation unit 5 generates a three-dimensional point cloud 36 by three-dimensionally converting the point cloud of the specified object from the point cloud data, as shown in Fig. 4. The virtual camera setting unit 6 sets a virtual camera 37 to be placed at a virtual viewpoint position with respect to the three-dimensional point cloud 36. Here, the virtual camera is a virtual camera that is different from an actually installed camera, and virtually captures images from a virtual viewpoint set in a virtual space.

[0034] The rotation unit 8 rotates the virtual camera 37 around the three-dimensional point cloud relative to the three-dimensional point cloud as indicated by the arrow in FIG.

[0035] As shown in FIG. 5, the virtual viewpoint image creation unit 9 generates virtual viewpoint images 29A-29E at each of a plurality of different virtual viewpoints from which a virtual camera 37 views a three-dimensional point cloud 36. Specifically, the virtual viewpoint image creation unit 9 generates two-dimensional RGB images by rendering point cloud data into a two-dimensional coordinate system. Here, the virtual viewpoint images 29A-29B are two-dimensional RGB images of a designated object captured from virtual viewpoints set in a virtual space, and are images simulating captured images of the point cloud obtained by cameras assuming that cameras are located at the positions of virtual viewpoints A, B, C, D, and E (see FIG. 4) set in the space. Note that in FIG. 5, virtual viewpoint images 29A-29E of the designated object viewed from five viewpoints A to E (0°, 30°, 60°, 90°, and 120°) are generated by shifting the virtual viewpoint by 30°, for example.

[0036] The object identification means 11 will now be described. The object identification means 11 comprises a shape learning unit 12 that performs machine learning of the shape of a designated object as training data from a training image including the designated object, and an object identification unit 13 that identifies the type of object based on the machine-learned shape of the designated object from a plurality of point cloud data acquired from a plurality of viewpoints for the designated object.

[0037] The shape learning unit 12 is composed of an image reading unit 14, a teacher data creation unit 15, and a learning unit 16. The image reading unit 14 reads a teacher image including a specified object. The teacher data creation unit 15 performs annotation, creating a group of teacher data for a teacher model for each object from the multiple teacher images read into the image reading unit 14. The annotation is performed by adding information as a note to the image data, specifically by enclosing the range of the specified object. The learning unit 16 inputs the group of teacher data created by the teacher data creation unit 15 as learning data (teacher data) and learns the three-dimensional shape of each object as well as its features.

[0038] The object identification unit 13 includes a point cloud data recognition unit 17 and an identification unit 18. The point cloud data recognition unit 17 reads point cloud data (a virtual point cloud image, a two-dimensional RGB image, in this embodiment) for recognition (for generating a three-dimensional image) as input data. The identification unit 18 identifies the three-dimensional shape of a specified object in the input point cloud data (virtual viewpoint image) from the recognition results obtained using training data after learning, and identifies the type of object. A specific identification method will be described later.

[0039] The object calculation means 22 will now be described. The object calculation means 22 calculates the coordinates and dimensions of a specified object from the point cloud data of the specified object. The object calculation means 22 is, for example, Open3D software, and includes an object extraction unit 23, a point cloud division unit 24, and a calculation unit 25.

[0040] The object extraction unit 23 extracts only the point cloud corresponding to a specified object from the point cloud data viewed from a certain viewpoint. The object extraction unit 23 includes a range designation unit 26. When the user designates the range of the point cloud data of the specified object from the entire displayed point cloud via the input means 106, the range designation unit 26 recognizes the range 48 as shown in Fig. 6(a), and the object extraction unit 23 extracts only the point cloud corresponding to the specified object as shown in Fig. 6(b) (see Fig. 6).

[0041] When point clouds 29A and 29B of multiple (two in the illustrated example) designated objects (designated object A and designated object B) are included in a range 48 designated by the user as shown in Fig. 7(a) based on the type information of the designated object in the point cloud data identified by the identification unit 18, the point cloud division unit 24 divides the point cloud data for each designated object identified by the object identification means 11, as shown by the dotted lines in Fig. 7(b). That is, the point cloud division unit 24 divides the point cloud data into the number of designated objects, with point cloud data including designated object A as the first data and point cloud data including designated object B as the second data.

[0042] The calculation unit 25 calculates the coordinates and dimensions of the extracted designated object and stores the calculation results. That is, by calculating the average values ​​of the x, y, and z coordinates of the point cloud of the extracted designated object, it is possible to obtain position information of the point cloud (coordinates of the center point). Also, the distance from the center point of the point cloud to the farthest point is calculated and used as dimensional information (object radius). Note that the calculation of coordinates and object radius is performed for each designated object. That is, if the point cloud data is divided into two, for example, the coordinates and dimensions of the point cloud are calculated using the first data, and the coordinates and dimensions of the point cloud are calculated using the second data.

[0043] The three-dimensional image generating means 31 will now be described. 3D CAD software such as Plant3D can be used as the three-dimensional image generating means 31. The three-dimensional image generating means 31 includes a model registration unit 32, a selection unit 33, a dimension adjustment unit 34, and an arrangement unit 35.

[0044] Model images of designated objects are registered in advance in the model registration unit 32. In this embodiment, the model images registered in the model registration unit 32 are images of a pump and an image of a valve.

[0045] The selection unit 33 selects a model image that matches the type of the designated object identified by the object identification means 11 from the model images registered in the model registration unit 32. For example, if the identification unit 18 identifies the designated object in the point cloud data as a valve, the selection unit 33 selects a model image of the valve stored in the model registration unit 32.

[0046] The size adjustment unit 34 adjusts the size of the selected model image based on the size information of the specified object calculated by the calculation unit 26 of the object position calculation means 21. In other words, the size adjustment unit 34 calculates conversion parameters to enlarge or reduce the model image so that the model image, when displayed in CAD, represents a size corresponding to the calculated size.

[0047] The placement unit 35 places the model image, whose dimensions have been adjusted by the dimension adjustment unit 34, at a three-dimensional coordinate position based on the coordinate information. That is, the placement unit 35 calculates transformation parameters so that the model image is placed at the calculated coordinates when the model image is displayed in CAD, and inserts the model image at the calculated position. In this way, a three-dimensional image can be generated.

[0048] The processing performed by the point cloud image generating means 3, object identification means 11, object calculation means 23, and three-dimensional image generating means 31 is displayed as an image, and the display means is equipped with a point cloud enlargement unit (not shown). The point cloud enlargement unit enlarges the size of each of the points that make up the point cloud data. In other words, if the point cloud data is unclear when displayed on a screen, by enlarging each point, for example, five times, the individual points will overlap and become clear when displayed as a point cloud.

[0049] Next, a method for generating a three-dimensional image using this three-dimensional image generating device will be described with reference to Fig. 8. As an example, a case where a three-dimensional image of equipment in a plant facility is generated will be described, with pumps and valves being the specified objects.

[0050] First, machine learning is performed to learn the shapes of a specified object viewed from multiple viewpoints (step S1). That is, the image reading unit 14 reads teacher images (in this embodiment, images of a pump and an image of a valve) including the specified object, in which the specified object is viewed from multiple viewpoints in each direction. This results in reading multiple surface shapes for one object. Note that the teacher images can be images (panoramic images or rendering images) of pumps or valves taken at an actual plant facility, or web images of pumps or valves. After extracting RGB data from the panoramic image's spherical coordinate system, the data is automatically converted and mapped to two-dimensional coordinates viewed from various angles to create teacher images.

[0051] The teacher data creation unit 15 performs annotation and creates a group of teacher data for a teacher model for each individual object from the many teacher images read by the image reading unit 14.

[0052] The learning unit 16 inputs the group of teacher data created by the teacher data creation unit 15 as learning data (teaching data) and learns the shape and features of each object. In this case, the surface shape and features at different surface positions of one object are read.

[0053] Meanwhile, a point cloud acquisition means 2, which is a laser scanner, is installed in the space where a three-dimensional image is to be generated. The laser scanner is installed on a tripod, for example, in the center of the space, and acquires point cloud data of the space (step S2). As mentioned above, point cloud data contains a collection of tens of millions of points, each of which stores coordinate values ​​(X, Y, Z) and color information (R, G, B), so that the collection of points forms an image similar to a photograph. Note that steps S1 and S2 may be interchanged, or they may be performed simultaneously.

[0054] The point cloud image generating means 3 generates a plurality of (five) virtual viewpoint images for each virtual viewpoint when viewing a three-dimensional point cloud 36 of the designated object from the acquired point cloud data. Specifically, assuming that cameras exist at the positions of virtual viewpoints A, B, C, D, and E (see FIG. 4) set in space, the point cloud as viewed in the coordinate system of the virtual camera 37 is calculated. The display size of the point cloud may be enlarged as necessary. The resulting virtual viewpoint image is then rendered onto a two-dimensional coordinate system to generate a two-dimensional RGB image. Specifically, as shown in FIG. 5, virtual viewpoint images 29A to 29E are generated as two-dimensional RGB images of the designated object as viewed from five viewpoints (0°, 30°, 60°, 90°, and 120°) shifted by 30° each. The virtual viewpoint images 29A to 29B in FIG. 5 are generated by imaging only the portion of the point cloud data 27 corresponding to the designated object 28 (the portion surrounded by a bold line in FIG. 5).

[0055] When the point cloud data recognition unit 17 reads the virtual viewpoint images 29A to 29E as input data, the identification unit 18 identifies the object type of the objects in these point cloud data based on the shape of the specified object learned by machine learning (step S4).

[0056] The object type is identified as follows: From the five point cloud data mentioned above, the object identification result of the point cloud data that outputs the highest confidence score is used. The confidence score is an index that indicates the probability that each bounding box detected by the model contains an object of a specific class, and in Figure 9(a) it is 0.85. It is calculated as shown in Equation 1 by combining the confidence P(Object) that an object exists, the probability P(Class Object) that the object belongs to a specific class (Valve in Figure 9(a)), and IoU (Intersection over Union).

number

[0057] As shown in Figure 9(a), this score numerically represents the reliability of the bounding box 45 output by the recognition model to include the target object. The IoU in Equation 1 is a number between 0 and 1 that indicates the degree of overlap between the predicted bounding box and the correct bounding box, with a larger value indicating a greater overlap. In other words, it indicates that the prediction is closer to the actual situation.

[0058] As shown in Figure 9(b), we can measure not only the overlap (i.e., IoU) between the predicted bounding box (Predicted(P)) 46 and the actual bounding box (Ground-truth(G)) 47, but also the centroid distance d and aspect ratio w between the bounding boxes. G / h G , w P / h P By also taking into consideration the consistency of the above, a more comprehensive accuracy evaluation can be achieved (Equation 2 and Equation 3). In this embodiment, the overall score for each object type is calculated by summing up the confidence scores of that type of output recognition result (Equation 4), and the object type with the highest overall score is determined as the final identification result (Equation 5).

number

number

number

number

[0059] Meanwhile, point cloud data viewed from any virtual viewpoint and including a specified object (e.g., a valve) is displayed using software such as Open3D (step S5), as shown in Fig. 6(a). In this case, the size of each point constituting the point cloud data may be enlarged (e.g., 5 times) by a point cloud enlargement unit. Furthermore, as long as a point cloud of the target object exists, point cloud data viewed from any virtual viewpoint may be displayed.

[0060] The user extracts only the point cloud corresponding to the specified object by specifying a range 48 of the target valve from the displayed point cloud data using the range specifying unit 26 (step S6). In this case, as shown in Fig. 6(b), the portion outside the specified range may be deleted so that only the specified range 48 is displayed.

[0061] As shown in Fig. 7(a), when point clouds 29A and 29B of multiple designated objects (designated object A, designated object B) are included in a range 48 designated by the user, the point cloud division unit 24 divides the point cloud data for each designated object identified by the object identification means 11, as shown by the dotted lines in Fig. 7(b) (step S7). That is, the point cloud division unit 24 divides the point cloud data by treating the point cloud data including designated object A as first data and the point cloud data including designated object B as second data. Note that if there is only one designated object within the designated range 48, the point cloud division unit 24 does not divide the point cloud data.

[0062] The calculation unit 25 calculates the coordinates and dimensions of the designated object in the recognized point cloud data and stores the calculation results (step S8). That is, the center point (coordinates) of the point cloud is found by calculating the average values ​​of the x, y, and z coordinates of the point cloud of the recognized designated object. The coordinates of this center point become the placement point of the model image of the designated object, which will be described later in step S11. In addition, the distance from the center point of the point cloud to the farthest point is calculated and used as the dimension (object radius). In this case, accurate dimensions can be obtained by calculating the distances of multiple points and filtering them. Note that the calculation of coordinates and dimension is performed for each designated object. That is, if there are multiple designated objects in the point cloud data, the point cloud division unit 24 divides the data into first data, second data, etc., and the calculation is performed for each of these data.

[0063] Model images of the designated objects (images of the pump and the valve) are registered in advance in the model registration unit 32 of the three-dimensional image generation means 31. The selection unit 33 selects a model image of the type of designated object identified by the object identification means 11 (step S9). For example, if the identification unit 18 identifies the type of the designated object in the point cloud data as a valve, the selection unit 33 selects a model image of the valve stored in the model registration unit 32.

[0064] The size adjustment unit 34 adjusts the size of the selected model image based on the size information of the specified object calculated by the calculation unit 25 of the object position calculation means 21 (step S10). That is, the size adjustment unit 34 calculates conversion parameters to enlarge or reduce the model image so that the model image, when displayed in CAD, represents a size corresponding to the calculated size.

[0065] The placement unit 35 places the model image, whose dimensions have been adjusted by the dimension adjustment unit 34, at a three-dimensional coordinate position based on the calculated coordinates (step S11). That is, the placement unit 35 calculates transformation parameters so that the model image is placed at the calculated coordinates when displayed in CAD, and inserts the model image at the calculated position.

[0066] In this way, a three-dimensional image can be generated by the three-dimensional image generating means 31 (3D CAD software) using the dimensional information (object radius) and coordinate information (center point of the point group) calculated by the calculation unit 25 as parameters.

[0067] In the 3D image generation device, 3D image generation method, and 3D image generation program of the first embodiment, the type of a specified object is identified from multiple point cloud images obtained by viewing the specified object from multiple different viewpoints. This allows the specified object to be recognized from multiple viewpoints, and the type of the specified object can be determined even if the object has a complex shape. Furthermore, the object calculation means acquires coordinate information and dimensional information for the specified object separately from identifying the object's type. This allows the coordinate information and dimensional information to be added to the type information of the specified object, enabling the generation of a 3D image. In other words, this eliminates the need for model readjustment and the registration of multiple models with different dimensions, which were previously required, and allows for the generation of a 3D image easily and accurately.

[0068] Furthermore, in the first embodiment, the virtual viewpoint image is a two-dimensional RGB image, which has higher resolution and clarity than a point cloud image, improving the accuracy of recognizing objects with complex shapes such as valves and pumps. Point clouds can be made denser and their resolution increased by changing the scanner settings, but the data size becomes very large, which means that point cloud processing takes time. Therefore, analyzing two-dimensional RGB images can improve accuracy and shorten processing time.

[0069] The 3D image generation device of the second embodiment will be described with reference to Fig. 10. The 3D image generation device of the second embodiment differs from the 3D image generation device of the first embodiment in that the 3D image generation device of the second embodiment does not include a point cloud image generation means. In this case, the point cloud data recognition unit 17 can directly input point cloud data.

[0070] In the first embodiment, in step S4, a virtual viewpoint image (two-dimensional RGB image) is input as data for recognition by the object identification means 11, and the point cloud data recognition unit 17 reads the virtual viewpoint image, but in the second embodiment, the point cloud data recognition unit 17 reads the point cloud data itself as input data.

[0071] In this way, the 3D image generation device of the second embodiment can generate a 3D image from the acquired point cloud data without generating a point cloud image as a 2D image. In the 3D image generation device shown in Fig. 10, the same components as those in the 3D image generation device of the first embodiment are assigned the same reference numerals as in Fig. 3, and their description will be omitted.

[0072] The 3D image generation device of the third embodiment will be described with reference to Fig. 11. The object calculation means 51 of the 3D image generation device of the third embodiment is different from the object calculation means 22 of the 3D image generation device of the first embodiment.

[0073] The object calculation means 51 includes an object extraction unit 52, a point cloud division unit 53, and a calculation unit 54. In the third embodiment, the object extraction unit 52 automatically extracts point cloud data of an object whose type has been identified by the identification unit 18 of the object identification means 11, as shown in Fig. 9(b). The point cloud division unit 53 and the calculation unit 54 are similar to the point cloud division unit 24 and the calculation unit 25 of the first embodiment.

[0074] In the first embodiment, in step S6, the user specifies the range of the target object from the displayed point cloud data and extracts only the point cloud corresponding to the specified object. However, in the third embodiment, the object extraction unit 52 extracts only the point cloud corresponding to the specified object based on information from the identification unit 18 of the object identification means 11. In other words, the specified object is not extracted by the user, and the range 48 in FIG. 6(a) can be extracted automatically. In the 3D image generation device shown in FIG. 11, components similar to those in the 3D image generation device of the first embodiment are assigned the same reference numerals as those in FIG. 3, and their description will be omitted.

[0075] Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and various modifications are possible. For example, the specified object is not limited to a valve or a pump, but can be other equipment in a plant facility, such as piping, support structures, racks, meters, electrical conduits, cable trays, bull boxes, electrical panels, ducts, room equipment (doors, fluorescent lights, geared trolleys, surveillance cameras, etc.). Furthermore, the specified object is not limited to a plant facility, and can be a three-dimensional image of the inside of another building.

[0076] In the embodiment, when the point cloud of the specified object is extracted by the point cloud extraction unit, other point clouds are erased, but calculations may be performed without erasing them on the screen. The point cloud image creation means and object calculation means can be performed using Open3D, but known software may also be used. The three-dimensional image generation means can be performed using Plant3D, but other 3D CAD software may also be used.

[0077] In the embodiment, the virtual viewpoint images are acquired from five viewpoints (0°, 30°, 60°, 90°, and 120°) by shifting the virtual viewpoint by 30° increments, but the shift angle and the number of point cloud data can be set arbitrarily. Furthermore, the virtual viewpoint does not have to be shifted at equal intervals and can be shifted randomly. The number of virtual viewpoint images can also be set in various ways.

[0078] The method for identifying the object type is not limited to the reliability calculation method of the embodiment, and any other method may be used as long as it can identify the object type. The teacher image may be both an image of the specified object taken in an actual plant facility (a panoramic image or a rendering image) and a web image of the specified object, or only one of them. In other words, any image acquisition method or image processing method may be used as long as the image represents the specified object. In the embodiment, the dimension of the specified object is defined as the length from the center coordinate of the specified object to the coordinate position farthest from the center coordinate of the specified object, but it may also be the length to another coordinate position depending on the shape of the specified object. [Explanation of symbols]

[0079] 1 Three-dimensional image generation device 3 Point cloud image generator 5. 3D virtual model generation unit 6 Virtual camera settings section 8 Rotating part 9 Virtual viewpoint image generation unit 11 Object identification means 23, 51 Object calculation means 24, 53 Point group division part 31 Three-dimensional image generation means 36 3D point cloud

Claims

1. A three-dimensional image generating device that generates a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, a point cloud image generating means for generating point cloud images of a designated object from a plurality of different viewpoints from point cloud data that is three-dimensional information; an object identification means for performing machine learning on the shape of a designated object as viewed from a plurality of viewpoints from a teacher image including the designated object as teacher data, and for identifying the type of object based on the machine-learned shape of the designated object from the point cloud image generated by the point cloud image generation means; an object calculation means for calculating coordinates and dimensions of a designated object from point cloud data of the designated object; a three-dimensional image generating means for registering model images of designated objects in advance, selecting a model image that matches the type of designated object identified by said object identification means, adjusting the dimensions of the selected model image based on dimensional information of the designated object calculated by said object position calculation means, and arranging the dimension-adjusted model image at a three-dimensional coordinate position based on said coordinate information to generate a three-dimensional image.

2. The three-dimensional image generating device according to claim 1, characterized in that the point cloud image generating means comprises a three-dimensional point cloud generating unit that generates a three-dimensional point cloud by three-dimensionalizing the point cloud of a specified object from point cloud data, a virtual camera setting unit that sets a virtual camera to be placed at a virtual viewpoint position with respect to the three-dimensional point cloud, a rotation unit that rotates the virtual camera relatively around the three-dimensional point cloud, and a virtual viewpoint image creation unit that generates virtual viewpoint images for each of a plurality of different virtual viewpoints when the virtual camera views the three-dimensional point cloud from different virtual viewpoints.

3. 3. The three-dimensional image generating device according to claim 2, wherein the virtual viewpoint image is a two-dimensional RGB image.

4. A three-dimensional image generating device that generates a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, an object identification means for performing machine learning of the shapes of a designated object viewed from a plurality of viewpoints as training data from a training image including the designated object, and for identifying the type of object based on the machine-learned shapes of the designated object from point cloud data for each of the plurality of different viewpoints viewed from the designated object; an object calculation means for calculating coordinates and dimensions of a designated object from point cloud data of the designated object; a three-dimensional image generating means for registering model images of designated objects in advance, selecting a model image that matches the type of designated object identified by said object identification means, adjusting the dimensions of the selected model image based on dimensional information of the designated object calculated by said object position calculation means, and arranging the dimension-adjusted model image at a three-dimensional coordinate position based on said coordinate information to generate a three-dimensional image.

5. The three-dimensional image generating device according to claim 1 or claim 4, characterized in that the object calculation means includes a point cloud division unit that divides the point cloud data for each designated object identified by the object identification means when the point cloud data includes multiple designated objects.

6. 5. The three-dimensional image generating device according to claim 1, further comprising a point cloud enlargement unit that, when the point cloud data is displayed, enlarges and displays the size of each of the points that make up the point cloud data.

7. 5. The three-dimensional image generating device according to claim 1, wherein the dimension is a length from the center coordinate of the designated object to the coordinate position farthest from the center coordinate of the designated object.

8. 5. The three-dimensional image generating apparatus according to claim 1, wherein the specified object is a facility within a plant facility.

9. A three-dimensional image generation method for generating a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, comprising: From the point cloud data, which is three-dimensional information, a point cloud image is generated for each of the viewpoints when the specified object is viewed from a plurality of different viewpoints; from a training image including the designated object, machine learning is performed on the shapes of the designated object as viewed from a plurality of viewpoints as training data, and a type of object is identified from the point cloud image based on the machine-learned shape of the designated object; Calculate the coordinates and dimensions of the specified object from the point cloud data of the specified object; A three-dimensional image generation method characterized in that model images of designated objects are registered in advance, a model image that matches the type of the identified designated object is selected, the dimensions of the selected model image are adjusted based on calculated dimensional information of the designated object, and the dimension-adjusted model image is placed at a three-dimensional coordinate position based on the coordinate information to generate a three-dimensional image.

10. A three-dimensional image generation program for generating a three-dimensional image from three-dimensional information of a space having a designated object whose type is specified in advance, A process of generating point cloud images of a specified object from a plurality of different viewpoints from point cloud data, which is three-dimensional information; A process of performing machine learning on the shape of a specified object as viewed from a plurality of viewpoints from a training image including the specified object as training data, and identifying the type of object from the point cloud image based on the machine-learned shape of the specified object; A process of calculating coordinates and dimensions of the designated object from the point cloud data of the designated object; a process of selecting a model image that matches the type of the specified object from pre-registered model images of the specified object, adjusting the dimensions of the selected model image based on the calculated dimension information of the specified object, and arranging the dimension-adjusted model image at a three-dimensional coordinate position based on the coordinate information to generate a three-dimensional image; A three-dimensional image generating program that causes a computer to execute each of the above processes.

Images