Point cloud processing device, point cloud processing method, program, and point cloud processing system

Abstract

To estimate the orientation of detected partial regions and appropriately replace the partial regions representing objects with three-dimensional models.SOLUTION: A point cloud processing device is equipped with a detection unit that detects partial regions based on three-dimensional point clouds obtained by measuring a target, an estimation unit that estimates the orientation of the partial regions based on the three-dimensional models corresponding to the partial regions and the three-dimensional point clouds, and a replacement unit that replaces the partial regions with the three-dimensional models based on the orientation estimated by the estimation unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a point cloud processing device, a point cloud processing method, a program, and a point cloud processing system. [Background technology]

[0002] Conventionally, a technique is known in which a partial region such as a bounding box surrounding an object is detected from a three-dimensional point cloud, and the point cloud included in the detected bounding box is replaced with a model shape.

[0003] Patent document 1 also describes a three-dimensional shape generation device that uses model shape information indicating a three-dimensional model shape to generate three-dimensional shape information indicating a three-dimensional shape corresponding to a three-dimensional point cloud, and the three-dimensional shape generation device is equipped with a three-dimensional information generation means that generates the three-dimensional shape information using the multiple model shape information set by the model setting operation based on point cloud information indicating the three-dimensional point cloud and a model setting operation on a reception screen that accepts the model setting operation to set multiple model shape information used to generate the three-dimensional shape information from the multiple model shape information. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art does not describe estimating the orientation of the detected bounding box.

[0005] The present invention has been made in view of the above, and has an object to estimate the orientation of a detected partial region and to appropriately replace the partial region with a three-dimensional model. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the point cloud processing device of the present invention includes a detection unit that detects a partial area representing an object based on a three-dimensional point cloud obtained by measuring the target, an estimation unit that estimates the orientation of the partial area based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud, and a replacement unit that replaces the partial area with the three-dimensional model based on the orientation estimated by the estimation unit. [Effects of the Invention]

[0007] According to the present invention, the orientation of the detected partial region can be estimated, and the partial region can be appropriately replaced with a three-dimensional model. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a point cloud processing system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a hardware configuration diagram of the terminal device and management server according to this embodiment. [Figure 3] FIG. 3 is a functional block diagram of the point cloud processing system according to this embodiment. [Figure 4] FIG. 4 is a functional block diagram of the 3D model processing unit according to this embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of point cloud data. [Figure 6] FIG. 6 is a diagram showing an example of a detected bounding box. [Figure 7] FIG. 7 is a diagram showing an example of bounding box data detected for each class. [Figure 8] FIG. 8 is a diagram showing an example in which the bounding box is replaced with a 3D model. [Figure 9] FIG. 9 is a diagram showing an example in which a bounding box is replaced with a 3D model. [Figure 10] FIG. 10 is a diagram showing an example of a mesh of a 3D model and a model point cloud. [Figure 11] FIG. 11 is a diagram illustrating the orientation of a 3D model. [Figure 12] FIG. 12 is a flowchart showing an example of the procedure of 3D model processing according to this embodiment. [Figure 13] FIG. 13 is a sequence diagram showing an example of 3D model processing according to this embodiment. [Figure 14] FIG. 14 is an explanatory diagram showing an example of a setting screen according to this embodiment. [Figure 15] FIG. 15 is an explanatory diagram showing an example of a selection screen according to this embodiment. [Figure 16] FIG. 16 is a functional block diagram of a 3D model processing unit according to the second embodiment. [Figure 17] FIG. 17 is an explanatory diagram illustrating an example of a setting screen according to the second embodiment. [Figure 18] FIG. 18 is a flowchart illustrating an example of a procedure for 3D model processing according to the second embodiment. [Figure 19] FIG. 19 is a diagram illustrating instance segmentation. DETAILED DESCRIPTION OF THE INVENTION

[0009] In industries such as civil engineering and construction, BIM (Building Information Modeling) / CIM (Construction Information Modeling) is being adopted with the aim of improving labor productivity.

[0010] BIM is a solution that uses a building database that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereinafter referred to as a three-dimensional model or 3D model) created on a computer, allowing information to be utilized in all processes from building design and construction to maintenance and management.

[0011] CIM is a solution for the civil engineering field (general infrastructure such as roads, electricity, gas, and water) that was proposed following the example of BIM, which was being promoted in the construction field. Like BIM, it is being worked on as a way to improve the efficiency and sophistication of the entire construction production system by sharing information between stakeholders, mainly through 3D models.

[0012] An important aspect of promoting BIM / CIM is how to easily obtain 3D information about objects such as buildings, public facility spaces, etc. 3D information here refers to three-dimensional point clouds (hereafter sometimes simply referred to as point clouds) that hold distance information to objects obtained by measuring the object using a laser scanner (hereafter referred to as LS), as well as mesh objects and 3D models generated based on point cloud data that indicates three-dimensional point clouds.

[0013] When constructing a structure from scratch, BIM / CIM software can be used to design the finished product from scratch, making it easy to introduce BIM / CIM. However, in the case of existing buildings, the original blueprints for the building may no longer exist, or the current state may differ from the original blueprints due to renovations over time, making the hurdles to BIM / CIM implementation higher. Converting existing buildings to BIM in this way is called As-Build BIM, and is an important issue for promoting future BIM / CIM implementation.

[0014] One way to achieve As-Build BIM is to use the workflow described above to measure space using LS and create a 3D model from the measured point cloud data. Traditionally, this work has been done using photographs and tape measures, or by sketching by hand, but depending on the size of the space, the presence or absence of installed objects, and the complexity (such as the intricate layout of piping), this can result in significant work costs. For this reason, the introduction of LS, which can obtain 3D information about the space, is gaining attention as a promising way to solve this problem.

[0015] With As-Build using LS, it has become easier to obtain 3D information, but a new task arises: point cloud processing of point cloud data, which did not exist in previous work.Typical point cloud processing involves "multi-point measurement using LS," "generation of an integrated point cloud from each point cloud," "removal of unnecessary point clouds such as noise," "detection of bounding boxes," and "conversion to a 3D model."

[0016] In the process of generating a 3D model of an entire scene by converting bounding boxes corresponding to objects into 3D models, the challenges are to improve the accuracy of estimating the object's pose (orientation) and to enable users to easily modify the detection results. Here, a bounding box is a partial region that surrounds an object included in the target, and is an example of a partial region that indicates an object. The partial region is detected based on a point cloud acquired by measuring objects within a building. Furthermore, objects include inanimate objects such as building components as well as living objects such as humans and animals.

[0017] In view of the above problems, a first object of this embodiment is to accurately estimate the orientation of a detected bounding box.

[0018] A second object is to enable the user to easily modify the detected bounding box.

[0019] (First embodiment) 1 is a diagram showing the overall configuration of a point cloud processing system 1 according to a first embodiment of the present invention. The point cloud processing system 1 of this embodiment is constructed by a terminal device 3, which is an example of a communication terminal, and a management server 5.

[0020] The management server 5 is an example of a point cloud processing device that executes one or more point cloud processes on point cloud data representing a three-dimensional point cloud.

[0021] Here, a three-dimensional point cloud is a collection of coordinate points in the X, Y, and Z directions corresponding to measurement points on the surface of an object when the space in which the object is located is measured using LS or the like. Each coordinate point is represented as (1, 3, 5), for example. Color information may also be added to each coordinate point, and the RGB values ​​of each coordinate point may be added as color information. A three-dimensional point cloud is also sometimes called a point cloud. Point cloud data is data that can be handled by a computer or the like as a collection of coordinate points in a virtual three-dimensional space.

[0022] Although the above example shows the measurement of a 3D point cloud using LS, it is also possible to measure a 3D point cloud using other optical or mechanical measurement means. Other optical measurement means include a method using a stereo camera or a method using Visual SLAM (Simultaneous Localization And Mapping).

[0023] The terminal device 3 and the management server 5 can communicate with each other via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. The communication network 100 may include not only wired communication networks but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide interoperability for Microwave Access), LTE (Long Term Evolution), and 5G (5th Generation). The terminal device 3 can also communicate with each other using short-range communication technologies such as NFC (Near Field Communication) (registered trademark).

[0024] <Hardware configuration> 2 is a diagram showing the hardware configuration of the terminal device 3 and management server 5 according to this embodiment. Each piece of hardware configuration of the terminal device 3 is indicated by a reference number in the 300s. Each piece of hardware configuration of the management server 5 is indicated by a reference number in the 500s in parentheses.

[0025] The terminal device 3 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) 305, a recording medium 306, a media I / F 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a CD-RW (Compact Disc-Re Writable) drive 314, and a bus line 310.

[0026] Of these, the CPU 301 controls the overall operation of the terminal device 3. The ROM 302 stores programs used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD 305 controls the reading and writing of various data from and to the HD 304 under the control of the CPU 301. The media I / F 307 controls the reading and writing (storage) of data from and to a recording medium 306 such as a flash memory. The display 308 displays various information such as a cursor, menus, windows, characters, and images. The network I / F 309 is an interface for data communication using the communication network 100. The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, and the like. The mouse 312 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, and the like. The CD-RW drive 314 controls the reading and writing of various data from and to a CD-RW 313, an example of a removable recording medium. The terminal device 3 may further include a configuration for controlling reading or writing (storing) of data from or to an external PC (Personal Computer) or external device connected by wire or wirelessly such as Wi-Fi (Wireless Fidelity).

[0027] The management server 5 also includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD 505, a recording medium 506, a media I / F 507, a display 508, a network I / F 509, a keyboard 511, a mouse 512, a CD-RW drive 514, and a bus line 510. These components have the same configuration as those described above (CPU 301, ROM 302, RAM 303, HD 304, HDD 305, recording medium 306, media I / F 307, display 308, network I / F 309, keyboard 311, mouse 312, CD-RW drive 314, and bus line 310), and therefore description thereof will be omitted. The CD-RW drive 514 controls reading and writing of various data from and to a CD-RW 513, which is an example of a removable recording medium.

[0028] The CD-RW drive 314 (514) may be replaced with a CD-R drive, etc. The terminal device 3 and the management server 5 may each be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or memory unit) is divided and arbitrarily assigned.

[0029] FIG. 3 is a functional block diagram of the point cloud processing system 1 according to this embodiment.

[0030] <Functional configuration of terminal device> As shown in Fig. 3, the terminal device 3 has a transmitting / receiving unit 31, a receiving unit 32, a display control unit 34, and a storage / reading unit 39. Each of these units is a function or a means for performing the function, which is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 301 in accordance with a program loaded from the HD 304 onto the RAM 303. The terminal device 3 also has a storage unit 3000 constructed by the RAM 303 and HD 304 shown in Fig. 2.

[0031] (Functional configuration of terminal device) Next, each component of the terminal device 3 will be described.

[0032] The transmitter / receiver unit 31 is an example of a receiving means, and is realized by instructions from the CPU 301 shown in Figure 2 and the network I / F 309, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100.

[0033] The reception unit 32 is an example of a reception means, and is realized mainly by commands from the CPU 301 shown in FIG. 2, as well as the keyboard 311 and mouse 312, and receives various inputs from the user.

[0034] The display control unit 34 is an example of a display control means, and is realized by commands from the CPU 301 shown in FIG. 2, and causes the display 308, which is an example of a display unit, to display various images and screens.

[0035] The storage / reading unit 39 is an example of a storage control means, and is executed by commands from the CPU 301 shown in FIG. 2, as well as the HDD 305, the media I / F 307, the CD-RW drive 314, and an external PC or external device, and performs processes such as storing various data in the storage unit 3000, the recording media 306, the CD-RW 313, and an external PC or external device, and reading various data from the storage unit 3000, the recording media 306, the CD-RW 313, and an external PC or external device.

[0036] <Management server functional configuration> The management server 5 has a transmitting / receiving unit 51, a processing unit 53, a determining unit 55, a generating unit 57, and a storing / reading unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503. The management server 5 also has a memory unit 5000 constructed by the HD 504 shown in Fig. 2. The memory unit 5000 is an example of a memory means.

[0037] (Management server functional configuration) Next, we will explain each component of the management server 5. The management server 5 may be configured to distribute each function among multiple computers. Furthermore, although the management server 5 will be explained as a server computer existing in a cloud environment, it may also be a server existing in an on-premise environment.

[0038] The transmitter / receiver unit 51 is an example of a transmitting means, and is realized by instructions from the CPU 501 shown in Figure 2 and the network I / F 509, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100.

[0039] The processing unit 53 is realized by instructions from the CPU 501 shown in Fig. 2, and performs various processes including 3D model processing, which will be described later. The processing unit 53 is an example of a point cloud processing means.

[0040] The determination unit 55 is realized by instructions from the CPU 501 shown in FIG. 2, and makes various determinations.

[0041] The generation unit 57 is realized by instructions from the CPU 501 shown in Fig. 2, and generates various display screens, which will be described later. The generation unit 57 is an example of a display screen generation unit.

[0042] The generation unit 57 generates a display screen that displays a three-dimensional point cloud, a bounding box, a 3D model, etc. Furthermore, when estimating the orientation of the bounding box, the generation unit 57 can generate a display screen that displays a point cloud (hereinafter referred to as a model point cloud) obtained by converting a 3D model and rotates it relative to the three-dimensional point cloud.

[0043] 2, as well as by the HDD 505, media I / F 507, CD-RW drive 514, and external PCs and external devices, and performs processing to store various data in the storage unit 5000, recording medium 506, CD-RW 513, and external PCs and external devices, and to read various data from the storage unit 5000, recording medium 506, CD-RW 513, and external PCs and external devices. The storage unit 5000, recording medium 506, CD-RW 513, and external PCs and external devices are examples of storage means.

[0044] The storage unit 5000 stores a user information management DB 5001, a setting information management DB 5002, a storage processing management DB 5003, a point cloud management DB 5004, and a processing result management DB 5005, each of which is configured using a setting information management table.

[0045] The user information management DB 5001 stores and manages file names of three-dimensional point cloud data associated with user information, the setting information management DB 5002 stores and manages various setting information, the storage processing management DB 5003 stores and manages various processing programs and data for executing point cloud processing, the point cloud management DB 5004 stores and manages point cloud data, and the processing result management DB 5005 stores and manages processing result information that indicates the processing results of executing point cloud processing on point cloud data.

[0046] <Functional configuration of the 3D model processing section> The processing unit 53 includes a 3D model processing unit 530 that executes processing to generate a 3D model from a point cloud (3D model processing). Fig. 4 is a functional block diagram of the 3D model processing unit 530 according to this embodiment. The 3D model processing unit 530 includes a detection unit 531, an estimation unit 532, and a replacement unit 533.

[0047] The detection unit 531 detects a bounding box using AI (Artificial Intelligence) based on a three-dimensional point cloud, which is point cloud data acquired by measuring an object.

[0048] Figure 5 is a diagram showing an example of point cloud data. This example shows point cloud data acquired by measuring the area of ​​a restroom in a building, with Figure 5(a) showing the point cloud data viewed from diagonally above, and Figure 5(b) showing the point cloud data viewed from directly above. As shown in the figure, from the front of the area to the back, there is an entrance and wall, five urinals, five stalls, and toilet seats. The coordinate axes are defined as X on the horizontal plane, Y toward the back of the area, and Z as the vertical axis.

[0049] FIG. 6 is a diagram showing an example of detected bounding boxes. Of the bounding boxes detected based on the point cloud data of FIG. 5, FIG. 6 shows bounding boxes corresponding to toilet seats. Here, the bounding boxes are rectangular parallelepipeds, and five bounding boxes corresponding to five toilet seats are detected, as shown in FIGS. 6(a) and 6(b). Note that the detection unit 531 detects unoriented bounding boxes (axis aligned bounding boxes). An "unoriented bounding box" is a bounding box whose sides are aligned with coordinate axes, regardless of the orientation of the object to be detected, as shown in FIG. 6(c).

[0050] The AI ​​used in the detection unit 531 is the result of machine learning using input point cloud data and output (ground truth data) bounding boxes and classes (types of objects, such as toilet seats and toilet bowls) corresponding to the bounding boxes. The bounding boxes and classes used in machine learning can be detected using, for example, object detection or instance segmentation. Furthermore, the features of the input data (positional relationship of the coordinates of each point in the point cloud, color, normal, reflection intensity, etc.) can be used in common, and the output data can be used as bounding boxes and classes, allowing the AI ​​to perform machine learning to detect each of them.

[0051] 6 shows an example of a bounding box whose class is a toilet seat, but the AI ​​can also be trained to detect other types of objects such as doors and sinks as classes, and the bounding boxes of these classes can also be detected. In other words, the detection unit 531 detects a bounding box for each class.

[0052] FIG. 7 shows an example of bounding box data detected for each class. Here, coordinates (X, Y, Z) represent the center position of the bounding box, and L, W, and H represent the length of the bounding box in the X direction, the Y direction, and the Z direction (height), respectively. Note that bounding box data can also be expressed as the minimum and maximum values ​​in the X direction, the minimum and maximum values ​​in the Y direction, and the minimum and maximum values ​​in the Z direction. If these six values ​​are Xmin, Xmax, Ymin, Ymax, Zmin, and Zmax, respectively, they can be expressed by the following equation using (X, Y, Z) and L, W, and H. Therefore, bounding box data may be expressed using either (X, Y, Z, L, W, H) or (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax).

[0053] (Xmin,Ymin,Zmin)=(XL / 2,YW / 2,ZH / 2) (Xmax,Ymax,Zmax)=(X+L / 2,Y+W / 2,Z+H / 2)

[0054] The estimation unit 532 estimates the orientation of the bounding box based on the 3D model and three-dimensional point cloud corresponding to the class. The AI ​​can also be trained to learn the orientation of the bounding box. However, since the accuracy of detecting the orientation is low, in this embodiment, unoriented bounding boxes and classes are learned to detect them, and an oriented bounding box is obtained using the orientation of the bounding box estimated by the estimation unit 532.

[0055] The replacement unit 533 replaces the oriented bounding box with a 3D model corresponding to the bounding box, using the orientation estimated by the estimation unit 532. As a result, the orientation of the 3D model after replacement becomes the same as the orientation estimated by the estimation unit 532. Here, the 3D model corresponding to the bounding box is a CG (Computer Graphics) model corresponding to the class of the bounding box detected by the detection unit 531.

[0056] 8 and 9 are diagrams showing examples in which a bounding box has been replaced with a 3D model. Note that in FIG. 8, point cloud data and a 3D model are displayed simultaneously, resulting in some areas where points located in front of the 3D model appear to overlap with the 3D model. Also, FIG. 9(a) shows an example in which the point cloud data has been removed from FIG. 8(a), and FIG. 9(b) shows an example in which a 3D model of another class has been added to FIG. 9(a).

[0057] Next, the process of estimating the orientation of a bounding box in the estimation unit 532 will be described. Fig. 10 is a diagram showing an example of a 3D model mesh and a model point cloud. Here, an example of a 3D model mesh (Fig. 10(a)) when the bounding box class is a toilet seat, and an example of a model point cloud obtained by converting the 3D model (Fig. 10(b)) are shown. For example, the 3D model can be converted into a model point cloud by a technique such as Poisson Disk Sampling, which replaces surfaces with a point cloud of uniform density.

[0058] The estimation unit 532 compares the model point cloud with the point cloud included in the bounding box (a three-dimensional point cloud acquired by measuring the target) while changing the orientation of the 3D model, and estimates the orientation in which the index of the distance between the point clouds is smallest as the orientation of the bounding box. The index of the distance between the point clouds can be calculated using, for example, Chamfer Distance. Note that the comparison between the model point cloud and the point cloud included in the bounding box is performed with their positions normalized (aligned with the origin of the coordinate system). If the resolution of the orientation (angle) is, for example, 45 degrees, the 3D model is rotated in the XY plane by angles of 0 degrees (orientation along the coordinate axes), 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees to calculate the index of the distance between the point clouds.

[0059] FIG. 11 is a diagram illustrating the orientation of a 3D model. In FIGS. 11(a) and (b), black dots represent an example of a model point cloud when the orientation of the 3D model is 90 degrees, and in FIGS. 11(c) and (d), the orientation of the 3D model is 0 degrees. The gray dots represent point cloud data obtained by measurement. In this example, when the orientation is 90 degrees, the orientations of the model point cloud and the three-dimensional point cloud do not match, resulting in a large distance index calculation result. When the orientation is 0 degrees, the orientations of the model point cloud and the three-dimensional point cloud match, resulting in a small distance index calculation result. The estimation unit 532 similarly calculates distance indices for other angles, and in this example, estimates that the orientation of 0 degrees is the orientation of the bounding box.

[0060] 12 is a flowchart showing an example of the procedure for 3D model processing according to this embodiment. First, the detection unit 531 detects a bounding box based on a three-dimensional point cloud acquired by measuring the target (step S120).

[0061] Next, the estimation unit 532 acquires a three-dimensional point cloud included in the bounding box (step S121), and converts a 3D model of a class corresponding to the bounding box into a point cloud (model point cloud) (step S122). Subsequently, the estimation unit 532 calculates a distance index between the three-dimensional point cloud and the model point cloud while changing the orientation of the 3D model (while rotating the 3D model) (step S123). Then, the estimation unit 532 estimates that the orientation with the smallest distance index is the orientation of the bounding box (step S124).

[0062] Next, the replacement unit 533 obtains a bounding box with an orientation using the estimated orientation and replaces it with the 3D model (step S125). In this way, by rotating the 3D model corresponding to the bounding box relative to the three-dimensional point cloud included in the bounding box, the orientation of the bounding box can be easily estimated and replaced with the 3D model in the estimated orientation. Note that the process of rotating the 3D model and calculating the index may be performed automatically according to a predetermined resolution (angle), or may be performed according to an angle or timing specified by a user operation.

[0063] FIG. 13 is a sequence diagram showing an example of 3D model processing according to this embodiment.

[0064] The reception unit 32 of the terminal device 3 receives an input operation related to user information (step S1). The transmission / reception unit 31 transmits a request for a setting screen including the user information received in step S1 to the management server 5 of the terminal device 3, and the transmission / reception unit 51 of the management server 5 receives the request transmitted from the terminal device 3 (step S2).

[0065] Next, the memory / read unit 59 of the management server 5 searches the user information management DB 5001 using the user information included in the request received in step S2 as a search key, thereby reading out the file name of the three-dimensional point cloud data associated with the user information included in the request, and the generation unit 57 of the management server 5 generates a display screen including a setting screen based on the file name read out by the memory / read unit 59 (step S3).

[0066] This setting screen is a GUI (Graphical User Interface) screen on which means for inputting point cloud information (such as the file name of point cloud data), means for selecting point cloud processing, and the like are arranged.

[0067] The transmitter / receiver 51 transmits display screen information including setting screen information relating to the setting screen generated in step S3 to the terminal device 3, and the transmitter / receiver 31 of the terminal device 3 receives the display screen information transmitted from the management server 5 (step S4).

[0068] Next, the display control unit 34 of the terminal device 3 causes the display 308 to display a display screen including the setting screen received in step S4 (step S5). The accepting unit 32 of the terminal device 3 accepts a predetermined input operation by the user on the displayed setting screen. This input operation includes an operation to input point cloud information.

[0069] The transmitter / receiver 31 transmits input information relating to the input operation received by the reception unit 32 to the management server 5, and the transmitter / receiver 51 of the management server 5 receives the input information transmitted from the terminal device 3 (step S6).

[0070] The storage / reading unit 59 of the management server 5 searches the point cloud management DB 5004 using the point cloud setting information included in the input information received in step S6 as a search key, thereby reading out the three-dimensional point cloud data to be subjected to 3D model processing.

[0071] In addition, the storage / reading unit 59 reads out a point cloud processing program (in this embodiment, a program for processing a 3D model) by searching the setting information management DB 5002 using the point cloud processing information included in the input information received in step S6 as a search key.

[0072] The processing unit 53 of the management server 5 generates 3D model processing information based on the three-dimensional point cloud data and the 3D model processing program read from the storage / readout unit 59 (step S7). The 3D model processing information includes information on the result of detecting the bounding box, information on the result of replacing the bounding box with a 3D model, etc.

[0073] The generation unit 57 of the management server 5 generates a display screen including a selection screen for displaying 3D model processing information and allowing the user to select whether to continue or terminate the processing. The selection screen is, for example, a GUI (Graphical User Interface) screen for accepting user instructions. The transmission / reception unit 51 also transmits the generated display screen information to the terminal device 3 (step S8).

[0074] The transmitter / receiver 31 of the terminal device 3 receives the display screen information transmitted from the management server 5, the display control unit 34 of the terminal device 3 displays the received display screen on the display 308, and the reception unit 32 of the terminal device 3 receives a predetermined input operation by the user on the displayed display screen (step S9). This input operation includes a selection operation for selecting whether to continue or end the 3D model processing.

[0075] The transmitter / receiver 31 transmits input information relating to the input operation received by the reception unit 32 to the management server 5, and the transmitter / receiver 51 of the management server 5 receives the input information transmitted from the terminal device 3 (step S10).

[0076] If this input information includes selection information indicating the end of the 3D model processing, the processing unit 53 of the management server 5 determines the processing result of the 3D model processing.

[0077] On the other hand, if the input information includes selection information indicating continuation of the 3D model processing, the processing unit 53 of the management server 5 continues the 3D model processing in step S7 based on this information.

[0078] The processing unit 53 converts the processing result information into a file format that can be read by point cloud processing software, a file format that can be read by 3D-CAD software, a file format that can be read by BIM / CIM software, etc., and the storage / reading unit 59 stores the converted processing result information in the processing result management DB 5005, the recording medium 506, or the CD-RW 513 (step S11). The transmitting / receiving unit 51 transmits the determined processing result information to the terminal device 3 (step S12).

[0079] The transmitter / receiver 31 of the terminal device 3 receives the processing result information transmitted from the management server 5, and the display controller 34 of the terminal device 3 displays the received processing result on the display 308 (step S13).

[0080] In the above, the functions of the management server 5 in FIG. 3 may be integrated into the terminal device 3, and the processing of the management server 5 in FIG. 13 may also be executed by the terminal device 3.

[0081] 14 is an explanatory diagram showing an example of a setting screen according to this embodiment. Fig. 14 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S5 of the sequence diagram shown in Fig. 13.

[0082] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a user information display screen 1100 and a setting screen 1200.

[0083] The setting screen 1200 includes a point cloud setting screen 1210 , a processing setting screen 1220 , and an execute button 1230 .

[0084] The point cloud setting screen 1210 is a screen that accepts a point cloud setting operation for setting point cloud data indicating a three-dimensional point cloud used to execute point cloud processing, and the display control unit 34 displays point cloud setting areas 1212, 1214 in association with the file names of the plurality of point cloud data read by the storage / reading unit 59. A plurality of point cloud setting areas 1212, 1214 can be set.

[0085] The processing setting screen 1220 is a screen that accepts setting operations for setting the type of point cloud processing, and the display control unit 34 displays processing setting areas 1221, 1222, and 1223 in association with the names of multiple point cloud processes. The display control unit 34 also displays an execute button 1230 for confirming various setting operations.

[0086] Furthermore, the display control unit 34 displays a pointer 1240 such as a mouse 312 for selecting the processing setting areas 1221 to 1223.

[0087] When the various setting areas are clicked with the pointer 1240 in the reception unit 32 of the terminal device 3, the display control unit 34 displays a black circle or a check mark in the various setting areas as shown in the figure. When the reception unit 32 receives the various setting operations and the execute button 1230 is operated, the various setting operations are completed and point cloud processing is executed.

[0088] Specifically, as described in steps S6 and S7 of FIG. 11, the transmitter / receiver 31 transmits input information including various setting information based on various setting operations received by the reception unit 32 to the management server 5, and the processing unit 53 performs point cloud processing (3D model processing in the example of FIG. 14).

[0089] 15 is an explanatory diagram showing an example of a selection screen according to this embodiment. A selection screen 1300 including a processing result area 1310, an up / down button 1320, a left / right button 1330, a continue button 1380, and an end button 1390 is displayed on the display screen 1000. The processing result area 1310 also includes a processing result screen 1311 (such as point cloud data including the detected bounding box and data in which the bounding box has been replaced with a 3D model). Note that FIG. 15 shows the display screen 1000 displayed on the display 308 of the terminal device 3 in step S9 of the sequence diagram shown in FIG. 13, and the display control unit 34 of the terminal device 3 causes the selection screen 1300 to be displayed on the display screen 1000.

[0090] The processing result screen 1311 in FIG. 15 displays point cloud data (similar to data in FIG. 6( a)) including the detected bounding box. The user can change the viewpoint up, down, left, or right by clicking an up / down button 1320 or a left / right button 1330 using a pointer 1340, such as the mouse 312, to view the processing result screen from the desired viewpoint. When the user clicks the continue button 1380 to continue the 3D model processing, the processing unit 53 executes the following processes: converting the 3D model into a point cloud, rotating the 3D model, estimating the orientation of the bounding box, and replacing the bounding box with a 3D model. The generation unit 57 generates a display screen that displays each of the processing results. The user can continue the 3D model processing, check the results of the replacement with the 3D model on the processing result screen 1311, and then click the end button 1390 to terminate the 3D model processing. The user can also click the end button 1390 to interrupt the 3D model processing midway through.

[0091] Note that the operation by the user to check the processing result screen 1311 is not limited to changing the viewpoint, and may include, for example, an operation to enlarge or reduce a part of the processing result. Also, instead of the up / down button 1320 or the left / right button 1330, the viewpoint may be changed by operating the mouse 312 (such as by rotating the mouse wheel).

[0092] In this way, by using the setting screen 1200 and the selection screen 1300, the user can perform 3D model processing on the desired point cloud data, and can easily check the results of the 3D model processing using the mouse 312, etc.

[0093] As described above, according to this embodiment, the orientation of the bounding box is estimated using a 3D model, so that the bounding box can be appropriately replaced with the 3D model.

[0094] (Second embodiment) Next, a second embodiment will be described.

[0095] The second embodiment allows the user to check the detected unoriented bounding box and modify (add, delete, etc.) the bounding box as necessary. When the user wants to modify the results of 3D model processing, modifying the bounding box or 3D model, including its orientation, is time-consuming and burdensome for the user. Therefore, in this embodiment, the burden on the user is reduced by modifying the unoriented bounding box at the stage when it is detected.

[0096] 16 is a functional block diagram of a 3D model processing unit 530 according to the second embodiment. The 3D model processing unit 530 in this embodiment has a detection unit 531, an estimation unit 532, a replacement unit 533, and a correction unit 534. In the following description of the second embodiment, descriptions of the same parts as in the first embodiment will be omitted, and differences from the first embodiment will be described.

[0097] The operations of the detection unit 531, the estimation unit 532, and the replacement unit 533 are the same as those in the first embodiment, and therefore will not be described. The correction unit 534 deletes the bounding box detected by the detection unit 531 or adds a new bounding box in accordance with a user operation.

[0098] FIG. 17 is a diagram showing an example of a selection screen according to this embodiment. The difference from the first embodiment is that a processing result area 1310 includes a delete button 1312 and an add button 1313. As shown in FIG. 17, when the processing result screen 1311 displays the detected bounding boxes, the user can operate the pointer 1340 to select a bounding box. For example, the selected bounding box changes color or flashes, allowing the user to visually identify which bounding box is selected. When the delete button 1312 is clicked while a bounding box is selected, the correction unit 534 deletes the selected bounding box.

[0099] Furthermore, after clicking add button 1313, the user can operate pointer 1340 to add a new bounding box to processing result screen 1311. For example, if the user selects the class of the bounding box they want to add from a pull-down menu or the like and clicks and specifies the desired coordinates on processing result screen 1311, correction unit 534 reads out the bounding box data corresponding to the selected class from storage processing management DB 5003 via storage / readout unit 59 and adds it to the specified coordinates.

[0100] Here, the storage processing management DB 5003 stores bounding box data, 3D model data, model point cloud data, etc. for each class. The pull-down menu can be displayed by operating the mouse 312 button, by providing a menu display button in the processing result area 1310, or by other means. The correction unit 534 can also perform correction processes other than the above-mentioned deletion and addition. For example, it may be possible to change the class or size of a selected bounding box.

[0101] 18 is a flowchart showing an example of the procedure for 3D model processing according to this embodiment. The difference from the first embodiment is that after a bounding box is detected in step S180, the bounding box is corrected in step S181. The other procedures are the same as those in the first embodiment, and therefore their explanations are omitted. The correction unit 534 deletes a bounding box displayed on the processing result screen 1311 or adds a new bounding box, in accordance with a user operation on the selection screen 1300 (step S181).

[0102] As described above, according to this embodiment, the user can easily modify the detected bounding box by simply adding or deleting an orientationless bounding box, without having to specify or modify the orientation of the bounding box. Furthermore, since the processing unit 53 estimates the orientation of the bounding box using a 3D model, the bounding box can be appropriately replaced with the 3D model.

[0103] In the above embodiments, a bounding box has been used as an example of a partial region, but the partial region is not limited to this. For example, a segmentation mask (hereinafter referred to as a mask) obtained by instance segmentation may be used as a partial region.

[0104] FIG. 19 is a diagram illustrating instance segmentation. FIG. 19(a) shows an example of an image divided into regions by normal segmentation, and FIG. 19(b) shows an example of an object extracted from the image by instance segmentation. In the example of FIG. 19(b), a fruit class and a cup class are extracted, and their respective masks are shown. When using masks as subregions, the detection unit 531 detects masks indicating objects based on a three-dimensional point cloud. Furthermore, the estimation unit 532 estimates the orientation of the mask based on a 3D model corresponding to the detected mask class and the three-dimensional point cloud, and the replacement unit 533 replaces the mask with the 3D model using the estimated orientation. In this way, the masks obtained by instance segmentation can be used as subregions.

[0105] The point cloud processing program executed by the point cloud processing device of each of the embodiments described above is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0106] The point cloud processing program executed by the point cloud processing device of each embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The point cloud processing program executed by the point cloud processing device of each embodiment may be provided or distributed via a network such as the Internet.

[0107] The point cloud processing program of each embodiment may be provided by being pre-installed in a ROM or the like.

[0108] The point cloud processing program executed by the point cloud processing device of each embodiment has a modular structure including the above-mentioned units (detection unit 531, estimation unit 532, replacement unit 533, etc.), and in actual hardware, the CPU (processor) reads and executes the point cloud processing program from the above-mentioned storage medium, thereby loading the above-mentioned units onto the main memory device, and the detection unit 531, estimation unit 532, replacement unit 533, etc. are generated on the main memory device.

[0109] Each function of each of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0110] Additionally, the devices described in each embodiment represent only one of multiple computing environments for implementing each embodiment disclosed herein.

[0111] In one embodiment, the management server 5 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, the terminal device 3 may include multiple computing devices configured to communicate with each other.

[0112] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These requirements can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form.

[0113] For example, aspects of the present invention are as follows. <1> This point cloud processing device includes a detection unit that detects a partial area representing an object based on a three-dimensional point cloud obtained by measuring the object, an estimation unit that estimates the orientation of the partial area based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud, and a replacement unit that replaces the partial area with the three-dimensional model based on the orientation estimated by the estimation unit. <2> the estimation unit estimates the orientation of the partial region by rotating the three-dimensional model relative to the three-dimensional point cloud included in the partial region; <1> The point cloud processing device is described in <3> the estimation unit estimates the orientation of the partial region by comparing a model point cloud, which is a point cloud obtained by converting the three-dimensional model, with the three-dimensional point cloud included in the partial region; <1> or <2> The point cloud processing device is described in <4> a display screen generation unit that generates a display screen that displays the three-dimensional point cloud and the partial region, <1> ~ <3> The point cloud processing device according to any one of the above items. <5> the display screen generation unit generates a display screen that further displays a model point cloud, which is a point cloud obtained by converting the three-dimensional model. <4> The point cloud processing device is described in <6> the display screen generation unit generates a display screen in which the three-dimensional model or the model point cloud is rotated relative to the three-dimensional point cloud and displayed; <4> or <5> The point cloud processing device is described in <7> a display control unit that displays the display screen on a display unit; <4> ~ <6> The point cloud processing device according to any one of the above items. <8> The image processing device further includes a correction unit that corrects the partial area detected by the detection unit, and the correction unit deletes the partial area or adds a new partial area. <1> ~ <6> The point cloud processing device according to any one of the above items. <9> The partial region is a bounding box obtained by object detection. <1> ~ <6> The point cloud processing device according to any one of the above items. <10> The subregion is a segmentation mask obtained by instance segmentation. <1> ~ <6> The point cloud processing device according to any one of the above items. <11> This point cloud processing method includes a detection step of detecting a partial region representing an object based on a three-dimensional point cloud obtained by measuring the object; an estimation step of estimating the orientation of the partial region based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud; and a replacement step of replacing the partial region with the three-dimensional model based on the orientation estimated in the estimation step. <12> This program causes a computer to function as a detection means that detects a partial area representing an object based on a three-dimensional point cloud obtained by measuring the object, an estimation means that estimates the orientation of the partial area based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud, and a replacement means that replaces the partial area with the three-dimensional model based on the orientation estimated by the estimation means. <13> The point cloud processing system includes a point cloud processing device and a terminal device capable of communicating with the point cloud processing device, and includes a detection unit that detects a partial area representing an object based on a three-dimensional point cloud obtained by measuring an object, an estimation unit that estimates the orientation of the partial area based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud, and a replacement unit that replaces the partial area with the three-dimensional model based on the orientation estimated by the estimation unit. [Explanation of symbols]

[0114] 1. Point cloud processing system 3 Terminal Devices 5 Management Server 32 Reception Department 34 Display control unit 39 Memory / readout section 53 Processing section 57 Generation part 100 Communication Network 308 Display 311 Keyboard 312 Mouse 530 3D model processing section 531 Detection unit 532 Estimation part 533 Substitution part 534 Correction section 1300 Selection Screen 1310 Processing Result Area 1311 Processing result screen 1320 Up and Down Buttons 1330 left and right buttons [Prior art documents] [Patent documents]

[0115] [Patent Document 1] Japanese Patent Application Publication No. 2023-103167

Claims

1. a detection unit that detects a partial region representing an object based on a three-dimensional point cloud acquired by measuring the object; an estimation unit that estimates the orientation of the partial region based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud; a replacement unit that replaces the partial region with the three-dimensional model based on the orientation estimated by the estimation unit.

2. The point cloud processing device according to claim 1 , wherein the estimation unit estimates the orientation of the partial region by rotating the three-dimensional model relative to the three-dimensional point cloud included in the partial region.

3. The point cloud processing device according to claim 2 , wherein the estimation unit estimates the orientation of the partial region by comparing a model point cloud, which is a point cloud obtained by converting the three-dimensional model, with the three-dimensional point cloud included in the partial region.

4. The point cloud processing device according to claim 3 , further comprising a display screen generation unit that generates a display screen for displaying the three-dimensional point cloud and the partial region.

5. The point cloud processing apparatus according to claim 4 , wherein the display screen generation unit generates a display screen that further displays the three-dimensional model or the model point cloud.

6. The point cloud processing apparatus according to claim 5 , wherein the display screen generation unit generates a display screen in which the three-dimensional model or the model point cloud is rotated relative to the three-dimensional point cloud and displayed.

7. The point cloud processing device according to claim 4 , further comprising a display control unit that causes the display screen to be displayed on a display unit.

8. a correction unit that corrects the partial region detected by the detection unit, The point cloud processing apparatus according to claim 1 , wherein the correction unit deletes the partial region or adds a new partial region.

9. The point cloud processing device according to claim 1 , wherein the partial region is a bounding box obtained by object detection.

10. The point cloud processing device according to claim 1 , wherein the partial region is a segmentation mask obtained by instance segmentation.

11. a detection step of detecting a partial region representing an object based on a 3D point cloud obtained by measuring the object; an estimation step of estimating the orientation of the partial region based on a 3D model corresponding to the type of object and the 3D point cloud; a replacement step of replacing the partial region with the three-dimensional model based on the orientation estimated in the estimation step.

12. Computer, a detection means for detecting a partial region representing an object based on a three-dimensional point cloud acquired by measuring the object; an estimation means for estimating the orientation of the partial region based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud; a replacement means for replacing the partial region with the three-dimensional model based on the orientation estimated by the estimation means.

13. A point cloud processing system comprising a point cloud processing device and a terminal device capable of communicating with the point cloud processing device, a detection unit that detects a partial region representing an object based on a three-dimensional point cloud acquired by measuring the object; an estimation unit that estimates the orientation of the partial region based on a three-dimensional model corresponding to the type of object and the three-dimensional point cloud; a replacement unit that replaces the partial region with the three-dimensional model based on the orientation estimated by the estimation unit.

Images