Point cloud processing apparatus, point cloud processing method, recording medium, and point cloud processing system

Abstract

A point cloud processing apparatus includes display screen generating means for generating a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object.

Description

The present disclosure relates to a point cloud processing apparatus, a point cloud processing method, a recording medium, and a point cloud processing system. [Background Art]

[0002] PTL 1 describes a point cloud data processing apparatus that includes a highlight control unit. When a first point cloud including multiple markers for alignment and a second point cloud including multiple markers for alignment are displayed, in response to the designation of a maker on one of the first and second point clouds, the highlight control unit highlights the corresponding marker on the other point cloud. Accordingly, the efficiency of a task of aligning a plurality of point cloud data is enhanced. [Citation List] [Patent Literature]

[0003] [PTL 1] Japanese Patent No. 6910820 [Summary of Invention] [Technical Problem]

[0004] An object of the present disclosure is to align multiple point clouds appropriately. [Solution to Problem]

[0005] The present disclosure described herein provides a point cloud processing apparatus including display screen generating means for generating a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object. The present disclosure described herein provides a point cloud processing method including generating a display screen in which a plurality of point clouds representing a threedimensional point cloud obtained by measuring an object are superimposed on an image representing the object. The present disclosure described herein provides a recording medium storing a program code for causing a computer to generate a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object. The present disclosure described herein provides a point cloud processing system including a point cloud processing apparatus and a terminal apparatus communicable with the point cloud processing apparatus. The point cloud processing apparatus includes display screen generating means for generating a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object and transmitting means for transmitting display screen information indicating the display screen to the terminal apparatus. The terminal apparatus includes receiving means for receiving the display screen information transmitted from the point cloud processing apparatus and display controlling means for displaying the display screen on a display. [Advantageous Effects of Invention]

[0006] According to one or more embodiments of the present disclosure, the alignment of multiple point clouds is performed appropriately. [Brief Description of Drawings]

[0007] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings. [FIG. 1] FIG. 1 is a diagram illustrating an overall configuration of a point cloud processing system according to a first embodiment of the present disclosure. [FIG. 2] FIG. 2 is a block diagram illustrating a hardware configuration of a terminal apparatus or a management server of the point cloud system of FIG. 1. [FIG. 3] FIG. 3 is a block diagram illustrating a functional configuration of the point cloud processing system of FIG. 1. [FIG. 4] FIG. 4 is a block diagram illustrating a functional configuration of an alignment processing unit according to the first embodiment. [FIG. 5] FIG. 5 is a diagram illustrating an example of drawing data. [FIG. 6] FIG. 6 is a diagram illustrating an example of the matching candidate image. [FIG. 7] FIG. 7 is a diagram illustrating an example of a matching candidate point cloud. [FIG. 8A] FIG. 8A is a diagram illustrating examples of end points or intersection points of straight lines. [FIG. 8B] FIG. 8B is a diagram illustrating examples of end points or intersection points of straight lines. [FIG. 9] FIG. 9 is a diagram illustrating an example of a selection screen according to the first embodiment. [FIG. 10] FIG. 10 is a flowchart of an operation of alignment processing of a point cloud according to the first embodiment. [FIG. 11] FIG. 11 is a sequence diagram illustrating point cloud processing according to the first embodiment. [FIG. 12] FIG. 12 is a diagram for describing an example of a setting screen according to the first embodiment. [FIG. 13] FIG. 13 is a block diagram illustrating a functional configuration of an alignment processing unit according to a second embodiment. [FIG. 14A] FIG. 14A is a diagram illustrating an example of attribute information. [FIG. 14B] FIG. 14B is a diagram illustrating an example of attribute information. [FIG. 14C] FIG. 14C is a diagram illustrating an example of attribute information. [FIG. 15] FIG. 15 is a diagram illustrating a part of a superimposition screen on which the attribute information of FIG. 14A to FIG. 14C is displayed. [FIG. 16] FIG. 16 is a block diagram illustrating a functional configuration of an alignment processing unit according to a third embodiment. [FIG. 17] FIG. 17 is a flowchart of an operation of alignment processing of a point cloud according to the third embodiment. The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views. [Description of Embodiments]

[0008] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result. Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0009] In the fields of civil engineering and architecture, the implementation of building information modeling (BIM) / construction information modeling (CIM) has been promoted for coping with, for example, the demographic shift towards an older population and enhancing labor efficiency and productivity. BIM is a solution that involves utilizing a database of buildings, in which attribute data such as cost, finishing details, and management information, is added to a three-dimensional (3D) digital model of a building. This model is created on a computer and utilized throughout every stage of the architectural process, including design, construction, and maintenance. In the following description, the three-dimensional digital model may be referred to as a “3D model.” CIM is a solution that has been proposed for the field of civil engineering (covering general infrastructure such as roads, electricity, gas, water supply, etc.,) following BIM that has been advancing in the field of architecture. CIM is being pursued, similar to BIM, to aim for efficiency and advancement in a series of construction production systems by sharing information through centralized 3D models among participants.

[0010] A matter of concern for promoting BIM / CIM implementation is how to easily obtain 3D information on an object such as a building or a public facility. The term “3D information” used in the following description refers to, for example, a three-dimensional point cloud retaining distance information of a distance to the object acquired by measuring the object with, for example, a laser scanner, a mesh object generated based on point cloud data representing a three-dimensional point cloud, or a 3D computer-aided design (CAD) model. In the following description, the three-dimensional point cloud may also be referred to simply as a “point cloud.”

[0011] When a structure is constructed from scratch, BIM / CIM software can be used to design a completed structure from scratch. Accordingly, the introduction of BIM / CIM is easier. On the other hand, regarding the existing buildings, there is a case where, for example, a design drawing at the time of construction does not remain, or a current situation is different from a drawing at the time of design due to modification with the lapse of time, and the barriers for the BIM / CIM implementation are high. Such BIM of an existing building is called, for example, “As-Build BIM,” and is a challenge for promoting BIM / CIM in the future.

[0012] As a way to achieve the As-Build BIM, a workflow of performing spatial measurement using the above-described laser scanner and creating a 3D-CAD model from point cloud data obtained by the spatial measurement. This task has been performed in the related art by using, for example, a method of measuring with photographs, a method of measuring manually, or a method of sketching by hand. However, in such an approach using the known method described above, significant costs may arise due to factors such as the size of a space, the presence of an object in a space, or the complexity of a space (e.g., complexity of pipe arrangement). As an effective way to address such an issue, the introduction of a laser scanner that acquires 3D information on a space has been gaining attention.

[0013] In the As-Build using the laser scanner, acquisition of the 3D information is facilitated, however, a task of point cloud processing performed on point cloud data, which has not been present in the known art, is added. Typical point cloud processing includes multipoint measurement using a laser scanner, aligning individual point clouds to construct an integrated point cloud, removing unnecessary points such as noise, converting the point cloud into a mesh, mapping a texture on the mesh, converting the point cloud into a 3D-CAD model, and alignment processing.

[0014] In the processing of aligning a first point cloud with a second point cloud, when the two point clouds are acquired (image) with a wall interposed therebetween, there is no common part between the two point clouds. As a result, the alignment is not performed properly. Further, in order to create a common part (overlap) between the point clouds, one has to intentionally create the overlap at the time of imaging, such as imaging in fragments between one room and the other room, or imaging with the door open so that the point cloud of the adjacent room can be obtained.

[0015] In view of the above-described issue, a first object of the present disclosure is to align multiple point clouds appropriately even when there is no common part between the point clouds.

[0016] A second object of the present disclosure is to reduce the burden on a user in post-processing after the alignment processing is performed when the result of the alignment does not achieve the desired quality.

[0017] First Embodiment FIG. 1 is a diagram illustrating an overall configuration of a point cloud processing system 1 according to a first embodiment of the present disclosure. The point cloud processing system 1 according to the present embodiment includes a terminal apparatus 3 and a management server 5. The terminal apparatus 3 is an example of a communication terminal.

[0018] FIG. 1 is a diagram illustrating an overall configuration of the point cloud processing system 1 according to the present embodiment. The point cloud processing system 1 according to the present embodiment includes the terminal apparatus 3 and the management server 5. The terminal apparatus 3 is an example of a communication terminal.

[0019] The management server 5 is an example of a point cloud processing apparatus that performs one or more point cloud processing operations on point cloud data representing a threedimensional point cloud.

[0020] The three-dimensional point cloud is defined as a collection of, or a set of coordinate points in directions such as the X, Y, and Z directions that correspond to measurement points on the surface of an object when a certain space in which the object is present is measured using, for example, a laser scanner. For example, each of the coordinate points is indicated as (1,3,5). Further, color information may be added to each of the coordinate points. The color information may be a red-green-blue (RGB) value of the coordinate point. The threedimensional point cloud may also be referred to as a “point cloud.” The point cloud data is data that represents a collection of, or a set of coordinate points, which corresponds to a threedimensional point cloud, in a virtual three-dimensional space, and that can be processed by, for example, a computer.

[0021] The description given above is of a case where the three-dimensional point cloud is measured using the laser scanner. Alternatively, the three-dimensional point cloud may be measured using another optical measurement means or a mechanical measurement means. Examples of another optical measuring means include a method using a stereo camera and a method using visual simultaneous localization and mapping (SLAM).

[0022] The terminal apparatus 3 and the management server 5 can communicate with each other through a communication network 100. The communication network 100 includes, for example, the Internet, a mobile communication network, and a local area network (LAN). The communication network 100 may include, in addition to a wired network, a wireless network in compliance with such as 3rd Generation (3G), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE), 5th Generation (5G), etc. Further, the terminal apparatus 3 can establish communication using a short-range communication technology such as NEAR FIELD COMMUNICATION (NFC).

[0023] Hardware Configuration FIG. 2 is a block diagram illustrating a hardware configuration of each of the terminal apparatus 3 and the management server 5 according to the present embodiment. The hardware components of the terminal apparatus 3 are denoted by reference numerals in the 300s. The hardware components of the management server 5 are denoted by parenthesized numerals in the 500s.

[0024] The terminal apparatus 3 includes a central processing unit (CPU) 301, a read-only memory (ROM) 302, a random-access memory (RAM) 303, a hard disk (HD) 304, a hard disk drive (HDD) 305, a storage medium 306, a medium interface (FF) 307, a display 308, a network FF 309, a keyboard 311, a mouse 312, a compact disc-rewritable (CD-RW) drive 314, and a bus line 310.

[0025] The CPU 301 controls the overall operation of the terminal apparatus 3. The ROM 302 stores a program for driving the CPU 301. The RAM 303 is used as a working area for the CPU 301. The HD 304 stores various data such as a program. The HDD 305 controls the reading and writing of various data to and from the HD 304 under the control of the CPU 301. The medium FF 307 controls the reading and writing of various data from and to the storage medium 306. The display 308 displays various information such as a cursor, a menu, a window, a character, or an image. The network FF 309 is an interface for transmitting and receiving data (performing communications) through the communication network 100. The keyboard 311 is an input device including multiple keys that allow a user to input, for example, characters, numerals, or various instructions. The mouse 312 is another input device that allows a user to perform an operation for, for example, selecting or executing various instructions, selecting objects to be processed, or moving a cursor being displayed. The CD-RW drive 314 controls the reading and writing of various data from and to the CD-RW 313, which is an example of a removable storage medium. The terminal apparatus 3 may have a configuration that controls the reading and writing (storing) of data from and to an external personal computer (PC) or an external device connected by wire or wirelessly such as wireless fidelity (Wi-Fi).

[0026] The management server 5 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD 505, a medium FF 507 that controls the reading and writing of data from and to a storage medium 506, a display 508, a network FF 509, a keyboard 511, a mouse 512, a CD-RW drive 514, and a bus line 510. Since these hardware elements are the same or substantially the same as the CPU 301, the ROM 302, the RAM 303, the HD 304, the HDD 305, the storage medium 306, the medium I / F 307, the display 308, the network I / F 309, the keyboard 311, the mouse 312, the CD-RW drive 314, and the bus line 310, redundant descriptions thereof are omitted below.

[0027] The CD-RW drive 314 (514) may be replaced with, for example, a compact disc-recordable (CD-R) drive. Each of the terminal apparatus 3 and the management server 5 may be implemented by a single computer or multiple computers to which divided units (functions, means, or storage units) are allocated as desired.

[0028] FIG. 3 is a block diagram illustrating the functional configuration of the point cloud processing system 1 according to the present embodiment.

[0029] Functional Configuration of Terminal Apparatus As illustrated in FIG. 3, the terminal apparatus 3 includes a transmission / reception unit 31, a reception unit 32, a display control unit 34, and a storing / reading unit 39. These units of functions are implemented by or caused to function by one or more of the components illustrated in FIG. 2 operating according to instructions from the CPU 301 according to the program loaded from the HD 304 to the RAM 303. The terminal apparatus 3 further includes a storage unit 3000 implemented by the RAM 303 and the HD 304 illustrated in FIG. 2.

[0030] Functional Units of Terminal Apparatus Each functional unit of the terminal apparatus 3 is described below.

[0031] The transmission / reception unit 31 is implemented by instructions from the CPU 301 and the network PF 309 illustrated in FIG. 2 and transmits and receives various data (or information) to and from another terminal, apparatus, or system through the communication network 100. The transmission / reception unit 31 is an example of receiving means.

[0032] The reception unit 32 is implemented by instruction from the CPU 301, the keyboard 311, and the mouse 312 illustrated in FIG. 2, and receives various inputs from a user. The reception unit 32 is an example of accepting means.

[0033] The display control unit 34 is implemented by instructions from the CPU 301 illustrated in FIG. 2 and controls the display 308 to display various images and screens. The display control unit 34 is an example of display controlling means. The display 308 is an example of a display unit.

[0034] The storing / reading unit 39, which is an example of storage controlling means, is implemented by instructions from the CPU 301, the HDD 305, the medium PF 307, and the CD-RW drive 314 illustrated in FIG. 2 and external PCs or external devices. The storing / reading unit 39 writes and reads various types of data to and from the storage unit 3000, the storage medium 306, the CD-RW 313, and the external PCs or external devices.

[0035] Functional Configuration of Management Server The management server 5 includes a transmission / reception unit 51, a processing unit 53, a determination unit 55, a generation unit 57, and a storing / reading unit 59. These units of functions are implemented by or caused to function by one or more of the components illustrated in FIG. 2 operating according to instructions from the CPU 501 according to the program loaded from the HD 504 to the RAM 503. The management server 5 further includes a storage unit 5000 implemented by the HD 504 illustrated in FIG. 2. The storage unit 5000 is an example of storing means.

[0036] Functional Units of Management Server Each functional unit of the management server 5 is described below. The management server 5 may be implemented by multiple computers to which the functions thereof are allocated. Although the management server 5 is a server computer that resides in a cloud environment in the following description, alternatively, the management server 5 may be a server that resides in an on-premises environment.

[0037] The transmission / reception unit 51 is implemented by instructions from the CPU 501 and the network RF 509 illustrated in FIG. 2 and transmits and receives various data (or information) to and from another terminal, apparatus, or system through the communication network 100. The transmission / reception unit 51 is an example of transmitting means.

[0038] The processing unit 53 is implemented by instructions from the CPU 501 illustrated in FIG. 2 and performs various processing including alignment processing. The processing unit 53 is an example of point cloud processing means.

[0039] The determination unit 55 is implemented by instructions from the CPU 501 illustrated in FIG. 2 and performs various types of determination.

[0040] The generation unit 57 is implemented by instructions from the CPU 501 illustrated in FIG. 2 and performs various types of generation such as screen generation described below.

[0041] The storing / reading unit 59, which is an example of storage controlling means, is implemented by instructions from the CPU 501, the HDD 505, the medium RF 507, and the CD-RW drive 514 illustrated in FIG. 2 and external PCs or external devices. The storing / reading unit 59 writes and reads various types of data to and from the storage unit 5000, the storage medium 506, the CD-RW 513, and the external PCs or external devices. The storage unit 5000, the storage medium 506, the CD-RW 513, and the external PCs or the external devices are examples of storing means.

[0042] The storage unit 5000 includes a user information management database (DB) 5001, a setting information management DB 5002, a storage management DB 5003, a point cloud management DB 5004, and a processing result management DB 5005 that are implemented by a setting information management table.

[0043] In the user information management DB 5001, a file name of three-dimensional point cloud data is stored and managed in association with user information. In the setting information management DB 5002, various setting information is stored and managed. In the storage management DB 5003, various processing programs for performing point cloud processing are stored and managed. In the point cloud management DB 5004, point cloud data is stored and managed. In the processing result management DB 5005, processing result information indicating a processing result of point cloud processing performed on point cloud data is stored and managed.

[0044] Functional Configuration of Alignment Processing Unit The processing unit 53 includes an alignment processing unit 530 that performs alignment processing on a point cloud. FIG. 4 is a block diagram illustrating the functional configuration of the alignment processing unit 530 according to the present embodiment. The alignment processing unit 530 includes a drawing analysis unit 531, a point cloud analysis unit 532, a pair selection unit 533, a position estimation unit 534, and a display image generation unit 535.

[0045] The drawing analysis unit 531 analyzes drawing data and generates a matching candidate image of a wall, a pillar, etc., in the drawing. The drawing data that is input to the drawing analysis unit 531 is image data obtained by converting a drawing such as a plan view of a building or a public facility for which a point cloud data is to be acquired into an image. The drawing data is provided in, for example, a bitmap format or a joint photographic experts group (JPEG) format. The drawing data may be image data obtained by reading a paper drawing with, for example, a scanner.

[0046] FIG. 5 is a diagram illustrating an example of the drawing data. The drawing data illustrated in FIG. 5 is a plan view, in which walls, pillars, etc. of a building are represented by straight lines, and doors and windows are represented by predetermined symbols. The walls are often represented by a combination of rectangles having thick sides.

[0047] FIG. 6 is a diagram illustrating an example of the matching candidate image. The drawing analysis unit 531 analyzes the drawing data using edge detection and straight line detection and generates the matching candidate image as illustrated in FIG. 6. Specifically, when a pixel detected as an edge is on pixels detected as a straight line, the pixel is a matching candidate pixel. In FIG. 6, a white area indicates pixels (candidate pixels) detected as matching candidates, and a black area indicates pixels other than the candidate pixels. As described above, in the generated matching candidate image, the surfaces of walls and pillars are detected as the matching candidate pixels. As illustrated in FIG. 6, the drawing analysis unit 531 determines an area in which the symbol representing a door is present in the plan view and excludes such an area from an object on which the edge detection is to be performed.

[0048] In the present embodiment, it is assumed that the scale (reduction scale ratio) of the drawing data is known. The length on the matching candidate image and the length of a point cloud on which the alignment processing is to be performed are compared using such a scale. For example, when it is known from a known scale that the length of 1000 pixels of the matching candidate image corresponds to 10 meters (m) on the drawing, the comparison between the length on the matching candidate image and the point cloud is performed with the length of one pixel being set to 10 / 1000 = 0.01 m. When the scale is not known, a tentative scale is set and the actual scale is estimated in the alignment processing, as in the embodiment described below.

[0049] The point cloud analysis unit 532 extracts a matching candidate point cloud belonging to a wall, a pillar, etc. from point cloud data obtained by measuring, for example, a room. It is assumed that the point cloud data to be input is already corrected so that the horizontality is maintained with the vertical coordinate (Z-coordinate) of the floor surface being zero meter. Further, it is assumed that the value of the point cloud data is acquired in a known unit such as meter.

[0050] Drawing data to be compared with point cloud data is generally set as a cross section assuming a certain height from the floor surface. For this reason, the point cloud analysis unit 532 extracts a point cloud at a certain height from the floor surface from the point cloud data and sets the extracted point cloud as the matching candidate point cloud. For example, when a plan view corresponding to drawing data is created as a cross section at the height of one meter from the floor surface, only a point cloud near the height of one meter from the floor surface is extracted from the point cloud data. When the coordinates of points belonging to the point cloud data represented by three axes of X, Y, and Z are (px, py, pz), it is preferable to extract only points satisfying the following two inequalities as the matching candidate point cloud in view of the accuracy of the point cloud data. The range h of the height is preferably obtained experimentally according to the density and accuracy of the point cloud.

[0051] Pz < l+h / 2 pz > l-h / 2

[0052] FIG. 7 is a diagram illustrating an example of the matching candidate point cloud. By appropriately setting the range h of the height in the above inequality, the matching candidate point cloud is generated as point cloud data on an XY plane that is parallel to the floor surface.

[0053] In the present embodiment, the matching candidate image generated by the drawing analysis unit 531 is matched with each of multiple matching candidate point clouds generated by the point cloud analysis unit 532 to align multiple pieces of point cloud data. The matching candidate point clouds are, for example, point clouds acquired in multiple rooms in a building. There is no common part between the point clouds. Alignment between the point clouds is enabled by aligning each of the point clouds with the position on the drawing data.

[0054] A known technique called “registration” is used for the alignment of point cloud data. For example, a correspondence point is obtained using an iterative closest point (ICP) algorithm or an algorithm using a fast point feature histogram (FPFH), and then rigid transformation is obtained for alignment using a random sample consensus (RANSAC) algorithm. Although such methods are primarily intended for a three-dimensional point cloud, the method can be used in a degenerate two-dimensional form. Also in the present embodiment, alignment for multiple pieces of point cloud data can be performed using these methods.

[0055] However, since the ICP algorithm highly depends on an initial value, the ICP algorithm mistakenly converge to a local minimum. For this reason, a drawback that a point cloud is not arranged at a desired position arises. In the algorithm using the FPFH feature, point cloud data is acquired only inside a wall. On the other hand, the drawing data includes data on both the outside and inside of the wall. Accordingly, a drawback that a common feature is not extracted, and a point cloud is not arranged at a desired position arises. In view of such drawbacks, the alignment processing unit 530 described above with reference to FIG. 4 uses a method of detecting a straight line part from the drawing data or the point cloud data and arranging a point cloud at a desired position by alignment processing using the end points or intersection points of the straight lines.

[0056] The pair selection unit 533 selects a pair of the end points or the intersection points of straight lines in each of the matching candidate image and the matching candidate point cloud. FIG. 8A and FIG. 8B are diagrams illustrating examples of the end points or the intersection points of the straight lines. FIG. 8A illustrates an example of the matching candidate image, and white points (white circles) in the figure indicate the end points and the intersection points of the straight lines. The pair selection unit 533 selects two points from these white points and sets the selected white points as a candidate pair of matching candidate images. FIG. 8B illustrates an example of the matching candidate point cloud, and black points (black circles) in the figure indicate the end points and the intersection points of the straight lines. The pair selection unit 533 selects two black points from these black points and sets the selected black points as a candidate pair of the matching candidate point cloud.

[0057] The position estimation unit 534 matches the candidate pair of the matching candidate image and the candidate pair of the matching candidate point cloud with each other and estimates a position at which the point cloud data to be relatively moved to the corresponding part in the drawing data using a rigid transformation unit 536 included in the position estimation unit 534. In the present embodiment, as will be described below, the selection of a candidate pair by the pair selection unit 533 and the estimation of a position by the position estimation unit 534 are repeatedly performed, and a position at which the point cloud that is relatively moved is to be superimposed on drawing data is determined.

[0058] The display image generation unit 535 generates an image in which each of the point clouds (matching candidate point clouds) is superimposed on the drawing data in accordance with the determined position of the point cloud. FIG. 9 is a diagram illustrating an example of a screen displayed on the display 308 of the terminal apparatus 3. On a display screen 1000, a selection screen 1300 including a processing result area 1310, a parallel movement button 1331, a rotation button 1332, and a confirmation button 1390 is displayed. The processing result area 1310 includes a superimposition screen 1350 in which the point clouds are superimposed on the drawing data.

[0059] On the superimposition screen 1350 illustrated in FIG. 9, three pieces of point cloud data (a first point cloud, a second point cloud, and a third point cloud) are superimposed on the drawing data. A user can confirm the result of the alignment processing by clicking the confirmation button 1390 using a pointer 1340 corresponding to, for example, the mouse 312. Further, as described below, the result of the alignment processing can be corrected for each of the point clouds using the parallel movement button 1331, the rotation button 1332, and the pointer 1340.

[0060] Although the description given above with reference to FIG. 9 is of a case where the multiple pieces of point cloud data are concurrently superimposed on the drawing data, the point clouds may be separately superimposed. For example, only the first point cloud is displayed superimposed on the drawing data at first. Then, after the user clicks the confirmation button 1390, only the second point cloud is displayed superimposed on the drawing data. Then, after the user clicks the confirmation button 1390, only the third point cloud is displayed superimposed on the drawing data.

[0061] FIG. 10 is a flowchart of the operation of the alignment processing of a point cloud. In step S100, the pair selection unit 533 selects a first candidate pair from the end points and the intersection points of a matching candidate point cloud for one point cloud data. In step S101, the pair selection unit 533 selects a second candidate pair from the end points and the intersection point of a matching candidate image.

[0062] In step S102, the position estimation unit 534 calculates a length LI of the first candidate pair and a length L2 of the second candidate pair. As described above, since the scale of the drawing data is known and the matching candidate point cloud and the matching candidate image are created at the same scale, the lengths of the two pairs can be calculated and compared in the same units (e.g., meter).

[0063] In step S103, when the difference between the lengths of the two pairs is large, the operation returns to S101, and when the difference is small, the operation proceeds to SI04. When the operation returns to S101, the pair selection unit 533 selects another second candidate pair in step S101. When the lengths of pairs are compared, the data contains errors. In view of this, for example, when the following two inequalities are satisfied, it is determined that the difference between the lengths of the two pairs is small, and when the following two inequalities are not satisfied, it is determined that the difference between the lengths of the two pairs is large. The value of th is experimentally determined taking into consideration data errors.

[0064] L2-th<Ll LI <L2+th

[0065] In step S104, based on the determination result indicating that the difference between the lengths of the two pairs is small, the position estimation unit 534 obtains a rigid transformation from the matching candidate point cloud to the matching candidate image so that the two pairs overlap each other. The rigid transformation is limited to parallel movement and rotation in an XY plane (horizontal plane).

[0066] Subsequently, the rigid transformation unit 536 moves the matching candidate point cloud relative to the matching candidate image using the derived rigid transformation. Through this relative movement, all the points belonging to the matching candidate point cloud are moved. In step S105, the position estimation unit 534 counts, for all the candidate pixels of the matching candidate image and all the points belonging to the matching candidate point cloud after the relative movement, the number of combinations of the candidate pixel and the point between which distance is equal to or less than a constant value d. The constant value d is preferably determined experimentally according to the resolution of the drawing data and the accuracy of the point cloud.

[0067] When it is determined that the number of combinations counted in SI05 is larger than the maximum value of the number of combinations in step S106, the candidate transformation and the maximum value of the number of combinations are updated in step SI07. Then, the operation proceeds to step S108. In step SI07, the candidate transformation is updated to the rigid transformations derived in step S104, and the maximum value of the number of combinations is updated to the number of combinations counted in step S105. At the start of this operation, the candidate transformation is set to identity transformation (where not all points are moved), and the maximum value of the number of combinations is set to zero. The initial settings of the candidate transformation and the maximum value of the number of combinations may be other than those described above. The description given above is of a case where the distances between the candidate pixels and the points are checked for all combinations of the candidate pixels and the points in the S105. However, the checking of the distance may be performed on a part of the combinations. For example, the position estimation unit 534 counts, for all the points belonging to the first candidate pair and all the candidate pixels belonging to the second candidate pair, the number of combinations of the candidate pixel and the point between which distance is equal to or less than the constant value d.

[0068] When it is determined that the counted number of combinations is equal to or less than the maximum value of the number of combinations in step S106, the candidate transformation and the maximum value of the number of combinations are not updated, and the operation proceeds to step S108. In step S108, the pair selection unit 533 determines whether all the second candidate pairs have been selected. When there is any second candidate pair that has not been selected, the operation returns to step S101. When there is no second candidate pair that has not been selected, the operation proceeds to step S109.

[0069] In step S109, the pair selection unit 533 determines whether all the first candidate pairs have been selected. When there is any first candidate pair that has not been selected, the operation returns to step S100. When there is no first candidate pair that has not been selected, the operation ends.

[0070] As described above, the pair selection unit 533 and the position estimation unit 534 repeat the processes from step S100 to step S109 until there is no combination of the first candidate pair and the second candidate pair on which the processes have not been performed. The rigid transformation set as the candidate transformation at the time when the operation ends is determined as the relative movement to be used for the alignment. When the above-described known algorithms are used in the alignment processing, the alignment processing unit 530 may be configured by replacing the pair selection unit 533 and the position estimation unit 534 with an alignment unit using the known method. In this case, the registration unit using the known method obtains a rigid transformation of the matching candidate point cloud. The obtained rigid transformation is used for relative movement described below by the display image generation unit 535.

[0071] The display image generation unit 535 generates an image in which the matching candidate point cloud that has been relatively moved (transformed) using the rigid transformation obtained by the pair selection unit 533 and the position estimation unit 534 is superimposed on the drawing data. The user can individually correct the result of the alignment processing by clicking the parallel movement button 1331 or the rotation button 1332 and then dragging the position of the point cloud using the pointer 1340. For example, when a certain point is dragged after the parallel movement button 1331 is clicked, the point cloud to which the certain point belongs may be moved in parallel. Further, when a certain point is dragged after the rotation button 1332 is clicked, the point cloud to which the certain point belongs may be rotated around a particular point. The user can correct the result of the alignment processing by performing these operations in combination while viewing the superimposed image. When the user determines that the result of the alignment is correct on the superimposed image, the user can fix the final position of the point cloud data by clicking the confirmation button 1390.

[0072] Although the matching candidate point cloud is two-dimensional point cloud data on a horizontal plane, the alignment of the height has been completed for the drawing data. Accordingly, as described above, the alignment processing is completed also for the threedimensional point cloud data by performing the parallel movement and the rotation around the vertical axis in the horizontal plane. Further, by performing the alignment processing on the pieces of point cloud data respectively corresponding to the rooms of the drawing data, the alignment processing for the three-dimensional point cloud data is completed for all the rooms.

[0073] FIG. 11 is a sequence diagram illustrating point cloud processing according to the present embodiment.

[0074] In step SI, the reception unit 32 of the terminal apparatus 3 receives an input operation relating to user information. In step S2, the transmission / reception unit 31 of the terminal apparatus 3 transmits, to the management server 5, a request for a setting screen including the user information received in step SI. The transmission / reception unit 51 of the management server 5 receives the request transmitted from the terminal apparatus 3.

[0075] In step S3, the storing / reading unit 59 of the management server 5 searches the user information management DB 5001 using the user information included in the request received in step S2 as a search key to read a file name of three-dimensional point cloud data associated with the user information included in the request, and the generation unit 57 of the management server 5 generates a display screen including a setting screen based on the file name read by the storing / reading unit 59.

[0076] The setting screen is a graphical user interface (GUI) screen including images of inputting means for inputting point cloud information (e.g., a file name of point cloud data), selecting means for selecting point cloud processing, inputting means for inputting drawing information (e.g., a file name of drawing data) used for alignment processing.

[0077] In step S4, the transmission / reception unit 51 transmits to the terminal apparatus 3 display screen information including setting screen information relating to the setting screen generated in step S3. The transmission / reception unit 31 of the terminal apparatus 3 receives the display screen information transmitted from the management server 5.

[0078] In step S5, the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the display screen including the setting screen received in step S4. The reception unit 32 of the terminal apparatus 3 receives an input operation performed by the user on the displayed setting screen. The input operation includes an operation of inputting point cloud information, information about point cloud processing, and drawing information. In the present embodiment, the user inputs information instructing alignment processing and drawing information used for the alignment processing as the information about the point cloud processing.

[0079] In step S6, the transmission / reception unit 31 transmits to the management server 5 input information relating to the input operation received by the reception unit 32. The transmission / reception unit 51 of the management server 5 receives the input information transmitted from the terminal apparatus 3. The input information includes the point cloud information, the information about point cloud processing, and the drawing information that are input.

[0080] The storing / reading unit 59 of the management server 5 searches the point cloud management DB 5004 using point cloud setting information included in the input information received in step S6 as a search key to read three-dimensional point cloud data and drawing data on which the alignment processing is to be performed.

[0081] Further, the storing / reading unit 59 searches the setting information management DB 5002 using the information about point cloud processing included in the input information received in step S6 as a search key to read a point cloud processing program. In the present embodiment, a program for alignment processing is read as the point cloud processing program.

[0082] In step S7, the processing unit 53 of the management server 5 generates point cloud processing information based on the three-dimensional point cloud data, the drawing data, and the alignment processing program that are read from the storing / reading unit 59. The point cloud processing information is information of a result obtained by performing the alignment processing on the point cloud data.

[0083] The step S7 is an example of a point cloud processing step of performing alignment processing on point cloud data representing three-dimensional point cloud.

[0084] The generation unit 57 of the management server 5 generates a display screen including the display of the point cloud processing information (the result of the alignment processing) and a selection screen that allows the user to select whether to end the alignment processing or to perform the alignment processing again. The selection screen is a GUI screen such as the selection screen 1300 as illustrated in FIG. 9. In step S8, the transmission / reception unit 51 transmits the generated display screen information to the terminal apparatus 3. Step S8 is an example of a generation step.

[0085] The transmission / reception unit 31 of the terminal apparatus 3 receives the display screen information transmitted from the management server 5. In step S9, the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the display screen based on the received display screen information. The reception unit 32 of the terminal apparatus 3 receives an operation input by the user on the display screen.

[0086] The input operation includes a selection operation of selecting whether to end the alignment processing or to perform the alignment processing again and an adjustment operation of adjusting the processing result of the alignment processing. The selection operation may be an operation of selecting one result from multiple results of the alignment processing.

[0087] In step S10, the transmission / reception unit 31 transmits input information relating to the input operation received by the reception unit 32 to the management server 5. The transmission / reception unit 51 of the management server 5 receives the input information transmitted from the terminal apparatus 3.

[0088] When the input information includes selection information indicating the end of the alignment processing, the processing unit 53 of the management server 5 determines the processing result of the alignment processing.

[0089] By contrast, when the input information includes selection information indicating that the alignment processing is to be performed again or adjustment information according to the adjustment operation, the processing unit 53 of the management server 5 performs the alignment processing in step S7 again based on the selection information or the adjustment information.

[0090] In step Sil, the processing unit 53 converts the processing result information into, for example, a file format readable by the point cloud processing software, a file format readable by the 3D-CAD software, or a file format readable by the BIM / CIM software, and the storing / reading unit 59 stores the converted processing result information in the processing result management DB 5005, the storage medium 506, or the CD-RW 513. Step S11 is an example of a storing step.

[0091] In step S12, the transmission / reception unit 51 transmits the determined processing result information to the terminal apparatus 3.

[0092] In step S13, the transmission / reception unit 31 of the terminal apparatus 3 receives the processing result information transmitted from the management server 5, and the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the received processing result.

[0093] In the above-described operation, the functional units of the management server 5 described with reference to FIG. 3 may be integrated into the terminal apparatus 3, and the processes by the management server 5 described with reference to FIG. 11 may be also performed by the terminal apparatus 3.

[0094] FIG. 12 is a diagram illustrating an example of the setting screen according to the present embodiment. FIG. 12 illustrates a display screen 1000 displayed on the display 308 of the terminal apparatus 3 in step S5 of the sequence diagram illustrated in FIG. 11.

[0095] The display control unit 34 of the terminal apparatus 3 displays a user information display screen 1100 and a setting screen 1200 on the display screen 1000.

[0096] The setting screen 1200 includes a point cloud setting screen 1210, a processing setting screen 1220, and an execution button 1230.

[0097] The point cloud setting screen 1210 is a screen for receiving a point cloud setting operation for setting point cloud data representing a three-dimensional point cloud to be used for the execution of the point cloud processing. The display control unit 34 displays point cloud setting areas 1212 and 1214 which are respectively associated with file names of multiple pieces of point cloud data read by the storing / reading unit 59. Multiple point cloud setting areas including the point cloud setting areas 1212 and 1214 can be set.

[0098] The processing setting screen 1220 is a screen for receiving a setting operation for setting a type of point cloud processing. The display control unit 34 displays processing setting areas 1221, 1222, and 1223 in association with the names of types of point cloud processing. When setting the alignment processing, drawing setting areas 1225 and 1226 are displayed in association with the file names of the drawing data. Multiple drawing setting areas including the drawing setting areas 1225 and 1226 can be set. The display control unit 34 further displays the execution button 1230 for confirming various setting operations.

[0099] Further, the display control unit 34 displays a pointer 1240 corresponding to, for example, the mouse 312 for selecting a desired one of the processing setting areas 1221 to 1223 and selecting a desired one of the drawing setting areas 1225 and 1226.

[0100] When the reception unit 32 of the terminal apparatus 3 receives the clicking operation on the desired ones of the above-described setting areas with the pointer 1240, the display control unit 34 displays a black circle or check mark in the setting areas as illustrated in FIG. 12. When the reception unit 32 receives an operation performed on the execution button 1230 after receiving the setting operations, the setting operations end, and the alignment processing is performed.

[0101] Specifically, as described with reference to steps S6 and S7 of FIG. 11, the transmission / reception unit 31 transmits input information including pieces of setting information corresponding to the setting operations received by the reception unit 32 to the management server 5, and the processing unit 53 performs the alignment processing.

[0102] As described above, FIG. 9 is a diagram illustrating an example of the selection screen 1300 according to the present embodiment. FIG. 9 illustrates the display screen 1000 displayed on the display 308 of the terminal apparatus 3 in step S9 of the sequence diagram of FIG. 11, and the display control unit 34 of the terminal apparatus 3 displays the selection screen 1300 on the display screen 1000.

[0103] As described above, by using the setting screen 1200 or the selection screen 1300, a user can perform the alignment processing on desired point cloud data and can easily correct the alignment result using, for example, the mouse 312.

[0104] In the present embodiment, an example is described in which, for each point cloud data, only one alignment result of one point cloud data is displayed superimposed on drawing data. Alternatively, multiple alignment candidates of one point cloud data may be presented to a user. For example, multiple rigid transformations corresponding to the larger number of combinations counted in step S105 of FIG. 10 may be left as candidates, and pieces point cloud data obtained by performing relative movement with the rigid transformations may be displayed on the superimposition screen 1350 concurrently or in a switching manner. The user can fix the position of the point cloud data by checking the positions of the pieces of point group data and selecting desired one of the positions on the superimposition screen 1350.

[0105] As described above, according to the present embodiment, by aligning point clouds with drawing data, the point clouds are aligned even when there is no common part between the point clouds. Further, when the alignment does not achieve a desired quality after the alignment processing of the point cloud is executed, the burden on a user for post-processing is reduced by adjusting the position of the point cloud while checking the point cloud superimposed on the drawing data.

[0106] Second Embodiment In the first embodiment, when multiple rooms having the same shape are present in drawing data, it is difficult to determine with which room a matching candidate point cloud having a shape similar to the shapes of the rooms is to be aligned. In a second embodiment, when the positions of the doors or the positions of the windows of such rooms are not the same, a matching candidate point cloud can be aligned.

[0107] FIG. 13 is a block diagram illustrating the functional configuration of the alignment processing unit 530 according to the second embodiment. The alignment processing unit 530 according to the present embodiment includes the drawing analysis unit 531, the point cloud analysis unit 532, an attribute addition unit 537, the pair selection unit 533, the position estimation unit 534, and the display image generation unit 535. In the second embodiment, elements, members, components, functions, or operations that are different from those of the first embodiment are described.

[0108] The drawing analysis unit 531 and the point cloud analysis unit 532 operate in the same or substantially the same manner as the first embodiment. The attribute addition unit 537 adds attribute information such as information about, for example, a wall, a door, and a window to the matching candidate image generated by the drawing analysis unit 531 and the matching candidate point cloud generated by the point cloud analysis unit 532. In the case of drawing data, the attribute information is determined according to which part of the drawing a candidate pixel of the matching candidate image is extracted from. FIG. 14A and FIG. 14B are diagrams illustrating examples of the attribute information added to each of the candidate pixels of the matching candidate image. In FIG. 14A and FIG. 14B, a circle indicates attribute information added to a pixel classified as a wall, a cross indicates attribute information added to a pixel classified as a door, and a square indicates attribute information added to a pixel classified as a window. These pieces of attribute information are associated with category numbers such as 1,2, and 3.

[0109] Regarding point cloud data, the attribute information can be classified using a technique called a point cloud segmentation technique. FIG. 14C is a diagram illustrating an example of attribute information added to points of the matching candidate point cloud. In FIG. 14C, a circle indicates attribute information added to a point classified as a wall, a cross indicates attribute information added to a point classified as a door, and a square indicates attribute information added to a point classified as a window frame. These pieces of attribute information are associated with category numbers such as 1,2, and 3 in the same or substantially the same manner as the attribute information added to the matching candidate image. The attribute information in the point cloud data is an example of a point cloud marker that allows a user to align point cloud data with drawing data.

[0110] The pair selection unit 533 and the position estimation unit 534 operate in substantially the same manner as the first embodiment. However, in step S105 of FIG. 10, the condition that the attribute information of a point that has been relatively moved using rigid transformation and the attribute information of a candidate pixel have the same category number is added to the condition for counting the number of combinations. Thus, even when the shape of the matching candidate point cloud is the same as the shapes of rooms in the drawing data, the counted number of combinations reduces for a room in which the attribute information of the candidate pixel and the attribute information of the point corresponding to the candidate pixel are different, and alignment is not performed for the room. Further, since alignment is performed on a room in which the attribute information of the candidate pixel and the attribute information of the point match most (i.e., the counted number of combinations is the maximum value) among the rooms, alignment processing is performed appropriately.

[0111] The display image generation unit 535 operates in the same or substantially the same manner as the first embodiment. As illustrated in FIG. 9, the display image generation unit 535 generates the superimposition screen 1350 in which pieces of point cloud data are aligned with rooms of drawing data and superimposed and displays the superimposition screen 1350 on the display screen 1000.

[0112] At the display image generation unit 535, the display control unit 34 of the terminal apparatus 3 may display the attribute information at a point superimposed on the drawing data on the superimposition screen 1350. For example, an icon image representing a circle is superimposed on a point classified as a wall, an icon image representing a cross is superimposed on a point classified as a door, and an icon image representing a square is superimposed on a point classified as a window frame, thus presenting the attribute information added to the points. FIG. 15 is a diagram illustrating a part of the superimposition screen 1350 on which the attribute information is displayed. The user can determine whether the point cloud matches the position of the door or the window on the drawing by checking the displayed attribute information. When the user determines that the point cloud does not match the position of the door or the window on the drawing, the user can correct the result of the alignment processing using the parallel movement button 1331 or the rotation button 1332.

[0113] As described above, according to the present embodiment, by adding attribute information to drawing data and point cloud data and performing alignment with reference to the attribute information in addition to the shapes of the drawing data and the point cloud data, point clouds are aligned with the drawing data even when rooms having similar shapes are present in the drawing data. When the user aligns point cloud data with drawing data, the user can visually check the attribute of the drawing data in some cases. Accordingly, as a modification of the present embodiment, although attribute information is added to point cloud data, attribute information does not have to be added to drawing data.

[0114] Third Embodiment In the above-described embodiment, the scale of drawing data is known. However, in some actual cases, the scale is unknown. In a third embodiment, even when the scale of drawing data is unknown, the alignment processing of a point cloud can be accurately performed by setting a tentative scale.

[0115] FIG. 16 is a block diagram illustrating the functional configuration of the alignment processing unit 530 according to the third embodiment. FIG. 17 is a flowchart of the operation of the alignment processing of a point cloud according to the third embodiment. The alignment processing unit 530 according to the present embodiment is different from that of the first embodiment in that the position estimation unit 534 further includes a correction value derivation unit 538. The configuration and operation according to the embodiment are described below with reference to FIG. 16 and FIG. 17. Redundant descriptions of functional units and processes that are the same as those of the first embodiment described above are omitted below.

[0116] The drawing analysis unit 531, the point cloud analysis unit 532, and the pair selection unit 533 operate in the same or substantially the same manner as the first embodiment. In the present embodiment as well, the selection of the candidate pair by the pair selection unit 533 and the estimation of the position by the position estimation unit 534 are repeatedly performed to estimate the position at which the point cloud is to be superimposed on the drawing data.

[0117] First, the pair selection unit 533 selects the first candidate pair in step S170 and selects the second candidate pair in step S171. Then, in step S172, the position estimation unit 534 calculates the length of the second candidate pair based on a tentative scale. For example, when the size of one pixel of the matching candidate image based on the tentative scale is 0.01 m, the length of 120 pixels is calculated as 120x0.01=1.2 m. When the tentative scale is too large or too small with respect to the correct value, the reference of the distance obtained for the pair of candidate pixels and the reference of the distance obtained for the pair of points of the matching candidate point cloud are significantly different. This makes it difficult to compare the lengths by the position estimation unit 534. For this reason, it is preferable to use an appropriate value as the tentative scale. For example, since the width of a door of a room is approximately determined as a standard, the tentative scale can be set based on the width of the door in the drawing data.

[0118] In step S173, the position estimation unit 534 compares the length LI of the first candidate pair with the length L2 of the second candidate pair. Taking into consideration the accuracy of the tentative scale, the threshold value used for the comparison of the lengths of the pairs is preferably set to a value larger than that used in the first embodiment. When the following two inequalities are satisfied, it is determined that the difference between the lengths of the two pairs is small. When the following two inequalities are not satisfied, it is determined that the difference between the lengths of the two pairs is large. In the following inequalities, a (where a>l) is a coefficient for setting the threshold value used for the comparison of the lengths of the pairs to a value larger than that used in the first embodiment and is experimentally determined taking into consideration the accuracy of the tentative scale.

[0119] L2-axth < LI LI < L2+axth

[0120] When it is determined that the difference between the lengths of the two pairs is small, in step S174, the position estimation unit 534 assumes that the length LI of the first candidate pair and the length L2 of the second candidate pair are equal to each other, and obtains a correction value k for correcting the tentative scale using the following equation.

[0121] k = L1 / L2

[0122] The corrected tentative scale is obtained by multiplying the tentative scale by k. For example, when the size of one pixel of the matching candidate image based on the tentative scale is 0.01 m and the correction value k is 1.3, the length of 120 pixels is calculated as 120x0.01x1.3=1.56 m.

[0123] In step S175, the position estimation unit 534 obtains a rigid transformation from the matching candidate point cloud to the matching candidate image so that the two pairs overlap each other based on the corrected tentative scale.

[0124] In step S176, when the position estimation unit 534 counts the number of combinations of the candidate pixel and the point between which distance is equal to or less than the constant value d, the position estimation unit 534 calculates the distance based on the corrected tentative scale. When it is determined that the counted number of combinations is larger than the maximum value of the number of combinations in step S177, the candidate correction value, the candidate transformation, and the maximum value of the number of combinations are updated in step S178. Then, the operation proceeds to step S179. In step S178, the candidate correction value is updated to the correction value derived in step S174, the candidate transformation is updated to the rigid transformations derived in step S175, and the maximum value of the number of combinations is updated to the number of combinations counted in step S176. It is assumed that the candidate correction value is set to 1 at the start of the operation.

[0125] The pair selection unit 533 and the position estimation unit 534 perform the above-described operation for all combinations of the first candidate pair and the second candidate pair and determine the candidate correction value at the end of this operation as the correction value for correcting the tentative scale. Thus, the correction value for correcting the tentative scale is obtained for one point cloud.

[0126] The alignment processing unit 530 obtains correction values ki respectively for point clouds i on which alignment is to be performed by the above-described operation and obtains a final correction value kF using the correction values ki. i (i=l, 2, 3...) is the sequential number of each point cloud. The final correction value kF can be obtained by averaging the correction values obtained for the respective point clouds. In order to eliminate the influence of a correction value having a large error, the final correction value kF may be obtained by taking the median.

[0127] Subsequently, the alignment processing unit 530 sets a scale obtained by multiplying the tentative scale by kF as a final scale and performs the alignment processing on each of the point clouds again using the final scale.

[0128] The display image generation unit 535 operates in the same or substantially the same manner as the first embodiment. As illustrated in FIG. 9, the display image generation unit 535 generates the superimposition screen 1350 in which pieces of point cloud data are aligned with rooms of drawing data and superimposed and displays the superimposition screen 1350 on the display screen 1000. The display image generation unit 535 displays the drawing data of which size is changed (enlarged or reduced) relative to the point cloud data using the final scale. Instead of changing the size of the drawing data, the display image generation unit 535 may change (enlarge or reduce) the size of the point cloud data relative to the drawing data using the reciprocal of the final scale. In the present embodiment, since the scale of the drawing data is unknown, a button for performing enlargement / reduction may be arranged in the selection screen 1300 so that the user can correct the scale using the button while checking the superimposition screen 1350.

[0129] The description given above is of a case where the third embodiment is implemented by adding the correction value derivation unit 538 to the configuration according to the first embodiment. Alternatively, the third embodiment may be implemented by adding the correction value derivation unit 538 to the configuration according to the second embodiment. In this case, the attribute addition unit 537 adds attribute information corresponding to the category number to the matching candidate image and the matching candidate point cloud, and in step S176 of FIG. 17, the condition that the attribute information of a point that has been relatively moved using rigid transformation and the attribute information of a candidate pixel have the same category number is added to the condition for counting the number of combinations.

[0130] As described above, according to the present embodiment, even when the scale of drawing data is unknown, a tentative scale is set and the tentative is corrected in the alignment processing. This enables performing the alignment processing of point clouds accurately.

[0131] Aspects of the present disclosure are, for example, as follows. Aspect 1 According to Aspect 1, a point cloud processing apparatus includes display screen generating means for generating a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object. Aspect 2 According to Aspect 2, in the point cloud processing apparatus of Aspect 1, the display screen generating means generates the display screen by using a relative movement that moves at least one of the plurality of point clouds relative to the image. Aspect 3 According to Aspect 3, in the point cloud processing apparatus of Aspect 1 or 2, the display screen generating means generates the display screen by using a size change that changes at least one of the plurality of point clouds relative to the image. Aspect 4 According to Aspect 4, in the point cloud processing apparatus of any one of Aspects 1 to 3, the display screen generating means generates the display screen on which a point cloud marker for alignment with the image is displayed for at least one of the plurality of point clouds. Aspect 5 According to Aspect 5, in the point cloud processing apparatus of any one of Aspects 1 to 3, the display screen generating means generates the display screen on which attribute information is displayed for at least one of the plurality of point clouds. Aspect 6 According to Aspect 6, the point cloud processing apparatus of any one of Aspects 1 to 5 further includes display controlling means for displaying the display screen. Aspect 7 According to Aspect 7, in the point cloud processing apparatus of any one of Aspects 2 to 6, the relative movement includes at least one of parallel movement in a horizontal plane or rotation in the horizontal plane. Aspect 8 According to Aspect 8, in the point cloud processing apparatus of any one of Aspects 3 to 7, the size change includes at least one of enlargement or reduction. Aspect 9 According to Aspect 9, the point cloud processing apparatus of any one of Aspects 5 to 8, the attribute information is information indicating at least one of a door or a window frame. Aspect 10 According to Aspect 10, a point cloud processing method includes a display screen generating step of generating a display screen in which a plurality of point clouds representing a threedimensional point cloud obtained by measuring an object are superimposed on an image representing the object. Aspect 11 According to Aspect 11, a recording medium stores a program for causing a computer to generate a display screen in which a plurality of point clouds reprinting a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object. Aspect 12 According to Aspect 12, a point cloud processing system includes a point cloud processing apparatus and a terminal apparatus communicable with the point cloud processing apparatus. The point cloud processing apparatus includes display screen generating means for generating a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object. The point cloud processing apparatus includes transmitting means for transmitting display screen information indicating the display screen to the terminal apparatus. The terminal apparatus includes receiving means for receiving the display screen information transmitted from the point cloud processing apparatus. The terminal apparatus includes display controlling means for controlling a display to display the display screen.

[0132] The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may be combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

[0133] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Intemet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD-ROM), a magnetic tape device, or a solid state memory device.

[0134] The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality. There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of an FPGA or ASIC.

[0135] This patent application is based on and claims priority to Japanese Patent Application No. 2024-006119, filed on January 18, 2024, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein. [Reference Signs List]

[0136] 1 Point cloud processing system 3 Terminal apparatus 5 Management server 32 Reception unit 34 Display control unit 39 Storing / reading unit 53 Processing unit 57 Generation unit 100 Communication network 308 Display 311 Keyboard 312 Mouse 530 Alignment processing unit 531 Drawing analysis unit 532 Point cloud analysis unit 533 Pair selection unit 534 Position estimation unit 535 Display image generation unit 536 Rigid transformation unit 537 Attribute addition unit 538 Correction value derivation unit 1300 Selection screen 1310 Processing result area 1331 Parallel movement button 1332 Rotation button 1350 Superimposition screen

Claims

[CLAIMS]1. A point cloud processing apparatus, comprising display screen generating means for generating a display screen in which a plurality of point clouds representing a threedimensional point cloud obtained by measuring an object are superimposed on an image representing the object.

2. The point cloud processing apparatus according to claim 1, wherein the display screen generating means generates the display screen by performing a relative movement that moves at least one of the plurality of point clouds relative to the image.

3. The point cloud processing apparatus according to claim 1 or 2, wherein the display screen generating means generates the display screen by performing a size change that changes at least one of the plurality of point clouds relative to the image.

4. The point cloud processing apparatus according to any one of claims 1 to 3, wherein the display screen generating means generates the display screen on which a point cloud marker for alignment with the image is displayed for at least one of the plurality of point clouds.

5. The point cloud processing apparatus according to any one of claims 1 to 3, wherein the display screen generating means generates the display screen on which attribute information is displayed for at least one of the plurality of point clouds.

6. The point cloud processing apparatus according to any one of claims 1 to 5, further comprising display controlling means for displaying the display screen.

7. The point cloud processing apparatus according to any one of claims 2 to 6, wherein the relative movement includes at least one of parallel movement in a horizontal plane or rotation in the horizontal plane.

8. The point cloud processing apparatus according to claim 3, wherein the size change includes at least one of enlargement or reduction.

9. The point cloud processing apparatus according to any one of claims 5 to 8, wherein the attribute information is information indicating at least one of a door or a window frame.

10. A point cloud processing method, comprising:generating a display screen in which a plurality of point clouds representing a threedimensional point cloud obtained by measuring an object are superimposed on an image representing the object.

11. A recording medium storing a program code for causing a computer to generate a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object.

12. A point cloud processing system, comprising:a point cloud processing apparatus; anda terminal apparatus communicable with the point cloud processing apparatus,the point cloud processing apparatus including:display screen generating means for generating a display screen in which a plurality of point clouds representing a three-dimensional point cloud obtained by measuring an object are superimposed on an image representing the object; andtransmitting means for transmitting display screen information indicating the display screen to the terminal apparatus,the terminal apparatus including:receiving means for receiving the display screen information transmitted from the point cloud processing apparatus; anddisplay controlling means for displaying the display screen on a display.

Images