IMAGE POINT CLOUD DATA PROCESSING APPARATUS, IMAGE POINT CLOUD DATA PROCESSING METHOD, AND IMAGE POINT CLOUD DATA PROCESSING PROGRAM
Patent Information
- Application Number
- JP2024532198
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-01-30
- Publication Date
- 2025-01-07
- Estimated Expiration
- 2043-01-30
AI Technical Summary
Conventional devices for converting image data and three-dimensional point cloud data into a common coordinate system struggle with precision when there are objects other than the calibration board within the shooting range of the camera and detection range of the ranging sensor.
An image point cloud data processing device that detects image feature points from a reference object, calculates representative positions, and uses a registration processing unit to convert image and point cloud data into a common coordinate system using methods like NDT or ICP, even in environments with multiple objects.
Enables high-precision conversion of image and three-dimensional point cloud data into a common coordinate system, allowing accurate superimposition and color addition of point clouds on images, enhancing understanding of real-space data.
Smart Images
Figure 00000009_0000 
Figure 00000009_0001 
Figure 00000010_0000
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image point cloud data processing device, an image point cloud data processing method, and an image point cloud data processing program. [Background technology]
[0002] There has been a proposal for a composite calibration device for converting image data obtained by camera photography and three-dimensional point cloud data detected by a distance measuring sensor into data in a common coordinate system (for example, see Patent Document 1). In this device, a robot arm is used to move the camera and distance measuring sensor in front of a calibration board, and registration is performed using the image data and three-dimensional point cloud data obtained at that time. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2022-039906 Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the above conventional device, it is assumed that the camera and the distance measuring sensor are moved by a robot arm, and therefore, in a real environment where objects other than the calibration board exist within the shooting range of the camera and the detection range of the distance measuring sensor, there is a problem that it is not possible to perform coordinate conversion between image data and 3D point cloud data with high accuracy.
[0005] An object of the present disclosure is to convert image data and three-dimensional point cloud data into data in a common coordinate system with high accuracy. [Means for solving the problem]
[0006] The image point cloud data processing device disclosed herein is characterized by having an image data processing unit that detects multiple image feature points from a reference object included in image data generated by a camera, and calculates image representative positions, which are multiple three-dimensional positions representing the reference object, based on the multiple image feature points; a point cloud data processing unit that detects an intra-area point cloud, which is a three-dimensional point cloud existing within a detection area determined based on the image representative positions, from three-dimensional point cloud data indicating the three-dimensional point cloud generated by a ranging sensor, and detects a representative point cloud representing the reference object based on the intra-area point cloud; and a registration processing unit that converts the image data and the three-dimensional point cloud data into data in a common coordinate system based on the image representative position and the representative point cloud.
[0007] The image point cloud data processing method disclosed herein is a method executed by an image point cloud data processing device, and is characterized by comprising the steps of: detecting a plurality of image feature points from a reference object included in image data generated by a camera, and calculating image representative positions which are a plurality of three-dimensional positions representing the reference object based on the plurality of image feature points; detecting an intra-area point cloud which is a three-dimensional point cloud existing within a detection area determined based on the image representative positions from three-dimensional point cloud data indicating the three-dimensional point cloud generated by a ranging sensor, and detecting a representative point cloud representing the reference object based on the intra-area point cloud; and converting the image data and the three-dimensional point cloud data into data in a common coordinate system based on the image representative position and the representative point cloud. Effect of the Invention
[0008] According to the device, method, and program of the present disclosure, image data and three-dimensional point cloud data can be converted into data in a common coordinate system with high accuracy. [Brief description of the drawings]
[0009] [Figure 1] 1 is a functional block diagram illustrating an outline of a configuration of an image point cloud data processing device according to a first embodiment. [Diagram 2]1 is a diagram illustrating an example of a hardware configuration of an image point cloud data processing device according to a first embodiment. [Diagram 3] 4 is a flowchart showing the operation of the image point cloud data processing device according to the first embodiment. [Figure 4] 5A to 5C are explanatory diagrams showing the operation of the image data processing unit in the first embodiment. [Diagram 5] 5A to 5D are explanatory diagrams showing the operation of the point cloud data processing unit in the first embodiment. [Figure 6] 5(A) and 5(B) are explanatory diagrams showing the operation of a registration processing unit in the first embodiment. [Figure 7] 11 is a flowchart showing the operation of the image point cloud data processing device according to the second embodiment. [Figure 8] 13(A) to 13(C) are explanatory diagrams showing the operation of an image data processing unit in the second embodiment. [Figure 9] 13(A) to 13(D) are explanatory diagrams showing the operation of a point cloud data processing unit in the second embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of a hardware configuration of an image point cloud data processing device according to a third embodiment. [Figure 11] 13 is a flowchart showing the operation of the image point cloud data processing device according to the third embodiment. [Figure 12] 5A to 5C are explanatory diagrams showing the operation of an image data processing unit in the embodiment. [Figure 13] 13(A) to 13(C) are explanatory diagrams showing the operation of a point cloud data processing unit in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an image point cloud data processing device, an image point cloud data processing method, and an image point cloud data processing program according to embodiments will be described with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and each embodiment can be appropriately modified.
[0011] 1. First embodiment 《1-1》Configuration 1 is a functional block diagram showing a schematic configuration of an image point cloud data processing device 1 according to the first embodiment. The image point cloud data processing device 1 is a device capable of implementing an image point cloud data processing method according to the first embodiment, for example, a computer capable of executing an image point cloud data processing program according to the first embodiment. The image point cloud data processing device 1 is a device capable of deriving a transformation formula (i.e., a transformation matrix of the transformation formula) for converting image data obtained by shooting with a camera 51 as an imaging device and three-dimensional point cloud data detected by a LiDAR 52 (Light Detection And Ranging) as a distance measuring sensor into data in a common coordinate system.
[0012] 1, the image point cloud data processing device 1 has an image data processing unit 10, a point cloud data processing unit 20, and a registration processing unit 30. The image point cloud data processing device 1 may further have one or more of a data storage unit 40 which is a storage device that stores image data and three-dimensional point cloud data, a communication unit that communicates with an external device, and a display device that displays an image.
[0013] The image point cloud data processing device 1 determines a transformation matrix included in a transformation formula for converting image data obtained by the camera 51 and three-dimensional point cloud data detected by the LiDAR 52 into data in a common coordinate system in a real environment. In the real space, an object (i.e., another object) other than the calibration board as a reference object used for camera calibration may exist. In FIG. 1, the camera 51 and the LiDAR 52 are described as devices different from the image point cloud data processing device 1, but the camera 51 and the LiDAR 52 may be part of the image point cloud data processing device 1.
[0014] The image data processing unit 10 detects a plurality of image feature points (e.g., intersections of a checkered pattern) from the calibration pattern of the calibration board included in the image data generated by the camera 51, and calculates image representative positions, which are a plurality of three-dimensional positions representing the calibration board, based on the plurality of image feature points. The plurality of image representative positions are, for example, corner positions of the calibration board.
[0015] The point cloud data processing unit 20 detects an intra-area point cloud, which is a three-dimensional point cloud existing in a detection area determined based on an image representative position (for example, a position of a corner of the calibration board), from the three-dimensional point cloud data indicating a three-dimensional point cloud generated by the LiDAR 52, and detects a representative point cloud representing the calibration board, which is a reference object, based on the intra-area point cloud. The calibration board has a predetermined calibration pattern. The calibration pattern is a checker pattern in which a plurality of rectangular white and black areas are alternately arranged two-dimensionally. The calibration pattern may be a grid pattern composed of a plurality of straight lines arranged in a square shape, a dot pattern composed of a plurality of dots arranged two-dimensionally, or the like.
[0016] The registration processing unit 30 is a feature point matching processing unit that derives a transformation matrix of a transformation formula used for transforming image data and three-dimensional point cloud data by matching based on the image representative position calculated by the image data processing unit 10 and the representative point cloud detected by the point cloud data processing unit 20. The registration processing unit 30 calculates a transformation matrix using feature points from multiple images (e.g., the four corner points of the calibration board) and feature points of the three-dimensional point cloud measured by the LiDAR 52 (e.g., the four corner points of the calibration board) by using an NDT (Normal Distribution Transform) method or an ICP (Iterative Closest Point) method.
[0017] 2 is a diagram showing an example of a hardware configuration of the image point cloud data processing device 1 according to the first embodiment. The image point cloud data processing device 1 has, for example, a processor 101 such as a CPU (Central Processing Unit), a memory 102 as a storage unit, a storage device 103, and an interface 104. Each part constituting the image point cloud data processing device 1 is constituted by, for example, a processing circuit. The processing circuit may be dedicated hardware, or may include a CPU that executes a program (for example, an image point cloud data processing program) stored in the memory 102. The processor 101 realizes each functional block shown in FIG. 1.
[0018] The memory 102 is, for example, a semiconductor memory such as a RAM (Random Access Memory). The storage device 103 is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The image point cloud data processing device 1 may include both a component part consisting of a circuit and a component part consisting of a processor. A part or all of the image point cloud data processing device 1 may be a server computer on a network. The image point cloud data processing program is provided by downloading via the network or by a storage medium such as a USB memory that stores information. The interface 104 is connected to a camera 51, a LiDAR 52, an input device for performing user operations, a display device such as a liquid crystal display that presents information, and the like.
[0019] As shown in FIG. 2, a calibration board 60 exists as a reference object within the shooting range of the camera 51 and within the measurement range of the LiDAR 52, but other objects other than the calibration board 60 may exist. That is, the image point cloud data processing device 1 can convert image data and three-dimensional point cloud data into data of a common coordinate system with high accuracy by shooting and measuring a real space in which many objects exist. In addition, the image point cloud data processing device 1 can display an image in which a point cloud based on the three-dimensional point cloud data generated by the LiDAR 52 is superimposed on an image based on image data of the same coordinate system on a display device. In addition, the image point cloud data processing device 1 can display an image in which a color based on image data is added to a surface constituted by a point cloud based on the three-dimensional point cloud data generated by the LiDAR 52 on a display device.
[0020] 《1-2》Operation Fig. 3 is a flowchart showing the operation of the image point cloud data processing device 1 according to embodiment 1. Figs.
[0021] First, the image data processing unit 10 acquires image data generated by the camera 51 (step S101), detects the calibration board 60 as a reference object included in the image data, performs camera calibration, and determines the camera internal parameters (step S102). FIG. 4(A) shows a transformation matrix as the camera internal parameters generated by the image data processing unit 10. In FIG. 4(A), (c x ,c y ) denotes the coordinates of the principal point (usually the image center), and f x and f y denotes the focal length expressed in pixels.
[0022] Next, the image data processing unit 10 detects a plurality of image feature points 61 from the calibration pattern of the calibration board 60 (step S103), and calculates image representative positions 63, which are a plurality of three-dimensional positions representing the calibration board 60, based on the plurality of image feature points 61 (step S104). FIG. 4(B) shows the plurality of image feature points 61. FIG. 4(C) shows image representative positions 63 at the four corner positions of the calibration board 60 obtained based on the plurality of image feature points 61. At this time, the image data processing unit 10 calculates a rotation angle α of the calibration board 60 with respect to a reference direction 65. Note that instead of the image representative positions 63, corner positions 62 of a checker pattern, which is the calibration pattern of the calibration board 60, may be used.
[0023] 5(A) to (D) are explanatory diagrams showing the operation of the point cloud data processing unit 20. The point cloud data processing unit 20 acquires a three-dimensional point cloud 71 that is a cluster generated by the LiDAR 52 and divided by a method such as clustering (step S105), and detects an intra-area point cloud 71a that is a three-dimensional point cloud existing in a detection area 72 determined based on an image representative position 63 acquired from the image data processing unit 10 (step S106). FIG. 5(A) shows the three-dimensional point cloud 71 generated by the LiDAR 52. FIG. 5(B) shows an intra-area point cloud 71a that is a three-dimensional point cloud existing in the detection area 72. Each point of the three-dimensional point cloud 71 is a measurement point irradiated with a laser beam.
[0024] Next, the point cloud data processing unit 20 detects a representative point cloud 73 representing the calibration board 60 based on the intra-area point cloud 71a (step S107). The representative point cloud is a point cloud consisting of points indicating the positions of the corners of the calibration board 60. FIG. 5(C) shows the representative point cloud 73 calculated based on the intra-area point cloud 71a, which is a three-dimensional point cloud existing in the detection area 72. FIG. 5(D) shows the distribution characteristic of the number of points with respect to reflection intensity used when detecting the calibration board 60. That is, the point cloud data processing unit 20 can determine that an object having a distribution of the number of points similar to that of FIG. 5(D) (a distribution having two peaks) is the calibration board 60.
[0025] 6(A) and (B) are explanatory diagrams showing the operation of the registration processing unit 30. The registration processing unit 30 performs matching between the image representative position 63 indicated by the three-dimensional data calculated by the image data processing unit 10 and the representative point group 73 indicated by the three-dimensional point group data calculated by the point group data processing unit 20, that is, performs registration (step S108), and calculates a transformation matrix of a transformation formula for converting the image data and the three-dimensional point group data into data of a common coordinate system using the result (step S109). FIG. 6(A) shows registration, and FIG. 6(B) shows a transformation formula of a three-dimensional affine transformation between the image data and the three-dimensional point group data. By determining the elements a, b, ..., k, and l, it is possible to set rotation around each axis, enlargement / reduction, and parallel movement (translation). However, the transformation formula is not limited to the three-dimensional affine transformation.
[0026] 1-3 Effects As described above, in embodiment 1, the rotation angle α of the calibration board 60 is estimated based on the image data generated by the camera 51, the detection area of the three-dimensional point cloud data generated by the LiDAR 52 is rotated using the rotation angle α to extract the three-dimensional point cloud within the detection area, and registration is performed between the image representative position 63 and the representative point cloud 73, so that a highly accurate transformation matrix can be calculated.
[0027] Furthermore, by determining that an object having the characteristics (two peaks) shown in FIG. 5(D) is the calibration board 60, a highly accurate transformation matrix can be derived using image data and three-dimensional point cloud data of a real space in which objects other than the calibration board 60 exist.
[0028] 2. Second embodiment 2-1. Configuration In the first embodiment, an example is described in which the image representative position 63 is a coordinate point at a corner of the calibration board 60, but in the second embodiment, an example is described in which the image representative position is a coordinate point on an edge (side) of the calibration board 60. Other points in the second embodiment are common to the first embodiment, so FIG. 1 and FIG. 2 will be referred to in the description of the second embodiment.
[0029] 《2-2》Operation 7 is a flowchart showing the operation of the image point cloud data processing device according to embodiment 2. The image point cloud data processing device according to embodiment 2 differs from the image point cloud data processing device 1 according to embodiment 1 in the processes of steps S204 and S207.
[0030] 8(A) to (C) are explanatory diagrams showing the operation of the image data processing unit 10 in the second embodiment. First, the image data processing unit 10 acquires image data generated by the camera 51 (step S101), detects the calibration board 60 as a reference object included in the image data, performs camera calibration, and determines the camera internal parameters (step S102). Fig. 8(A) shows a transformation matrix as the camera internal parameters generated by the image data processing unit 10, similar to Fig. 4(A).
[0031] Next, the image data processing unit 10 detects a plurality of image feature points 61 from the calibration pattern of the calibration board 60 (step S103), and calculates edges (sides) 64, which are a plurality of three-dimensional positions representing the calibration board 60, based on the plurality of image feature points 61 (step S204). FIG. 8(B) shows a plurality of image feature points 61, and FIG. 8(C) calculates the position of the edge 64 of the calibration board 60 obtained based on the plurality of image feature points 61 as the image representative position. At this time, the image data processing unit 10 calculates a rotation angle α of the calibration board 60 with respect to the reference direction. Note that, instead of the image representative position 63, the position of the edge of a checker pattern, which is the calibration pattern of the calibration board 60, may be used.
[0032] 9(A) to 9(C) are explanatory diagrams showing the operation of the point cloud data processing unit 20 in the second embodiment. The point cloud data processing unit 20 acquires a three-dimensional point cloud 71 that is generated by the LiDAR 52 and is a cluster divided by a method such as clustering (step S105), and detects an intra-area point cloud 71a that is a three-dimensional point cloud existing in a detection area 72 determined based on the image representative position 63 acquired from the image data processing unit 10 (step S106). FIG. 9(A) shows the three-dimensional point cloud 71 generated by the LiDAR 52. FIG. 9(B) shows the intra-area point cloud 71a that is a three-dimensional point cloud existing in the detection area 72.
[0033] Next, the point cloud data processing unit 20 detects a representative point group 74 of an edge that represents the calibration board 60 based on the intra-area point group 71a (step S207). The representative point group 74 is a point group that indicates the positions of the edges of the calibration board 60. Fig. 9(C) shows the representative point group 74 of an edge calculated based on the intra-area point group 71a, which is a three-dimensional point group that exists in the detection area 72. Fig. 9(D) shows the distribution characteristic of the number of points with respect to reflection intensity, which is used when detecting the calibration board 60.
[0034] The operation of the registration processing unit 30 is similar to that in the first embodiment.
[0035] 2-3 Effects In the second embodiment, the same effects as those described in the first embodiment can be obtained.
[0036] 3. Third embodiment 3-1. Configuration Fig. 10 is a diagram showing an example of a hardware configuration of an image point cloud data processing device 3 according to the third embodiment. In Fig. 10, components identical to or corresponding to those shown in Fig. 2 are given the same reference numerals as those shown in Fig. 2. In the first and second embodiments, an example has been described in which the reference object is a calibration board 60 having a calibration pattern, but in the third embodiment, the reference object is a detection object (for example, a vehicle) 80. The detection object 80 does not need to have a calibration pattern that is a predetermined pattern. In addition, in the description of the third embodiment, reference will be made to Fig. 1, which is a functional block diagram.
[0037] 《3-2》Operation 11 is a flowchart showing the operation of the image point cloud data processing device 3 according to embodiment 2. The image point cloud data processing device 3 according to embodiment 3 differs from the image point cloud data processing device 1 according to embodiment 1 in the processes of steps S303, S304, S306, and S307.
[0038] 12(A) to 12(C) are explanatory diagrams showing the operation of the image data processing unit 10 in embodiment 3. First, the image data processing unit 10 acquires image data generated by the camera 51 (step S101), detects a detection object 80 (a vehicle in the figure) as a reference object included in the image data, performs camera calibration, and determines camera internal parameters (step S102).
[0039] Next, the image data processing unit 10 detects the detected object 80 from the image data using the color distribution of the image data as a feature amount, and estimates the depth of the detected object 80 (step S303). At this time, the object may be detected from the image data obtained by shooting with a camera by deep learning, the depth may be estimated, and feature points may be obtained.
[0040] Next, the image data processing unit 10 extracts the edges, four corner points, and color distribution of the detected object 80 as feature quantities, and converts the image data from two-dimensional image data to three-dimensional image data (step S304). Figures 12(A) and (B) show the process of calculating the detected object 80, multiple image representative positions 83, and the edges (sides) 81 of the detected object. Figure 12(C) shows that since the detected object has light and dark shades, there is a specific tendency in the distribution of the number of pixels relative to the grayscale (tone) of the detected object (for example, there is a peak in the number of pixels at high gradations and a peak in the number of pixels at low gradations).
[0041] 13(A) to 13(C) are explanatory diagrams showing the operation of the point cloud data processing unit 20 in the third embodiment. The point cloud data processing unit 20 acquires a three-dimensional point cloud 91, which is a cluster generated by scanning the laser of the LiDAR 52 and divided by a method such as clustering (step S105), and detects an intra-area point cloud 91a, which is a three-dimensional point cloud existing in a detection area 92 determined based on the image representative position 83 or edge 81 acquired from the image data processing unit 10 (step S306). FIG. 13(A) shows the three-dimensional point cloud 91 generated by the LiDAR 52. FIG. 13(B) shows the intra-area point cloud 91a, which is a three-dimensional point cloud existing in the detection area 92. FIG. 13(C) shows that there is a specific tendency in the distribution of the number of pixels for the grayscale (shade) of the detected object since the detected object has shades.
[0042] Next, the point cloud data processing unit 20 detects the edge representative point group 74 representing the detected object 80 based on the intra-region point group 91a (step S307). The representative point group 94 is a point group indicating the positions of the edges of the detected object 80. Fig. 13(C) shows the edge representative point group 74 calculated based on the intra-region point group 71a, which is a three-dimensional point group existing in the detection region 92. Fig. 13(D) shows the distribution characteristics of the number of points with respect to reflection intensity, which are used when detecting the calibration board 60.
[0043] Steps S108 and S109, which are the operations of the registration processing unit 30, are the same as those in the first embodiment.
[0044] 3-3 Effects In the third embodiment, the same effects as those described in the first embodiment can be obtained.
[0045] Furthermore, in the third embodiment, a calibration board having a calibration pattern is not required, so that by photographing and measuring the real space, the image data and the three-dimensional point cloud data can be converted into data in a common coordinate system.
[0046] (4) Industrial Applicability By using any one of the first to third embodiments, it is possible to photograph and measure a real space in which many objects exist, and to display, for example, an image in which a three-dimensional point cloud generated by the LiDAR 52 is superimposed with high accuracy on a color image on a display device. As a result, it becomes possible to display the depth (distance to each part) of each part of an object displayed in a color image in a form understandable to humans (for example, workers in the field of civil engineering) by the points of the three-dimensional point cloud.
[0047] Furthermore, by using any one of the first to third embodiments, it is possible to photograph and measure a real space in which many objects exist, and to add color based on a color image to a surface formed by a three-dimensional point cloud generated by the LiDAR 52. As a result, it becomes possible to display the surface formed by the three-dimensional point cloud in a format that is understandable to humans (for example, workers in the field of civil engineering).
[0048] Furthermore, by using any one of the first to third embodiments, it is possible to photograph and measure a real space in which many objects exist, and to add colors based on a color image to the points of the three-dimensional point cloud generated by, for example, the LiDAR 52. As a result, it becomes possible to display the three-dimensional point cloud in a format that is understandable to humans (for example, workers in the field of civil engineering). [Explanation of symbols]
[0049] 1, 3 Image point cloud data processing device, 10 Image data processing unit, 20 Point cloud data processing unit, 30 Registration processing unit, 51 Camera, 52 LiDAR (ranging sensor), 60 Calibration board (reference object), 61 Image feature point, 63, 83 Image representative position, 64 Edge (image representative position), 71, 91 3D point cloud, 71a, 91a Area point cloud, 72, 92 Detection area, 73, 74, 94 Representative point cloud, 80 Detected object (reference object), 81 Edge (image feature point), 101 Processor, 102 Memory, 103 Storage device, 104 Interface.
Claims
1. an image data processing unit that detects a plurality of image feature points from a reference object included in image data generated by a camera, and calculates image representative positions that are a plurality of three-dimensional positions that represent the reference object based on the plurality of image feature points; a point cloud data processing unit that detects an intra-area point cloud, which is a three-dimensional point cloud existing within a detection area determined based on the image representative position, from three-dimensional point cloud data indicating a three-dimensional point cloud generated by a distance measuring sensor, and detects a representative point cloud representing the reference object based on the intra-area point cloud; a registration processing unit that converts the image data and the three-dimensional point cloud data into data in a common coordinate system based on the image representative position and the representative point cloud; An image point cloud data processing device comprising:
2. The reference object is a calibration board having a predetermined calibration pattern. The image point cloud data processing device according to claim 1 .
3. The image representative position is a coordinate point of a corner of the calibration board. The image point cloud data processing apparatus according to claim 2 .
4. The image representative position is a coordinate point on the edge of the calibration board. The image point cloud data processing apparatus according to claim 2 .
5. The reference object is a detected object detected from the image data. The image point cloud data processing device according to claim 1 .
6. The image representative position is one or both of a coordinate point at a corner of the detected object and a coordinate point on an edge of the detected object. The image point cloud data processing apparatus according to claim 5 .
7. The image data processing unit estimates the image representative position of the reference object using deep learning.
7. The image point cloud data processing device according to claim 5 or 6.
8. The camera; The distance measuring sensor; The image point cloud data processing apparatus according to any one of claims 1 to 6, further comprising:
9. An image point cloud data processing method executed by an image point cloud data processing device, comprising: detecting a plurality of image feature points from a reference object included in image data generated by a camera, and calculating image representative positions, which are a plurality of three-dimensional positions representing the reference object, based on the plurality of image feature points; a step of detecting an intra-area point cloud, which is a three-dimensional point cloud existing within a detection area determined based on the image representative position, from three-dimensional point cloud data indicating a three-dimensional point cloud generated by a distance measuring sensor, and detecting a representative point cloud representing the reference object based on the intra-area point cloud; A step of converting the image data and the three-dimensional point cloud data into data in a common coordinate system based on the image representative position and the representative point cloud; An image point cloud data processing method comprising the steps of:
10. detecting a plurality of image feature points from a reference object included in image data generated by a camera, and calculating image representative positions, which are a plurality of three-dimensional positions representing the reference object, based on the plurality of image feature points; a step of detecting an intra-area point cloud, which is a three-dimensional point cloud existing within a detection area determined based on the image representative position, from three-dimensional point cloud data indicating a three-dimensional point cloud generated by a distance measuring sensor, and detecting a representative point cloud representing the reference object based on the intra-area point cloud; A step of converting the image data and the three-dimensional point cloud data into data in a common coordinate system based on the image representative position and the representative point cloud; An image point cloud data processing program characterized by causing a computer to execute the above.