Map generation device, map generation method, and map generation system

Abstract

To suppress the occurrence of color shifts in the map of the observation area. [Solution] The map generation device 3 is configured to include a point cloud data acquisition unit 11 that repeatedly acquires 3D point cloud data indicating the distance to 3D points from a sensor 1 that measures the distance to one or more 3D points on an object present in a partial region, switching between partial regions, which are a part of the observation area, as time progresses, and an image data acquisition unit 12 that repeatedly acquires image data of the observation area from a camera 2 that photographs the observation area. The map generation device 3 also includes a data extraction unit 13 that extracts image data corresponding to the time of measurement of the distance in the partial region by the sensor 1 from among a plurality of image data acquired by the image data acquisition unit 12, and a map generation unit 14 that generates a map of the partial region based on the 3D point cloud data relating to the partial region and adds image data relating to the partial region to the map of the partial region.

Description

Technical Field

[0001] The present disclosure relates to a map generation device, a map generation method, and a map generation system.

Background Art

[0002] There is a map generation device that generates a map of an observation area. As such a map generation device, for example, Patent Document 1 discloses a map generation device including a data acquisition unit, a data addition unit, and a map generation unit. The data acquisition unit acquires 3D point cloud data indicating the distance to each of one or more 3D points on an object existing in the observation area from a sensor that measures the distance to the 3D points, and also acquires shooting data of the observation area from a camera that shoots the observation area. The data addition unit adds the shooting data to the 3D point cloud data. The map generation unit generates a map of the observation area based on the 3D point cloud data to which the shooting data is added.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The map generation device disclosed in Patent Document 1 has a problem that due to the deviation between the distance measurement timing by the sensor and the shooting timing of the observation area by the camera, there may be a color deviation in the map of the observation area.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to obtain a map generation device capable of suppressing the occurrence of color deviation in the map of the observation area.

Means for Solving the Problems

[0006] The map generation device according to this disclosure includes a point cloud data acquisition unit that repeatedly acquires 3D point cloud data indicating the distance to each 3D point from a sensor that measures the distance to one or more 3D points on an object present in a sub-region, while switching between sub-regions, which are a part of the observation area, as time progresses, and an image data acquisition unit that repeatedly acquires image data of the observation area from a camera that photographs the observation area. The map generation device also includes a data extraction unit that extracts image data corresponding to the time of shooting when the sensor measures the distance in each sub-region from among a plurality of image data acquired by the image data acquisition unit, as image data related to each sub-region, and a map generation unit that generates a map of each sub-region based on the 3D point cloud data related to each sub-region and adds the image data related to each sub-region to the map of each sub-region. [Effects of the Invention]

[0007] According to this disclosure, it is possible to suppress the occurrence of areas with color shifts in the map of the observation area. [Brief explanation of the drawing]

[0008] [Figure 1] This is a configuration diagram showing a map generation system including a map generation device 3 according to Embodiment 1. [Figure 2] This is a hardware configuration diagram showing the hardware of the map generation device 3 according to Embodiment 1. [Figure 3] This is a hardware configuration diagram of a computer when the map generation device 3 is implemented by software or firmware, etc. [Figure 4] This is a flowchart showing the map generation method, which is the processing procedure of the map generation device 3. [Figure 5] This is an explanatory diagram showing a captured image containing pixels corresponding to the 3D points of an object included in 3D point cloud data. [Figure 6]This is a diagram showing a map generation system including a map generation device 3 according to Embodiment 2. [Figure 7] This is a hardware configuration diagram showing the hardware of the map generation device 3 according to Embodiment 2. [Figure 8] This is an explanatory diagram illustrating an example where the same object is located within three sub-regions of a map. [Figure 9] This is a configuration diagram showing a map generation system including a map generation device 3 according to Embodiment 3. [Figure 10] This is a hardware configuration diagram showing the hardware of the map generation device 3 according to Embodiment 3. [Modes for carrying out the invention]

[0009] To provide a more detailed explanation of this disclosure, the forms for implementing this disclosure will be described below with reference to the attached drawings.

[0010] Embodiment 1. Figure 1 is a diagram showing a map generation system including a map generation device 3 according to Embodiment 1. Figure 2 is a hardware configuration diagram showing the hardware of the map generation device 3 according to Embodiment 1. The map generation system shown in Figure 1 includes a sensor 1, a camera 2, and a map generation device 3.

[0011] Sensor 1 is implemented, for example, by LiDAR (Light Detection and Ranging). Sensor 1 measures the distance to one or more three-dimensional points on an object present within a sub-region, switching between sub-regions, which are parts of the observation area, as time progresses. Sensor 1 repeatedly outputs 3D point cloud data, indicating the distance to each 3D point, to the map generation device 3. The 3D point cloud data contains distance information for each 3D point, representing the distance to each 3D point of an object present within a sub-region. Camera 2 will photograph the observation area described above. Camera 2 repeatedly outputs the image data of the observed area to the map generation device 3. The map generation device 3 includes a point cloud data acquisition unit 11, a photography data acquisition unit 12, a data extraction unit 13, and a map generation unit 14.

[0012] The point cloud data acquisition unit 11 is implemented, for example, by the point cloud data acquisition circuit 21 shown in Figure 2. The point cloud data acquisition unit 11 repeatedly acquires 3D point cloud data from the sensor 1, indicating the distance to each 3D point. The point cloud data acquisition unit 11 outputs each 3D point cloud data to the data extraction unit 13.

[0013] The image data acquisition unit 12 is implemented, for example, by the image data acquisition circuit 22 shown in Figure 2. The shooting data acquisition unit 12 repeatedly acquires shooting data from the camera 2. The image data acquisition unit 12 outputs each image data to the data extraction unit 13.

[0014] The data extraction unit 13 is implemented, for example, by the data extraction circuit 23 shown in Figure 2. The data extraction unit 13 acquires 3D point cloud data for each sub-region from the point cloud data acquisition unit 11 and acquires the corresponding image data from the image data acquisition unit 12. The data extraction unit 13 extracts from the multiple shooting data acquired by the shooting data acquisition unit 12 the shooting data for each sub-region, which corresponds to the shooting time of the distance measurement time in each sub-region by the sensor 1. The data extraction unit 13 outputs 3D point cloud data and image data related to each sub-region to the map generation unit 14.

[0015] The map generation unit 14 is implemented, for example, by the map generation circuit 24 shown in Figure 2. The map generation unit 14 acquires 3D point cloud data and image data related to each sub-region from the data extraction unit 13. The map generation unit 14 generates maps of each sub-region based on the 3D point cloud data pertaining to each sub-region, and adds the image data pertaining to each sub-region to the map of each sub-region.

[0016] In Figure 1, the map generation device 3 is assumed to be implemented by dedicated hardware as shown in Figure 2, with each of its components being the point cloud data acquisition unit 11, the image data acquisition unit 12, the data extraction unit 13, and the map generation unit 14. Specifically, the map generation device 3 is assumed to be implemented by the point cloud data acquisition circuit 21, the image data acquisition circuit 22, the data extraction circuit 23, and the map generation circuit 24. Each of the point cloud data acquisition circuit 21, image data acquisition circuit 22, data extraction circuit 23, and map generation circuit 24 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0017] The components of the map generation device 3 are not limited to those implemented by dedicated hardware; the map generation device 3 may also be implemented by software, firmware, or a combination of software and firmware. Software or firmware is stored as a program in the computer's memory. A computer refers to the hardware that executes programs, and includes, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor).

[0018] Figure 3 is a hardware configuration diagram of a computer when the map generation device 3 is implemented by software or firmware, etc. If the map generation device 3 is implemented by software or firmware, a program is stored in memory 31 that causes the computer to execute the respective processing procedures in the point cloud data acquisition unit 11, the image data acquisition unit 12, the data extraction unit 13, and the map generation unit 14. The computer's processor 32 then executes the program stored in memory 31.

[0019] Furthermore, Figure 2 shows an example in which each component of the map generation device 3 is implemented by dedicated hardware, and Figure 3 shows an example in which the map generation device 3 is implemented by software or firmware, etc. However, this is only one example, and some components of the map generation device 3 may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0020] Next, we will explain the operation of the map generation system shown in Figure 1. Sensor 1 measures the distance to one or more three-dimensional points on an object present within a sub-region, switching between sub-regions, which are parts of the observation area, as time progresses. If sensor 1 is implemented by, for example, LiDAR, sensor 1 receives reflected light from a laser pulse irradiated onto an object in the observation area, while switching the irradiation direction of the laser pulse at predetermined measurement intervals. The irradiation range of the laser pulse by sensor 1 is within a portion of the observation area. Sensor 1 calculates the distance to the 3D point of the object to which the laser pulse was irradiated, based on the time from when the laser pulse is irradiated until the reflected light is received. Sensor 1 outputs 3D point cloud data, indicating the distance to each 3D point, to the map generation device 3.

[0021] Camera 2 captures the same observation area as sensor 1 at predetermined observation intervals. The camera 2 outputs the photographed data of the observation area to the map generation device 3 for each observation period. The photographing range by the camera 2 is, for example, the entire observation area including the irradiation range of the laser pulse by the sensor 1.

[0022] FIG. 4 is a flowchart showing a map generation method which is a processing procedure of the map generation device 3. The point cloud data acquisition unit 11 receives, from the sensor 1, three-dimensional point cloud data PD n (n = 1, ···, N) at the measurement time t n and acquires it (step ST1 in FIG. 4). N is an integer of 2 or more. The point cloud data acquisition unit 11 outputs the three-dimensional point cloud data PD n at the measurement time t n to the data extraction unit 13.

[0023] The photographed data acquisition unit 12 receives, from the camera 2, photographed data SD m (m = 1, ···, M) at the photographing time t m and acquires it (step ST2 in FIG. 4). M is an integer of 2 or more. The photographed data acquisition unit 12 outputs the photographed data SD m at the photographing time t m to the data extraction unit 13. Here, after the point cloud data acquisition unit 11 acquires the three-dimensional point cloud data PD n , the photographed data acquisition unit 12 acquires the photographed data SD m . However, this is only an example, and after the photographed data acquisition unit 12 acquires the photographed data SD m , the point cloud data acquisition unit 11 may acquire the three-dimensional point cloud data PD n , or the acquisition of the three-dimensional point cloud data PD n by the point cloud data acquisition unit 11 and the acquisition of the photographed data SD m by the photographed data acquisition unit 12 may be performed in parallel.

[0024] The data extraction unit 13 receives, from the point cloud data acquisition unit 11, the measurement time t n3D point cloud data PD (n=1,···,N) n The system acquires the shooting time t from the shooting data acquisition unit 12. m (m=1,···,M) shooting data SD m Obtain it. The data extraction unit 13 extracts N 3D point cloud data PD1 to PD N From this, extract any 3D point cloud data that has not yet been extracted, which pertains to any sub-region (Step ST3 in Figure 4). For the sake of explanation, here, the data extraction unit 13 is defined as the measurement time t n 3D point cloud data PD n Assume that the following is being extracted.

[0025] The data extraction unit 13 extracts M images of data SD1 to SD M From among these, the shooting data for each sub-region is the measurement time t of the distance in each sub-region by sensor 1. n Extract the shooting data corresponding to the shooting time (Step ST4 in Figure 4). Specifically, the data extraction unit 13 processes the measurement time t n 3D point cloud data PD n If extracted, M shooting data SD1~SD M From among them, measurement time t n Then, the shooting data with the closest shooting time is extracted. Measurement time t n The closest shooting time is, for example, t m In that case, the data extraction unit 13 extracts M images of shooting data SD1 to SD M From among them, shooting time t m Shooting data SD m Extract it. Measurement time t n The closest shooting time is, for example, t m-1 In that case, the data extraction unit 13 extracts M images of shooting data SD1 to SD M From among them, shooting time t m-1 Shooting data SD m-1 Extract it. Measurement time t n The closest shooting time is, for example, t m+1In that case, the data extraction unit 13 extracts M images of shooting data SD1 to SD M From among them, shooting time t m+1 Shooting data SD m+1 Extract it. For the sake of explanation, here, the data extraction unit 13 is defined as the shooting time t m Shooting data SD m Assume that the following is being extracted. Each time the data extraction unit 13 extracts shooting data corresponding to the measurement time, it outputs the 3D point cloud data for the measurement time and the shooting data for the corresponding shooting time to the map generation unit 14.

[0026] The map generation unit 14 receives 3D point cloud data PD from the data extraction unit 13. n and shooting data SD m To obtain and The map generation unit 14 generates 3D point cloud data PD. n Based on this, a map of the subregion is generated (step ST5 in Figure 4). The process of generating a map based on 3D point cloud data is a well-known technique, so a detailed explanation is omitted.

[0027] The map generation unit 14 applies the captured data SD to the map of the generated subregion. m Add (Step ST6 in Figure 4). Specifically, the map generation unit 14 generates 3D point cloud data PD n From this, we extract the 3D points of the objects contained within the sub-region. These 3D points are, for example, the feature points of the objects. The process of extracting the feature points of objects is a well-known technique, so a detailed explanation is omitted. As shown in Figure 5, the map generation unit 14 receives the captured data SD m From this data, color information of pixels corresponding to the object's 3D points is extracted. Since the process of extracting color information of pixels corresponding to 3D points is a well-known technique, a detailed explanation is omitted. Figure 5 is an explanatory diagram showing a captured image containing pixels corresponding to the 3D points of an object included in the 3D point cloud data.

[0028] The map generation unit 14 adds color information to distance information indicating the distance to the 3D points of an object, thereby generating 3D point cloud data PD. n For maps generated based on the image data SD m Add it. Distance information indicates the distance from sensor 1 to the 3D point, and color information indicates the color of the 3D point; therefore, the 3D point cloud data PD has color information added. n This data includes information indicating the distance from sensor 1 to the 3D point and information indicating the color of the 3D point. 3D point cloud data PD n Measurement time t related to n and shooting data SD m Time of shooting related to t m Because the time periods are close together, there are almost no areas with color shifts in the generated subregion maps.

[0029] The data extraction unit 13 extracts N 3D point cloud data PD1 to PD N If there is any 3D point cloud data that has not yet been extracted (step ST7 in Figure 4: YES), the 3D point cloud data that has not yet been extracted is extracted (step ST3 in Figure 4). For the sake of explanation, here, the data extraction unit 13 is defined as the measurement time t n+1 3D point cloud data PD n+1 Assume that the following is being extracted. The data extraction unit 13 extracts M images of data SD1 to SD M From among these, the imaging data relating to a partial region is measured at time t n+1 Extract the shooting data corresponding to the shooting time (Step ST4 in Figure 4). For the sake of explanation, here, the data extraction unit 13 is defined as the shooting time t m+1 Shooting data SD m+1 Assume that the following is being extracted.

[0030] The map generation unit 14 receives 3D point cloud data PD from the data extraction unit 13. n+1 and shooting data SD m+1 To obtain and The map generation unit 14 generates 3D point cloud data PD. n+1 Based on this, a map of the subregion is generated (step ST5 in Figure 4). The map generation unit 14 applies the captured data SD to the map of the generated subregion. m+1 Add (Step ST6 in Figure 4). N 3D point cloud data PD1~PD N If there is no remaining 3D point cloud data that has not yet been extracted (in the case of step ST7:NO in Figure 4), the series of processes by the map generation device 3 will be completed.

[0031] In the above embodiment 1, the map generation device 3 is configured to include a point cloud data acquisition unit 11 that repeatedly acquires 3D point cloud data indicating the distance to each 3D point from a sensor 1 that measures the distance to one or more 3D points on an object present in a sub-region, while switching between sub-regions, which are a part of the observation area, as time progresses, and an image data acquisition unit 12 that repeatedly acquires image data of the observation area from a camera 2 that photographs the observation area. Furthermore, the map generation device 3 includes a data extraction unit 13 that extracts image data corresponding to the time of shooting of each sub-region by the sensor 1 from among a plurality of image data acquired by the image data acquisition unit 12, as image data related to each sub-region, and a map generation unit 14 that generates a map of each sub-region based on the 3D point cloud data related to each sub-region and adds the image data related to each sub-region to the map of each sub-region.Therefore, the map generation device 3 can suppress the occurrence of areas with color misalignment in the map of the observation area.

[0032] Embodiment 2. In Embodiment 2, the map generation unit 15 generates maps of multiple sub-regions, and when there are multiple maps of sub-regions containing the same object, the map generation device 3 selects the map from among the multiple maps of sub-regions containing the same object that has the closest distance from sensor 1 to the 3D point of the same object.

[0033] Figure 6 is a configuration diagram showing a map generation system including a map generation device 3 according to Embodiment 2. In Figure 6, the same reference numerals as in Figure 1 indicate the same or corresponding parts, so a detailed explanation is omitted. Figure 7 is a hardware configuration diagram showing the hardware of the map generation device 3 according to Embodiment 2. In Figure 7, the same reference numerals as in Figure 2 indicate the same or corresponding parts, so a detailed explanation is omitted. The map generation system shown in Figure 6 includes a sensor 1, a camera 2, and a map generation device 3.

[0034] The map generation unit 15 is implemented, for example, by the map generation circuit 25 shown in Figure 7. The map generation unit 15 acquires 3D point cloud data and image data related to each sub-region from the data extraction unit 13. The map generation unit 15, similar to the map generation unit 14 shown in Figure 1, generates maps of each sub-region based on the 3D point cloud data related to each sub-region, and adds the image data related to each sub-region to the maps of each sub-region. Unlike the map generation unit 14 shown in Figure 1, the map generation unit 15 generates maps of multiple sub-regions. If there are multiple maps of sub-regions that contain the same object, it selects the map from among the multiple maps of sub-regions that contain the same object that has the shortest distance from sensor 1 to the 3D point of the same object.

[0035] In Figure 6, the map generation device 3 is assumed to be implemented by dedicated hardware as shown in Figure 7, with each of its components being the point cloud data acquisition unit 11, the image data acquisition unit 12, the data extraction unit 13, and the map generation unit 15. Specifically, the map generation device 3 is assumed to be implemented by the point cloud data acquisition circuit 21, the image data acquisition circuit 22, the data extraction circuit 23, and the map generation circuit 25. Each of the point cloud data acquisition circuit 21, image data acquisition circuit 22, data extraction circuit 23, and map generation circuit 25 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0036] The components of the map generation device 3 are not limited to those implemented by dedicated hardware; the map generation device 3 may also be implemented by software, firmware, or a combination of software and firmware. If the map generation device 3 is implemented by software or firmware, a program that causes a computer to execute the respective processing procedures in the point cloud data acquisition unit 11, the image data acquisition unit 12, the data extraction unit 13, and the map generation unit 15 is stored in the memory 31 shown in Figure 3. Then, the processor 32 shown in Figure 3 executes the program stored in the memory 31.

[0037] Furthermore, Figure 7 shows an example in which each component of the map generation device 3 is implemented by dedicated hardware, while Figure 3 shows an example in which the map generation device 3 is implemented by software or firmware, etc. However, this is merely one example, and some components of the map generation device 3 may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0038] Next, the operation of the map generation system shown in Figure 6 will be explained. Except for the map generation unit 15, it is the same as the map generation system shown in Figure 1. Therefore, only the operation of the map generation unit 15 will be explained here. For example, if sensor 1 is mounted on a moving object such as an automobile, sensor 1 moves over time, so even if the laser pulse from sensor 1 is shone on the same object, the distance from sensor 1 to the object may differ. When comparing the distance measurement accuracy when the distance from Sensor 1 is far with the distance measurement accuracy when the distance from Sensor 1 is close, the distance measurement accuracy when the distance from Sensor 1 is close is higher.

[0039] FIG. 8 is an explanatory diagram showing an example where the same object is present in a partial area of three maps. In the example of FIG. 8, the distance from Sensor 1 to the object in the map based on the three-dimensional point cloud data PD3 at the measurement time t3 is L3, the distance from Sensor 1 to the object in the map based on the three-dimensional point cloud data PD4 at the measurement time t4 is L4, and the distance from Sensor 1 to the object in the map based on the three-dimensional point cloud data PD5 at the measurement time t5 is L5. In the example of FIG. 8, L4 < L5 < L3.

[0040] After generating maps of a plurality of partial areas, when there are a plurality of maps of partial areas containing the same object among the maps of the plurality of partial areas, the map generation unit 15 selects the map with the shortest distance from Sensor 1 to the three-dimensional point of the object from among the plurality of maps. In the example of FIG. 8, since L4 < L5 < L3, the map generation unit 15 selects the map based on the three-dimensional point cloud data PD4 from among the maps based on the three-dimensional point cloud data PD3, the map based on the three-dimensional point cloud data PD4, and the map based on the three-dimensional point cloud data PD5. The map generation unit 15 discards each of the map based on the three-dimensional point cloud data PD3 and the map based on the three-dimensional point cloud data PD5. The map generation unit 15 adds the captured data related to the partial area to the selected map.

[0041] In the above Embodiment 2, after the map generation unit 15 generates maps of a plurality of partial areas, when there are a plurality of maps of partial areas containing the same object among the maps of the plurality of partial areas, the map generation device 3 is configured to select the map with the shortest distance from Sensor 1 to the three-dimensional point of the same object from among the maps of the plurality of partial areas containing the same object. Therefore, the map generation device 3 can suppress the occurrence of color deviation in the map of the observation area and can generate a map with high distance measurement accuracy.

[0042] Embodiment 3. Embodiment 3 describes a map generation device 3 in which the map generation unit 16 performs adjustment processing to match the maps of each sub-region to the self-position and orientation of the sensor 1.

[0043] Figure 9 is a configuration diagram showing a map generation system including a map generation device 3 according to Embodiment 3. In Figure 9, the same reference numerals as in Figures 1 and 6 indicate the same or corresponding parts, so a detailed explanation is omitted. Figure 10 is a hardware configuration diagram showing the hardware of the map generation device 3 according to Embodiment 3. In Figure 10, the same reference numerals as in Figures 2 and 7 indicate the same or corresponding parts, so a detailed explanation is omitted. The map generation system shown in Figure 9 includes a sensor 1, a camera 2, and a map generation device 3.

[0044] The map generation unit 16 is implemented, for example, by the map generation circuit 26 shown in Figure 10. The map generation unit 16 acquires 3D point cloud data and image data related to each sub-region from the data extraction unit 13. The map generation unit 16 generates maps of each sub-region based on 3D point cloud data related to each sub-region, similar to the map generation unit 14 shown in Figure 1. The map generation unit 16 performs adjustment processing to match the map of each sub-region to the self-position and orientation of the sensor 1. The map generation unit 16 adds the image data related to each sub-region to the map of each sub-region after the adjustment process.

[0045] The map generation system shown in Figure 9 is an example in which the map generation unit 16 is applied to the map generation system shown in Figure 1. However, this is merely one example, and the map generation unit 16 may also be applied to the map generation system shown in Figure 6.

[0046] In Figure 9, the map generation device 3 is assumed to be implemented by dedicated hardware as shown in Figure 10, with each of its components—the point cloud data acquisition unit 11, the image data acquisition unit 12, the data extraction unit 13, and the map generation unit 16—being implemented by dedicated hardware. Specifically, the map generation device 3 is assumed to be implemented by a point cloud data acquisition circuit 21, an image data acquisition circuit 22, a data extraction circuit 23, and a map generation circuit 26. Each of the point cloud data acquisition circuit 21, image data acquisition circuit 22, data extraction circuit 23, and map generation circuit 26 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0047] The components of the map generation device 3 are not limited to those implemented by dedicated hardware; the map generation device 3 may also be implemented by software, firmware, or a combination of software and firmware. If the map generation device 3 is implemented by software or firmware, a program that causes a computer to execute the respective processing procedures in the point cloud data acquisition unit 11, the image data acquisition unit 12, the data extraction unit 13, and the map generation unit 16 is stored in the memory 31 shown in Figure 3. Then, the processor 32 shown in Figure 3 executes the program stored in the memory 31.

[0048] Furthermore, Figure 10 shows an example in which each component of the map generation device 3 is implemented by dedicated hardware, while Figure 3 shows an example in which the map generation device 3 is implemented by software or firmware, etc. However, this is merely one example, and some components of the map generation device 3 may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0049] Next, the operation of the map generation system shown in Figure 9 will be explained. Except for the map generation unit 16, it is the same as the map generation system shown in Figure 1. Therefore, only the operation of the map generation unit 16 will be explained here.

[0050] The map generation unit 16 acquires 3D point cloud data and image data related to each sub-region from the data extraction unit 13. The map generation unit 16 generates maps of each sub-region based on 3D point cloud data related to each sub-region, similar to the map generation unit 14 shown in Figure 1. The map generation unit 16 performs adjustment processing to match the maps of each sub-region to the self-position and orientation of sensor 1. This adjustment processing includes, for example, matching the tilt of the map to the tilt of the distance image indicated by the 3D point cloud data, as well as correcting map distortion to match the shape of the map to the shape of the distance image. Since the adjustment processing itself is a well-known technique, a detailed explanation is omitted. The map generation unit 16 adds the image data related to each sub-region to the map of each sub-region after the adjustment process.

[0051] In the above embodiment 3, the map generation unit 16 performs an adjustment process to match the map of each sub-region to the self-position and orientation of the sensor 1, and the map generation device 3 is configured to add the image data related to each sub-region to the map of each sub-region after the adjustment process. Therefore, the map generation device 3 can suppress the occurrence of areas with color misalignment in the map of the observation area.

[0052] Furthermore, this disclosure allows for free combination of each embodiment, modification of any component in each embodiment, or omission of any component in each embodiment. [Explanation of Symbols]

[0053] 1 Sensor, 2 Camera, 3 Map generation device, 11 Point cloud data acquisition unit, 12 Shooting data acquisition unit, 13 Data extraction unit, 14, 15, 16 Map generation unit, 21 Point cloud data acquisition circuit, 22 Shooting data acquisition circuit, 23 Data extraction circuit, 24, 25, 26 Map generation circuit, 31 Memory, 32 Processor.

Claims

1. A point cloud data acquisition unit repeatedly acquires 3D point cloud data indicating the distance to each 3D point from a sensor that measures the distance to one or more 3D points on an object present in a sub-region, while switching between sub-regions, which are a part of the observation area, over time. A camera that photographs the aforementioned observation area is used to repeatedly acquire photographic data of the aforementioned observation area from the camera, and a photographic data acquisition unit is used to acquire photographic data of the aforementioned observation area. A data extraction unit extracts from among multiple shooting data acquired by the aforementioned shooting data acquisition unit the shooting data corresponding to the shooting time of the distance measurement time in each sub-region by the sensor, as shooting data related to each sub-region. A map generation unit generates a map of each sub-region based on 3D point cloud data related to each sub-region, and adds photographic data related to each sub-region to the map of each sub-region. A map generation device equipped with this device.

2. The map generation unit, The map generation device according to claim 1, characterized in that, after generating maps of multiple sub-regions, if there are multiple maps of sub-regions that contain the same object, the map that has the closest distance from the sensor to the 3D point of the same object is selected from among the multiple maps of sub-regions that contain the same object.

3. The map generation unit, From the 3D point cloud data for each sub-region, the 3D points of objects existing within each sub-region are extracted. From the captured image shown by the captured data for each sub-region, color information of pixels corresponding to the three-dimensional points of the object is extracted. The map generation apparatus according to claim 1, characterized in that it adds the color information to the information indicating the distance from the sensor to the three-dimensional point of the object, thereby adding the image data relating to each sub-region to the map of each sub-region.

4. The map generation unit, The map generation device according to claim 1, characterized in that it performs an adjustment process to match the map of each sub-region to the self-position and orientation of the sensor, and adds the image data relating to each sub-region to the map of each sub-region after the adjustment process.

5. The point cloud data acquisition unit repeatedly acquires 3D point cloud data indicating the distance to each 3D point from sensors that measure the distance to one or more 3D points on an object present within a sub-region, while switching between sub-regions, which are a part of the observation area, as time progresses. The image data acquisition unit repeatedly acquires image data of the observation area from the camera that photographs the observation area. The data extraction unit extracts from the multiple shooting data acquired by the shooting data acquisition unit the shooting data corresponding to the shooting time of the distance measurement time in each sub-region by the sensor, as shooting data related to each sub-region. The map generation unit generates maps of each sub-region based on the 3D point cloud data for each sub-region, and then adds the image data for each sub-region to the maps of each sub-region. Map generation method.

6. A sensor that measures the distance to one or more three-dimensional points on an object present within a sub-region, while switching between sub-regions, which are parts of the observation area, over time. A camera for photographing the aforementioned observation area, A point cloud data acquisition unit repeatedly acquires 3D point cloud data from the aforementioned sensor, indicating the distance to each 3D point. A data acquisition unit that repeatedly acquires image data of the observation area from the camera, A data extraction unit extracts from among multiple shooting data acquired by the aforementioned shooting data acquisition unit the shooting data corresponding to the shooting time of the distance measurement time in each sub-region by the sensor, as shooting data related to each sub-region. A map generation unit generates a map of each sub-region based on 3D point cloud data related to each sub-region, and adds photographic data related to each sub-region to the map of each sub-region. A map generation system equipped with this feature.

Images