OPTICAL DATA PROCESSING APPARATUS, OPTICAL DATA PROCESSING METHOD, AND OPTICAL DATA PROCESSING PROGRAM

Abstract

To ensure synchronization between a plurality of pieces of optical data by a simple method.SOLUTION: A processing device 108 includes an optical data acquisition unit 301 for acquiring laser scan data obtained by a laser scanner mounted on a moving mobile and image data of a pickup image captured by a camera mounted on the moving mobile, and a delay time acquisition unit 305 for calculating a delay of Δt when the camera is commanded to pick up an image at a specific time T and the camera perform pick-up of an image with the delay of Δt with respect to a point group image viewed from a viewpoint at the specific time T which are created on the basis of, the laser scan data. A relationship between exterior orientation elements of the laser scanner and the camera is known, and pieces of optical data are processed to acquire the Δt which is obtained under a condition that a difference in a degree of overlap between the captured image and the point group image on a time axis is minimized.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a technique for synchronizing a plurality of optical data. [Background technology]

[0002] There is a known technology in which a moving object is equipped with a laser scanner, a camera, a GNSS position measuring device, an IMU, and other measuring devices, and three-dimensionally measures the surroundings while moving (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2016-57108A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above technology, for example, in order to compare the laser scan point cloud (laser scan data) acquired by a laser scanner with the image captured by a camera, and further to integrate the two, a process is required to synchronize the two.

[0005] To achieve this synchronization, a shooting command signal is output to the camera, and an exposure signal (a signal indicating the actual timing of the shutter release) is output from the camera, and the shooting time of the captured image is managed based on this exposure signal.

[0006] The method using the exposure signal described above requires the camera to have a function to output the exposure signal, and the control side also needs to be set up to handle the exposure signal. A signal transmission means for handling the exposure signal is also required.

[0007] This leads to high costs overall and reduced versatility as a system. In addition, there are many restrictions on using cameras provided by users, making the system less user-friendly.

[0008] In this context, an object of the present invention is to provide a technique capable of ensuring synchronization between multiple optical data in a simple manner. [Means for solving the problem]

[0009] The present invention includes an optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object in a moving state, a delay time acquisition unit that acquires Δt when an image is captured by the camera with a delay of Δt when an image capture command is issued to the camera at time T, and a projection unit that creates a projected image by superimposing a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image while aligning the line of sight. a camera position calculation unit that calculates the position of the camera at the time of capturing the captured image by single-photo orientation based on the correspondence between the laser scan point cloud and the captured image; The relationship between the exterior orientation elements of the laser scanner and the camera in the moving body is known, and the projection is performed multiple times by changing the viewpoint position of one or both of the point cloud image and the captured image, thereby creating multiple projected images, and Δt is calculated under a condition that a difference in the degree of overlap between the point cloud image and the captured image in the multiple projected images created is minimum or equal to or less than a threshold value; A reference value of Δt is calculated based on the difference between the time when the image capture command was issued and the calculated time of the camera position, and a range of the viewpoint position is selected based on the reference value of Δt. An optical data processing device.

[0011] In the present invention, a specific range centered on a position corresponding to the reference value of Δt is selected as the range of the viewpoint. In the present invention, the acquisition of Δt is periodically repeated. In the present invention, the acquisition of Δt is triggered by a change in the settings of the camera.

[0012] The present invention includes acquiring a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object in a moving state, acquiring Δt when a command to capture an image is issued to the camera at time T and the camera captures the image with a delay of Δt, and creating a projected image by superimposing and projecting a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image with the line of sight aligned; A single-photo orientation based on the correspondence between the laser scan point cloud and the captured image is used to calculate the position of the camera at the time of capturing the captured image. The relationship between the exterior orientation elements of the laser scanner and the camera in the moving body is known, and the projection is performed multiple times by changing the viewpoint position of one or both of the point cloud image and the captured image, thereby creating multiple projected images, and Δt is obtained under a condition that a difference in the degree of overlap between the point cloud image and the captured image in the multiple projected images created is minimum or equal to or less than a threshold value; A reference value of Δt is calculated based on the difference between the time when the image capture command was issued and the calculated time of the camera position, and a range of the viewpoint position is selected based on the reference value of Δt. An optical data processing method.

[0013] The present invention is a program to be read and executed by a computer, which includes the following steps: acquiring a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object in a moving state; acquiring Δt when a command to capture an image is issued to the camera at time T and the camera captures the image with a delay of Δt; creating a projected image in which a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image are superimposed and projected with the line of sight aligned; A single-photo orientation based on the correspondence between the laser scan point cloud and the captured image is used to calculate the position of the camera at the time of capturing the captured image. a relationship between the exterior orientation elements of the laser scanner and the camera in the moving body is known, and the projection is performed multiple times by changing the viewpoint position of one or both of the point cloud image and the captured image, thereby creating multiple projected images, and Δt is found under a condition that a difference in the degree of overlap between the point cloud image and the captured image in the multiple projected images created is minimum or equal to or less than a threshold value; A reference value of Δt is calculated based on the difference between the time when the image capture command was issued and the calculated time of the camera position, and a range of the viewpoint position is selected based on the reference value of Δt. A program for optical data processing. Effect of the Invention

[0014] According to the present invention, a technique is provided that can ensure synchronization between a plurality of optical data in a simple manner. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a conceptual diagram of an embodiment. [Diagram 2] FIG. 13 is an image diagram showing a misalignment between a captured image and a point cloud image. [Diagram 3] FIG. 13 is an image diagram showing a point cloud image from a different viewpoint. [Figure 4] FIG. 13 is an image diagram showing the deviation between a captured image and a point cloud image over time. [Diagram 5] FIG. 1 is a block diagram of an embodiment. [Figure 6] FIG. 1 is a diagram showing the principle of determining exterior orientation parameters of a camera (single photo orientation (resection)). [Figure 7] 11 is a flowchart illustrating an example of a processing procedure. [Figure 8] 11 is a flowchart illustrating an example of a processing procedure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (overview) Fig. 1 is a conceptual diagram. Fig. 2 is an image diagram showing how a point cloud image and a captured image are superimposed. In this example, a vehicle 100 is equipped with a camera 101 and a laser scanner 102. While the vehicle 100 is traveling, an image of an object 200 is captured by the camera 101, and a laser scan is performed by the laser scanner 102.

[0017] Here, a shooting command signal is output to the camera 101, and upon receiving this signal, the camera 101 performs shooting. The camera 101 does not output an exposure signal or a signal equivalent thereto, that is, a signal that can determine the timing at which shooting was performed. It is also possible to adopt a camera 101 that outputs an exposure signal or a signal equivalent thereto. In this case, the exposure signal is not used, and no hardware or settings for that purpose are required.

[0018] There is a delay time Δt between when a shooting command signal is received and when the camera 101 takes a picture. The delay time Δt is unknown at the initial stage. The delay is caused by the time required for the processing required for shooting inside the camera 101. Δt differs depending on the type and model of the camera. Also, even for the same camera, Δt may differ depending on the operating mode and shooting conditions.

[0019] The laser scanner 102 manages the time when the laser scanning light is emitted and the time when the laser scanning light reflected from the target is received. The clock that keeps these times is, for example, a clock built into the GNSS position measuring device 103.

[0020] Here, it is assumed that the relationship between the positions and attitudes of the camera 101, the laser scanner 102, the GNSS position measuring device 103, and the IMU 104 in the vehicle 100 is acquired and known in advance. When the exterior orientation parameters (position and attitude) of the camera 101 are unknown, the position and attitude of the camera 101 in the vehicle 100 are acquired first by a method to be described later, and then the processing to be described below is performed. Of course, the exterior orientation parameters of the camera 101 in the vehicle 100 may be known from the beginning.

[0021] First, assume that while the vehicle 100 is moving, a shooting command signal is output to the camera 101 at time T. In other words, assume that at time T, the camera 101 is instructed to shoot.

[0022] Then, through post-processing, a point cloud image is generated based on the laser scan point cloud acquired by the laser scanner 102, with the viewpoint being the position at time T. This point cloud image is an image that shows the apparent distribution state of the point cloud as seen from position X at time T. Here, the position X is the position of the projection origin (optical origin) of the camera 101 at time T. Fig. 3 shows an image of obtaining a point cloud image with position X at time T as the viewpoint.

[0023] The position of the viewpoint and the time are measured. That is, the position of the vehicle 100 is measured by the GNSS position measuring device 103, the changes in speed and direction are measured by the IMU 106, and the rotation of the wheels of the vehicle 100 is measured by the wheel encoder 107. From these measured values, the position of the viewpoint of the point cloud image can be known. In addition, since the time of positioning can be obtained with the GNSS, the time associated with the positioning can also be obtained. Therefore, the position of the viewpoint on the moving vehicle 100 and the time at that position can be obtained.

[0024] The time when the shooting command signal is output from the arithmetic device 108 and the time when it is received by the camera 101 can be considered to be the same. Here, it is assumed that the shooting command signal is issued at time T. In this case, the camera 101 shoots an image with a delay of Δt. Note that the shooting time is the time when exposure starts. The center time of the exposure time or the time when exposure ends can also be used as the shooting time.

[0025] Figure 2 shows a state in which a point cloud image created based on the laser scan point cloud acquired by the laser scanner 102, with the viewpoint being a position at time T, is superimposed (superimposed and projected) on an image captured by the camera 101 at time T+Δt upon a shooting command being issued at time T.

[0026] Here, the exterior orientation parameters of the laser scanner 102 and the camera 101 on the vehicle 100 are known. Therefore, a point cloud image can be generated based on the laser scan point cloud acquired by the laser scanner 102, aligned with the optical axis direction of the image captured by the camera 101 and with a viewpoint at a position assumed to be the projection origin (optical origin) of the camera 101.

[0027] If the camera 101 captures an image at time T, the point cloud image and the captured image will ideally overlap completely.

[0028] In the case of Fig. 2, since there is a delay time Δt in the operation of the camera 101, when a point cloud image based on the laser scan point cloud obtained by the laser scanner 102 with respect to time T is superimposed on an image captured by the camera 101, a misalignment occurs between the two. Note that if the vehicle 100 is not moving, the misalignment in Fig. 2 does not occur even if 0<Δt.

[0029] If an exposure signal is output from the camera 101, Δt can be grasped, and so by moving one of the images on the time axis, that is, by moving the viewpoint position of one of the images by a distance corresponding to Δt, the two images can be superimposed without any misalignment. In other words, they can be synchronized. In conventional technology, this method ensured synchronization between the image captured by the camera and the point cloud image derived from the laser scan.

[0030] In this embodiment, since the exposure signal is not used, Δt is searched for by the following method. Here, Δt is estimated by moving one image on the time axis and evaluating the degree of overlap of the two images. Here, a case where a point cloud image based on a laser scan point cloud is moved on the time axis is described.

[0031] 1, Δt corresponds to a movement amount Δx of the vehicle 100 during that time. The movement of the vehicle 100 is measured by the functions of the GNSS position measurement device 103, the IMU 106, and the wheel encoder 107. Based on these measurement values, the movement trajectory of the vehicle 100 associated with time can be calculated.

[0032] Based on this movement trajectory, the viewpoint position for viewing the laser scan point cloud is calculated for each 1 ms, for example, the viewpoint position at T+1 ms, the viewpoint position at T+2 ms, the viewpoint position at T+3 ms, ... Note that the viewpoint position can also be calculated more precisely to improve accuracy.

[0033] Then, point cloud images are created by shifting the viewpoint position every 1 ms: a point cloud image viewed from a position at T+1 ms, a point cloud image viewed from a position at T+2 ms, a point cloud image viewed from a position at T+3 ms, etc. The viewpoint position is set as the position of the projection origin of the camera 101, and the line of sight direction is aligned with the direction of the optical axis of the camera 101.

[0034] Since the laser scan point cloud is distributed on an absolute coordinate system and the position of each point is fixed, the point cloud image with the viewpoint position changed can be obtained by calculation. Figure 3 shows an image of the point cloud image when the viewpoint position is shifted.

[0035] The point cloud images with the above-mentioned viewpoint position gradually shifted are created, for example, every 1 ms between T and T+30 ms. Here, the upper limit is set to 30 ms because the upper limit of Δt is estimated to be about 30 ms. This value is determined by the performance of the camera used. Usually, this upper limit value is selected from the range of about 25 ms to 50 ms.

[0036] Then, a superimposed projection image is created by superimposing and projecting a point cloud image viewed from a position at T+1 ms, a point cloud image viewed from a position at T+2 ms, a point cloud image viewed from a position at T+3 ms, etc., and an image captured by camera 101 obtained by issuing a command (instruction) to perform shooting at time T. In this case, a total of 30 superimposed projection images are created.

[0037] 4 is an image diagram of a superimposed projection image showing the state of superimposition of the point cloud image and the photographed image when the shift time is changed. Here, the point cloud image is obtained based on the laser scan point cloud obtained by the laser scanner 102 and according to the principle of FIG. 3, and the photographed image is an image taken by the camera 101.

[0038] For ease of understanding, Fig. 4 shows a case where point cloud images are generated by shifting the viewpoint position every 5 ms, and then projected by superimposing them on the captured image. By shifting the viewpoint position of the point cloud image, the degree of deviation between the point cloud image and the captured image taken at time T + Δt changes.

[0039] Figure 4 shows the case where the difference between the point cloud image seen from the viewpoint position corresponding to Δt = 20 ms and the captured image is the smallest. In this case, it is estimated that the actual capture was performed about 20 ms after the capture command signal was output, and Δt = 20 ms is acquired as the delay time.

[0040] Incidentally, when two captured images are taken from the same viewpoint with the same line of sight, the two captured images will overlap. Here, the superimposed projection image in FIG. 4 is obtained by gradually shifting the viewpoint position of the point cloud image while keeping the line of sight aligned. Therefore, in a situation where the deviation between the point cloud image and the captured image is minimized, the deviation (difference) between the viewpoint position of the point cloud image and the viewpoint position of the captured image will also be minimized. For this reason, the task in FIG. 4 can also be considered as a task of searching for the viewpoint position (camera position) of the captured image by searching for the viewpoint position of the point cloud image where the point cloud image and the captured image match.

[0041] In this way, by creating a point cloud image with a minute shift in the viewpoint position and comparing it with the image captured by the camera 101, an approximation of the actual Δt can be obtained. Also, the camera position of the captured image can be obtained. Note that by making the difference in the shift time more minute, Δt can be obtained more precisely.

[0042] Here, the method of shifting the point cloud image on the time axis has been shown, but the captured images may also be shifted on the time axis. Also, a method of shifting both the point cloud image and the captured images on the time axis is possible.

[0043] In this manner, the actual Δt is estimated, and the delay time Δt from when an image capture command is issued until when the image capture is actually performed is obtained. Δt may be obtained periodically, and the value of Δt may be updated periodically.

[0044] Once Δt is known, synchronization processing is performed. For example, the point cloud image based on the laser scan is shifted on the time axis, and the position at time T+Δt is set as the viewpoint. Figure 3 shows a case where the viewpoint is shifted from the viewpoint position at time T to the viewpoint position at time T+Δt, that is, from position X corresponding to time T to position X+Δx corresponding to time T+Δt, to generate a point cloud image. This makes it possible to align the viewpoint positions of the point cloud image based on the laser scan and the captured image on the time axis, i.e., to achieve synchronization.

[0045] In the above explanation, the case where Δt is obtained by post-processing after laser scanning and photographing, and the viewpoint position of the point cloud image and / or photographed image is corrected based on the obtained Δt has been explained, but it is also possible to obtain Δt in parallel while performing the laser scanner and photographing. In this case, when Δt is obtained, a photographing command signal can be sent to the camera 101 early, taking into account the delay of Δt.

[0046] For example, if it is desired to have the camera 101 capture an image at time T, a capture command signal is output to the camera 101 at time T-Δt, that is, at a time Δt before time T. In this way, capture is performed with a delay of Δt from the output of the capture command signal, and capture by the camera 101 is performed exactly at time T. This allows a captured image to be obtained that is synchronized with the point cloud image created based on time T.

[0047] Alternatively, if a command to capture is issued at time T, the actual capture is performed at time T+Δt, and the point cloud image is generated based on time T+Δt. This allows the captured image and the point cloud image to be synchronized.

[0048] 1. First embodiment 1 shows a vehicle 100, which is an example of a moving body. The vehicle 100 is equipped with a camera 101, a laser scanner 102, a GNSS position measuring device 103, an IMU (inertial measurement unit) 106, a wheel encoder 107, and a computing device 108.

[0049] The camera 101 is a digital still camera that captures still images. A camera that can capture moving images can also be used. In this example, the camera 101 repeatedly captures still images at specific time intervals. When capturing moving images, frame images that make up the moving image are used.

[0050] The laser scanner 102 obtains laser scan data by scanning a wide range or a specific range with a laser beam for distance measurement. For example, a pulsed laser beam is scanned linearly along a vertical plane at a repetition frequency of several kHz to several hundred kHz. A specific range is laser scanned by performing the above scan while the vehicle 100 is traveling. It is also possible to use a laser scanner that simultaneously irradiates multiple laser distance measurement beams distributed in a planar shape to simultaneously obtain laser scan data for a certain range.

[0051] The GNSS position measuring device 103 measures the position in an absolute coordinate system (global coordinate system) based on a navigation signal transmitted from a navigation satellite such as a GPS satellite. The absolute coordinate system is a coordinate system used to describe map information. In the absolute coordinate system, a position is specified by, for example, latitude, longitude, and altitude. The IMU (Inertial Measurement Unit) 106 measures the change in acceleration and orientation. The wheel encoder 107 detects the rotation of the wheels of the vehicle 100 and measures the travel distance (movement amount) of the vehicle 100.

[0052] The movement path and movement amount of the vehicle 100 linked to the time and position are calculated from changes in the measurement values ​​of the GNSS position measurement device 103, the acceleration and direction change of the vehicle 100 measured by the IMU 106, and the traveled distance of the vehicle 100 measured by the wheel encoder 107. The GNSS position measurement device 103 is equipped with a high-precision clock, and the time on the vehicle 100 is determined using this clock.

[0053] FIG. 5 shows a block diagram of the arithmetic device 108. The arithmetic device 108 is a computer and includes a CPU, a data storage device, an input / output interface, and a communication device. A general-purpose PC (personal computer) can be used as the arithmetic device 108. The arithmetic device 108 may be configured with dedicated hardware. A form in which the processing in the arithmetic device 108 is performed by a server is also possible. A form in which the functions of the arithmetic device 108 are distributed and performed using multiple computers is also possible.

[0054] The calculation device 108 has an optical data acquisition unit 300, a shooting command signal output unit 301, a movement amount calculation unit 302, a point cloud generation unit 303, a point cloud feature point calculation unit 304, a delay time (Δt) acquisition unit 305, a camera shooting time calculation unit 306, a camera position and orientation calculation unit 307, a point cloud feature point image projection unit 308, an image feature point calculation unit 309, an inter-feature point image residual calculation unit 310, and a synchronization processing unit 312.

[0055] These functional units are realized by executing software on a computer constituting the arithmetic device 108. It is also possible to configure one or more of the functional units shown in Fig. 5 with dedicated hardware.

[0056] The optical data acquisition unit 300 acquires image data of the image captured by the camera 101 and laser scan data acquired by the laser scanner 102. In addition, the optical data acquisition unit 300 acquires laser scan point cloud data based on the laser scan data acquired by the laser scanner 102.

[0057] The photographing command signal output unit 301 outputs a signal commanding (instructing) the camera 101 to photograph. For example, at time T in FIG.

[0058] The movement amount calculation unit 302 calculates the movement amount and movement direction of the vehicle 100 based on the change in position of the vehicle 100 measured by the GNSS position measurement device 103, the change in speed and change in direction of the vehicle 100 measured by the IMU 106, and the number of rotations of the wheels of the vehicle 100 measured by the wheel encoder 107. For example, the movement amount and movement direction of the vehicle 100 for Δx in Fig. 1 or every 1 ms are calculated. Since the GNSS position measurement device 103 has a clock, the calculated movement amount and movement direction are linked to the time.

[0059] The point cloud generating unit 303 generates a laser scan point cloud based on the laser scan data acquired by the laser scanner 102. The laser scanner 102 measures the direction of a reflection point of the laser scan light (the direction as seen from the laser scanner) and the distance to that point, and outputs data on the direction and distance to this reflection point as laser scan data. Based on this direction and distance, the three-dimensional coordinates of the reflection point (laser scan point) are calculated. This process is performed by the point cloud generating unit 303. A collection of the reflection points whose three-dimensional coordinates have been calculated becomes a laser scan point cloud. Note that the laser scanner 102 may be one that directly outputs a laser scan point cloud.

[0060] The point cloud feature point calculation unit 304 calculates feature points of the object described by the laser scan point cloud based on the laser scan point cloud generated by the point cloud generation unit 303. For example, in the case of FIG. 1, feature points that characterize the shape of the building 200 are calculated based on the laser scan point cloud of the building 200. An image of these feature points becomes a point cloud feature point image. This point cloud feature point image is an example of a point cloud image, and is an image of the feature points of the object resulting from the laser scan data.

[0061] There are two methods for calculating feature points that characterize the shape of the building 200. The first method is a method for extracting feature points that characterize the shape of the building 200 from a laser scan point cloud that targets the building 200. The second method is a method for creating a three-dimensional model of the building 200 based on a laser scan point cloud that targets the building 200, and extracting feature points that characterize the shape of the building 200 from the three-dimensional model.

[0062] The delay time (Δt) acquisition unit 305 acquires the delay time Δt from when a shooting command signal is issued to the camera 101 to shoot an image (when a shooting command signal is output) to when the camera 101 actually shoots an image. Δt is acquired using the method described in relation to Figures 1 to 4. Specifically, the method illustrated in Figure 4 evaluates the deviation between the shot image and the point cloud image, and acquires the delay time Δt from the condition that minimizes this deviation.

[0063] The camera shooting time calculation unit 306 calculates the shooting time of the camera 101 based on the above-mentioned Δt. For example, assume that a shooting command signal is output to the camera 101 at time T on the clock used in the arithmetic device 108. In this case, the time T+Δt is the time when the camera 101 captures the image (shooting time).

[0064] The shooting time of the camera 101 can also be obtained from the viewpoint position of the point cloud image searched for using the principle of FIG. 4. In the projected image exemplified in FIG. 4, when the point cloud image and the captured image match, the viewpoint positions of the point cloud image and the captured image match. Therefore, the viewpoint position of the point cloud image at this time becomes the shooting position (camera position) of the camera 101. Since the position of the camera 101 in the vehicle 100 is known, and the relationship between the movement trajectory of the vehicle 100 and time has been acquired, if the shooting position of the camera 101 is known, the time at that position can be known. The time at this position becomes the time when the image was captured by the camera 101. In this case, if the shooting time by the camera 101 is T+Δt and the time when the camera 101 is commanded to capture is T, Δt can be obtained.

[0065] The camera position and orientation calculation unit 307 calculates the exterior orientation parameters (position and orientation) of the camera 101 on the vehicle 100. This processing will be described later.

[0066] The point cloud feature point image projection unit 308 projects the point cloud feature point image calculated by the point cloud feature point calculation unit 304 based on the laser scan point cloud generated by the point cloud generation unit 303 onto the captured image, and creates a superimposed projection image by superimposing both images. Figures 2 and 4 are examples of superimposed projection images. In the process of Figure 4, a point cloud image based on the laser scan point cloud is projected onto a captured image captured by a camera, and the degree of overlap between the two images is examined. However, in this case, instead of the point cloud image itself, the point cloud feature point image in which the target feature points are extracted from the laser scan point cloud is used as the point cloud image, and this image is projected onto the captured image. Note that the captured image to be projected at this time is an image feature point image obtained by extracting feature points from the captured image.

[0067] At the above projection processing stage, the exterior orientation parameters of the laser scanner 102 and the camera 101 on the vehicle 100 are known. Therefore, a point cloud image can be generated based on the laser scan point cloud acquired by the laser scanner 102, in accordance with the optical axis direction of the image captured by the camera 101 and with the position of the viewpoint set as the projection origin (optical origin) of the camera 101.

[0068] At this stage, the position of the camera 101 at the time of shooting is unknown due to the unknown delay time Δt. Therefore, as shown below, multiple viewpoint positions of the point cloud image that become tentative camera positions on the time axis are set, and the camera 101 is assumed to be located at those positions to generate the point cloud image.

[0069] For example, the projection of the above point cloud feature point image onto the captured image (image feature point image) is performed by shifting the corresponding viewpoint position in 1 ms increments. Note that Fig. 4 shows a case where the viewpoint is shifted in 5 ms increments. In this case, since the direction of the optical axis of the camera 101 is known at this stage, the point cloud image (point cloud feature point image) is generated with the line of sight aligned with the direction of this optical axis.

[0070] The inter-feature point on-image residual calculation unit 310 calculates the residual between the superimposed point cloud image (point cloud feature point image) and the captured image (image feature point image). Specifically, it calculates the degree of misalignment between the two images shown in FIG.

[0071] The image synchronization processing unit 312 performs synchronization processing to synchronize the point cloud image based on the laser scan point cloud acquired by the laser scanner 102 with the captured image captured by the camera 101 based on the delay time acquired by the delay time acquisition unit 305.

[0072] There are several methods for synchronization processing. The first method is to move the point cloud image on the time axis and synchronize it with the captured image. For example, assume that when a command is given to the camera 101 to capture an image at time T, the image is captured with a delay of Δt. In this case, the point cloud image and the captured image can be synchronized by creating a point cloud image with a viewpoint corresponding to the position corresponding to the time T+Δt.

[0073] The second synchronization method is to move the captured image on the time axis. In this case, the image captured at time T+Δt is converted into an image seen from the viewpoint at time T. This conversion is performed, for example, by projective transformation. This makes it possible to synchronize the point cloud image derived from the laser scan data at the viewpoint at time T with the captured image captured by the camera 101. Note that it is also possible to move both the point cloud image and the captured image on the time axis (movement on the spatial axis of the viewpoint).

[0074] The third synchronization method is to take into account the delay in the shooting timing of the camera 101 and issue a shooting command at a timing advanced by Δt in advance. For example, if shooting is to be performed at time T, a shooting command signal is output to the camera 101 at time T-Δt. In this case, shooting by the camera 101 is performed at time T, which is delayed by Δt after the shooting command signal is output. In this case, synchronization between the shot image and the point cloud image created from the viewpoint at time T is ensured.

[0075] (Example of processing) An example of the processing performed by the arithmetic device 108 will be described below. Figs. 7 and 8 are flowcharts showing an example of the processing procedure. A program for executing the processing of Figs. 7 and 8 is stored in a storage device of a PC constituting the arithmetic device 108, and is executed by a CPU of the PC. The program may be stored in an appropriate storage medium. The program may be stored in a server connected to the Internet, and downloaded from there to a PC for implementing the arithmetic device 108.

[0076] Here, it is assumed that the exterior orientation parameters of the camera 101 on the vehicle 100 are unknown in the first stage, and the exterior orientation parameters of the camera 101 on the vehicle 100 are calculated first (FIG. 7). Note that if the exterior orientation parameters of the camera 101 on the vehicle 100 are known, the process of FIG. 7 is not necessary.

[0077] Here, it is assumed that the position and orientation of the camera 101 on the vehicle 100 are roughly known at an initial stage. This assumes, for example, that a user prepares the camera 101 and attaches it to the vehicle 100. In this case, the position to attach the camera is specified in advance, and the user sets the camera 101 there.

[0078] It is assumed that the relationship between the positions and attitudes of the laser scanner 102, the GNSS position measuring device 103, and the IMU 106 in the vehicle 100 is determined in advance and is known.

[0079] First, while the vehicle 100 is traveling in the direction of the X-axis in Fig. 1, a laser scanner 102 performs a laser scan of an object (e.g., a building 200), and a camera 101 photographs the same object. At this time, a GNSS position measuring device 103 measures the position of the vehicle 100 in an absolute coordinate system and its change, and determines the travel route of the vehicle 100 linked to the time. Measurement values ​​of the IMU 106 and the wheel encoder 107 are also used to determine this travel route. Furthermore, from these measurement values, the velocity vector of the vehicle 100 at each point on the travel route and at a specified time can be obtained.

[0080] After acquiring the captured image and the laser scan data, the following processing is performed as post-processing: First, the captured image of the same object by the camera 101 and the laser scan data by the laser scanner 102 are acquired by the optical data acquisition unit 301 (step S101).

[0081] Next, based on the acquired laser scan data, a laser scan point cloud is generated by the point cloud generation unit 303. Next, as a preparation for calculating the exterior orientation parameters of the camera 101, a viewpoint for generating a point cloud image is provisionally set (step S102).

[0082] This viewpoint is the initial value for calculating the camera position of the target captured image. At this stage, Δt is unknown, so the position of the viewpoint is also unknown at this stage, so here, an approximate value is set as the initial value. The camera position is understood as the position of the projection origin of the camera to be used.

[0083] For example, consider a case where the camera 101 is commanded to capture an image at time T. Here, it is assumed that the maximum delay time from the command (instruction) to capture an image to the actual capture is 30 ms. In this case, it is assumed that the center value of the 30 ms range is assumed, and the image is captured at T+15 ms. In other words, it is assumed that T+15 ms is the capture time.

[0084] Then, the position where the camera 101 is estimated to be located at the time T+15 ms is set as a tentative viewpoint position X01.

[0085] Here, it is assumed that the approximate mounting position of the camera 101 on the vehicle 100 is known. In this case, the approximate position X0 of the camera 101 at time T is known based on the approximate offset position of the camera 101 relative to the IMU 106. Here, the position X01 of the camera 101 at time T+15 ms is calculated from the position of the vehicle 100 at time T and the velocity vector V of the vehicle at time T, with the position X0 of the camera 101 at time T as an initial value. Specifically, X01 is calculated by X01=X0+(V×15 ms).

[0086] After provisionally setting the viewpoint position X01, a point cloud image is created by viewing the laser scan point cloud created earlier from that position.

[0087] Next, the correspondence between the point cloud image of the laser scan point cloud viewed from the viewpoint X01 and the captured image obtained when an image capture command is issued to the camera 101 at time T is obtained.

[0088] Next, the exterior orientation parameters of the camera 101 at the time of capturing the captured image are calculated. This process will be described below.

[0089] By determining the correspondence between the point cloud image and the captured image, the positions of many points in the captured image in the absolute coordinate system can be determined. Here, the position of the camera 101 in the absolute coordinate system is calculated by the intersection method using the many points in the captured image whose coordinates are known as reference points (orientation points).

[0090] In addition, by examining the relationship between the optical axis direction of the camera 101 and the direction of each point as viewed from the projection origin, the orientation of the camera 101 in the absolute coordinate system can be calculated. This method is a basic method of single-photo orientation. Details of this process are described in, for example, JP 2016-57108 A.

[0091] The principle of the method for determining the position and orientation of the camera will be briefly explained below. In Fig. 6, the camera is at position X, p1 to p6 are feature points in the screen of the image captured by the camera 101, and P1 to P6 are points of the laser scan point cloud corresponding to p1 to p6. Note that the camera position X is unknown, and the interior orientation parameters of the camera are known. Also, the camera position X is the projection origin (optical origin) of the camera.

[0092] Here, a direction line passing through P1 and p1, a direction line passing through P2 and p2, a direction line passing through P3 and p3, ... are created. The point where these direction lines intersect is the camera position X. Using this principle, the position (viewpoint of shooting) X1 of the camera 101 when shooting the target image is calculated. In addition, the line passing through position X1 and the center of the screen is the optical axis of the camera. From the relationship between this optical axis and the above direction lines, the attitude of the camera at camera position X1 can be found.

[0093] If the intersection point of the multiple direction lines cannot be determined, or if the range of intersection of the multiple direction lines is larger than a predetermined range, the value of the camera position X0 at time T is changed and recalculated. Instead of or in addition to changing the value of the camera position X0 and recalculating, there is also a method of reviewing the correspondence between the feature points on the captured image and the feature points on the point cloud image and recalculating. By determining the intersection point of the multiple direction lines or searching for X1 where the range of intersection of the multiple direction lines falls within a predetermined range, it is possible to obtain the position X1 of the camera 101 at time T+Δt (true shooting time) that is closer to the true value.

[0094] In this way, when an image capturing command is issued to the camera 101 at time T, the exterior orientation parameters (position and orientation) of the camera 101 at the time of image capturing performed with a delay of Δt are obtained (step S103).

[0095] The exterior orientation parameters of the camera 101 obtained at this stage are values ​​in an absolute coordinate system.

[0096] At this stage, the exterior orientation parameters of the camera 101 on the vehicle 100 are unknown. This is because, at this stage, Δt is unknown, the shooting time of the captured image is unknown, and the position of the vehicle 100 at this shooting time is unknown.

[0097] Next, Δt is calculated (step S104). Δt is calculated as follows.

[0098] Here, if the position of camera 101 at time T when the shooting command is issued is X0, the time when the shooting is taken by camera 101 is T+Δt, and the camera position at that time is X1, so the time it takes for vehicle 100 (camera 101) to move from X0 to X1 is Δt.

[0099] Here, if the speed of the vehicle 100 at time T is speed V, then V=(X1-X0) / Δt holds. That is, Δt can be calculated by Δt=(X1-X0) / V. This calculation is performed by the delay time (Δt) acquisition unit 305.

[0100] Here, X1 is the shooting position (camera position) of the camera 101 calculated according to the principle in Fig. 6. X0 is the position of the camera 101 at time T assumed as an initial condition for the calculation of X1. V is the speed of the vehicle 100 at time T.

[0101] Furthermore, since the velocity vector of the vehicle 100 at time T can be obtained based on the measurement values ​​obtained from the GNSS position measurement device 103, the IMU 106, and the wheel encoder 108, the above V can be obtained from these measurement values.

[0102] After Δt is calculated, the exterior orientation parameters (position and attitude) of the camera 101 on the vehicle 100 at the time T1=T+Δt when the image of interest is captured are obtained (step S105).

[0103] That is, by determining Δt, the actual shooting time T1=T+Δt of the camera 101 is determined. As a result, the position of the vehicle 100 at time T1, that is, the position of the vehicle 100 when the image was captured by the camera 101, is determined. In addition, the attitude of the vehicle 100 is determined from the measurement data of the IMU 106 at time T1. Then, the position of the camera 101 on the vehicle 100 is determined from the relationship between the position of the vehicle 100 at time T1 and the position X1 of the camera 101 at time T1.

[0104] Moreover, the attitude of the camera 101 in the absolute coordinate system at time T1 is calculated in step S106. Therefore, the attitude of the camera 101 in the vehicle 100 is obtained from the relationship between the attitude of the vehicle 100 in the absolute coordinate system at time T1 and the attitude of the camera 101 in the absolute coordinate system at time T1. In this manner, the exterior orientation parameters (position and attitude) of the camera 101 in the vehicle 100 are obtained. These processes are performed by the camera position and attitude calculation unit 307.

[0105] Next, a process of synchronizing an image captured by the camera 101 with a point cloud image based on laser scan data acquired by the laser scanner 102 will be described with reference to Fig. 8. Note that the process of Fig. 8 is performed in a state where the exterior orientation parameters of the camera 101 in the vehicle 100 are known.

[0106] First, the data of the laser scan point cloud obtained by the laser scanner 102 and the data (image data) of the captured image captured by the camera 101 based on a command issued at a specific time T are acquired (step S211). Here, the corresponding laser scan point cloud and captured image are acquired if they overlap for the same object.

[0107] Next, multiple point cloud images from multiple viewpoints are created based on the laser scan point cloud (step S212). For example, a point cloud image with a viewpoint at time T+1 ms, a point cloud image with a viewpoint at time T+2 ms, a point cloud image with a viewpoint at time T+3 ms, ..., a point cloud image with a viewpoint at time T+30 ms are created.

[0108] As a result, a point cloud image is created that is shifted slightly on the time axis, as shown in Fig. 4. Next, the point cloud image created in step S212 is projected onto the captured image captured by the camera 101 (step S213). This projection creates a superimposed projection image. In actual processing, a point cloud feature point image in which feature points are extracted from the point cloud image derived from the laser scan is projected onto an image of image feature points obtained from the captured image.

[0109] Next, the residual on the projection screen between the point cloud image derived from the laser scan by the laser scanner 102 and the image captured by the camera 101 is calculated (step S214), and the condition under which this residual becomes minimum is obtained (step S215). For example, in the case of the four patterns of projection images in Fig. 4, the case of T+20 ms is the case in which the residual is minimum.

[0110] Then, the value of Δt under the conditions acquired in step S215 is acquired (step S216). In the case of FIG. 4, Δt=20 ms is acquired. Finally, based on Δt acquired in step S216, a synchronization process is performed (step S217). This synchronization process ensures synchronization between the point cloud image derived from the laser scan by the laser scanner 102 and the image captured by the camera 101.

[0111] (Advantages) In this embodiment, no exposure signal is required from the camera 101. The camera 101 simply receives a shooting signal that commands shooting. Therefore, a variety of cameras can be used as the camera 101. In addition, no hardware is required to handle the exposure signal, which reduces costs. In addition, the degree of freedom and ease of setting are improved when using a camera prepared by the user.

[0112] (others) The interval between shots can be set arbitrarily. Frame images constituting a video can also be used as the shot images handled in the present invention. The delay time (time offset) Δt can be calculated periodically. In this case, Δt is updated periodically.

[0113] The moving body is not limited to a vehicle, and may be an aircraft or a ship. The moving body may be manned or unmanned.

[0114] As a condition for judging the difference in the degree of overlap between the captured image and the point cloud image in the superimposed projection image in which the captured image captured by the camera shown in Fig. 4 is superimposed on the point cloud image derived from the laser scan point cloud, the difference can be set to a predetermined threshold or less. For example, it is also possible to obtain Δt under the condition that the difference in the degree of overlap between the two images is 1% or less. This threshold can be determined based on the density of the point cloud and the required resolution.

[0115] 2. Second embodiment The viewpoint range of the point cloud image created in step S212 in Fig. 8 can be determined based on the exterior orientation parameters of the camera 101 obtained in the process of Fig. 7. As described above, by obtaining the exterior orientation parameters of the camera 101, the delay time Δt from when the camera 101 is commanded to capture images until the image is actually captured can be obtained.

[0116] This Δt is not always constant, but it is unlikely to fluctuate significantly. Therefore, the range of the time position of the viewpoint of the point cloud image (range of the viewpoint position) shown in FIG. 4 is determined using the Δt obtained from the exterior orientation parameters as a reference value.

[0117] For example, assume that Δt for the camera 101 obtained from the exterior orientation parameters at a certain time is Δt=15 ms. Also assume that the range of variation of Δt is about 10 ms. In this case, the time setting range of the viewpoint position illustrated in FIG. 4 is T+10 ms to T+20 ms.

[0118] According to this aspect, it is possible to narrow the range of the viewpoint (the range of the delay time to be temporarily set) in creating a point cloud image derived from a laser scan point cloud with a different viewpoint position. In addition, by narrowing this range, it is possible to set the viewpoint position more precisely, and it is possible to improve the accuracy of the finally obtained Δt. In addition, it is possible to reduce unnecessary calculations.

[0119] 3. Third embodiment Depending on the camera, Δt may change when the settings are changed, such as exposure time, burst speed, resolution, optical magnification, power consumption mode, etc.

[0120] When such a setting is changed, the process for acquiring Δt is executed as a trigger, thereby making it possible to respond to changes in Δt.

[0121] In addition, when a plurality of cameras are used, it is also effective to execute the process for obtaining Δt when the camera to be used is switched. [Explanation of symbols]

[0122] 100...vehicle, 101...camera, 102...laser scanner, 103...GNSS position measurement device, 106...IMU, 107...wheel encoder, 108...computing device.

Claims

1. an optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object; a delay time acquisition unit that acquires a delay time Δt when an image is captured by the camera with a delay of Δt after a command to the camera to capture an image at a time T; a projection unit that creates a projected image by superimposing a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image while aligning the line of sight; a camera position calculation unit that calculates the position of the camera at the time of capturing the captured image by single-photo orientation based on the correspondence between the laser scan point cloud and the captured image; Equipped with A relationship between the exterior orientation parameters of the laser scanner and the camera in the moving body is known; The projection is performed a plurality of times by changing the viewpoint position of one or both of the point cloud image and the captured image, thereby generating a plurality of projected images; The Δt is calculated under a condition that a difference in an overlapping state between the point cloud image and the captured image in the plurality of generated projection images is minimized or is equal to or smaller than a threshold value; The reference value of Δt is calculated based on the difference between the time when the command to capture the captured image was issued and the calculated time of the position of the camera; An optical data processing device in which a range of positions of the viewpoint is selected based on the reference value of Δt.

2. 2. The optical data processing device according to claim 1, wherein a specific range centered on a position corresponding to the reference value of .DELTA.t is selected as the range of the viewpoint.

3. 3. The optical data processing device according to claim 1, wherein the Δt is repeatedly acquired periodically.

4. 4. The optical data processing device according to claim 1, wherein acquisition of said Δt is triggered by a change in a setting of said camera.

5. Acquiring a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image taken by a camera mounted on the moving object in a moving state; Acquiring Δt when an image is captured by the camera with a delay of Δt after a command to the camera to capture an image at time T; Creating a projected image by superimposing a point cloud image of the laser scan point cloud from a specific viewpoint and the captured image with the line of sight aligned; A single-photo orientation based on the correspondence between the laser scan point cloud and the captured image is used to calculate the position of the camera at the time of capturing the captured image. having A relationship between the exterior orientation parameters of the laser scanner and the camera in the moving body is known; The projection is performed a plurality of times by changing the viewpoint position of one or both of the point cloud image and the captured image, thereby generating a plurality of projected images; The Δt is calculated under a condition that a difference in an overlapping state between the point cloud image and the captured image in the plurality of generated projection images is minimized or is equal to or smaller than a threshold value; The reference value of Δt is calculated based on the difference between the time when the command to capture the captured image was issued and the calculated time of the position of the camera; A method for optical data processing, in which a range of viewpoint positions is selected based on the reference value of Δt.

6. A program to be read and executed by a computer, To your computer Acquiring a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image taken by a camera mounted on the moving object in a moving state; Acquiring Δt when an image is captured by the camera with a delay of Δt after a command to the camera to capture an image at time T; Creating a projected image by superimposing a point cloud image of the laser scan point cloud from a specific viewpoint and the captured image with the line of sight aligned; A single-photo orientation based on the correspondence between the laser scan point cloud and the captured image is used to calculate the position of the camera at the time of capturing the captured image. Run the command, A relationship between the exterior orientation parameters of the laser scanner and the camera in the moving body is known; The projection is performed a plurality of times by changing the viewpoint position of one or both of the point cloud image and the captured image, thereby generating a plurality of projected images; The Δt is calculated under a condition that a difference in an overlapping state between the point cloud image and the captured image in the plurality of generated projection images is minimized or is equal to or smaller than a threshold value; The reference value of Δt is calculated based on the difference between the time when the command to capture the captured image was issued and the calculated time of the position of the camera; A program for optical data processing, in which the range of the viewpoint position is selected based on the reference value of Δt.

Images