OPTICAL DATA PROCESSING APPARATUS, OPTICAL DATA PROCESSING METHOD, AND OPTICAL DATA PROCESSING PROGRAM

Abstract

To provide an optical data processing device, a method, and a program for synchronizing a plurality of pieces of optical data that secure synchronization among the plurality of pieces of optical data by a simple method.SOLUTION: An operation device 108 mounted on a vehicle 100 includes an optical data acquisition unit for acquiring a laser scan point group obtained by a laser scanner 102 mounted on a moving vehicle and image data of a picked up image captured by a camera 101 mounted on the moving vehicle, a corresponding relationship identifying unit for acquiring a corresponding relationship between a point group image obtained by viewing a laser scan point group from a specific viewpoint, and a captured image, a camera position / posture calculation unit for obtaining a position of the camera at the time of capturing an image by single photograph standardization on the basis of, the corresponding relationship, and a delay time (Δt) calculation unit for calculating a delay time of Δt when the camera performs imaging with delay of Δt upon a command of imaging on the camera at a time T. The delay time (Δt) calculation unit calculates Δt from a difference between the time corresponding to the position of the camera and the time T.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for synchronizing a plurality of optical data. [Background technology]

[0002] There is a known technology in which a moving object is equipped with a laser scanner, a camera, a GNSS position measuring device, an IMU, and other measuring devices, and three-dimensionally measures the surroundings while moving (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2016-57108A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above description, for example, a process of synchronizing the laser scan point cloud acquired by the laser scanner with the image captured by the camera is required to compare and further integrate the two.

[0005] To achieve this synchronization, a shooting command signal is output to the camera, and an exposure signal (a signal indicating the actual timing of the shutter release) is output from the camera, and the shooting time of the captured image is managed based on this exposure signal.

[0006] The method using the exposure signal described above requires the camera to have a function to output the exposure signal, and the control side also needs to be set up to handle the exposure signal. In addition, a signal transmission means for handling the exposure signal is also required.

[0007] This leads to high costs overall and reduced versatility as a system. In addition, there are many restrictions on using cameras provided by users, making the system less user-friendly.

[0008] In this context, an object of the present invention is to provide a technique capable of ensuring synchronization between multiple optical data in a simple manner. [Means for solving the problem]

[0009] The present invention includes an optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving body in a moving state and image data of a photographed image captured by a camera mounted on the moving body in the moving state; a correspondence relationship specification unit that determines a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the photographed image; a camera position calculation unit that determines a position X1 of the camera at the time of photographing the photographed image by single photograph orientation based on the correspondence relationship; and a delay time calculation unit that calculates Δt when photographing by the camera is delayed by Δt when photographing is commanded to the camera at time T, The time T is the time when the camera is commanded to capture the image, and is calculated by Δt=(X1-X0) / V, where X0 is the position of the camera at the time T and V is the speed of the moving object at the time T. An optical data processing device.

[0011] In the present invention, an embodiment in which the initial condition in the single photo orientation is adopted as X0 is given.

[0012] The present invention relates to The present invention includes an optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving body in a moving state and image data of a captured image captured by a camera mounted on the moving body in a moving state; a correspondence relationship specification unit that determines a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image; a camera position calculation unit that determines a position X1 of the camera at the time of capturing the captured image by single photo orientation based on the correspondence relationship; and a delay time calculation unit that calculates Δt when the camera captures the image with a delay of Δt when the camera is commanded to capture the image at time T, wherein the time T is the time when the camera is commanded to capture the image, The relationship between the position and time of the moving body is acquired, and the position of the camera on the moving body is known. The time T1 corresponding to the camera position X1 is calculated based on X1, the relationship between the position and time of the moving body, and the position of the camera on the moving body. The Δt is calculated from the difference between the time T and the time T1. Light It is an academic data processing device.

[0013] The present invention includes an optical data acquisition step of acquiring a laser scan point cloud obtained by a laser scanner mounted on a moving body in a moving state and image data of a photographed image captured by a camera mounted on the moving body in the moving state; a correspondence relationship determination step of determining a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the photographed image; a camera position calculation step of determining a position X1 of the camera at the time of photographing the photographed image by single-photo orientation based on the correspondence relationship; and a delay time calculation step of calculating Δt when photographing by the camera is delayed by Δt when photographing is commanded to the camera at time T, The time T is the time when the camera is commanded to capture the image, and is calculated by Δt=(X1-X0) / V, where X0 is the position of the camera at the time T and V is the speed of the moving object at the time T. An optical data processing method.

[0014] The present invention is a program to be read and executed by a computer, the program being operated as an optical data acquisition unit that acquires image data of a laser scan point cloud obtained by a laser scanner mounted on a moving object and a photographed image captured by a camera mounted on the moving object in the moving state, a correspondence relationship specification unit that determines a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the photographed image, a camera position calculation unit that determines a position X1 of the camera at the time of photographing the photographed image by single photograph orientation based on the correspondence relationship, and a delay time calculation unit that calculates Δt when photographing by the camera is delayed by Δt when photographing is commanded to the camera at time T, The time T is the time when the camera is commanded to capture the image, and is calculated by Δt=(X1-X0) / V, where X0 is the position of the camera at the time T and V is the speed of the moving object at the time T. A program for optical data processing. Effect of the Invention

[0015] According to the present invention, a technique is provided that can ensure synchronization between a plurality of optical data in a simple manner. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a conceptual diagram of an embodiment. [Diagram 2] FIG. 1 is a block diagram of an embodiment. [Diagram 3]FIG. 13 is an image diagram showing a point cloud image when the viewpoint is changed. [Figure 4] FIG. 1 is a diagram showing the principle of determining exterior orientation parameters of a camera (single photo orientation (resection)). [Diagram 5] 11 is a flowchart illustrating an example of a processing procedure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] 1. First embodiment (composition) Fig. 1 is a conceptual diagram of an embodiment. Fig. 1 shows a vehicle 100, which is an example of a moving body. The vehicle 100 is equipped with a camera 101, a laser scanner 102, a GNSS position measuring device 103, an IMU (inertial measurement unit) 106, a wheel encoder 107, and a computing device 108.

[0018] In this example, while a vehicle 100 is traveling, an image of an object 200 is captured by a camera 101, and a laser scanner 102 performs laser scanning on the object 200.

[0019] The relationship between the positions and attitudes of the laser scanner 102, the GNSS position measuring device 103, and the IMU 106 in the vehicle 100 is acquired and known in advance. The approximate position and attitude of the camera 101 in the vehicle 100 is known. This assumes, for example, that the camera 101 is attached to the vehicle by a user.

[0020] The camera 101 is a digital still camera that captures still images. A camera that can capture moving images can also be used. In this example, the camera 101 repeatedly captures still images at specific time intervals. When capturing moving images, frame images that make up the moving image are used.

[0021] A shooting command signal is output to the camera 101, and upon receiving this signal, the camera 101 performs shooting. The camera 101 does not output an exposure signal or an equivalent signal, that is, a signal that can determine the timing at which shooting was performed. It is also possible to adopt a camera 101 that outputs an exposure signal or an equivalent signal. In this case, the exposure signal is not used, and no hardware or settings for that purpose are required.

[0022] There is a delay time Δt between when a shooting command signal is received and when the camera 101 takes a picture. The delay time Δt is unknown at the initial stage. The delay is caused by the time required for the processing required for shooting inside the camera 101. Δt differs depending on the type and model of the camera. Also, even for the same camera, Δt may differ depending on the operating mode and shooting conditions.

[0023] The shooting time is the time when exposure starts. The shooting time can also be the center time of the exposure time or the time when exposure ends. The position of the camera 101 is understood as the position of the projection origin (optical origin) of the camera 101.

[0024] The laser scanner 102 obtains laser scan data by scanning and irradiating a wide or specific range with a laser beam for distance measurement. For example, a linear laser scan is performed by scanning and irradiating a pulsed laser beam at a repetition frequency of several kHz to several hundred kHz along a vertical plane. The above laser scan is performed while the vehicle 100 is traveling, thereby performing a laser scan of a specific range. It is also possible to use a laser scanner that simultaneously irradiates multiple laser scan beams distributed in a planar shape to simultaneously obtain laser scan data of a certain range.

[0025] The position of the laser scanner 102 is understood as the position that is the origin of the light emission point and light reception point of the laser scanning light.

[0026] The laser scanner 102 manages the time when the laser scanning light is emitted and the time when the laser scanning light reflected from the target is received. For example, a clock built into the GNSS position measuring device 103 is used as a clock for keeping these times.

[0027] The GNSS position measuring device 103 measures the position in an absolute coordinate system (global coordinate system) based on a navigation signal transmitted from a navigation satellite such as a GPS satellite. The absolute coordinate system is a coordinate system used to describe map information. In the absolute coordinate system, a position is specified by, for example, latitude, longitude, and altitude. The IMU (Inertial Measurement Unit) 106 measures the change in acceleration and orientation. The wheel encoder 107 detects the rotation of the wheels of the vehicle 100 and measures the travel distance (movement amount) of the vehicle 100.

[0028] The movement trajectory of the vehicle 100 linked to time is calculated from changes in the measurement values ​​of the GNSS position measuring device 103, changes in acceleration and orientation of the vehicle 100 measured by the IMU 106, and the distance traveled by the vehicle 100 measured by the wheel encoder 107.

[0029] The GNSS position measuring device 103 is equipped with a highly accurate clock, which is used to determine the time in the vehicle 100 .

[0030] FIG. 2 shows a block diagram of the arithmetic device 108. The arithmetic device 108 performs calculations related to the delay time Δt of the camera 101 described above. The arithmetic device 108 is a computer, and includes a CPU, a data storage device, an input / output interface, and a communication device. A general-purpose PC (personal computer) can also be used as the arithmetic device 108. The arithmetic device 108 may be configured with dedicated hardware. A form in which the processing of the arithmetic device 108 is performed by a server is also possible. A form in which the functions of the arithmetic device 108 are distributed and performed using multiple computers is also possible.

[0031] The calculation device 108 has an optical data acquisition unit 301, an imaging command signal output unit 302, a velocity vector calculation unit 303, a laser scan point cloud creation unit 304, a viewpoint setting unit 305, a point cloud image creation unit 306, a correspondence determination unit 307, a camera position and orientation calculation unit 308, a delay time (Δt) calculation unit 309, and a synchronization processing unit 310.

[0032] These functional units are realized by executing software on a computer constituting the arithmetic unit 108. It is also possible to configure one or more of the functional units shown in Fig. 2 with dedicated hardware.

[0033] The optical data acquisition unit 301 acquires image data of an image captured by the camera 101 and laser scan data acquired by the laser scanner 102. In addition, the optical data acquisition unit 301 acquires laser scan point cloud data based on the laser scan data acquired by the laser scanner 102.

[0034] The photographing command signal output unit 302 outputs a signal commanding (instructing) the camera 101 to photograph. For example, at time T in FIG.

[0035] The velocity vector calculation unit 303 calculates a velocity vector of the vehicle 100 based on the change in position of the vehicle 100 measured by the GNSS position measurement device 103, the change in speed and the change in direction of the vehicle 100 measured by the IMU 106, and the number of rotations of the wheels of the vehicle 100 measured by the wheel encoder 107. For example, the velocity vector of the vehicle 100 at a specific time and position is calculated by the velocity vector calculation unit 303.

[0036] The laser scan point cloud creation unit 304 creates a laser scan point cloud based on the laser scan data acquired by the laser scanner 102. The laser scanner 102 measures the direction of the reflection point of the laser scan light (the direction as seen from the laser scanner) and the distance to that point, and outputs data on the direction and distance to this reflection point as laser scan data. Based on this direction and distance, the three-dimensional coordinates of the reflection point (laser scan point) are calculated. This process is performed by the laser scan point cloud creation unit 304. A collection of the reflection points whose three-dimensional coordinates have been calculated becomes the laser scan point cloud.

[0037] The position and orientation of the laser scanner 102 on the vehicle 100 are known, and the position of the vehicle 100 in the absolute coordinate system is measured by a GNSS position measuring device. Therefore, the above-mentioned laser scan point cloud is point cloud data described in the absolute coordinate system.

[0038] A viewpoint setting unit 305 sets a viewpoint for viewing the laser scan point cloud created by the laser scan point cloud creation unit 304. An image of the apparent distribution of points when the laser scan point cloud is viewed from this viewpoint becomes a point cloud image.

[0039] The point cloud image creation unit 306 creates a point cloud image that is an image of the laser scan point cloud viewed from a specific viewpoint. Fig. 3 shows a point cloud image created with position X as the viewpoint, and a point cloud image created with position X+Δx as the viewpoint. A point cloud image is an image that shows the state of a collection of points as they appear when a laser scan point cloud, which is a collection of points distributed in a three-dimensional space, is viewed from a specific viewpoint. Naturally, if the position of the viewpoint is different, the way the collection of points arranged in a three-dimensional space appears changes.

[0040] The correspondence specification unit 307 specifies the correspondence between the captured image and the point cloud image. The correspondence is specified using a known matching method such as template matching. There are multiple methods for specifying the correspondence between the captured image and the point cloud image.

[0041] The first method is to extract feature points from a captured image, create a feature point image, and identify the correspondence between this feature point image and the point cloud image. The second method is to create a three-dimensional model based on the laser scan point cloud that is the basis of the point cloud image, obtain an image of this three-dimensional model viewed from the viewpoint of the point cloud image as a three-dimensional model image, and identify the correspondence between this three-dimensional model image and the captured image. The third method is to convert the captured image into a pointillism image drawn with points, and identify the correspondence between this pointillism image and the point cloud image.

[0042] The camera position and orientation calculation unit 308 is an example of a camera position calculation unit. The camera position and orientation calculation unit 308 calculates exterior orientation elements (position and orientation) of the camera 101 in the vehicle 100. Details of this process will be described later.

[0043] A delay time (Δt) calculation unit 309 calculates a delay time Δt from when an image capture command is issued to the camera 101 until the image capture is actually performed. Details of this process will be described later.

[0044] The synchronization processing unit 310 performs a synchronization process to synchronize a point cloud image based on the laser scan point cloud acquired by the laser scanner 102 with an image captured by the camera 101 based on the calculated delay time Δt.

[0045] There are several methods for synchronization processing. The first method is to move the point cloud image on the time axis and synchronize it with the captured image. For example, assume that when a command is given to the camera 101 to capture an image at time T, the image is captured with a delay of Δt. In this case, the point cloud image and the captured image can be synchronized by creating a point cloud image with a viewpoint corresponding to the position corresponding to the time T+Δt.

[0046] The second synchronization method is to move the captured image on the time axis. In this case, the image captured at time T+Δt is converted into an image seen from the viewpoint at time T. This conversion is performed, for example, by projective transformation. This makes it possible to synchronize the point cloud image derived from the laser scan data at the viewpoint at time T with the captured image captured by the camera 101. Note that it is also possible to move both the point cloud image and the captured image on the time axis (movement on the spatial axis of the viewpoint).

[0047] The third synchronization method is to take into account the delay in the shooting timing of the camera 101 and issue a shooting command at a timing advanced by Δt in advance. For example, if shooting is to be performed at time T, a shooting command signal is output to the camera 101 at time T-Δt. In this case, shooting by the camera 101 is performed at time T, which is delayed by Δt after the shooting command signal is output. In this case, synchronization between the shot image and the point cloud image created from the viewpoint at time T is ensured.

[0048] (Example of processing) An example of the processing performed by the arithmetic device 108 will be described below. Fig. 5 is a flow chart showing an example of the processing procedure. A program for executing the processing of Fig. 5 is stored in a storage device of a PC constituting the arithmetic device 108, and executed by a CPU of the PC. The program may be stored in an appropriate storage medium. The program may be stored in a server connected to the Internet, and downloaded from there to a PC for implementing the arithmetic device 108.

[0049] Here, it is assumed that the position and orientation of the camera 101 on the vehicle 100 are roughly known at an initial stage. This assumes, for example, that a user prepares the camera 101 and attaches it to the vehicle 100. In this case, the position to attach the camera is specified in advance, and the user sets the camera 101 there.

[0050] It is assumed that the relationship between the positions and attitudes of the laser scanner 102, the GNSS position measuring device 103, and the IMU 106 in the vehicle 100 is determined in advance and is known.

[0051] First, while the vehicle 100 is traveling in the direction of the X-axis in Fig. 1, a laser scanner 102 performs a laser scan of an object (e.g., a building 200), and a camera 101 photographs the same object. At this time, a GNSS position measuring device 103 measures the position of the vehicle 100 in an absolute coordinate system and its change, and determines the travel route of the vehicle 100 linked to the time. Measurement values ​​of the IMU 106 and the wheel encoder 107 are also used to determine this travel route. Furthermore, from these measurement values, the velocity vector of the vehicle 100 at each point on the travel route and at a specified time can be obtained.

[0052] After acquiring the captured image and the laser scan data, the following processing is performed as post-processing: First, the captured image of the same object by the camera 101 and the laser scan data by the laser scanner 102 are acquired by the optical data acquisition unit 301 (step S101).

[0053] Next, based on the laser scan data acquired in step S101, the laser scan point cloud creation unit 304 creates a laser scan point cloud (step S102). Next, as a preliminary step for calculating the exterior orientation parameters of the camera 101, a viewpoint for creating a point cloud image is provisionally set (step S103). This setting is performed by the viewpoint setting unit 304.

[0054] This viewpoint is the initial value for calculating the camera position of the target captured image. At this stage, Δt is unknown, so the position of the viewpoint is also unknown at this stage, so here, an approximate value is set as the initial value. The camera position is understood as the position of the projection origin of the camera to be used.

[0055] For example, consider a case where the camera 101 is commanded to capture an image at time T. Here, it is assumed that the maximum delay time from the command (instruction) to capture an image to the actual capture is 30 ms. In this case, it is assumed that the center value of the 30 ms range is assumed, and the image is captured at T+15 ms. In other words, it is assumed that T+15 ms is the capture time.

[0056] Then, the position where the camera 101 is estimated to be located at the time T+15 ms is set as a tentative viewpoint position X01.

[0057] Here, it is assumed that the approximate mounting position of the camera 101 on the vehicle 100 is known. In this case, the approximate position X0 of the camera 101 at time T is known based on the approximate offset position of the camera 101 relative to the IMU 106. Here, the position X01 of the camera 101 at time T+15 ms is calculated from the position of the vehicle 100 at time T and the velocity vector V of the vehicle at time T, with the position X0 of the camera 101 at time T as an initial value. Specifically, X01 is calculated by X01=X0+(V×15 ms).

[0058] After the viewpoint position X01 is provisionally set, a point cloud image is created by viewing the laser scan point cloud created in step S102 from that position (step S104). The point cloud image is created by the point cloud image creation unit 306.

[0059] Next, a correspondence relationship is obtained between the point cloud image of the laser scan point cloud viewed from the viewpoint X01 and the captured image obtained when an image capture command is issued to the camera 101 at time T (step S105). This process is performed by the correspondence relationship identifying unit 307.

[0060] Next, the exterior orientation parameters of the camera 101 at the time of capturing the captured image are calculated (step S106). This process will be described below.

[0061] By determining the correspondence between the point cloud image and the captured image, the positions of many points in the captured image in the absolute coordinate system can be determined. Here, the position of the camera 101 in the absolute coordinate system is calculated by the intersection method using the many points in the captured image whose coordinates are known as reference points (orientation points).

[0062] In addition, by examining the relationship between the optical axis direction of the camera 101 and the direction of each point as viewed from the projection origin, the orientation of the camera 101 in the absolute coordinate system can be calculated. This method is a basic method of single-photo orientation. Details of this process are described in, for example, JP 2016-57108 A.

[0063] The principle of the method for determining the position and orientation of the camera will be briefly explained below. In Fig. 4, the camera is at position X, p1 to p6 are feature points in the screen of the image captured by the camera 101, and P1 to P6 are points of the laser scan point cloud corresponding to p1 to p6. Note that the camera position X is unknown, and the interior orientation parameters of the camera are known. Also, the camera position X is the projection origin (optical origin) of the camera.

[0064] Here, a direction line passing through P1 and p1, a direction line passing through P2 and p2, a direction line passing through P3 and p3, ... are created. The point where these direction lines intersect is the camera position X. Using this principle, the position (viewpoint of shooting) X1 of the camera 101 when shooting the target image is calculated. In addition, the line passing through position X1 and the center of the screen is the optical axis of the camera. From the relationship between this optical axis and the above direction lines, the attitude of the camera at camera position X1 can be found.

[0065] If the intersection point of the multiple direction lines cannot be determined, or if the range of intersection of the multiple direction lines is larger than a predetermined range, the value of the camera position X0 at time T is changed and recalculated. Instead of or in addition to changing the value of the camera position X0 and recalculating, there is also a method of reviewing the correspondence between the feature points on the captured image and the feature points on the point cloud image and recalculating. By determining the intersection point of the multiple direction lines or searching for X1 where the range of intersection of the multiple direction lines falls within a predetermined range, it is possible to obtain the position X1 of the camera 101 at time T+Δt (true shooting time) that is closer to the true value.

[0066] In this way, when an image capturing command is issued to the camera 101 at time T, the exterior orientation parameters (position and orientation) of the camera 101 at the time of image capturing performed with a delay of Δt are obtained (step S106). This process is performed by the camera position and orientation calculation unit 308.

[0067] The exterior orientation parameters of the camera 101 obtained at this stage (step S106) are values ​​in an absolute coordinate system.

[0068] At this stage, the exterior orientation parameters of the camera 101 on the vehicle 100 are unknown. This is because, at this stage, Δt is unknown, the shooting time of the captured image is unknown, and the position of the vehicle 100 at this shooting time is unknown.

[0069] Next, Δt is calculated (step S107). Δt is calculated as follows.

[0070] Here, if the position of camera 101 at time T when the shooting command is issued is X0, the time when the shooting is taken by camera 101 is T+Δt, and the camera position at that time is X1, so the time it takes for vehicle 100 (camera 101) to move from X0 to X1 is Δt.

[0071] Here, if the speed of the vehicle 100 at time T is speed V, then V=(X1-X0) / Δt holds. That is, Δt can be calculated by Δt=(X1-X0) / V. This calculation is performed by the delay time (Δt) calculation unit 309.

[0072] Here, X1 is the shooting position (camera position) of the camera 101 calculated according to the principle in Fig. 4. X0 is the position of the camera 101 at time T assumed as an initial condition for the calculation of X1. V is the speed of the vehicle 100 at time T.

[0073] Furthermore, since the velocity vector of the vehicle 100 at time T can be obtained based on the measurement values ​​obtained from the GNSS position measurement device 103, the IMU 106, and the wheel encoder 108, the above V can be obtained from these measurement values.

[0074] After Δt is calculated, the exterior orientation parameters (position and attitude) of the camera 101 on the vehicle 100 at the time T1=T+Δt when the image of interest is captured are obtained (step S108).

[0075] That is, by determining Δt, the actual shooting time T1=T+Δt of the camera 101 is determined. As a result, the position of the vehicle 100 at time T1, that is, the position of the vehicle 100 when the image was captured by the camera 101, is determined. In addition, the attitude of the vehicle 100 is determined from the measurement data of the IMU 106 at time T1. Then, the position of the camera 101 on the vehicle 100 is determined from the relationship between the position of the vehicle 100 at time T1 and the position X1 of the camera 101 at time T1.

[0076] Moreover, the attitude of the camera 101 in the absolute coordinate system at time T1 is calculated in step S106. Therefore, the attitude of the camera 101 in the vehicle 100 is obtained from the relationship between the attitude of the vehicle 100 in the absolute coordinate system at time T1 and the attitude of the camera 101 in the absolute coordinate system at time T1. In this manner, the exterior orientation parameters (position and attitude) of the camera 101 in the vehicle 100 are obtained. These processes are performed by the camera position and attitude calculation unit 308.

[0077] After Δt is obtained, a synchronization process is performed (step S109). By performing the synchronization process, the image captured by the camera 101 and the laser scan point cloud acquired by the laser scanner 102 can be synchronized.

[0078] This synchronization process ensures synchronization between the point cloud image derived from the laser scan by the laser scanner 102 and the image captured by the camera 101.

[0079] (Advantages) In this embodiment, no exposure signal is required from the camera 101. The camera 101 simply receives a shooting signal that commands shooting. Therefore, a variety of cameras can be used as the camera 101. In addition, no hardware is required to handle the exposure signal, which reduces costs. In addition, the degree of freedom and ease of setting are improved when using a camera prepared by the user.

[0080] (others) The interval between shots can be set arbitrarily. Frame images constituting a video can also be used as the shot images handled in the present invention. The delay time (time offset) Δt can be calculated periodically. In this case, Δt is updated periodically.

[0081] The moving body is not limited to a vehicle, and may be an aircraft or a ship. The moving body may be manned or unmanned.

[0082] 2. Second embodiment Here, an example will be described in which the position of the camera 101 on the vehicle 100 is known. Note that the orientation of the camera 101 on the vehicle 100 may be unknown or known.

[0083] In this case, the movement trajectory of the vehicle 100 is converted into the movement trajectory of the camera 101 based on the positional relationship between the vehicle 100 and the camera 101, and the movement trajectory of the camera 101 is obtained. Then, at the stage where the camera position is calculated in step S106, the time on the movement trajectory of the camera that is closest to this calculated camera position is obtained. This time becomes the actual shooting time T1=T+Δt of the camera 101. And the difference between the time T when shooting was commanded and the above T1=T+Δt becomes Δt.

[0084] 3. Third embodiment When there is a difference in the vertical direction between the camera position obtained by the intersection shown in FIG. 4 and the position of the viewpoint that was initially temporarily set, it is more likely that an error in the vertical direction existed when the viewpoint was initially temporarily set, rather than that this is due to Δt. In such a case, the viewpoint is initialized again, taking into account the obtained difference in the vertical direction, and the camera position is calculated again using the intersection described in the first embodiment. Then, Δt is calculated again. In this way, the accuracy of the exterior orientation parameters of the camera 101 and the accuracy of Δt can be improved.

[0085] The same can be said about the case where there is a difference between the camera position in the width direction of the vehicle calculated by the intersection method and the viewpoint position that was initially provisionally set. In other words, when there is a difference between the camera position calculated by the intersection method and the viewpoint position that was initially provisionally set in the direction perpendicular to the traveling direction of the vehicle 100, the initial value that corrects the difference is reset, and the camera position is calculated again according to the principle of Fig. 4. In addition, Δt is calculated based on this recalculated camera position.

[0086] 4. Fourth embodiment Depending on the camera, Δt may change when the settings are changed, such as exposure time, burst speed, resolution, optical magnification, power consumption mode, etc.

[0087] When such a setting is changed, the process for acquiring Δt is executed as a trigger, thereby making it possible to respond to changes in Δt.

[0088] In addition, when a plurality of cameras are used, it is also effective to execute the process for obtaining Δt when the camera to be used is switched.

[0089] 5. Fifth embodiment It is also possible to set a plurality of viewpoint positions to be provisionally set initially, and select a viewpoint position where the calculated value by the rear intersection method (single photo orientation) converges better. For example, the viewpoint positions are provisionally set at 1 ms intervals, such as T+1 ms, T+2 ms, T+3 ms, ...T+30 ms. Then, for the viewpoints at each provisionally set position, the exterior orientation parameters of the camera 101 are calculated by the method described in the first embodiment.

[0090] At this time, the provisional setting value with good convergence of the calculated value of the camera position is adopted as the provisional setting value closer to the true value. The calculated value obtained at this time is adopted as the exterior orientation parameters (position and attitude) of the camera 101.

[0091] The provisional set values ​​may be set in small increments in both the vertical and width directions of the vehicle 100 . [Explanation of symbols]

[0092] 100...vehicle, 101...camera, 102...laser scanner, 103...GNSS position measurement device, 106...IMU, 107...wheel encoder, 108...computing device.

Claims

1. an optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object; a correspondence relationship determining unit for determining a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image; A camera position calculation unit that calculates a position X1 of the camera at the time of taking the captured image by single-photo orientation based on the correspondence relationship; a delay time calculation unit that calculates a delay of Δt when an image is captured by the camera after a command to capture an image at a time T; Equipped with the time T is a time when a command is issued to the camera to capture the image, The position of the camera at the time T is X0, Let V be the velocity of the moving body at the time T, An optical data processing device, calculated by Δt=(X1-X0) / V.

2. 2. The optical data processing apparatus according to claim 1, wherein an initial condition in the single-photo orientation is adopted as X0.

3. An optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image taken by a camera mounted on the moving object in a moving state; a correspondence relationship determining unit for determining a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image; A camera position calculation unit that calculates a position X1 of the camera at the time of taking the captured image by single-photo orientation based on the correspondence relationship; a delay time calculation unit that calculates a delay of Δt when an image is captured by the camera after a command to capture an image at a time T; Equipped with the time T is a time when a command is issued to the camera to capture the image, A relationship between the position of the moving object and time is acquired; The position of the camera on the moving object is known; A time T1 corresponding to the position X1 of the camera is calculated based on the position X1, a relationship between the position and time of the moving body, and a position of the camera on the moving body; An optical data processing device in which the time Δt is calculated from the difference between the time T and the time T1.

4. an optical data acquisition step of acquiring a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object in a moving state; a correspondence determination step of determining a correspondence between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image; A camera position calculation step of determining a position X1 of the camera at the time of photographing the photographed image by single photograph orientation based on the correspondence relationship; a delay time calculation step of calculating a delay time Δt when the camera takes an image with a delay of Δt when the camera takes an image with a command to take an image at a time T; Equipped with the time T is a time when a command is issued to the camera to capture the image, The position of the camera at the time T is X0, Let V be the velocity of the moving body at the time T, An optical data processing method in which Δt is calculated by (X1-X0) / V.

5. A program to be read and executed by a computer, Computer an optical data acquisition unit that acquires a laser scan point cloud obtained by a laser scanner mounted on a moving object in a moving state and image data of a captured image captured by a camera mounted on the moving object; a correspondence relationship determining unit for determining a correspondence relationship between a point cloud image of the laser scan point cloud viewed from a specific viewpoint and the captured image; A camera position calculation unit that calculates a position X1 of the camera at the time of taking the captured image by single-photo orientation based on the correspondence relationship; a delay time calculation unit that calculates a delay of Δt when an image is captured by the camera after a command to capture an image at a time T; and run it, the time T is a time when a command is issued to the camera to capture the image, The position of the camera at the time T is X0, Let V be the velocity of the moving body at the time T, An optical data processing program that calculates Δt=(X1-X0) / V.

Images