Readout device

Abstract

A readout device (10) according to the present disclosure comprises: a detection unit (11) that detects a traveling direction on a reference map during capturing of a plurality of query images sequentially over time by a moving camera; a calculation unit (12) that, on the basis of optical flows of the plurality of query images, calculates the orientation of the camera with respect to the traveling direction; and a readout unit (13) that, from a database that stores, as reference data, a plurality of reference images and the reference map in which feature points of the plurality of reference images are associated with three-dimensional points on a three-dimensional map, reads out the reference data while restricting the range of the reference data on the basis of the orientation of the camera with respect to the traveling direction.

Description

Readout device

[0001] The present disclosure relates to a reading device.

[0002] Non-Patent Document 1 describes a technology for estimating the camera pose (imaging position and imaging direction at the time of image capture) of a query image captured by a camera. In the technology described in Non-Patent Document 1, a reference database (hereinafter referred to as a "reference DB") is prepared in advance. The reference database stores a plurality of reference images whose camera poses at the time of image capture are known, and a reference map in which feature points (two-dimensional feature points) of each of the plurality of reference images are associated with three-dimensional points on the three-dimensional map. Then, in the technology described in Non-Patent Document 1, feature points are matched between the reference images stored in the reference DB and the query image, and the camera pose of the query image is estimated from the three-dimensional points on the reference map that correspond to the feature points of the query image. In the technology described in Non-Patent Document 1, the reference data (reference images and reference map) stored in the reference DB typically has a large capacity. Therefore, when referencing the reference data, it is desirable to load (read) as little data as possible from the reference DB.

[0003] P. Sarlin et al., From Coarse to Fine: Robust Hierarchical Localization at Large Scale, Computer Vision and Pattern Recognition, 2019, [online], [Retrieved July 24, 2024], Internet<URL:https: / / arxiv.org / pdf / 1812.03506.pdf>

[0004] As a reference image with a known camera orientation at the time of image capture, an image captured by a camera mounted on a Mobile Mapping System (MMS) vehicle or an Unmanned Aerial Vehicle (UAV), a dashcam, a smartphone, or the like can be used. The camera and dashcam mounted on an MMS vehicle or UAV do not change their imaging direction relative to the housing. Furthermore, even when a smartphone is permanently installed at any position on the housing to capture images, the imaging direction relative to the housing does not change. In this case, the imaging range of multiple images (image sequence) captured consecutively in chronological order is limited to the Z-axis direction (camera orientation direction) of the camera coordinate system. Therefore, it may not be necessary to refer to all of the reference data stored in the reference DB to estimate the camera orientation. For example, if the general imaging direction of the query image is known, the reference data can be referenced within a predetermined range that includes that imaging direction.

[0005] One method for obtaining the imaging direction is to obtain the camera's orientation relative to the housing by calibrating the camera. Calibration is a process for calculating camera parameters using a spatial point whose position is known in advance and the projection of that point onto the image captured by the camera. Therefore, if the query image is an image captured using an MMS or UAV, it is also possible to reference reference data by narrowing it down to a predetermined range that includes the imaging direction of the query image. However, calibration requires a calibration target such as a checkerboard that allows the coordinates of spatial points to be determined, which is time-consuming. Therefore, there is a need for a technology that can more easily reduce the amount of reference data read from a reference DB.

[0006] The object of the present disclosure, made in consideration of the above-mentioned problems, is to provide a reading device that can more easily reduce the amount of data read from a database that stores multiple reference images and reference maps as reference data.

[0007] A reading device according to one embodiment reads data from a database that stores, as reference data, a plurality of reference images whose imaging positions and imaging directions at the time of imaging are known, and a reference map in which feature points of each of the plurality of reference images are associated with three-dimensional points on a three-dimensional map, and the reading device includes: a detection unit that detects the direction of travel on the reference map when capturing a plurality of query images that are captured successively in time series while moving with a camera; a calculation unit that calculates the orientation of the camera with respect to the direction of travel based on the optical flow of the plurality of query images; and a reading unit that limits and reads the range of the reference data based on the orientation of the camera with respect to the direction of travel.

[0008] According to the present disclosure, it is possible to more easily reduce the amount of data read from a database that stores a plurality of reference images and reference maps as reference data.

[0009] FIG. 2 is a diagram illustrating an example of the configuration of a readout device according to an embodiment of the present disclosure. FIG. 3 is a diagram for explaining calculation of the optical flow of a query image by the calculation unit shown in FIG. 1. FIG. 4 is a diagram illustrating an example of the optical flow of feature points according to the orientation of a camera. FIG. 5 is a diagram illustrating another example of the optical flow of feature points according to the orientation of a camera. FIG. 6 is a diagram illustrating yet another example of the optical flow of feature points according to the orientation of a camera. FIG. 7 is a diagram for explaining reading of a reference map by the readout unit shown in FIG. 1. FIG. 8 is a flowchart illustrating an example of the operation of the readout device shown in FIG. 1. FIG. 9 is a diagram illustrating an example of the configuration of a computer that functions as a readout device according to the present disclosure.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] 1 is a diagram illustrating an example of a configuration of a reading device 10 according to an embodiment of the present disclosure. The reading device 10 according to the present disclosure is a device that loads (reads) data from a database (reference DB) that stores, as reference data, a plurality of reference images whose imaging positions and imaging directions at the time of image capture are known, and a reference map in which feature points (two-dimensional feature points) of each of the plurality of reference images are associated with three-dimensional points on a three-dimensional map. The reading of data by the reading device 10 is performed, for example, to estimate the camera attitude (imaging position and imaging direction) of a plurality of query images that are captured continuously in time series while the camera is moving.

[0012] 1, a readout device 10 according to this embodiment includes a detection unit 11, a calculation unit 12, and a readout unit 13. The readout device 10 receives as input a plurality of query images captured successively in time series while moving using a camera.

[0013] The detection unit 11 detects the traveling direction on a reference map when capturing multiple query images captured consecutively in time series while the camera is moving. For example, when location information at the time of capturing acquired by a GPS (Global Positioning System) or the like is linked to each query image, the detection unit 11 detects the traveling direction when capturing the query images from the location information of the multiple query images captured in chronological order. Alternatively, the detection unit 11 detects, for example, features that appear in common in the multiple query images and detects the traveling direction when capturing the query images from the order in which the features appear. The detection unit 11 detects the features using, for example, a technique such as semantic segmentation.

[0014] The calculation unit 12 calculates the orientation of the camera relative to the traveling direction when capturing the query images based on the optical flows of the multiple query images. Specifically, the calculation unit 12 calculates the optical flow for each feature point that appears in common in the multiple query images. The optical flow is a vector that indicates the amount of movement of corresponding points in successive images in a time series. The calculation unit 12 calculates the optical flow using, for example, the Lucas-Kanade method described in Reference 1. [Reference 1] B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 674-679, 1981.

[0015]

[0016]

[0017] In formula (1), x ｉ , y ｉ are the x and y components of the optical flow (vector) of the i-th feature point, respectively, and N is the number of all feature points.

[0018] Next, the calculation unit 12 calculates the orientation θ of the camera relative to the traveling direction when capturing the query image using the following equation (2).

[0019]

[0020] 3A to 3C are diagrams showing examples of the optical flows of multiple feature points according to the camera direction θ. For example, when the camera direction θ is 90° leftward relative to the traveling direction, the optical flows of the multiple feature points indicated by the solid arrows are mainly leftward vectors, as shown in FIG. 3A. Therefore, the average vector of the optical flows of the multiple feature points indicated by the hollow arrows is also a leftward vector. Therefore, it can be seen from the average vector of the optical flows of the multiple feature points that the camera direction θ is approximately 90° leftward relative to the traveling direction.

[0021] Furthermore, for example, when the camera direction θ is 45° leftward relative to the traveling direction, the optical flows of the multiple feature points indicated by the solid arrows are mainly vectors pointing diagonally downward to the left, as shown in Figure 3B. Therefore, the average vector of the optical flows of the multiple feature points indicated by the white arrows is also a vector tilted approximately 45° downward to the left. Therefore, from the average vector of the optical flows of the multiple feature points, it can be seen that the camera direction θ is approximately 45° leftward relative to the traveling direction.

[0022] Furthermore, for example, when the camera direction θ is facing forward relative to the traveling direction, the optical flows of the multiple feature points indicated by the solid arrows are mainly downward vectors, as shown in Figure 3C. Therefore, the average vector of the optical flows of the multiple feature points indicated by the white arrows is also a downward vector. Therefore, from the average vector of the optical flows of the multiple feature points, it can be seen that the camera direction θ is facing roughly forward relative to the traveling direction.

[0023] In this way, the calculation unit 12 calculates the approximate camera orientation θ with respect to the traveling direction when capturing the query image, using the average vector of the optical flows of the multiple feature points. That is, in this embodiment, the traveling direction when capturing the query image is detected based on position information of the capturing position of the query image or the appearance (feature points) of the query image. Then, the calculation unit 12 calculates the approximate camera orientation θ with respect to the traveling direction when capturing the query image, based on the optical flow of the feature points of the query image. Therefore, it is possible to more easily calculate the camera orientation θ when capturing the query image, without performing calibration or the like. Furthermore, because calibration is not required, there is no restriction on the camera, such as a dashcam or smartphone, as long as the image sequence is captured by a camera fixed to a moving object.

[0024] Referring back to FIG. 1, the reading unit 13 reads out the reference data by restricting the range of the reference data based on the camera direction θ relative to the traveling direction when the query image was captured, which is calculated by the calculation unit 12 .

[0025] FIG. 4A is a diagram showing an example of a reference map. Assume that the traveling direction when capturing the query image is from the front side of the paper to the back side of the paper in FIG. 4A, and the camera direction θ is facing leftward relative to the traveling direction as shown in FIG. 4A. In this case, the reading unit 13 reads the reference map by deleting a certain range from the right end of the reference map, as shown in FIG. 4B, for example. In this way, the reading unit 13 may delete unnecessary portions of the reference map according to the camera direction θ relative to the traveling direction before reading it. This makes it possible to reduce the amount of reference data read from the reference DB compared to reading the entire reference map.

[0026] 4A , when the camera direction θ is facing left with respect to the traveling direction (e.g., θ = 45°), the reading unit 13 may read from the reference DB, among the reference images stored in the reference DB, reference images whose imaging direction is oriented from 0° to 90° and a reference map of a predetermined range including three-dimensional points associated with feature points of the reference images. In this way, the reading unit 13 may read reference images whose imaging direction is within a predetermined range including the camera direction θ with respect to the traveling direction and a reference map of a predetermined range including three-dimensional points associated with feature points of the reference images. In this way, it is possible to narrow down the reference images stored in the reference DB to be read only to images whose imaging direction is within a predetermined range including the camera direction θ with respect to the traveling direction, thereby reducing the amount of reference data read from the reference DB.

[0027] Next, the operation of the reading device 10 according to this embodiment will be described. Fig. 5 is a flowchart showing an example of the operation of the reading device 10 according to this embodiment, and is a diagram for explaining a reading method executed by the reading device according to this embodiment.

[0028] The detection unit 11 detects the traveling direction on the reference map when capturing a plurality of query images captured successively in time series while the camera is moving (step S11). As described above, the detection unit 11 detects the traveling direction when capturing the query images, for example, from position information associated with the plurality of query images captured in time series. Furthermore, the detection unit 11 detects the traveling direction when capturing the query images, for example, from the appearance order of feature points commonly appearing in the plurality of query images.

[0029] The calculation unit 12 calculates the orientation of the camera relative to the traveling direction based on the optical flows of the multiple query images. The calculation unit 12 calculates the average of the optical flows for each feature point commonly captured in the multiple query images, for example, according to the above-mentioned formula (1). Then, the calculation unit 12 calculates the orientation θ of the camera relative to the traveling direction when capturing the query image, according to the above-mentioned formula (2) (step S12).

[0030] The reading unit 13 reads the reference data by restricting the range of the reference data based on the camera orientation θ relative to the traveling direction (step S13). For example, as shown in Fig. 4B, the reading unit 13 deletes unnecessary parts of the reference map according to the camera orientation θ relative to the traveling direction and reads the reference data. Furthermore, the reading unit 13 reads, for example, reference images within a predetermined range whose imaging direction includes the camera orientation θ relative to the traveling direction, and a reference map within a predetermined range that includes three-dimensional points associated with feature points of the reference images.

[0031] As described above, the reading device 10 according to this embodiment includes a detection unit 11, a calculation unit 12, and a reading unit 13. The detection unit 11 detects the traveling direction on the reference map when capturing multiple query images that are captured continuously in time series while the camera is moving. The calculation unit 12 calculates the orientation θ of the camera relative to the traveling direction based on the optical flows of the multiple query images. The reading unit 13 limits the range of reference data and reads it out based on the orientation θ of the camera relative to the traveling direction.

[0032] By using optical flow, it is possible to easily calculate the camera orientation θ relative to the traveling direction when capturing multiple query images without performing calibration. Then, by limiting and reading the range of reference data based on the camera orientation θ relative to the traveling direction, it is possible to read reference data excluding unnecessary data (e.g., reference data that deviates from the camera orientation θ). Therefore, according to the present disclosure, it is possible to more easily reduce the amount of data read from a reference DB that stores multiple reference images and reference maps as reference data. Furthermore, by reducing the amount of reference data read, it is possible to speed up processes such as estimating the camera orientation of a query image using the reference data and reduce the amount of memory used for processing.

[0033] One possible method for reading reference data from a reference DB is to limit the range of reference data read from the reference DB based on data obtained by calibrating a camera that captures a query image. However, this method requires that the camera that captures the query image be calibrated in advance. Therefore, this method cannot be applied to query images captured by a camera that has not been calibrated.

[0034] Another method for reading reference data from the reference DB is to detect the traveling direction when capturing the query image based on position information linked to the query image, calculate the camera orientation θ relative to the traveling direction based on the optical flows of multiple query images, and then limit and read the range of the reference data based on the camera orientation θ relative to the traveling direction. This method requires position information linked to the query image, but does not require camera calibration, and is therefore applicable to query images captured by any camera.

[0035] Another method for reading reference data from the reference DB is to detect the traveling direction when capturing a query image based on feature extraction, calculate the camera orientation θ relative to the traveling direction based on the optical flows of multiple query images, and then limit and read the range of reference data based on the camera orientation θ relative to the traveling direction. This method does not require location information linked to the query image or camera calibration, and is therefore applicable to query images captured by any camera.

[0036] The reading device 10 described above can be realized by a computer 20 shown in FIG. 6. A program for causing the computer 20 to function as the reading device 10 may be provided. The program may be stored in a storage medium or provided via a network. FIG. 6 is a block diagram showing a schematic configuration of the computer 20 functioning as the reading device 10. The computer 20 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be program code, code segments, or the like for performing necessary tasks.

[0037] 6, the computer 20 includes a processor 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage 24, an input unit 25, a display unit 26, and a communication interface (I / F) 27. Each component is communicably connected to one another via a bus 29. The processor 21 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be configured with multiple processors of the same or different types.

[0038] The processor 21 is a control unit that controls each component and performs various arithmetic processing. That is, the processor 21 reads a program from the ROM 22 or the storage 24 and executes the program using the RAM 23 as a work area. The processor 21 controls each component and performs various arithmetic processing in accordance with the program stored in the ROM 22 or the storage 24. In this embodiment, the ROM 22 or the storage 24 stores a program for operating the computer 20 as the reading device 10 according to the present disclosure. The program is read and executed by the processor 21 to realize each component of the reading device 10.

[0039] The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), a USB (Universal Serial Bus) memory, etc. The program may also be provided in a form downloaded from an external device via a network.

[0040] The ROM 22 stores various programs and various data. The RAM 23 temporarily stores programs or data as a working area. The storage 24 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0041] The input unit 25 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0042] The display unit 26 is, for example, a liquid crystal display, and displays various information. The display unit 26 may be a touch panel type and function as the input unit 25.

[0043] The communication interface 27 is an interface for communicating with other devices, for example, an interface for a LAN.

[0044] The following additional notes are provided regarding the above-described embodiments.

[0045] [Supplementary Item 1] A reading device that reads data from a database that stores, as reference data, a plurality of reference images whose imaging positions and imaging directions at the time of imaging are known, and a reference map in which feature points of each of the plurality of reference images are associated with three-dimensional points on a three-dimensional map, the reading device comprising a control unit that is configured to: detect a direction of travel on the reference map when a plurality of query images are captured that are captured consecutively in time series while moving with a camera; calculate an orientation of the camera with respect to the direction of travel based on the optical flow of the plurality of query images; and limit and read out the range of the reference data based on the orientation of the camera with respect to the direction of travel.

[0046] [Supplementary Item 2] The readout device according to Supplementary Item 1, wherein the control unit deletes unnecessary portions of the reference map according to the orientation of the camera relative to the traveling direction and reads the reference map.

[0047] [Supplementary Item 3] In the reading device described in Supplementary Item 1, the control unit reads out a reference image within a predetermined range whose imaging direction includes the orientation of the camera relative to the traveling direction, and a reference map within a predetermined range including three-dimensional points to which feature points of the reference image are associated.

[0048] [Supplementary Item 4] A reading method executed by a reading device that reads data from a database that stores, as reference data, a plurality of reference images whose imaging positions and imaging directions at the time of imaging are known, and a reference map in which feature points of each of the plurality of reference images are associated with three-dimensional points on a three-dimensional map, the reading method comprising: detecting a direction of travel on the reference map when capturing a plurality of query images that are captured continuously in time series while moving with a camera; calculating an orientation of the camera relative to the direction of travel based on the optical flow of the plurality of query images; and limiting and reading the range of the reference data based on the orientation of the camera relative to the direction of travel.

[0049] [Supplementary Item 5] A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to operate as the reading device described in any one of Supplementary Items 1 to 3.

[0050] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-described embodiments, and various modifications and alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the block diagrams of the embodiments can be combined into one, or one building block can be divided.

[0051] REFERENCE SIGNS LIST 10 Reading device 11 Detection unit 12 Calculation unit 13 Reading unit 20 Computer 21 Processor 22 ROM 23 RAM 24 Storage 25 Input unit 26 Display unit 27 Communication I / F 29 Path

Claims

1. A reading device that reads data from a database that stores, as reference data, a plurality of reference images whose imaging positions and imaging directions at the time of imaging are known, and a reference map in which feature points of each of the plurality of reference images are associated with three-dimensional points on a three-dimensional map, the reading device comprising: a detection unit that detects the direction of travel on the reference map when capturing a plurality of query images that are captured continuously in time series while moving with a camera; a calculation unit that calculates the orientation of the camera relative to the direction of travel based on the optical flow of the plurality of query images; and a reading unit that limits the range of the reference data and reads it based on the orientation of the camera relative to the direction of travel.

2. A readout device according to claim 1, wherein the readout unit deletes unnecessary parts of the reference map according to the orientation of the camera relative to the traveling direction and reads the reference map.

3. A reading device as described in claim 1, wherein the reading unit reads out a reference image within a predetermined range whose imaging direction includes the orientation of the camera relative to the traveling direction, and a reference map within a predetermined range including three-dimensional points to which feature points of the reference image are associated.

Images