Change point detection device, control method, program, and storage medium

Abstract

To provide a change point detection device that can appropriately detect a change point related to a characteristic object.SOLUTION: A change point detection device 4 has first acquisition means, second acquisition means, alignment means, and a controller 45 functioning as change point detection means. The first acquisition means acquires a first ortho image that is created based on a first image group, and a first projection position of a characteristic object included in at least one image of the first image group, the characteristic object being projected on the first ortho image. The second acquisition means acquires a second ortho image that is created based on a second image group captured in a period different from that for the first image group, and a second projection position of a characteristic object included in at least one image of the second image group, the characteristic object being projected on the second ortho image. The alignment means performs alignment between the first ortho image and the second ortho image. The change point detection means detects a change point related to the characteristic object on the basis of the first projection position and the second projection position at least one of which is corrected through the alignment.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a technique for detecting change points.

Background Art

[0002] Conventionally, a technique for detecting change points of a map based on the output of sensors installed in a vehicle is known. For example, Patent Document 1 discloses a system in which when each vehicle detects a change point in map data using a sensor, the map data is updated by transmitting data related to the change point to a map management server.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When measuring the road periphery with a measurement vehicle, if feature point-based SLAM (Simultaneous Localization and Mapping) is applied, the position estimation accuracy may be low in a road section where there are few feature objects serving as landmarks, etc., and the accuracy of the generated measurement data may be low. In this case, there is a risk that change points cannot be appropriately detected.

[0005] The present invention has been made to solve the above problems, and a main object thereof is to provide a change point detection device capable of preferably detecting change points.

Means for Solving the Problems

[0006] The invention described in the claims is a change point detection device comprising: alignment means for aligning a first composite image including a first projection position on which a feature object is projected and a second composite image including a second projection position on which the feature object is projected; and change point detection means for detecting a change point relating to the feature object based on the first projection position and the second projection position, which have been corrected by the alignment.

[0007] Furthermore, the invention described in the claims is a control method that uses a computer to align a first composite image including a first projection position on which a feature object is projected with a second composite image including a second projection position on which the feature object is projected, and detects a change point relating to the feature object based on the first projection position and the second projection position, which have been corrected by the alignment.

[0008] Furthermore, the invention described in the claims is a program that causes a computer to function as a change point detection means for aligning a first composite image including a first projection position on which a feature object is projected with a second composite image including a second projection position on which the feature object is projected, and for detecting change points relating to the feature object based on the first projection position and the second projection position, which have been corrected by the alignment. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic configuration of the change point detection system. [Figure 2] The block configuration of the in-vehicle unit and the change point detection device is shown. [Figure 3] This shows a functional block diagram of the change point detection device. [Figure 4] This shows an aerial view of a certain road section. [Figure 5] (A) Shows the first orthoimage generated based on the first dataset. (B) Shows the second orthoimage generated based on the second dataset. (C) This is the image obtained by overlaying the first and second orthoimages after alignment. [Figure 6]This is an example of a flowchart showing the procedure for change point detection processing according to the embodiment. [Figure 7] This is an example of a flowchart showing the procedure for detecting change points related to modified versions. [Modes for carrying out the invention]

[0010] According to a preferred embodiment of the present invention, the change point detection device includes: a first acquisition means for acquiring a first composite image generated based on a first group of images and a first projection position when a feature object included in at least one image of the first group of images is projected onto the first composite image; a second acquisition means for acquiring a second composite image generated based on a second group of images taken during a different period than the first group of images and a second projection position when the feature object included in at least one image of the second group of images is projected onto the second composite image; an alignment means for aligning the first composite image and the second composite image; and a change point detection means for detecting a change point relating to the feature object based on the first projection position and the second projection position, which have been corrected by the alignment. In this embodiment, the change point detection device can suitably detect a change point of a feature object based on a first group of images and a second group of images taken during different periods.

[0011] In one embodiment of the change point detection device described above, the first image group and the second image group consist of images taken from a vehicle traveling on a predetermined road section, and the alignment means performs the alignment using the road in the road section as a reference. In this embodiment, the change point detection device can accurately align the first composite image and the second composite image and suitably detect change points of feature objects.

[0012] In another embodiment of the change point detection device described above, the change point detection device further includes composite image generation means for generating a first composite image based on the first image group and generating a second composite image based on the second image group. In this embodiment, the change point detection device can suitably acquire a first composite image and a second composite image for alignment.

[0013] In another embodiment of the change point detection device described above, the composite image generation means generates the first composite image based on the first image group and the shooting position and shooting direction of each image constituting the first image group, and generates the second composite image based on the second image group and the shooting position and shooting direction of each image constituting the second image group. In this embodiment, the change point detection device can suitably generate a comparable first composite image and a second composite image.

[0014] In another embodiment of the change point detection device described above, the first image group and the second image group consist of images taken from a vehicle, and the change point detection device further includes a shooting position calculation means that calculates the shooting position and shooting direction based on data output by a sensor provided on the vehicle during the image shooting period. In this embodiment, the change point detection device can suitably identify the shooting position and shooting direction of each image necessary for generating a composite image.

[0015] In another embodiment of the change point detection device described above, the composite image generation means generates orthophotos as the first composite image and the second composite image. This allows the change point detection device to suitably generate the first composite image and the second composite image suitable for alignment.

[0016] In another embodiment of the change point detection device described above, the change point detection device further includes projection means for calculating a first projected position obtained by projecting the spatial position of the feature object, which is identified based on the images in the first group of images containing the feature object, onto the first composite image, and for calculating a second projected position obtained by projecting the spatial position of the feature object, which is identified based on the images in the second group of images containing the feature object, onto the second composite image. In this embodiment, the change point detection device can suitably calculate the first projected position of the feature object on the first composite image and the second projected position of the feature object on the second composite image.

[0017] According to another preferred embodiment of the present invention, the control method includes a computer obtaining a first composite image generated based on a first image group and a first projection position when a feature object included in at least one image of the first image group is projected onto the first composite image, obtaining a second composite image generated based on a second image group captured in a period different from the first image group and a second projection position when the feature object included in at least one image of the second image group is projected onto the second composite image, aligning the first composite image and the second composite image, and detecting a change point regarding the feature object based on the first projection position and the second projection position in which at least one of them is corrected by the alignment. By executing this control method, the computer can suitably detect a change point of the feature object based on the first image group and the second image group captured in different periods.

[0018] According to another preferred embodiment of the present invention, the program causes a computer to function as a first acquisition means for obtaining a first composite image generated based on a first image group and a first projection position when a feature object included in at least one image of the first image group is projected onto the first composite image, a second acquisition means for obtaining a second composite image generated based on a second image group captured in a period different from the first image group and a second projection position when the feature object included in at least one image of the second image group is projected onto the second composite image, an alignment means for aligning the first composite image and the second composite image, and a change point detection means for detecting a change point regarding the feature object based on the first projection position and the second projection position in which at least one of them is corrected by the alignment. By executing this program, the computer can suitably detect a change point of the feature object based on the first image group and the second image group captured in different periods. Preferably, the above program is stored in a storage medium.

Example

[0019] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0020] (1) System Overview FIG. 1 is a schematic configuration diagram of a change point detection system according to this embodiment. The change point detection system shown in FIG. 1 is a system for detecting change points of feature objects existing around a road, and mainly includes a measurement vehicle that travels on the road while performing measurement, and a change point detection device 4 that performs change point detection processing based on data measured by the measurement vehicle. Note that the feature object is an object on or around the road that can be detected by a camera, and examples thereof include signs, billboards, postal posts, guardrails, and the like.

[0021] The measurement vehicle mainly includes an in-vehicle device 1, a camera 2, and a sensor unit 3. The in-vehicle device 1 is electrically connected to the camera 2 and the sensor unit 3 by wire or wirelessly, and stores data generated by the camera 2 and the sensor unit 3. Further, the in-vehicle device 1 is capable of data communication with the change point detection device 4, and transmits data generated by the camera 2 and the sensor unit 3 to the change point detection device 4 as measurement data “Im”.

[0022] The camera 2 is installed on the measurement vehicle and generates an image (also referred to as a “captured image”) of the scenery from the measurement vehicle. The captured image includes date and time data indicating the date and time of capture (generation time) of the captured image as metadata. The camera 2 supplies the captured image including the date and time data to the in-vehicle device 1.

[0023] Sensor unit 3 is a group of sensors that detect information regarding the position and attitude (direction of travel) of the measurement vehicle. Sensor unit 3 includes, for example, multiple sensors such as a GPS receiver, an accelerometer, a gyroscope, and an IMU (Inertial Measurement Unit). The GPS receiver may generate highly accurate positional information indicating the absolute position of the measurement vehicle (e.g., a three-dimensional position of latitude, longitude, and altitude) based on the RTK positioning method (i.e., interferometric positioning method). Sensor unit 3 may also be sensors provided on camera 2 to directly detect the position and shooting direction of camera 2. Sensor unit 3 supplies information (also called "sensor information") that associates the detection results from the sensors with date and time data indicating the date and time of detection to the in-vehicle device 1.

[0024] The in-vehicle unit 1 transmits measurement data Im, which includes the captured image output by camera 2 and sensor information output by sensor unit 3, to the change point detection device 4 at a predetermined timing. In this case, the in-vehicle unit 1 performs processing to appropriately correct the data output by camera 2 and sensor unit 3. For example, the in-vehicle unit 1 may detect the difference in reference time within each device, camera 2 and sensor unit 3, and correct at least one of these date and time data based on the above time difference so that the date and time data of the captured image and sensor information are synchronized. As a result, even if there is a difference in the internal reference time between camera 2 and sensor unit 3, the in-vehicle unit 1 generates measurement data Im that appropriately associates the captured image with the shooting position and shooting direction of the captured image.

[0025] The change point detection device 4 receives measurement data Im, which includes captured images and sensor information, from the in-vehicle unit 1 and stores the received measurement data Im. The change point detection device 4 then detects changes in a feature object between measurement timings based on the measurement data Im measured at different time periods (measurement timings) within the same section. In this case, the change point detection device 4 detects changes such as changes in the position or shape of a feature object, the disappearance of a feature object, or the creation of a new feature object as a change point.

[0026] Note that the configuration of the change point detection system shown in Figure 1 is just one example, and various modifications can be made to the configuration shown in Figure 1. For example, at least two of the in-vehicle unit 1, camera 2, and sensor unit 3 may be integrated into a single unit. In this case, the in-vehicle unit 1, camera 2, and sensor unit 3 may be configured as a single drive recorder. Alternatively, instead of acquiring measurement data Im through data communication with the in-vehicle unit 1, the change point detection device 4 may acquire measurement data Im by reading the measurement data Im stored in a storage medium by the in-vehicle unit 1. In this case, the storage medium is electrically connected to the in-vehicle unit 1 during measurement of the vehicle, so that the in-vehicle unit 1 can write the measurement data Im. After measurement of the vehicle, the storage medium is electrically connected to the change point detection device 4, so that the change point detection device 4 can read the measurement data Im. Alternatively, the camera 2 and sensor unit 3 may each independently store log data generated on a storage medium, and the measurement data Im may be supplied to the change point detection device 4 by having the storage medium read by the change point detection device 4. In this case, the change point detection system does not need to have an in-vehicle unit 1. Also, there may be multiple measurement vehicles. Furthermore, the change point detection device 4 may consist of multiple devices. In this case, the multiple devices execute pre-assigned processes and exchange necessary data with each other.

[0027] (2) Device configuration Figure 2(A) is a block diagram showing the functional configuration of the in-vehicle unit 1. The in-vehicle unit 1 mainly consists of an interface 11, a memory 12, and a controller 15. Each of these elements is interconnected via a bus line.

[0028] Interface 11 performs interface operations related to the exchange of data between the in-vehicle unit 1 and external devices. In this embodiment, interface 11 acquires data output from the camera 2 and sensor unit 3, etc., and supplies it to memory 12.

[0029] Memory 12 is composed of various types of memory, including RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including hard disk drives, flash memory, etc.). Memory 12 stores programs for the controller 15 to execute predetermined processes. Memory 12 is also used as the controller 15's working memory. Note that the programs executed by the controller 15 may be stored in storage media other than memory 12.

[0030] Furthermore, the memory 12 functionally includes a captured image storage unit 16 and a sensor information storage unit 17. The captured image storage unit 16 stores captured images generated by the camera 2. These captured images include date and time data indicating the date and time of capture as metadata. The sensor information storage unit 17 stores sensor information output by the sensor unit 3. The sensor information includes date and time data indicating the date and time of detection by the sensor unit 3.

[0031] Furthermore, at least one of the captured image storage unit 16 and the sensor information storage unit 17 may be stored in an external storage device of the in-vehicle device 1, such as a hard disk connected to the in-vehicle device 1 via the interface 11.

[0032] The controller 15 includes processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the entire in-vehicle device 1. In this case, the controller 15 executes programs stored in memory 12, etc., to perform data storage processing for the captured image storage unit 16 and the sensor information storage unit 17, and transmission processing for the measurement data Im to the change point detection device 4.

[0033] Figure 2(B) is a block diagram showing the functional configuration of the change point detection device 4. The change point detection device 4 includes an interface 41, a memory 42, and a controller 45. These elements are interconnected via a bus line.

[0034] Interface 41 performs interface operations related to the exchange of data between the change point detection device 4 and an external device. In this embodiment, interface 41 receives the measurement data Im generated by the in-vehicle device 1. Interface 41 may be a wireless interface for wireless communication with the in-vehicle device 1, or it may be a hardware interface for reading the measurement data Im from a storage medium or the like that stores the measurement data Im.

[0035] Memory 42 is composed of various types of memory, including RAM, ROM, and other non-volatile memory (including hard disk drives, flash memory, etc.). Memory 42 stores programs for the controller 45 to execute predetermined processes. Memory 42 is also used as working memory for the controller 45. Note that the programs executed by the controller 45 may be stored in storage media other than memory 42.

[0036] Furthermore, the memory 42 functionally includes a measurement data storage unit 46, an ortho-image storage unit 47, a feature information storage unit 48, and a change point information storage unit 49. The measurement data storage unit 46 stores the measurement data Im received from the in-vehicle device 1. Preferably, the change point detection device 4 stores in the measurement data storage unit 46 a group of captured images and corresponding sensor information for each road section divided according to a predetermined rule. Road section identification is performed automatically or manually, for example, based on the information of the shooting location included in the sensor information.

[0037] The orthoimage storage unit 47 stores the orthoimage generated by the change point detection device 4 by synthesizing a series of continuously captured images. The feature information storage unit 48 stores feature information about feature objects, which are characteristic features detected from the captured images. The feature information includes, for example, the projection position when the 3D data of the feature object extracted from the captured image is projected onto the generated orthoimage, and the identification information of the corresponding orthoimage. The change point information storage unit 49 stores information about change points of feature objects detected by the change point detection device 4.

[0038] Furthermore, at least one of the measurement data storage unit 46, orthoimage storage unit 47, feature information storage unit 48, and change point information storage unit 49 may be stored in an external storage device of the change point detection device 4, such as a hard disk connected to the change point detection device 4 via the interface 41. The above storage device may be a server device that communicates with the change point detection device 4. In addition, the above storage device may consist of multiple devices.

[0039] The controller 45 includes a processor such as a CPU and GPU, and controls the entire change point detection device 4. In this case, the controller 45 performs processing related to the detection of change points of feature objects by executing a program stored in memory 42, etc. The controller 45 functions as a "first acquisition means," a "second acquisition means," a "composite image generation means," a "shooting position calculation means," a "alignment means," a "projection means," a "change point detection means," and a computer that executes the program.

[0040] Furthermore, the processing performed by the controller 45 is not limited to being implemented by software through a program; it may also be implemented by a combination of hardware, firmware, and software. Additionally, the processing performed by the controller 45 may be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program executed by the controller 45 in this embodiment may be implemented using this integrated circuit. Thus, the controller 45 may be implemented using hardware other than a processor.

[0041] (3) Functional Blocks Figure 3 is a block diagram showing the functional configuration of the controller 45 of the change point detection device 4 in this embodiment. As shown in Figure 3, the controller 45 functionally includes a feature extraction unit 51, a shooting position calculation unit 52, a spatial position calculation unit 53, an orthomosaic image generation unit 54, a projection unit 55, a positioning unit 56, and a change point detection unit 57.

[0042] Figure 3 illustrates an example of detecting change points of feature objects based on measurement data Im generated when a measurement vehicle travels and measures a certain driving section (the target driving section) during different measurement periods (referred to as the "first measurement period" and the "second measurement period," respectively). Hereafter, a group of images captured during the first measurement period will be called the "first image group," and the first image group and the sensor information corresponding to the first image group will also be called the "first data set." Similarly, a group of images captured during the second measurement period will be called the "second image group," and the second image group and the sensor information corresponding to the second image group will also be called the "second data set." Note that the change point detection device 4 does not need to process the first data set and the second data set simultaneously; it may process the previously obtained data set first and store the generated orthoimage and feature information in the orthoimage storage unit 47 and the feature information storage unit 48.

[0043] The feature extraction unit 51 extracts feature objects (which may be multiple) from each captured image that makes up the first image group and the second image group, respectively. In this case, for example, the feature extraction unit 51 uses image recognition technology to extract the pixel regions that make up the feature objects from each captured image. In this case, the feature extraction unit 51 may extract feature objects based on pattern matching, or it may extract feature objects using an inference unit that outputs information on the type of feature object present in the image and its pixel region from the input image. The inference unit is, for example, a learning model based on machine learning such as deep learning, and the feature extraction unit 51 constructs the inference unit by reading the parameters of the inference unit obtained by prior machine learning from the memory 42, etc., and inputs the captured image to the inference unit. The feature extraction unit 51 then supplies the two-dimensional coordinates of the pixel regions of the feature objects extracted from each captured image of the first image group and the second image group (also called "feature object two-dimensional coordinates") to the shooting position calculation unit 52. The 2D coordinates of the feature object indicate the coordinates in the captured image corresponding to each pixel that makes up the pixel region of the extracted feature object.

[0044] The shooting position calculation unit 52 calculates the shooting position and shooting direction of each image constituting the first image group and the second image group, respectively. In this case, the shooting position calculation unit 52 refers to sensor information that includes date and time data that matches the date and time data attached to each image, and calculates the shooting position and shooting direction of the target image. If the referenced sensor information indicates the position and direction of travel of the measurement vehicle, the shooting position calculation unit 52 further refers to camera installation information that indicates the relative position and orientation of the camera 2 with respect to the measurement vehicle (more specifically, with respect to the sensor unit 3), and identifies the position and shooting direction of the camera 2 that captured the target image. The shooting position calculation unit 52 may receive the above camera installation information from the in-vehicle device 1 together with the measurement data Im, or it may be stored in the memory 42 in advance. The shooting position calculation unit 52 supplies the information indicating the shooting position and shooting direction of each calculated image to the spatial position calculation unit 53. The shooting position calculation unit 52 also supplies the information indicating the shooting position and shooting direction of each calculated image together with the corresponding image to the orthomosaic image generation unit 54.

[0045] The spatial position calculation unit 53 calculates a model of each feature object (also called a "feature object model") that represents the three-dimensional coordinates of the spatial position, based on the two-dimensional coordinates of each feature object extracted from the captured images by the feature object extraction unit 51 and the shooting position and shooting direction of each captured image identified by the shooting position calculation unit 52. In this case, the shooting position calculation unit 52 calculates a feature object model, which is point cloud data representing the three-dimensional shape of a feature object, from the two-dimensional coordinates of the feature objects that are common to multiple captured images in the first image group, based on a stereo measurement method using triangulation. The shooting position calculation unit 52 then supplies the feature object model calculated using the first image group to the projection unit 55. Similarly, the shooting position calculation unit 52 calculates a feature object model from the two-dimensional coordinates of the feature objects that are common to multiple captured images in the second image group, based on a stereo measurement method using triangulation. The shooting position calculation unit 52 then supplies the feature object model, which is point cloud data indicating the three-dimensional coordinates of the feature object calculated using the second image group, to the projection unit 55.

[0046] The orthoimage generation unit 54 generates an orthoimage, which is a composite image of the captured images, based on the captured images constituting a series of images supplied from the shooting position calculation unit 52, and information indicating the shooting position and direction of these captured images. In this case, the orthoimage generation unit 54 generates an orthoimage by performing 3D modeling based on, for example, a stereo measurement method using triangulation, and then performing an orthorectification transformation on the generated 3D model. The above-mentioned 3D model may be point cloud data representing the shape of a feature object, or if the feature object is limited to a planar object such as a sign, it may be a combination of the center position of the feature object and a normal vector. In this way, the orthoimage generation unit 54 generates an orthoimage (also called the "first orthoimage") based on the first image group and generates an orthoimage (also called the "second orthoimage") based on the second image group. The orthoimage generation unit 54 stores the first orthoimage and the second orthoimage in the orthoimage storage unit 47.

[0047] The projection unit 55 calculates the position of the target feature object on the orthoimage (also called the "projected position") by projecting the feature object model generated by the spatial position calculation unit 53 onto the orthoimage generated by the orthoimage generation unit 54. In this case, the projection unit 55 calculates the projected position for each feature object by applying the same orthographic transformation that the orthoimage generation unit 54 performed on the 3D model when generating the orthoimage to the feature object model. The projection unit 55 then calculates the projected position of the feature object on the first orthoimage (referred to as the "first projected position") from the first orthoimage and feature object model based on the first dataset. The projection unit 55 also calculates the projected position of the feature object on the second orthoimage (referred to as the "second projected position") from the second orthoimage and feature object model based on the second dataset. The projection unit 55 stores the information indicating the calculated projected positions and the identification information of the corresponding orthoimage in the feature object information storage unit 48. Furthermore, the feature information storage unit 48 may also store the two-dimensional coordinates of the feature calculated by the feature extraction unit 51, and / or the feature model calculated by the spatial position calculation unit 53.

[0048] The alignment unit 56 aligns the first orthoimage with the second orthoimage. In this case, for example, the alignment unit 56 calculates the similarity between the first and second orthoimages while relatively translating one of them, and calculates the amount of translation that maximizes the similarity (match) as the shift amount. This shift amount corresponds to the shift between the coordinate system used as the reference in the first orthoimage and the coordinate system used as the reference in the second orthoimage. The alignment unit 56 supplies the calculated shift amount to the change point detection unit 57. The alignment unit 56 may extract the first and second orthoimages from the orthoimage storage unit 47 or obtain them from the orthoimage generation unit 54.

[0049] Preferably, the alignment unit 56 aligns the first orthoimage and the second orthoimage using the roads included in the first orthoimage and the second orthoimage, respectively, as a reference. In this case, the alignment unit 56 excludes areas other than roads from the first orthoimage and the second orthoimage by masking, and then aligns the first orthoimage and the second orthoimage after masking. In this case, the alignment unit 56 recognizes areas other than roads in the first orthoimage and the second orthoimage by extracting the road areas in each of the first orthoimage and the second orthoimage, for example, using any image recognition technology. This allows the alignment unit 56 to perform alignment using only the road portions with a high degree of confidence in their ability to change.

[0050] The change point detection unit 57 detects change points based on the first and second projection positions for each feature calculated by the projection unit 55 and the amount of displacement calculated by the alignment unit 56. In this case, the change point detection unit 57 corrects the first and second projection positions to indicate their positions in a common coordinate system based on the amount of displacement calculated by the alignment unit 56 relative to the first or second projection position. The change point detection unit 57 then determines the presence or absence of a change point by comparing the corrected first and second projection positions for each feature point. The change point detection unit 57 then stores change point information related to the detected change point in the change point information storage unit 49. In this case, for example, the change point detection unit 57 generates change point information including the projection position of the target feature after the change and the feature model used to calculate the projection position. Furthermore, if either the first projection position or the second projection position for a certain feature object does not exist, the change point detection unit 57 determines that the feature object has disappeared or been newly created, and stores the determination result in the change point information storage unit 49.

[0051] (4) Specific example A specific example of the processing of the change point detection device 4, as described in Figure 3, will be explained with reference to Figures 4 and 5.

[0052] Figure 4 shows an overhead view of a road section 50. Feature objects A and B are located near road section 50. At a certain date and time, the measurement vehicle travels along road section 50 following the solid line trajectory "Lt1" and conducts measurements, generating the first dataset shown in Figure 3. Subsequently, the measurement vehicle travels along road section 50 again following the dashed line trajectory Lt2, generating the second dataset shown in Figure 3.

[0053] Figure 5(A) shows the first orthoimage generated based on the first dataset. Figure 5(B) shows the second orthoimage generated based on the second dataset. Figure 5(C) is an image obtained by superimposing the first and second orthoimages after alignment. The first orthoimage clearly indicates the first projection positions of feature A and feature B, and the second orthoimage clearly indicates the second projection positions of feature A and feature B.

[0054] The alignment unit 56 performs alignment to determine the most matching relative position by performing translation, scaling, rotation in the roll, pitch, and yaw directions on at least one of the first orthoimage shown in Figure 5(A) and the second orthoimage shown in Figure 5(B). In this case, as shown in Figure 5(C), the alignment is performed so that the road section 50 matches. Then, the change point detection unit 57 compares the first and second projected positions, which have been corrected by the alignment to show the position in the common coordinate system, for each of feature object A and feature object B. In this case, the change point detection unit 57 determines that there has been a change in feature object A because there is a difference between the first and second projected positions of feature object A, and generates change point information for feature object A.

[0055] Here, we will provide a supplementary explanation regarding the comparison between the first projection position and the second projection position. The change point detection unit 57 determines that at least one of the position or shape of the target feature has changed when the centroid of the region corresponding to the first projection position and the centroid of the region corresponding to the second projection position are at least a predetermined distance apart, and generates change point information for that feature. In another example, the change point detection unit 57 determines that at least one of the position or shape of the target feature has changed when the degree of overlap between the region corresponding to the first projection position and the region corresponding to the second projection position is less than or equal to a predetermined ratio, and generates change point information for that feature.

[0056] (5) Processing flow Figure 6 is an example flowchart showing the procedure for change point detection processing performed by the change point detection device 4 in this embodiment. In Figure 6, the change point detection device 4 calculates orthomosaic images and projection positions by sequentially processing the captured images and sensor information datasets for each road section, as an example. Then, if the change point detection device 4 has already calculated orthomosaic images and projection positions based on another dataset corresponding to the same road section, it determines whether or not there is a change point in that road section. The change point detection device 4 repeatedly executes the process shown in the flowchart in Figure 6.

[0057] First, the change point detection device 4 acquires and stores the measurement data Im (step S10). In this case, the change point detection device 4 acquires the measurement data Im generated by the in-vehicle unit 1 and stores it in the measurement data storage unit 46. Subsequently, the change point detection device 4 extracts a dataset from the measurement data storage unit 46 that corresponds to a combination of the image group of captured images generated in the road section where the change point is to be detected and the sensor information corresponding to the captured images (step S11).

[0058] Next, the change point detection device 4 determines whether or not a feature exists in the captured image included in the dataset extracted in step S11 (step S12). If the change point detection device 4 determines that a feature exists (step S12; Yes), it proceeds to step S13. On the other hand, if the change point detection device 4 determines that a feature does not exist (step S12; No), it determines whether orthoimages for the same section are stored in the orthoimage storage unit 47 (step S20). If orthoimages for the same section are stored in the orthoimage storage unit 47 (step S20; Yes), the change point detection device 4 determines that the feature that was present in the target section may have been removed (step S21). In this case, the change point detection device 4 uses a method such as semantic segmentation to perform an occlusion determination to determine whether or not the feature is hidden by another object. If the change point detection device 4 determines that occlusion has not occurred, it considers the target feature object as a candidate change point related to removal, and stores or transmits information regarding the candidate change point to another device.

[0059] Next, the change point detection device 4 generates a feature model, which is a three-dimensional model of a feature in each captured image included in the dataset extracted in step S11 (step S13). In this case, the change point detection device 4 generates the feature model based on the two-dimensional coordinates of the feature corresponding to the pixel region of the feature extracted from multiple captured images, and the shooting position and shooting direction of these captured images calculated from sensor information.

[0060] Furthermore, the change point detection device 4 generates an orthomosaic image (step S14). In this case, the change point detection device 4 generates a 3D model of the area around the target road section based on each captured image of the target dataset and the shooting position and direction of each captured image identified from the sensor information. Then, the change point detection device 4 generates an orthomosaic image by performing an orthographic transformation on the generated 3D model. Furthermore, the change point detection device 4 calculates the projection position of feature objects on the orthomosaic image generated in step S14 (step S15). In this case, the change point detection device 4 calculates the above-mentioned projection position by performing a process of projecting the feature object model generated in step S13 onto the orthomosaic image generated in step S14.

[0061] Next, the change point detection device 4 determines whether or not an orthoimage for the same section as the orthoimage generated in step S14 is stored in the orthoimage storage unit 47 (step S16). If an orthoimage for the same section as the orthoimage generated in step S14 is stored in the orthoimage storage unit 47 (step S16; Yes), the change point detection device 4 aligns these orthoimages (step S17). As a result, the change point detection device 4 calculates the amount of shift required to correct the projection position of the feature corresponding to each compared orthoimage to its position in the common coordinate system.

[0062] Next, the change point detection device 4 extracts the projection position of the feature corresponding to the orthoimage, compared with the orthoimage generated in step S14, from the feature information storage unit 48. Then, the change point detection device 4 corrects at least one of the extracted feature projection position and the feature projection position calculated in step S15 based on the alignment in step S17, and detects change points based on these corrected projection positions (step S18). In this case, the change point detection device 4 determines whether a change has occurred by comparing the projection position of the feature extracted from the feature information storage unit 48 and the projection position of the feature calculated in step S15 in a common coordinate system. If the change point detection device 4 determines that a change has occurred, it considers the changed feature as a change point candidate and stores or transmits information about the change point candidate to another device. Furthermore, the change point detection device 4 stores the orthoimage generated in step S14 and the projection position of the feature calculated in step S15 in the orthoimage storage unit 47 and the feature information storage unit 48, respectively.

[0063] On the other hand, if the change point detection device 4 determines that an orthoimage of the same section as the orthoimage generated in step S14 is not stored in the orthoimage storage unit 47 (step S16; No), it stores the orthoimage generated in step S14 and the projection position of the feature calculated in step S15 in the orthoimage storage unit 47 and the feature information storage unit 48, respectively (step S19). In this case, the change point detection device 4 determines that the feature may have been newly installed, considers the target feature as a candidate change point related to the new installation, and stores or transmits information regarding the candidate change point. The data stored in the orthoimage storage unit 47 and the feature information storage unit 48 in step S19 will be used in steps S17 and S18 of this flowchart, which will be executed later using a dataset targeting the same road section.

[0064] In this way, the change point detection device 4 can suitably detect change points by comparing feature objects on an orthomosaic image synthesized from images captured by the measurement vehicle. As a result, the change point detection device 4 can accurately detect change points even in areas with few feature objects where real-time feature-point-based monocular SLAM (Simultaneous Localization and Mapping) cannot function accurately.

[0065] (6) Variation The processes of the feature extraction unit 51, the shooting position calculation unit 52, the spatial position calculation unit 53, the orthoimage generation unit 54, and the projection unit 55, as described in Figure 3, may be performed by a device other than the change point detection device 4. For example, the in-vehicle device 1 may have functions equivalent to the feature extraction unit 51, the shooting position calculation unit 52, the spatial position calculation unit 53, the orthoimage generation unit 54, and the projection unit 55. In this case, for example, the in-vehicle device 1 calculates the projection positions of orthoimages and feature objects based on a dataset of captured images and sensor information for each road section, and transmits the calculated orthoimages and feature object information to the change point detection device 4. The change point detection device 4 stores the orthoimages received from the in-vehicle device 1 in the orthoimage storage unit 47, and stores the feature object information received from the in-vehicle device 1 in the feature object information storage unit 48. Then, at a predetermined timing, the change point detection device 4 uses two sets of orthomosaic images and feature information for the same road section to perform alignment by the alignment unit 56 and change point detection by the change point detection unit 57.

[0066] Figure 7 is an example flowchart showing the procedure for change point detection processing performed by the change point detection device 4 in this modified example. The change point detection device 4 performs the processing shown in the flowchart in Figure 7 when there are two sets of orthomosaic images and feature information for the same road section.

[0067] First, the change point detection device 4 acquires a first orthoimage and a first projection position, and a second orthoimage and a second projection position, for the same road section (step S31). In this case, for example, the change point detection device 4 may extract this information from the orthoimage storage unit 47 and the feature information storage unit 48, or it may receive it from the in-vehicle device 1.

[0068] Next, the alignment unit 56 of the change point detection device 4 aligns the first orthoimage and the second orthoimage (step S32). Then, the change point detection unit 57 of the change point detection device 4 detects the change point based on the first projection position and the second projection position, which have been corrected by the alignment in step S32 (step S33).

[0069] Thus, in this modified example as well, the change point detection device 4 can suitably detect change points by comparing the projection positions of feature objects on an orthomosaic image synthesized from images captured by the measurement vehicle.

[0070] As described above, the change point detection device 4 according to this embodiment includes a first acquisition means, a second acquisition means, a alignment means, and a change point detection means. The first acquisition means acquires a first orthoimage generated based on a first image group and a first projection position when a feature object included in at least one image of the first image group is projected onto the first orthoimage. The second acquisition means acquires a second orthoimage generated based on a second image group taken at a different time than the first image group and a second projection position when a feature object included in at least one image of the second image group is projected onto the second orthoimage. The alignment means aligns the first orthoimage and the second orthoimage. The change point detection means detects change points related to feature objects based on the first projection position and the second projection position, which have been corrected by the alignment. As a result, the change point detection device 4 can suitably detect change points.

[0071] In the embodiments described above, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer, such as a controller. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (Random Access Memory)).

[0072] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of symbols]

[0073] 1 On-vehicle device 2 cameras 3 Sensor Unit 4. Change Point Detection Device 16 Image storage unit 17 Sensor Information Storage Unit 46 Measurement data storage unit 47 Ortho-image storage unit 48. Feature Information Storage Unit 49 Change Point Information Storage Unit

Claims

1. Alignment means for aligning a first composite image including a first projection position on which a feature object is projected and a second composite image including a second projection position on which the feature object is projected, A change point detection means for detecting change points relating to the feature object based on the first projection position and the second projection position, which have been corrected by the aforementioned alignment, A change point detection device having the following features.

2. The first composite image is generated based on a first set of images taken from a vehicle traveling on a predetermined road section. The second composite image is generated based on a second set of images taken from a vehicle traveling on the road section during a different period than the first set of images. The change point detection device according to claim 1, wherein the alignment means performs the alignment with respect to the road in the road section.

3. The change point detection device according to claim 1 or 2, further comprising a composite image generation means for generating a first composite image based on a first group of images, and generating a second composite image based on a second group of images taken during a different period from the first group of images.

4. The composite image generation means is Based on the first image group and the shooting position and shooting direction of each image constituting the first image group, the first composite image is generated. The change point detection device according to claim 3, which generates the second composite image based on a second group of images taken during a different period from the first group of images, and the shooting position and shooting direction of each image constituting the second group of images.

5. The first and second image groups consist of images taken from the vehicle. The change point detection device according to claim 4, further comprising a shooting position calculation means for calculating the shooting position and the shooting direction based on data output by a sensor provided on the vehicle during the period of image capture.

6. The change point detection device according to any one of claims 3 to 5, wherein the composite image generation means generates orthoimages as the first composite image and the second composite image.

7. The first composite image is generated based on the first image group, The second composite image is generated based on a second set of images taken at a different time period than the first set of images. A change point detection device according to any one of claims 1 to 6, further comprising projection means for calculating a first projection position obtained by projecting the spatial position of the feature object, which is identified based on the images of the first group of images containing the feature object, onto a first composite image, and for calculating a second projection position obtained by projecting the spatial position of the feature object, which is identified based on the images of the second group of images containing the feature object, onto a second composite image.

8. By computer, Alignment is performed between a first composite image including a first projection position on which the feature object is projected and a second composite image including a second projection position on which the feature object is projected. Based on the first projection position and the second projection position, which have been corrected by the aforementioned alignment, a change point relating to the feature object is detected. Control method.

9. Alignment means for aligning a first composite image including a first projection position on which a feature object is projected and a second composite image including a second projection position on which the feature object is projected, Change point detection means for detecting change points relating to the feature object based on the first projection position and the second projection position, which have been corrected by the aforementioned alignment. A program that makes a computer function.

10. A computer-readable storage medium storing the program described in claim 9.

Images