Change point detection device, control method, program, and storage medium

Abstract

To provide a change point detection device that can appropriately detect a change point related to a characteristic object.SOLUTION: A change point detection device 4 has first acquisition means, second acquisition means, alignment means, and a controller 45 functioning as change point detection means. The first acquisition means acquires a first ortho image that is created based on a first image group, and a first projection position of a characteristic object included in at least one image of the first image group, the characteristic object being projected on the first ortho image. The second acquisition means acquires a second ortho image that is created based on a second image group captured in a period different from that for the first image group, and a second projection position of a characteristic object included in at least one image of the second image group, the characteristic object being projected on the second ortho image. The alignment means performs alignment between the first ortho image and the second ortho image. The change point detection means detects a change point related to the characteristic object on the basis of the first projection position and the second projection position at least one of which is corrected through the alignment.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for detecting a change point. [Background technology]

[0002] Conventionally, there is known a technique for detecting a change point of a map based on the output of a sensor installed in a vehicle. For example, Patent Document 1 discloses a system in which, when each vehicle detects a change point in the map data by a sensor, the vehicle updates the map data by transmitting data related to the change point to a map management server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-156973 A Summary of the Invention [Problem to be solved by the invention]

[0004] When measuring the road surroundings using a measurement vehicle, if feature-based SLAM (Simultaneous Localization and Mapping) is applied, the accuracy of position estimation may be low in road sections with few landmark features, and the accuracy of the generated measurement data may be low. In this case, there is a risk that change points cannot be detected properly.

[0005] The present invention has been made in order to solve the above-mentioned problems, and has as its main object to provide a change-point detection device capable of suitably detecting a change-point. [Means for solving the problem]

[0006] The invention described in the claims is a change point detection device comprising: an alignment means for aligning a first composite image including a first projection position onto which a feature is projected, and a second composite image including a second projection position onto which the feature is projected; and a change point detection means for detecting a change point relating to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the alignment.

[0007] The invention described in the claims is a control method in which a computer aligns a first composite image including a first projection position onto which a feature is projected with a second composite image including a second projection position onto which the feature is projected, and detects a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the alignment.

[0008] The invention described in the claims is also a program that causes a computer to function as an alignment means for aligning a first composite image including a first projection position onto which a feature is projected and a second composite image including a second projection position onto which the feature is projected, and a change point detection means for detecting a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the alignment. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic configuration of a change-point detection system. [Diagram 2] 2 shows a block configuration of an on-board device and a change-point detection device. [Diagram 3] FIG. 2 shows a functional block diagram of a change-point detection device. [Figure 4] 1 shows an overhead view of a road section. [Diagram 5] (A) A first ortho-image generated based on the first dataset, (B) a second ortho-image generated based on the second dataset, and (C) an image obtained by overlaying the first and second ortho-images after aligning them. [Figure 6]11 is a flowchart illustrating a procedure of a change-point detection process according to an embodiment. [Figure 7] 13 is an example of a flowchart illustrating a procedure of a change-point detection process according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] According to a preferred embodiment of the present invention, the change point detection device includes a first acquisition means for acquiring a first composite image generated based on a first image group and a first projection position when a feature included in at least one image of the first image group is projected onto the first composite image, a second acquisition means for acquiring a second composite image generated based on a second image group captured in a different period from the first image group and a second projection position when the feature included in at least one image of the second image group is projected onto the second composite image, a position alignment means for aligning the first composite image and the second composite image, and a change point detection means for detecting a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the position alignment. With this aspect, the change point detection device can preferably detect a change point of the feature based on the first image group and the second image group captured in different periods.

[0011] In one aspect of the change point detection device, the first image group and the second image group are composed of images taken from a vehicle traveling on a predetermined road section, and the alignment means performs the alignment based on a road in the road section. With this aspect, the change point detection device can accurately align the first composite image and the second composite image, and preferably detect change points in features.

[0012] In another aspect of the above-mentioned change point detection device, the change point detection device further includes a composite image generating means for generating the first composite image based on the first image group and generating the second composite image based on the second image group. With this aspect, the change point detection device can suitably obtain the first composite image and the second composite image to be aligned.

[0013] In another aspect of the change point detection device, the composite image generating means generates the first composite image based on the first image group and the shooting positions and shooting directions of each image constituting the first image group, and generates the second composite image based on the second image group and the shooting positions and shooting directions of each image constituting the second image group. With this aspect, the change point detection device can suitably generate a first composite image and a second composite image that are comparable.

[0014] In another aspect of the above-mentioned change point detection device, the first image group and the second image group are composed of images taken from a vehicle, and the change point detection device further includes a photographing position calculation means for calculating the photographing position and the photographing direction based on data output from a sensor provided on the vehicle during a photographing period of the images. With this aspect, the change point detection device can suitably identify the photographing position and the photographing direction of each image required for generating a composite image.

[0015] In another aspect of the change point detection device, the composite image generating means generates orthoimages as the first composite image and the second composite image, whereby the change point detection device can suitably generate the first composite image and the second composite image suitable for alignment.

[0016] In another aspect of the above change point detection device, the change point detection device further comprises a projection means for calculating the first projection position by projecting a spatial position of the feature identified based on the images of the first image group including the feature onto the first composite image, and for calculating the second projection position by projecting a spatial position of the feature identified based on the images of the second image group including the feature onto the second composite image. With this aspect, the change point detection device can suitably calculate the first projection position of the feature on the first composite image and the second projection position of the feature on the second composite image.

[0017] According to another preferred embodiment of the present invention, the control method includes a computer that acquires a first composite image generated based on a first image group and a first projection position when a feature included in at least one image of the first image group is projected onto the first composite image, acquires a second composite image generated based on a second image group captured in a different period from the first image group and a second projection position when the feature included in at least one image of the second image group is projected onto the second composite image, aligns the first composite image with the second composite image, and detects a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the alignment. By executing this control method, the computer can preferably detect a change point of the feature based on the first image group and the second image group captured in different periods.

[0018] According to another preferred embodiment of the present invention, the program causes a computer to function as a first acquisition means for acquiring a first composite image generated based on a first image group and a first projection position when a feature included in at least one image of the first image group is projected onto the first composite image, a second acquisition means for acquiring a second composite image generated based on a second image group captured in a period different from that of the first image group and a second projection position when the feature included in at least one image of the second image group is projected onto the second composite image, a position adjustment means for aligning the first composite image and the second composite image, and a change point detection means for detecting a change point related to the feature based on the first projection position and the second projection position, at least one of which is corrected by the position adjustment. By executing this program, the computer can preferably detect a change point of the feature based on the first image group and the second image group captured in different periods. Preferably, the program is stored in a storage medium. EXAMPLES

[0019] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0020] (1) System Overview Fig. 1 is a schematic diagram of a change-point detection system according to the present embodiment. The change-point detection system shown in Fig. 1 is a system that detects change-points in feature objects present around a road, and mainly includes a measurement vehicle that travels along the road while taking measurements, and a change-point detection device 4 that performs a process of detecting change-points based on data measured by the measurement vehicle. The feature objects are objects on or around the road that can be detected by a camera, such as signs, billboards, postboxes, and guardrails.

[0021] The measurement vehicle mainly has an on-board device 1, a camera 2, and a sensor unit 3. The on-board device 1 is electrically connected to the camera 2 and the sensor unit 3 by wire or wirelessly, and stores data generated by the camera 2 and the sensor unit 3. The on-board device 1 is also capable of data communication with a change-point detection device 4, and transmits the data generated by the camera 2 and the sensor unit 3 to the change-point detection device 4 as measurement data "Im."

[0022] The camera 2 is installed on the measurement vehicle and generates images (also called "captured images") of the scenery captured from the measurement vehicle. The captured images contain date and time data indicating the capture date and time (generation date and time) of the captured images as metadata. The camera 2 supplies the captured images containing the date and time data to the vehicle-mounted device 1.

[0023] The sensor unit 3 is a group of sensors that detect information regarding the position and attitude (traveling direction) of the measurement vehicle. The sensor unit 3 includes a plurality of sensors, such as a GPS receiver, an acceleration sensor, a gyro sensor, and an IMU (Inertial Measurement Unit). The GPS receiver may generate highly accurate position information indicating the absolute position (e.g., three-dimensional position of latitude, longitude, and altitude) of the measurement vehicle based on an RTK positioning method (i.e., an interferometric positioning method). The sensor unit 3 may be a sensor provided in the camera 2 so as to directly detect the position and shooting direction of the camera 2. The sensor unit 3 supplies information (also called "sensor information") that associates the detection result by the sensor with date and time data indicating the detection date and time to the vehicle-mounted device 1.

[0024] The vehicle-mounted device 1 transmits the measurement data Im including the captured image output by the camera 2 and the sensor information output by the sensor unit 3 to the change-point detection device 4 at a predetermined timing. In this case, the vehicle-mounted device 1 performs a process of appropriately correcting the data output by the camera 2 and the sensor unit 3. For example, the vehicle-mounted device 1 may detect a difference in the time references in the camera 2 and the sensor unit 3, and correct at least one of these date and time data based on the time difference so that the date and time data of the captured image and the sensor information are synchronized. In this way, even if there is a difference between the internal times referenced by the camera 2 and the sensor unit 3, the vehicle-mounted device 1 generates measurement data Im in which the captured image is appropriately associated with the capture position and capture direction of the captured image.

[0025] The change-point detection device 4 receives measurement data Im including captured images and sensor information from the vehicle-mounted device 1 and accumulates the received measurement data Im. Then, based on the measurement data Im measured in different periods (measurement timings) in the same section, the change-point detection device 4 detects a change point of a feature between the measurement timings. In this case, the change-point detection device 4 detects a change in the position or shape of a feature, the disappearance of a feature, the addition of a feature, etc. as a change point.

[0026] The configuration of the change point detection system shown in FIG. 1 is an example, and various modifications may be made to the configuration shown in FIG. 1. For example, at least two of the vehicle-mounted device 1, the camera 2, and the sensor unit 3 may be integrated. In this case, the vehicle-mounted device 1, the camera 2, and the sensor unit 3 may be configured as one drive recorder. Instead of acquiring the measurement data Im through data communication with the vehicle-mounted device 1, the change point detection device 4 may acquire the measurement data Im by reading the measurement data Im stored in the storage medium by the vehicle-mounted device 1 from the storage medium. In this case, the above-mentioned storage medium is electrically connected to the vehicle-mounted device 1 during measurement of the measurement vehicle, so that the measurement data Im is written by the vehicle-mounted device 1. After measurement of the measurement vehicle, the above-mentioned storage medium is electrically connected to the change point detection device 4, so that the measurement data Im is read by the change point detection device 4. Furthermore, the camera 2 and the sensor unit 3 may each independently store the log data generated in a storage medium, and the storage medium may be read by the change-point detection device 4 to supply the measurement data Im to the change-point detection device 4. In this case, the change-point detection system does not need to have the vehicle-mounted device 1. Also, there may be a plurality of measurement vehicles. Also, the change-point detection device 4 may be composed of a plurality of devices. In this case, the plurality of devices executes pre-assigned processes, and transmits and receives necessary data between each other.

[0027] (2) Device configuration 2(A) is a block diagram showing the functional configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 mainly includes an interface 11, a memory 12, and a controller 15. These elements are connected to each other via a bus line.

[0028] The interface 11 performs an interface operation related to the transmission and reception of data between the vehicle-mounted device 1 and an external device. In this embodiment, the interface 11 acquires data output from the camera 2, the sensor unit 3, etc., and supplies the data to the memory 12.

[0029] The memory 12 is composed of various types of memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a non-volatile memory (including a hard disk drive, a flash memory, etc.). The memory 12 stores programs for the controller 15 to execute predetermined processes. The memory 12 is also used as a working memory for the controller 15. The programs executed by the controller 15 may be stored in a storage medium other than the memory 12.

[0030] Moreover, functionally, the memory 12 has a captured image storage section 16 and a sensor information storage section 17. The captured image storage section 16 stores the captured image generated by the camera 2. This captured image includes date and time data indicating the date and time of capture as metadata. Moreover, the sensor information storage section 17 stores the sensor information output by the sensor unit 3. The sensor information includes date and time data indicating the date and time when detection by the sensor unit 3 was performed.

[0031] At least one of the captured image storage unit 16 and the sensor information storage unit 17 may be stored in a storage device external to the vehicle-mounted device 1, such as a hard disk connected to the vehicle-mounted device 1 via the interface 11.

[0032] The controller 15 includes a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the entire vehicle-mounted device 1. In this case, the controller 15 executes a program stored in the memory 12, etc., to accumulate data stored in the captured image storage unit 16 and the sensor information storage unit 17, and transmit the measurement data Im to the change-point detection device 4.

[0033] 2(B) is a block diagram showing the functional configuration of the change-point detection device 4. The change-point detection device 4 has an interface 41, a memory 42, and a controller 45. These elements are connected to each other via a bus line.

[0034] The interface 41 performs interface operations related to data transmission and reception between the change-point detection device 4 and an external device. In this embodiment, the interface 41 receives the measurement data Im generated by the vehicle-mounted device 1. The interface 41 may be a wireless interface for wireless communication with the vehicle-mounted device 1, or may be a hardware interface for reading out the measurement data Im from a storage medium or the like in which the measurement data Im is stored.

[0035] The memory 42 is composed of various types of memory such as RAM, ROM, and other non-volatile memories (including hard disk drives, flash memories, etc.). The memory 42 stores programs for the controller 45 to execute predetermined processes. The memory 42 is also used as a working memory for the controller 45. The programs executed by the controller 45 may be stored in a storage medium other than the memory 42.

[0036] Moreover, the memory 42 functionally has a measurement data storage unit 46, an orthoimage storage unit 47, a feature information storage unit 48, and a change point information storage unit 49. The measurement data storage unit 46 stores the measurement data Im received from the vehicle-mounted device 1. Preferably, the change point detection device 4 associates a group of captured images with corresponding sensor information for each road section divided according to a predetermined rule and stores them in the measurement data storage unit 46. The road section is identified automatically or manually based on, for example, information on the shooting position included in the sensor information.

[0037] The orthoimage storage unit 47 stores the orthoimage generated by the change point detection device 4 by synthesizing a group of continuously captured photographed images. The feature information storage unit 48 stores feature information related to features, which are characteristic features detected from the photographed images. The feature information includes, for example, a projection position when three-dimensional data of the feature extracted from the photographed images is projected onto the generated orthoimage, and identification information of the corresponding orthoimage. The change point information storage unit 49 stores information related to the change points of the feature detected by the change point detection device 4.

[0038] At least one of the measurement data storage unit 46, the orthoimage storage unit 47, the feature information storage unit 48, and the change-point information storage unit 49 may be stored in an external storage device of the change-point detection device 4, such as a hard disk connected to the change-point detection device 4 via the interface 41. The above storage device may be a server device that communicates with the change-point detection device 4. The above storage device may be composed of multiple devices.

[0039] The controller 45 includes a processor such as a CPU, a GPU, etc., and controls the entire change-point detection device 4. In this case, the controller 45 performs processing related to change-point detection of feature objects by executing programs stored in the memory 42, etc. The controller 45 functions as a "first acquisition means," a "second acquisition means," a "synthetic image generation means," a "photographing position calculation means," a "position alignment means," a "projection means," a "change-point detection means," and a computer that executes programs, etc.

[0040] The processing executed by the controller 45 is not limited to being realized by software using a program, but may be realized by any combination of hardware, firmware, and software. The processing executed by the controller 45 may be realized by using an integrated circuit programmable by a user, such as a field-programmable gate array (FPGA) or a microcomputer. In this case, the program executed by the controller 45 in this embodiment may be realized by using this integrated circuit. In this way, the controller 45 may be realized by hardware other than a processor.

[0041] (3) Functional Blocks Fig. 3 is a block diagram showing a functional configuration of the controller 45 of the change point detection device 4 in this embodiment. As shown in Fig. 3, the controller 45 functionally includes a feature object extraction unit 51, a shooting position calculation unit 52, a spatial position calculation unit 53, an orthoimage generation unit 54, a projection unit 55, a position adjustment unit 56, and a change point detection unit 57.

[0042] FIG. 3 illustrates an example of detecting change points in features based on measurement data Im generated when a measurement vehicle travels and measures a certain travel section (a target travel section) in different measurement periods (referred to as a "first measurement period" and a "second measurement period", respectively). Hereinafter, a group of captured images generated in the first measurement period is referred to as a "first image group", and the first image group and the sensor information corresponding to the first image group are also referred to as a "first data set". In addition, a group of captured images generated in the second measurement period is referred to as a "second image group", and the second image group and the sensor information corresponding to the second image group are also referred to as a "second data set". Note that the change point detection device 4 does not need to process the first data set and the second data set simultaneously, and may process the previously obtained data set in advance and store the generated orthoimage and feature information in the orthoimage storage unit 47 and the feature information storage unit 48.

[0043] The feature extraction unit 51 extracts feature objects (may be a plurality of feature objects) included in each captured image from each captured image constituting the first image group and the second image group. In this case, for example, the feature extraction unit 51 performs a process of extracting pixel areas constituting the feature objects from each captured image using an image recognition technology. In this case, the feature extraction unit 51 may extract feature objects based on pattern matching, or may extract feature objects using an inference device that outputs information on the type of feature object present in an input image and its pixel area from the input image. The inference device is, for example, a learning model based on machine learning such as deep learning, and the feature extraction unit 51 configures the inference device by reading out parameters of the inference device obtained by prior machine learning from the memory 42 or the like, and inputs the captured image to the inference device. Then, the feature extraction unit 51 supplies the two-dimensional coordinates (also called "feature object two-dimensional coordinates") of the pixel areas of the feature objects extracted from each captured image of the first image group and the second image group to the photographing position calculation unit 52. The two-dimensional coordinates of a feature indicate the coordinates in the captured image corresponding to each pixel constituting the pixel region of the extracted feature.

[0044] The photographing position calculation unit 52 calculates the photographing position and the photographing direction of each photographed image constituting each of the first image group and the second image group. In this case, the photographing position calculation unit 52 refers to the sensor information including the date and time data that matches the date and time indicated by the date and time data added to each photographed image, and calculates the photographing position and the photographing direction of the photographed image of the target. Here, when the referred sensor information indicates the position and traveling direction of the measurement vehicle, the photographing position calculation unit 52 further refers to the camera installation information indicating the relative position and the orientation of the camera 2 with respect to the measurement vehicle (more specifically, with respect to the sensor unit 3), thereby specifying the position and the photographing direction of the camera 2 that photographed the photographed image of the target. The photographing position calculation unit 52 may receive the camera installation information from the vehicle-mounted device 1 together with the measurement data Im, or may store it in advance in the memory 42. The photographing position calculation unit 52 supplies the calculated information indicating the photographing position and the photographing direction of each photographed image to the spatial position calculation unit 53. The photographing position calculation unit 52 also supplies the calculated information indicating the photographing position and the photographing direction of each photographed image to the orthoimage generation unit 54 together with the corresponding photographed image.

[0045] The spatial position calculation unit 53 calculates a model of each feature (also called a "feature model") indicating three-dimensional coordinates, which are spatial positions, based on the two-dimensional coordinates of each feature extracted from the captured images by the feature extraction unit 51 and the capturing positions and capturing directions of each captured image specified by the capturing position calculation unit 52. In this case, the capturing position calculation unit 52 calculates a feature model, which is point cloud data indicating the three-dimensional shape of the feature, based on a stereo measurement method using triangulation from the two-dimensional coordinates of the feature for the feature that exists in common in the multiple captured images of the first image group. Then, the capturing position calculation unit 52 supplies the feature model calculated using the first image group to the projection unit 55. Similarly, the capturing position calculation unit 52 calculates a feature model based on a stereo measurement method using triangulation from the two-dimensional coordinates of the feature for the feature that exists in common in the multiple captured images of the second image group. Then, the photographing position calculation unit 52 supplies to the projection unit 55 a feature model that is point cloud data indicating the three-dimensional coordinates of the feature calculated using the second image group.

[0046] The orthoimage generating unit 54 generates an orthoimage, which is a composite image of the captured images, based on the captured images constituting a series of image groups supplied from the photographing position calculating unit 52 and information indicating the photographing positions and photographing directions of these captured images. In this case, the orthoimage generating unit 54 performs 3D modeling based on a stereo measurement method using triangulation, for example, and performs orthogonal transformation on the generated 3D model to generate an orthoimage. The above-mentioned 3D model may be point cloud data representing the shape of a feature, or may be a combination of the center position and normal vector of a feature when the feature is limited to a planar object such as a sign. In this way, the orthoimage generating unit 54 generates an orthoimage (also called a "first orthoimage") based on the first image group, and generates an orthoimage (also called a "second orthoimage") based on the second image group. The orthoimage generating unit 54 stores the first orthoimage and the second orthoimage in the orthoimage storage unit 47.

[0047] The projection unit 55 calculates the position of the target feature on the orthoimage (also referred to as the "projection position") by performing a process of projecting the feature model generated by the spatial position calculation unit 53 onto the orthoimage generated by the orthoimage generation unit 54. In this case, the projection unit 55 calculates the projection position for each feature by similarly applying to the feature model the orthotransformation performed on the 3D model when the orthoimage generation unit 54 generates the orthoimage. Then, the projection unit 55 calculates the projection position of the feature on the first orthoimage (referred to as the "first projection position") from the first orthoimage and the feature model based on the first data set. In addition, the projection unit 55 calculates the projection position of the feature on the second orthoimage (referred to as the "second projection position") from the second orthoimage and the feature model based on the second data set. The projection unit 55 stores information indicating the calculated projection position and identification information of the corresponding orthoimage in the feature information storage unit 48. The feature information storage unit 48 may further store the two-dimensional coordinates of the feature calculated by the feature extraction unit 51 and / or the feature model calculated by the spatial position calculation unit 53.

[0048] The positioning unit 56 performs positioning between the first ortho-image and the second ortho-image. In this case, for example, the positioning unit 56 calculates the similarity between the first ortho-image and the second ortho-image while relatively translating one of them, and calculates the amount of translation at which the similarity (match degree) is highest as the amount of deviation. This amount of deviation corresponds to the amount of deviation between the coordinate system that is the reference in the first ortho-image and the coordinate system that is the reference in the second ortho-image. The positioning unit 56 supplies the calculated amount of deviation to the change point detection unit 57. The positioning unit 56 may extract the first ortho-image and the second ortho-image from the ortho-image storage unit 47, or may acquire them from the ortho-image generation unit 54.

[0049] Preferably, the alignment unit 56 aligns the first and second ortho-images based on the roads included in the first and second ortho-images. In this case, the alignment unit 56 removes areas other than the roads from the first and second ortho-images by masking, and aligns the first and second ortho-images after the masking. In this case, the alignment unit 56 recognizes areas other than the roads in the first and second ortho-images by extracting the road areas of each of the first and second ortho-images using, for example, any image recognition technology. This allows the alignment unit 56 to align the roads using only the road parts that have a high reliability for changes.

[0050] The change point detection unit 57 detects a change point based on the first and second projection positions for each feature calculated by the projection unit 55 and the deviation calculated by the alignment unit 56. In this case, the change point detection unit 57 corrects the first and second projection positions to indicate positions in a common coordinate system based on the deviation calculated by the alignment unit 56 for the first or second projection position. Then, the change point detection unit 57 compares the corrected first and second projection positions for each feature point to determine the presence or absence of a change point. Then, the change point detection unit 57 stores change point information related to the detected change point in the change point information storage unit 49. In this case, for example, the change point detection unit 57 generates change point information including the projection position after the change of the target feature and the feature model used to calculate the projection position. In addition, when either the first projection position or the second projection position for a certain feature does not exist, the change point detection unit 57 determines that the feature has disappeared or been newly established, and stores the determination result in the change point information storage unit 49.

[0051] (4) Examples A specific example of the process of the change-point detection device 4 described with reference to FIG. 3 will be described with reference to FIG. 4 and FIG.

[0052] Fig. 4 shows an overhead view of a road section 50. Feature A and feature B are present near the road section 50. The measurement vehicle performs measurements while traveling along the solid trajectory line "Lt1" on a certain date and time, thereby generating a first data set shown in Fig. 3. The measurement vehicle then travels again along the road section 50 along the dashed trajectory line Lt2, thereby generating a second data set shown in Fig. 3.

[0053] Fig. 5(A) shows a first ortho-image generated based on the first data set. Fig. 5(B) shows a second ortho-image generated based on the second data set. Fig. 5(C) shows an image in which the first ortho-image and the second ortho-image are superimposed after alignment of the two. Note that the first ortho-image clearly indicates the first projection positions of feature A and feature B, and the second ortho-image clearly indicates the second projection positions of feature A and feature B.

[0054] The positioning unit 56 performs positioning to determine the most matching relative position while performing translation, enlargement / reduction, and rotation in the roll direction, pitch direction, and yaw direction on at least one of the first ortho-image shown in FIG. 5(A) and the second ortho-image shown in FIG. 5(B). In this case, as shown in FIG. 5(C), positioning is performed so that the road section 50 matches. Then, the change point detection unit 57 compares the first projection position and the second projection position corrected to indicate the position in the common coordinate system by the positioning with respect to each of the feature A and the feature B. Then, in this case, the change point detection unit 57 determines that a change has occurred in the feature A because there is a difference between the first projection position and the second projection position of the feature A, and generates change point information regarding the feature A.

[0055] Here, a supplementary explanation will be given regarding the comparison between the first projection position and the second projection position. For example, when the center of gravity of the area corresponding to the first projection position and the center of gravity of the area corresponding to the second projection position are greater than or equal to a predetermined distance, the change point detection unit 57 determines that at least one of the position or shape of the target feature has changed, and generates change point information regarding the feature. In another example, when the degree of overlap between the area corresponding to the first projection position and the area corresponding to the second projection position is less than or equal to a predetermined ratio, the change point detection unit 57 determines that at least one of the position or shape of the target feature has changed, and generates change point information regarding the feature.

[0056] (5) Processing flow Fig. 6 is an example of a flowchart showing the procedure of the change point detection process executed by the change point detection device 4 in this embodiment. In Fig. 6, the change point detection device 4 calculates an ortho-image and a projection position by sequentially processing the data set of the captured image and the sensor information for each road section, as an example. Then, when the ortho-image and the projection position based on another data set corresponding to the same road section have been calculated, the change point detection device 4 determines whether or not there is a change point in the road section. The change point detection device 4 repeatedly executes the process of the flowchart shown in Fig. 6.

[0057] First, the change point detection device 4 acquires and stores the measurement data Im (step S10). In this case, the change point detection device 4 acquires the measurement data Im generated by the vehicle-mounted device 1 and stores it in the measurement data storage unit 46. After that, the change point detection device 4 extracts a data set corresponding to a combination of an image group of captured images generated in the road section where the change point is detected and sensor information corresponding to the captured images from the measurement data storage unit 46 (step S11).

[0058] Next, the change point detection device 4 determines whether or not a feature exists in the captured image included in the data set extracted in step S11 (step S12). If the change point detection device 4 determines that a feature exists (step S12; Yes), the process proceeds to step S13. On the other hand, if the change point detection device 4 determines that a feature does not exist (step S12; No), the change point detection device 4 determines whether or not an orthoimage targeting the same section is stored in the orthoimage storage unit 47 (step S20). If an orthoimage targeting the same section is stored in the orthoimage storage unit 47 (step S20; Yes), the change point detection device 4 determines that a feature that existed in the target section may have been removed (step S21). In this case, the change point detection device 4 performs occlusion determination as to whether or not the feature is hidden by another object using a method such as Semantic Segmentation. Then, when it is determined that no occlusion has occurred, the change-point detection device 4 regards the target feature as a change-point candidate related to removal, and stores information about the change-point candidate or transmits it to another device.

[0059] Next, the change point detection device 4 generates a feature model, which is a 3D model of the feature in each captured image included in the data set extracted in step S11 (step S13). In this case, the change point detection device 4 generates the feature model based on the feature 2D coordinates corresponding to the pixel area of ​​the feature extracted from the multiple captured images and the shooting positions and shooting directions of these captured images calculated from the sensor information.

[0060] The change point detection device 4 also generates an ortho-image (step S14). In this case, the change point detection device 4 generates a 3D model of the surroundings of the target road section based on each captured image of the target dataset and the capturing position and capturing direction of each captured image identified from the sensor information. Then, the change point detection device 4 generates an ortho-image by performing orthogonal transformation on the generated 3D model. Furthermore, the change point detection device 4 calculates the projection position of the feature on the ortho-image generated in step S14 (step S15). In this case, the change point detection device 4 calculates the above-mentioned projection position by performing a process of projecting the feature model generated in step S13 onto the ortho-image generated in step S14.

[0061] Next, the change point detection device 4 determines whether or not an orthoimage of the same section as the orthoimage generated in step S14 is stored in the orthoimage storage unit 47 (step S16). If an orthoimage of the same section as the orthoimage generated in step S14 is stored in the orthoimage storage unit 47 (step S16; Yes), the change point detection device 4 aligns these orthoimages (step S17). As a result, the change point detection device 4 calculates a shift amount for correcting the projection positions of features corresponding to each of the compared orthoimages to positions in a common coordinate system.

[0062] Next, the change point detection device 4 extracts the projection positions of the features corresponding to the orthoimage compared with the orthoimage generated in step S14 from the feature information storage unit 48. Then, the change point detection device 4 corrects at least one of the projection positions of the extracted features and the projection positions of the features calculated in step S15 based on the alignment in step S17, and detects a change point based on these corrected projection positions (step S18). In this case, the change point detection device 4 determines whether or not there is a change by comparing the projection positions of the features extracted from the feature information storage unit 48 with the projection positions of the features calculated in step S15 in a common coordinate system. Then, when the change point detection device 4 determines that there is a change, it regards the features etc. in which the change has occurred as change point candidates, and stores information on the change point candidates or transmits them to another device. Furthermore, the change point detection device 4 stores the orthoimage generated in step S14 and the projection positions of the features calculated in step S15 in the orthoimage storage unit 47 and the feature information storage unit 48, respectively.

[0063] On the other hand, when the change point detection device 4 determines that the orthoimage of the same section as the orthoimage generated in step S14 is not stored in the orthoimage storage unit 47 (step S16; No), the change point detection device 4 stores the orthoimage generated in step S14 and the projection position of the feature calculated in step S15 in the orthoimage storage unit 47 and the feature information storage unit 48, respectively (step S19). In this case, the change point detection device 4 determines that the feature may have been newly installed, regards the target feature as a change point candidate related to the new installation, and stores information on the change point candidate or transmits it to another device. Note that the data stored in the orthoimage storage unit 47 and the feature information storage unit 48 in step S19 are used in steps S17 and S18 of this flowchart, which are executed using a dataset targeting the same road section.

[0064] In this way, the change-point detection device 4 can suitably detect change-points by comparing features on an orthoimage synthesized with images captured by a measurement vehicle. This allows the change-point detection device 4 to accurately detect change-points even in areas with few features where feature-based monocular SLAM (Simultaneous Localization and Mapping) that operates in real time does not function accurately.

[0065] (6) Variations The processes of the feature extraction unit 51, the photographing position calculation unit 52, the spatial position calculation unit 53, the orthoimage generation unit 54, and the projection unit 55 described in FIG. 3 may be executed by a device other than the change point detection device 4. For example, the vehicle-mounted device 1 may have functions corresponding to the feature extraction unit 51, the photographing position calculation unit 52, the spatial position calculation unit 53, the orthoimage generation unit 54, and the projection unit 55. In this case, for example, the vehicle-mounted device 1 calculates the orthoimage and the projection position of the feature based on the data set of the photographed image and the sensor information for each road section, and transmits the calculated orthoimage and feature information to the change point detection device 4. The change point detection device 4 stores the orthoimage received from the vehicle-mounted device 1 in the orthoimage storage unit 47, and stores the feature information received from the vehicle-mounted device 1 in the feature information storage unit 48. Then, at a predetermined timing, the change point detection device 4 executes alignment by the alignment unit 56 and change point detection by the change point detection unit 57 using two sets of orthoimages and feature information for the same road section.

[0066] Fig. 7 is an example of a flowchart showing the procedure of the change-point detection process executed by the change-point detection device 4 in this modified example. The change-point detection device 4 executes the process of the flowchart shown in Fig. 7 when there are two sets of ortho-images and feature information for the same road section.

[0067] First, the change point detection device 4 acquires a first ortho-image and a first projection position, and a second ortho-image and a second projection position, which are for the same road section (step S31). In this case, for example, the change point detection device 4 may extract these pieces of information from the ortho-image storage unit 47 and the feature information storage unit 48, or may receive them from the vehicle-mounted device 1.

[0068] Next, the alignment unit 56 of the change point detection device 4 aligns the first orthoimage and the second orthoimage (step S32). Then, the change point detection unit 57 of the change point detection device 4 detects a change point based on the first projection position and the second projection position, at least one of which has been corrected by the alignment in step S32 (step S33).

[0069] In this manner, even in this modified example, the change point detection device 4 can preferably detect change points by comparing the projection positions of features on an orthoimage synthesized from images taken by a measurement vehicle.

[0070] As described above, the change point detection device 4 according to the present embodiment includes a first acquisition means, a second acquisition means, a positioning means, and a change point detection means. The first acquisition means acquires a first ortho-image generated based on a first image group, and a first projection position when a feature included in at least one image of the first image group is projected onto the first ortho-image. The second acquisition means acquires a second ortho-image generated based on a second image group captured during a period different from that of the first image group, and a second projection position when a feature included in at least one image of the second image group is projected onto the second ortho-image. The positioning means performs positioning of the first ortho-image and the second ortho-image. The change point detection means detects a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the positioning. This allows the change point detection device 4 to preferably detect a change point.

[0071] In the above-described embodiment, the program can be stored using various types of non-transitory computer readable media and supplied to a controller or the like that is a computer. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).

[0072] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0073] 1 On-vehicle device 2 Camera 3 Sensor unit 4. Change point detection device 16 Photographed image storage section 17 Sensor information storage unit 46 Measurement data storage unit 47 Orthoimage storage unit 48 Feature information storage unit 49 Change point information storage unit

Claims

1. a position alignment unit that aligns a first composite image including a first projection position onto which the feature is projected with a second composite image including a second projection position onto which the feature is projected; a change point detection means for detecting a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the alignment; A change point detection device having the following.

2. the first composite image is generated based on a first group of images captured from a vehicle traveling along a predetermined road section; the second composite image is generated based on a second group of images captured from a vehicle traveling along the road section during a period different from that of the first group of images; 2. The change point detection device according to claim 1, wherein said position adjustment means performs said position adjustment using a road in said road section as a reference.

3. 3. The change point detection device according to claim 1, further comprising a composite image generation means for generating the first composite image based on a first image group, and generating the second composite image based on a second image group captured during a period different from that of the first image group.

4. The composite image generating means includes: generating the first composite image based on a first image group and the shooting positions and shooting directions of each image constituting the first image group; The change point detection device according to claim 3, wherein the second composite image is generated based on a second image group taken during a period different from that of the first image group and the shooting position and shooting direction of each image constituting the second image group.

5. the first image group and the second image group are composed of images taken from a vehicle; 5. The change point detection device according to claim 4, further comprising a photographing position calculation means for calculating the photographing position and the photographing direction based on data outputted by a sensor provided on the vehicle during a period in which the image is photographed.

6. The change point detection device according to any one of claims 3 to 5, wherein the composite image generating means generates orthoimages as the first composite image and the second composite image.

7. the first composite image is generated based on a first set of images; the second composite image is generated based on a second group of images captured during a period different from that of the first group of images; The change point detection device according to any one of claims 1 to 6, further comprising a projection means for calculating the first projection position by projecting onto the first composite image the spatial position of the feature identified based on the images of the first image group including the feature, and for calculating the second projection position by projecting onto the second composite image the spatial position of the feature identified based on the images of the second image group including the feature.

8. By computer, performing alignment between a first composite image including a first projection position onto which the feature is projected and a second composite image including a second projection position onto which the feature is projected; detecting a change point related to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the alignment; Control methods.

9. a position alignment unit that aligns a first composite image including a first projection position onto which the feature is projected with a second composite image including a second projection position onto which the feature is projected; a change point detection means for detecting a change point relating to the feature based on the first projection position and the second projection position, at least one of which has been corrected by the position adjustment; A program that makes a computer function as a

10. A storage medium storing the program according to claim 9.

Images