Program, information processing device, and information processing method

Abstract

To more easily compare a plurality of images obtained by imaging objects such as a road facility.SOLUTION: A program allows a computer to function as an acquisition unit, a detection unit, an estimation unit, and a generation unit. The acquisition unit acquires a first image and a second image. The detection unit detects an object imaged in the first image by using at least the first image. The estimation unit estimates the relative position of the object using the imaging position of an object image as a reference by using the object image which is at least one of the first image and the second image. The generation unit generates specific information in which object identification information for identifying the object, facility information based on the relative position, and image identification information for identifying the object image are associated with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] A system has been proposed that uses images captured by an onboard camera to update and manage a database of road facilities, with the aim of detecting abnormalities in the facilities. For example, such a system has a database containing multiple reference images for each piece of road facility, and selects and outputs from the database the reference image whose image conditions are closest to the most recent image of the road facility. By using the image capture position as the image capture condition, it is possible to compare the reference image captured at the same position with the most recent image, making it easy to determine changes in the road facility over time. Note that images captured at the same (or nearby) positions are used because if the image capture positions are not close, the appearance of the facility will change, making it difficult to determine whether the facility itself has changed over time or whether it is simply a difference in appearance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5551236 [Non-patent literature]

[0004] [Non-Patent Document 1] Arandjelovic, Relja, et al. "NetVLAD: CNN architecture for weakly supervised place recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a program, an information processing device, and an information processing method that can more easily compare multiple images of objects such as road facilities. [Means for solving the problem]

[0006] A program according to an embodiment causes a computer to function as an acquisition unit, a detection unit, an estimation unit, and a generation unit. The acquisition unit acquires a first image and a second image. The detection unit uses at least the first image to detect an object captured in the first image. The estimation unit uses a target image, which is at least one of the first image and the second image, to estimate a relative position of the object based on the capture position of the target image. The generation unit generates identification information that associates object identification information that identifies the object, equipment information based on the relative position, and image identification information that identifies the target image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an information processing system according to an embodiment. [Figure 2] 10 is a flowchart of a generation process according to an embodiment. [Figure 3] 10 is a flowchart of a display process according to an embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a display screen displayed on a display unit. [Figure 5] FIG. 4 is a diagram showing an example of a display screen displayed on a display unit. [Figure 6] FIG. 1 is a hardware configuration diagram of an information processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an information processing apparatus according to the present invention will be described in detail below with reference to the accompanying drawings.

[0009] In the following, an example will be mainly described in which the object to be detected from the image is a facility (road facility) on a road on which a moving body such as a vehicle travels. Road facility includes, for example, signs, billboards, and traffic lights. The object to be detected from the image is not limited to road facility, and may be any other object.

[0010] The images are captured by, for example, an on-board camera (an example of an imaging device) mounted on a moving body (such as a vehicle) traveling on a road, but may be captured by any other imaging device. For example, the imaging device may be an imaging device mounted on a railway vehicle traveling on a track, or an imaging device mounted on a moving body such as an AGV (Automated Guided Vehicle) traveling indoors. In these cases, the facility may include structures around the travel route of the railway vehicle or moving body.

[0011] As a mobile object equipped with an imaging device such as an on-board camera moves, image data (video) including multiple images taken at consecutive times is obtained by the imaging device. When detecting an abnormality in road facilities, for example, images containing the same road facilities are extracted from multiple videos taken at different times and displayed for comparison.

[0012] As described above, a system that outputs multiple images captured at the same location so that they can be compared has been proposed as a technology for detecting abnormalities in road facilities (Comparative Example 1). In this technology, for example, the image capture location is acquired using a GPS (Global Positioning System).

[0013] One method of selecting a reference image is to use the similarity between images (Comparative Example 2). Images that are highly similar to the captured image are expected to be captured at a nearby location. Another method of selecting a reference image is to use the size of the subject in the image (subject size) (Comparative Example 3). Images with similar subject sizes are expected to have similar distances to the subject and similar capture locations. By outputting the latest image so that it can be compared with a reference image captured at a nearby location, it is possible to easily determine changes in road facilities over time.

[0014] In Comparative Example 1, a reference image is selected based on the imaging position acquired by GPS. Therefore, when an inexpensive, low-accuracy GPS is used, it may be impossible to select a reference image with the same imaging position due to GPS positioning errors. Furthermore, in environments where GPS information cannot be received, such as inside a tunnel or indoors, the imaging position cannot be acquired, and the technology of Comparative Example 1 cannot be used.

[0015] Furthermore, in Comparative Example 2, if the scenery around the road changes due to an increase or decrease in the number of buildings, the images may no longer be similar, and therefore it is not always possible to select a reference image captured at the same position.

[0016] Furthermore, in Comparative Example 3, if the subject is obstructed by a tree or the like, the subject size may change, and therefore it may not be possible to select a reference image with the same imaging position.

[0017] An information processing device according to an embodiment estimates the relative position of an object (facility) based on the imaging position for each image (hereinafter referred to as "facility image") that includes equipment in video captured by a moving imaging device. The information processing device generates and outputs information (hereinafter referred to as "specific information") that associates information based on the estimated relative position (facility information) with identification information for the equipment image. Images in which the relative position of the equipment is the same are expected to be captured at the same position. This makes it possible to extract and compare equipment images captured at the same position even for videos captured at different dates and times.

[0018] In this embodiment, the relative position of the facility can be estimated from the image. This avoids the problems of positioning error and environmental dependency that can occur with technology using GPS (Comparative Example 1). Furthermore, as long as there is an image from which the relative position of the facility can be estimated, it is possible to extract an image captured at the same position regardless of changes in the scenery around the road, obstructions, etc.

[0019] In addition, in this embodiment, multiple images extracted from multiple videos with the same relative position are displayed in an easy-to-see format, making it easier to compare multiple images of objects such as road facilities.

[0020] 1 is a block diagram showing an example of the configuration of an information processing system 10 according to an embodiment. As shown in FIG. 1, the information processing system 10 has information processing devices 100 and 200 and an imaging device 300 connected via a network 400.

[0021] The network 400 may be of any type, and may be configured, for example, by the Internet, etc. The network 400 may be a wired network, a wireless network, or a network in which wired and wireless networks coexist.

[0022] The imaging device 300 is an imaging device such as an in-vehicle camera mounted on a moving body such as a vehicle. The imaging device 300 captures images while moving in accordance with the movement of the moving body. The imaging device 300 is realized by, for example, a drive recorder, a video camera, a stereo camera, an infrared sensor, or the like. Although one imaging device 300 is shown in FIG. 1, multiple imaging devices 300 may be provided as will be described in a modified example.

[0023] The information processing device 100 corresponds to a device that generates and outputs identification information for identifying from a video an image to be displayed for the purpose of detecting an abnormality, etc. The information processing device 200 corresponds to a device that extracts from a video an image identified by the identification information and displays it.

[0024] First, a description will be given of the functions of the information processing device 100. The information processing device 100 includes an acquisition unit 101, a detection unit 102, an estimation unit 103, a generation unit 104, an output control unit 105, and a storage unit 121.

[0025] The acquisition unit 101 acquires various information used in the information processing device 100. For example, the acquisition unit 101 acquires video captured by the imaging device 300. The video includes multiple images (time-series images) captured at different times. Each image included in the video is identified by identification information such as a time and a frame number (hereinafter referred to as image identification information).

[0026] The acquisition unit 101 may acquire video associated with additional information such as GPS information and vehicle speed information. The GPS information is, for example, information acquired by a GPS, indicating the location where each image included in the video was captured. The vehicle speed information is, for example, the speed of a moving object equipped with the imaging device 300 when each image included in the video was captured.

[0027] The video corresponds to information including image IA (first image) and image IB (second image). Image IA and image IB are, for example, images included in the video captured at time t1 and time t2, respectively. Image IB can be interpreted as corresponding to an image captured at a different imaging position from image IA by imaging device 300 that captures image IA.

[0028] The acquisition unit 101 may acquire information by any method, for example, a method of receiving information from an external device via the network 400, or a method of reading information from a storage medium.

[0029] The detection unit 102 detects an object (facility) captured in the video using the video acquired by the acquisition unit 101. For example, the detection unit 102 uses at least the image IA to detect an object captured in the image IA.

[0030] Any method may be used for detecting an object by the detection unit 102, but for example, a method (image recognition processing) using a machine learning model trained in advance using equipment images for training can be applied. The machine learning model is trained to input, for example, an image at a certain time, and output a label indicating the type of equipment, information indicating the area in the image that includes the equipment (such as a detected rectangle), and a score indicating the reliability of detection.

[0031] In addition to the output of the machine learning model (label, detected rectangle, score), the detection unit 102 outputs, as detection results, data that associates image identification information of the image (facility image) in which the equipment is detected and the imaging location (e.g., GPS information) obtained by GPS or the like.

[0032] The detection unit 102 may detect two or more pieces of equipment from multiple images captured at different times in the video, or from a single image in the video. The detection unit 102 assigns unique identification information (hereinafter, object identification information) to the one or more pieces of equipment detected. Hereinafter, the object identification information may be referred to as an equipment ID.

[0033] The detection unit 102 may perform a tracking process on the detected equipment and assign the same equipment ID to the same equipment detected in multiple images. The tracking process can be realized, for example, by matching detection rectangles of the equipment detected in two or more chronologically consecutive images.

[0034] The estimation unit 103 estimates the relative position of the object (facility) with respect to the imaging position. The relative position may be expressed in any format, for example, by a three-dimensional vector. The three-dimensional vector is a vector that represents the position of the facility with respect to the imaging position (base point).

[0035] For example, the estimation unit 103 uses a target image, which is at least one of image IA and image IB, to estimate the relative position of an object based on the capture position of the target image. In this embodiment, the estimation unit 103 estimates the relative position from time-series images using Structure from Motion (SfM). In this case, the target images are both image IA and image IB. In SfM, the relative position is estimated using one or more pairs of image IA and image IB (hereinafter, referred to as image pairs). That is, the estimation unit 103 estimates the relative position using one or more image pairs.

[0036] The method for estimating relative position using SfM will now be described in further detail. In SfM, camera motion and the three-dimensional relative position from the imaging position of imaging device 300 to the subject are estimated from the correspondence between points between multiple images (image IA, image IB) containing the same subject captured by moving imaging device 300. Therefore, estimation unit 103 extracts multiple chronologically adjacent images in which the same equipment is detected from the video, and performs SfM using the extracted multiple images to estimate the relative position of the equipment.

[0037] In SfM, the relative position from the imaging position is calculated for at least some of the pixels included in the image. The estimation unit 103 estimates one relative position for each facility using the calculated relative position for each pixel. Any method may be used to estimate the relative position for each facility, but for example, the following method can be applied. A representative pixel (representative pixel) is selected from among the multiple pixels corresponding to the equipment, and the relative position of the representative pixel is estimated as the relative position of the equipment. The representative pixel is, for example, the pixel closest to the center or center of gravity of the area (e.g., detection rectangle) that includes the equipment among the multiple pixels corresponding to the equipment. The statistical values ​​of the relative positions of multiple pixels corresponding to the equipment are estimated as the relative position of the equipment. The statistical values ​​are, for example, the average value and the median value.

[0038] Note that the relative position obtained by SfM for an image captured by one imaging device 300 (corresponding to a monocular camera) has an indefinite distance scale. Therefore, the estimation unit 103 corrects the relative position using known information that can calculate the distance scale. The known information is, for example, at least some of the following information: The height of the imaging device 300 relative to the traveling surface (road surface, etc.) of the moving body on which the imaging device 300 is mounted. Multiple imaging locations (e.g., multiple imaging locations at different times) The speed at which the imaging device 300 (a moving object on which the imaging device 300 is mounted) moves - The size of the object whose actual dimensions are known

[0039] The generation unit 104 generates identification information for identifying an image to be extracted from the video. The identification information is, for example, information that associates an equipment ID (object identification information) of the equipment (object), equipment information based on the estimated relative position, and image identification information of the equipment image.

[0040] The facility information based on the relative position may be the relative position itself (three-dimensional vector), or may be information in another format that can be calculated from the relative position. For example, the facility information may be the following information: Distance: Indicates the distance from the imaging position to the equipment position. The distance is calculated, for example, by the magnitude (absolute value) of a three-dimensional vector that indicates the relative position. Angle: Represents the angle of the direction from the imaging position toward the equipment position relative to the reference direction. The reference direction is, for example, the direction in which the subject at the center of the image is captured from the imaging position. The angle may be a rotation angle relative to a predetermined axis. For example, the angle may be a rotation angle (yaw angle) relative to the up-down axis.

[0041] The facility information can be used to identify facility images from multiple videos that capture the same facility and have similar imaging position values. For example, the imaging position values ​​being similar means that the imaging positions are within a predetermined range.

[0042] For example, when the relative position itself (three-dimensional vector) is used as facility information, facility images with similar relative position values ​​can be identified as facility images with similar imaging position values.

[0043] Distance is effective when the imaging device 300 is installed so as to capture images in front of or behind the moving object. Among multiple images of the same equipment captured by such an imaging device 300, the variance in distance is large, but the variance in angle is small. In other words, multiple equipment images with similar distance values ​​can be interpreted as images with similar relative position (imaging position) values.

[0044] The angle is effective when the imaging device 300 is installed so as to capture images of the side of a moving object. Among multiple images of the same equipment captured by such an imaging device 300, the variance in angle is large, but the variance in distance is small. In other words, multiple equipment images with similar angle values ​​can be interpreted as images with similar relative position (imaging position) values.

[0045] The facility information can also be used to specify the imaging position of a specific image. For example, distance and angle are expressed as scalar values. Therefore, a slider for specifying a scalar value or an input field for inputting a scalar value can be used as a user interface for specifying the imaging position.

[0046] The output control unit 105 controls the output of various information used in the information processing device 100. For example, the output control unit 105 outputs the generated specific information. Any method for outputting the information may be used, but for example, a method of transmitting the information to an external device (such as the information processing device 200) via the network 400 may be applied.

[0047] At least a part of each of the above units (acquisition unit 101, detection unit 102, estimation unit 103, generation unit 104, and output control unit 105) may be realized by one or more processing units. Each of the above units is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0048] The storage unit 121 stores various types of information used in the information processing device 100. For example, the storage unit 121 stores information (such as video) acquired by the acquisition unit 101, information on the machine learning model used by the detection unit 102, and processing results of each unit.

[0049] The storage unit 121 can be configured from any commonly used storage medium such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disk Drive), and an optical disk.

[0050] The information processing device 100 may be physically configured as one device or may be physically configured as multiple devices. For example, the information processing device 100 may be built on a cloud environment.

[0051] 1, the imaging device 300, the information processing device 100, and the information processing device 200 are shown separately, but two or all of the devices may be built in the same device. For example, the information processing device 100 and the information processing device 200 may be built in a single server device or in a cloud environment.

[0052] Next, a description will be given of the generation process performed by the information processing device 100. Fig. 2 is a flowchart showing an example of the generation process according to the embodiment.

[0053] The acquisition unit 101 acquires video captured by the imaging device 300 (step S101). The detection unit 102 detects equipment from each image included in the video (step S102). The detection unit 102 then performs a tracking process on the equipment detected from each image (step S103). In the tracking process, equipment IDs of one or more pieces of equipment can be detected from the video.

[0054] The estimation unit 103 determines whether or not a facility has been detected (step S104). If no facility has been detected in step S104 (step S104: No), the generation process ends.

[0055] If the equipment is detected (step S104: Yes), the estimation unit 103 estimates the relative position of the equipment with respect to the image capturing position of the image for each image in which the equipment is detected (step S105).

[0056] The generation unit 104 generates identification information for each facility ID detected in step S103 (step S106). For example, the generation unit 104 generates identification information that associates the facility ID of the detected facility, facility information based on the relative position estimated in step S105, and image identification information of the facility image (time, frame number, etc.).

[0057] The output control unit 105 outputs the generated specific information for each facility ID (step S107), and ends the generation process.

[0058] In images captured at different positions, the appearance of the equipment differs, making it difficult to grasp time-series changes in the equipment itself, such as deformation and deterioration. On the other hand, the information processing device 100 of this embodiment generates and outputs specific information that associates the relative position of the equipment with the detection result of the equipment.

[0059] As will be described later, the identification information can be used by the information processing device 200 to extract and display images. For example, by using the identification information, it becomes possible to extract equipment images with similar image capture positions (relative positions) from multiple videos captured on different dates and times. For example, by specifying an equipment image corresponding to a specific relative position, a user can compare multiple equipment images with the same image capture position that are extracted from multiple videos captured on different dates and times. Because the multiple equipment images are expected to have the same appearance of the equipment, the user can easily compare equipment captured at different times.

[0060] (Variation 1) In the above embodiment, an example has been described in which the relative position of equipment is estimated by SfM. The method of estimating the relative position is not limited to the method using SfM. In Modification 1, an example will be described in which disparity is estimated by stereo matching using stereo images captured by a plurality of imaging devices 300, and the relative position is estimated using the estimated disparity.

[0061] In this modification, the information processing system 10 includes a plurality of imaging devices 300. The plurality of imaging devices 300 correspond to a stereo camera, which is two cameras arranged horizontally on the left and right. For example, the acquisition unit 101 acquires time-series stereo images captured by each of the plurality of imaging devices 300.

[0062] In this modification, an image included in a video captured by one of the multiple imaging devices 300 (first imaging device) corresponds to image IA, and an image included in a video captured by the other imaging device 300 (second imaging device) corresponds to image IB. In other words, image IB is an image captured by an imaging device 300 different from the imaging device 300 that captures image IA.

[0063] The detection unit 102 may detect an object using only the image (image IA) captured by one of the imaging devices 300, or may detect an object using both the images (image IA, image IB) captured by each of the multiple imaging devices 300.

[0064] The estimation unit 103 estimates the parallax of the equipment (object) by stereo matching using stereo images, and estimates the relative position of the equipment using the parallax. In this modification, the target images are both the image IA and the image IB. That is, the estimation unit 103 estimates the relative position using the parallax between the image IA and the image IB obtained from the stereo camera.

[0065] If the stereo images are calibrated in advance, the relative positions of the equipment can be estimated using a distance scale that reflects reality. Therefore, distance scale correction is not necessary, and processing time can be shortened compared to the above embodiment. Furthermore, because this modification uses stereo images, that is, images IA and IB captured at the same time, processing time can be shortened compared to the above embodiment, which uses images IA and IB captured at different times.

[0066] (Variation 2) In Modification 2, an example will be described in which the relative position of equipment is estimated using a depth image. The depth image is an image that includes, as pixel values ​​for each pixel, a depth that indicates the distance from the imaging position to a subject corresponding to each pixel.

[0067] In this modification, the information processing system 10 includes an imaging device 300 capable of capturing depth images in addition to the imaging device 300 similar to that of the above embodiment. The imaging device 300 capable of capturing depth images can be realized by, for example, an infrared sensor. The following describes an example in which a depth image is captured by an infrared sensor.

[0068] In this modification, the acquisition unit 101 further acquires time-series depth images captured by an infrared sensor. In this modification, an image included in a video captured by an imaging device 300 other than an infrared sensor corresponds to image IA, and a depth image captured by the infrared sensor corresponds to image IB. In other words, image IB is a depth image including depth for each pixel.

[0069] The detection unit 102 may detect an object from an image (image IA) included in a video captured by an imaging device 300 that is not an infrared sensor, or may detect an object by further using a depth image (image IB) captured by an infrared sensor.

[0070] In this modification, the target image is a depth image (image IB). That is, the estimation unit 103 estimates the relative position of the equipment using the target image, which is a depth image. By using a perspective projection model or the like, the relative position of each pixel can be calculated from the pixel value (value indicating depth) of each pixel of the depth image, the coordinates of each pixel, and the focal length. Therefore, the estimation unit 103 can estimate the relative position of the equipment, similar to the above embodiment, by using the relative position of each pixel calculated in this way, instead of the relative position of each pixel in SfM.

[0071] The depth obtained by the infrared sensor is based on a distance scale that reflects reality. Therefore, as in Modification 1, correction of the distance scale is not required, and processing time can be reduced compared to the above embodiment.

[0072] (Variation 3) In the above embodiment, the identification information is, for example, information that associates a facility ID, facility information, and image identification information of a facility image. However, the data structure of the identification information is not limited to this, and the identification information may further include other information.

[0073] For example, the identification information may further include location information indicating the location where the target image was captured. The location information may be, for example, lane information indicating the lane on which the mobile object equipped with the imaging device 300 is traveling. The lane information can be obtained, for example, by analyzing the target image. In this modified example, the generation unit 104 generates identification information that is further associated with location information indicating the location where the target image was captured.

[0074] In a method of extracting corresponding images using only facility information such as relative position, distance, and angle, there is a possibility that images captured while traveling in different lanes may be extracted as corresponding images. By generating and outputting specific information including position information such as lane information, it becomes possible to extract corresponding images under the constraint that the position information matches or is similar. In other words, corresponding images can be extracted with higher accuracy.

[0075] Next, a description will be given of the function of the information processing device 200 that extracts and displays images using the specific information. The information processing device 200 uses the specific information output from the information processing device 100 to extract images of the same equipment that have similar values ​​for the relative position of the equipment from each of the multiple videos captured by the imaging device 300, and displays the extracted multiple images so that they can be compared.

[0076] As shown in FIG. 1, the information processing device 200 includes an acquisition unit 201, a determination unit 202, an extraction unit 203, a display control unit 204, and a display unit 221.

[0077] The display unit 221 is a display device that displays various information, and can be realized by, for example, a liquid crystal display. The display unit 221 does not need to be provided in the information processing device 200, but may be provided in another device (such as a terminal device) connected via the network 400, for example.

[0078] The acquisition unit 201 acquires various types of information used in the information processing device 200. For example, the acquisition unit 201 acquires multiple videos captured at different dates and times from the imaging device 300. The acquisition unit 201 also acquires specific information output from the information processing device 100 for each of the multiple videos.

[0079] When detecting an abnormality by comparing images, a configuration is conceivable in which two images, for example, an image from a video captured in the past and an image from the latest video, are displayed so that they can be compared. Below, an example will be described in which two images extracted from two videos captured at different times are displayed. The number of videos (images) to be displayed is not limited to two, and may be three or more.

[0080] In the following, an example will be described in which two images are used: an image MD1 captured at time t1 and an image MD2 captured at time t2. The images MD1 and MD2 may be images captured at different dates and times by the same imaging device 300 (imaging device 300 mounted on the same moving body), or may be images captured by multiple different imaging devices 300 (imaging devices 300 mounted on multiple different moving bodies). It is assumed that the images MD1 and MD2 include one or more pieces of equipment that are commonly captured by both images.

[0081] Hereinafter, the target images (at least one of image IA and image IB) captured at time t1 and time t2 will be referred to as target image PI_t1 (first target image) and target image PI_t2 (second target image), respectively. The specific information generated using target image PI_t1 and target image PI_t2 will be referred to as specific information SI_t1 (first specific information) and specific information SI_t2 (second specific information), respectively.

[0082] The acquisition unit 201 acquires the specific information SI_t1 and the specific information SI_t2 from the information processing device 100. The acquisition unit 201 also acquires, from the imaging device 300, an image MD1 captured at time t1 and an image MD2 captured at time t2.

[0083] The identification information SI_t1 and the identification information SI_t2 each include a facility ID that is assigned independently. Therefore, even if the facility IDs included in the identification information SI_t1 and the identification information SI_t2 have the same value, they do not necessarily indicate the same facility. Conversely, even if the facility IDs included in the identification information SI_t1 and the identification information SI_t2 have different values, they may indicate the same facility.

[0084] Therefore, the determination unit 202 determines whether multiple pieces of identification information indicate the same facility and associates the pieces of identification information indicating the same facility. For example, the determination unit 202 determines whether the facility (object) identified by the facility ID (object identification information) included in the identification information SI_t1 matches the facility identified by the facility ID included in the identification information SI_t2. The determination unit 202 then assigns new facility IDs (hereinafter, common facility IDs) to the facilities included in the identification information SI_t1 and SI_t2 so that the common facility IDs have the same value when the identification information SI_t1 and SI_t2 indicate the same facility and different values ​​when the identification information SI_t1 and SI_t2 indicate different facilities.

[0085] Whether the equipment matches, that is, whether it is the same equipment, can be determined by the following method, for example. If the image capture locations obtained by GPS or other methods are close (for example, if the difference in image capture locations is below a threshold), it is determined that the equipment is the same. If the similarity between equipment images corresponding to the same relative position is below a threshold, it is determined that they are the same equipment. -Displays multiple equipment images, and determines that equipment designated by the user as being the same by referring to the displayed equipment images is the same equipment.

[0086] The extraction unit 203 extracts images from the video MD1 and the video MD2, respectively, using one or more pieces of identification information SI_t1 and one or more pieces of identification information SI_t2 that include the equipment ID (common equipment ID) of the equipment that is determined to match.

[0087] For example, the extraction unit 203 extracts from the video MD1 a target image identified by image identification information included in the identification information SI_t1 including the specified facility information (relative position, distance, or angle). The extraction unit 203 also extracts from the video MD2 a target image identified by image identification information included in the identification information SI_t2 including the specified facility information.

[0088] As described above, specifying the facility information corresponds to specifying the imaging position (relative position). The facility information may be specified by any method, but for example, the following method can be applied. A list of facility images corresponding to different facility information is displayed, and the facility information corresponding to a facility image selected by a user or the like from the displayed facility images is used as the specified facility information. The list of facility images may be either a list of facility images obtained from video MD1 or a list of facility images obtained from video MD2. In the case of facility information expressed as a scalar value such as distance or angle, the facility information is specified using a user interface such as a slider for specifying the scalar value.

[0089] When the designated facility information is changed, the extraction unit 203 uses the changed facility information to further extract images from the video MD1 and the video MD2. For example, the extraction unit 203 extracts from the video MD1 target images identified by image identification information included in the identification information SI_t1 including the changed facility information. The extraction unit 203 also extracts from the video MD2 target images identified by image identification information included in the identification information SI_t2 including the changed facility information.

[0090] The display control unit 204 controls the display of various information on the display unit 221. For example, the display control unit 204 displays the extracted multiple images (target images) on the display unit 221 in association with each other so that they can be compared.

[0091] At least a part of each of the above units (acquisition unit 201, determination unit 202, extraction unit 203, and display control unit 204) may be realized by one or more processing units. Each of the above units is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU or GPU execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0092] Next, a description will be given of the display processing by the information processing device 200. Fig. 3 is a flowchart showing an example of the display processing in the embodiment.

[0093] The acquiring unit 201 acquires the images MD1 and MD2 from the imaging device 300 (step S201). The acquiring unit 201 receives the images MD1 and MD2, respectively, and acquires the identification information SI_t1 and the identification information SI_t2 output from the information processing device 100 (step S202). Note that steps S201 and S202 do not have to be performed in the order shown in FIG. 3, and may be performed in the reverse order or in parallel.

[0094] The determination unit 202 determines whether the plurality of pieces of specific information indicate common equipment (the same equipment), and assigns a common equipment ID to the common equipment (step S203).

[0095] The extraction unit 203 determines whether an equipment image to be displayed has been designated (step S204). As described above, an equipment image can be designated by the user selecting it from a list of multiple equipment images, for example.

[0096] If an equipment image is specified (step S204: Yes), the extraction unit 203 selects equipment information corresponding to the specified equipment image (step S205). For example, the extraction unit 203 obtains identification information corresponding to the specified equipment image, and selects the common equipment ID assigned to the equipment ID included in the obtained identification information and the equipment information (relative position, distance, or angle) included in the identification information.

[0097] If an equipment image is not specified (step S204: No), the extraction unit 203 determines whether equipment information is specified (step S206). For example, if the equipment information can be specified by a scalar value such as a distance or an angle, the extraction unit 203 determines whether a scalar value indicating the equipment information is specified.

[0098] If facility information is specified (step S206: Yes), the extraction unit 203 selects the specified facility information (step S207). If facility information is not specified (step S206: No), the extraction unit 203 selects facility information determined as a default value (step S208). The default value of the facility information is determined in advance as facility information that makes the facility easy to see, for example, according to the actual size of the facility and the angle of view of the imaging device 300.

[0099] If multiple pieces of equipment are detected from the equipment image, the extraction unit 203 may select a designated common equipment ID from among multiple common equipment IDs assigned to the multiple pieces of equipment. If no common equipment ID is designated, the extraction unit 203 selects one of the multiple common equipment IDs (for example, the common equipment ID with the smallest value).

[0100] Next, the extraction unit 203 extracts from the video MD1 an equipment image corresponding to the common equipment ID and equipment information specified in any of steps S205, S207, and S208, using the identification information SI_t1. Similarly, the extraction unit 203 extracts from the video MD2 an equipment image corresponding to the specified common equipment ID and equipment information, using the identification information SI_t2 (step S209).

[0101] The display control unit 204 displays the extracted facility images on the display unit 221 (step S210).

[0102] The display control unit 204 determines whether or not the end of the display has been designated (step S211). If the end of the display has not been designated (step S211: No), the process returns to step S204 and is repeated. If the end of the display has been designated (step S211: Yes), the display process ends.

[0103] 4 and 5 are diagrams showing examples of display screens displayed on display unit 221. As shown in Fig. 4, display screen 401 includes images 421 and 422 captured at different dates and times, and a slider 411 that can specify a distance as facility information. Fig. 4 shows an example in which 10 is specified as the distance using slider 411.

[0104] Image 421 corresponds to an image extracted from multiple images included in video MD1 captured at time t1 (January XX day) using specific information SI_t1 that includes a distance value of 10 as facility information. Image 422 corresponds to an image extracted from multiple images included in video MD2 captured at time t2 (February XX day) using specific information SI_t2 that includes a distance value of 10 as facility information.

[0105] 5 shows an example of the display screen 401 when the distance designation is changed to 20 by the slider 411. Image 521 corresponds to an image extracted from multiple images included in video MD1 captured at time t1 (January XX day) using specific information SI_t1 that includes a distance value of 20 as facility information. Image 522 corresponds to an image extracted from multiple images included in video MD2 captured at time t2 (February XX day) using specific information SI_t2 that includes a distance value of 20 as facility information.

[0106] In this way, the information processing device 200 extracts multiple images corresponding to the specified facility information (relative position, distance, or angle) from among images of the same facility included in multiple videos, and displays the extracted multiple images in a comparable manner. This makes it possible to easily grasp the time-series changes in the facility itself, excluding the influence of differences in the appearance of the facility due to differences in the imaging positions.

[0107] (Variation 4) In the above embodiment, an example has been described in which one facility image corresponding to one selected piece of facility information is extracted from each video (video MD1, video MD2), and a set of facility images extracted from each video is displayed. However, two or more sets of images may be extracted and displayed.

[0108] For example, the user specifies two or more types of facility information (relative position, distance, or angle). The extraction unit 203 extracts two or more sets of images using each of the specified two or more types of facility information. The display control unit 204 displays the two or more sets of images on the display unit 221 in association with each other so that they can be compared.

[0109] This makes it easier to understand, for example, differences in how equipment appears depending on the imaging position. For example, it is easy to understand situations where equipment is hidden by trees or signs only when viewed from a specific distance.

[0110] (Variation 5) In the above embodiment, it is possible to display equipment images of any equipment in response to a specification by a user, etc. However, there are cases where it is required to display only images of a specific type of equipment or equipment captured at a specific imaging position. In Modification 5, only equipment that has been specified to be displayed is subject to display.

[0111] For example, the extraction unit 203 extracts from the video MD1 a target image identified by image identification information included in identification information SI_t1, which includes the equipment ID of the specified type of equipment and the specified equipment information. Also, the extraction unit 203 extracts from the video MD2 a target image identified by image identification information included in identification information SI_t2, which includes the equipment ID of the specified type of equipment and the specified equipment information.

[0112] The imaging position may be specified by a range of imaging positions. In this case, the extraction unit 203 extracts from the video MD1 target images that were captured at imaging positions within the specified range and identified by image identification information included in the identification information SI_t1 that includes the specified facility information. The extraction unit 203 also extracts from the video MD2 target images that were captured at imaging positions within the specified range and identified by image identification information included in the identification information SI_t2 that includes the specified facility information.

[0113] This allows the user to display and compare images corresponding to the relative positions of only the desired type of equipment or equipment captured at the desired imaging position.

[0114] In this manner, in the embodiment, comparison of multiple images of an object such as road furniture can be more easily performed.

[0115] Next, the hardware configuration of the information processing apparatus according to the embodiment will be described with reference to Fig. 6. Fig. 6 is an explanatory diagram illustrating an example of the hardware configuration of the information processing apparatus according to the embodiment.

[0116] The information processing device of the embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication I / F 54 that connects to a network and communicates, and a bus 61 that connects each part.

[0117] The programs executed by the information processing apparatus according to the embodiment are provided in advance in the ROM 52 or the like.

[0118] The program executed by the information processing device of the embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0119] Furthermore, the program executed by the information processing apparatus of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the information processing apparatus of the embodiment may be provided or distributed via a network such as the Internet.

[0120] The programs executed by the information processing device of the embodiment can cause a computer to function as each unit of the information processing device described above. In this computer, the CPU 51 can read the programs from a computer-readable storage medium onto a main storage device and execute the programs.

[0121] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0122] 100 Information processing device 101 Acquisition Department 102 Detection unit 103 Estimation part 104 Generation part 105 Output control section 121 Storage section 200 Information processing device 201 Acquisition Department 202 Judgment section 203 Extraction part 204 Display control unit 221 Display section 300 Imaging device 400 Network

Claims

1. Computer, an acquisition unit that acquires a first image and a second image; a detection unit that detects an object captured in the first image using at least the first image; an estimation unit that estimates a relative position of the object based on an imaging position of the target image, the target image being at least one of the first image and the second image; a generating unit that generates identification information that associates object identification information that identifies the object, facility information based on the relative position, and image identification information that identifies the target image; A program to function as a

2. the second image is an image captured by an imaging device that captures the first image at an imaging position different from that of the first image, the estimation unit uses one or more image pairs, which are a set of the first image and the second image, to estimate the relative position from a correspondence relationship between points in the first image and the second image included in the image pair; The program according to claim 1.

3. The estimation unit the relative position is estimated by further using at least a part of the imaging position, the height of the imaging device, the speed at which the imaging device moves, and the size of the subject captured in the target image; The program according to claim 2.

4. the second image is an image captured by a second imaging device different from a first imaging device that captures the first image, the estimation unit estimates the relative position using a parallax between the first image and the second image. The program according to claim 1.

5. the second image is a depth image including depth for each pixel; the estimation unit estimates the relative position using the target image, which is the depth image; The program according to claim 1.

6. the generation unit generates the identification information further associated with position information indicating a position where the target image was captured. The program according to claim 1.

7. a determination unit that determines whether the object identified by the object identification information included in first identification information generated using the target image captured at time t1 matches the object identified by the object identification information included in second identification information generated using the target image captured at time t2; an extraction unit that uses one or more of the first identification information and one or more of the second identification information including the object identification information of the object determined to match to extract the target image identified by the image identification information included in the first identification information including the specified equipment information and the target image identified by the image identification information included in the second identification information including the specified equipment information; a display control unit that displays the extracted target images on a display device in association with each other; Further provided with The program according to claim 1.

8. Computer, an acquisition unit that acquires identification information that associates object identification information that identifies an object detected using at least the first image of a first image and a second image, and a relative position of the object estimated using a target image that is at least one of the first image and the second image with image identification information that identifies the target image, the relative position being based on the imaging position of the target image, the image identification information being generated using the first target image captured at time t1, and second identification information that is generated using a second target image captured at time t2; a determination unit that determines whether the object identified by the object identification information included in the first identification information and the object identified by the object identification information included in the second identification information match; an extraction unit that uses one or more of the first identification information and one or more of the second identification information including the object identification information of the object determined to match to extract the target image identified by the image identification information included in the first identification information including the specified equipment information among the equipment information based on the relative position, and the target image identified by the image identification information included in the second identification information including the specified equipment information; a display control unit that displays the extracted target images on a display device in association with each other; A program to function as a

9. When the designation of the facility information is changed, the extraction unit extracts the target image identified by the image identification information included in the first identification information including the changed facility information, and the target image identified by the image identification information included in the second identification information including the changed facility information. The program according to claim 8.

10. the extraction unit extracts the target image identified by the image identification information included in the first identification information, which includes the object identification information of a specified type of object and the specified equipment information, and the target image identified by the image identification information included in the second identification information, which includes the object identification information of a specified type of object and the specified equipment information. The program according to claim 8.

11. The extraction unit extracts the target image captured at an imaging position within a specified range and identified by the image identification information included in the first identification information including the specified facility information, and the target image captured at an imaging position within the range and identified by the image identification information included in the second identification information including the specified facility information. The program according to claim 8.

12. an acquisition unit that acquires a first image and a second image; a detection unit that detects an object captured in the first image using at least the first image; an estimation unit that estimates a relative position of the object based on an imaging position of the target image, the target image being at least one of the first image and the second image; a generating unit that generates identification information that associates object identification information that identifies the object, facility information based on the relative position, and image identification information that identifies the target image; An information processing device comprising:

13. An information processing method executed by an information processing device, an acquisition step of acquiring a first image and a second image; a detecting step of detecting an object captured in the first image using at least the first image; an estimation step of estimating a relative position of the object based on an imaging position of the target image, the target image being at least one of the first image and the second image; a generating step of generating identification information that associates object identification information that identifies the object, facility information based on the relative position, and image identification information that identifies the target image; An information processing method including:

14. an acquisition unit that acquires identification information that associates object identification information that identifies an object detected using at least the first image of a first image and a second image, and a relative position of the object estimated using a target image that is at least one of the first image and the second image with image identification information that identifies the target image, the relative position being based on the imaging position of the target image, the image identification information being generated using the first target image captured at time t1, and second identification information that is generated using a second target image captured at time t2; a determination unit that determines whether the object identified by the object identification information included in the first identification information and the object identified by the object identification information included in the second identification information match; an extraction unit that uses one or more of the first identification information and one or more of the second identification information including the object identification information of the object determined to match to extract the target image identified by the image identification information included in the first identification information including the specified equipment information among the equipment information based on the relative position, and the target image identified by the image identification information included in the second identification information including the specified equipment information; a display control unit that displays the extracted target images on a display device in association with each other; An information processing device comprising:

15. An information processing method executed by an information processing device, an acquiring step of acquiring identification information that associates object identification information that identifies an object detected using at least the first image of the first and second images, and image identification information that identifies the target image with a relative position of the object estimated using a target image that is at least one of the first and second images, with the image identification information that identifies the target image, the first identification information being generated using a first target image captured at time t1, and second identification information being generated using a second target image captured at time t2; a determining step of determining whether the object identified by the object identification information included in the first identification information matches the object identified by the object identification information included in the second identification information; an extraction step of extracting the target image identified by the image identification information included in the first identification information including the specified equipment information among the equipment information based on the relative position, and the target image identified by the image identification information included in the second identification information including the specified equipment information, using one or more of the first identification information and one or more of the second identification information including the object identification information of the object determined to match; a display control step of displaying the extracted plurality of target images on a display device in association with each other; An information processing method including:

Images