Information processing device, system, method, and program

Abstract

The present invention makes it possible to select an appropriate image, as an image to be compared, with respect to a captured image. A class detection means (21) analyzes a captured image captured using a camera (30) and classifies objects included in the captured image into a plurality of classes. A candidate extraction means (22) extracts, from an information accumulation unit (25), candidates for the image to be compared on the basis of imaging information of the captured image. An image selection means (23) selects the image to be compared from the candidates of the image to be compared on the basis of: at least one of the type of class included in the captured image, the size of a region of each class, and the percentage that the region of each class occupies in the image; and at least one of the type of class included in each of the candidates of the image to be compared, the size of a region of each class, and the percentage that the region of each class occupies in the image.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a system, a method, and a computer-readable medium. [Background technology]

[0002] As a related technique, Patent Document 1 discloses an information processing device that identifies the position of a vehicle with high accuracy. The information processing device described in Patent Document 1 receives a captured image for position identification taken by an imaging device from an in-vehicle device. The information processing device detects landmarks from the captured image for position identification and extracts landmark information from the captured image for position identification. The landmark information includes the shape, color, pattern, and texture of the landmark, as well as the coordinate range of the landmark on the image.

[0003] The information processing device stores location information, a combination of multiple landmarks, and information on each landmark for each registration image captured in advance. The information processing device compares the landmarks included in the location identification image with the landmarks included in the location registration image. The information processing device identifies a registration image whose landmarks match those of the location identification image, and identifies the location where the registration image was captured as the vehicle's location at the time the location identification image was captured. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-78700 A Summary of the Invention [Problem to be solved by the invention]

[0005] One possible system for detecting anomalies using video from an in-vehicle camera is to compare two images taken at different times at the same location and detect the differences to detect damaged signs or signs obscured by vegetation, etc. In such a system, it is considered important to accurately identify an image taken at a certain time from the same location but at a different time.

[0006] In Patent Document 1, an information processing device identifies the location where a location identification image was captured using landmarks such as signs, buildings, and billboards. By comparing a location identification image and a registration image captured at the same location, changes that occurred between the capture of the registration image and the capture of the location identification image can be detected. However, Patent Document 1 does not take into account the weather, season, and time when the image was captured. In Patent Document 1, if the weather or time of day when the location identification image was captured differs from the weather and time of day when the registration image was captured, there is a possibility that the change will not be detected correctly due to changes in the appearance of the image.

[0007] In view of the above circumstances, an object of the present disclosure is to provide an information processing device, a system, a method, and a computer-readable medium that can select an appropriate image as a comparison image for a captured image. [Means for solving the problem]

[0008] To achieve the above object, the present disclosure provides, as a first aspect, an information processing device. The information processing device includes: a class detection means for analyzing a captured image taken with a camera mounted on a moving object and classifying objects included in the captured image into a plurality of classes; a candidate extraction means for extracting, from an information storage unit that stores one or more past images taken in the past and shooting information including at least one of shooting date and time information and weather information at the time of shooting for each of the one or more past images, candidate comparison images to be compared with the captured image based on the shooting information of the captured image; and an image selection means for selecting a comparison image from the candidate comparison images based on at least one of the type of class included in the captured image, the size of an area of each class, and the proportion of the area of each class to the image, and at least one of the type of class included in each of the candidate comparison images, the size of the area of each class, and the proportion of the area of each class to the image.

[0009] The present disclosure provides, as a second aspect, an information processing system. The information processing system includes one or more cameras mounted on a mobile object and an information processing device that performs image processing on captured images captured using the one or more cameras. The information processing device includes: a class detection means that analyzes the captured images and classifies objects included in the captured images into a plurality of classes; a candidate extraction means that extracts candidate comparison images to be compared with the captured images based on the shooting information of the captured images from an information storage unit that stores one or more past images captured in the past and shooting information including at least one of shooting date and time information and weather information at the time of shooting for each of the one or more past images; and an image selection means that selects a comparison image from the candidate comparison images based on at least one of the class types included in the captured image, the size of each class area, and the proportion of the image occupied by each class area, and at least one of the class types included in each of the candidate comparison images, the size of each class area, and the proportion of the image occupied by each class area.

[0010] The present disclosure provides, as a third aspect, an information processing method, which includes: analyzing a captured image captured using a camera mounted on a moving object; classifying objects included in the captured image into a plurality of classes; extracting candidate comparison images to be compared with the captured image based on the shooting information of the captured image from an information storage unit that stores one or more previous images captured in the past and shooting information including at least one of shooting date and time information and weather information at the time of shooting for each of the one or more previous images; and selecting a comparison image from the candidate comparison images based on at least one of the type of class included in the captured image, the size of an area of each class, and the proportion of the area of each class to the image, and at least one of the type of class included in each of the candidate comparison images, the size of the area of each class, and the proportion of the area of each class to the image.

[0011] In a fourth aspect, the present disclosure provides a computer-readable medium storing a program for causing a computer to analyze captured images captured using a camera mounted on a moving object, classify objects included in the captured images into a plurality of classes, extract candidate comparison images to be compared with the captured images based on the captured image information from an information storage unit that stores one or more previous images captured in the past and image information including at least one of image capture date and time information and weather information at the time of capture for each of the one or more previous images, and select a comparison image from the candidate comparison images based on at least one of the class types included in the captured image, the size of an area of each class, and the proportion of the area of each class to the image, and at least one of the class types included in each of the candidate comparison images, the size of an area of each class, and the proportion of the area of each class to the image. [Effects of the Invention]

[0012] The information processing device, system, method, and computer-readable medium according to the present disclosure can select an appropriate image as a comparison target image for a captured image. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram illustrating an information processing system according to the present disclosure. [Figure 2] 1 is a block diagram showing an information processing system according to a first embodiment of the present disclosure. [Figure 3] FIG. 2 is a schematic diagram showing an example of an image captured by a camera. [Figure 4] FIG. 10 is a schematic diagram showing an example of selection of a comparison image. [Figure 5] 10 is a flowchart showing an operation procedure in the information processing device. [Figure 6] FIG. 10 is a block diagram showing an information processing system according to a second embodiment of the present disclosure. [Figure 7] FIG. 1 is a block diagram showing a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0014] Prior to describing embodiments of the present disclosure, an overview of the present disclosure will be described. Fig. 1 shows an information processing system according to the present disclosure. The information processing system 10 includes an information processing device 20 and one or more cameras 30. The camera 30 is mounted on a moving object. The information processing device 20 performs image processing on an image captured using the camera 30.

[0015] The information processing device 20 has a class detection means 21, a candidate extraction means 22, and an image selection means 23. The class detection means 21 analyzes an image captured by the camera 30, and classifies objects included in the captured image into a plurality of classes.

[0016] The information storage unit 25 stores one or more past images captured in the past and shooting information for each past image. The shooting information includes at least one of shooting date and time information and weather information at the time of shooting. The candidate extraction means 22 extracts, from the information storage unit 25, candidates for comparison images to be compared with the captured image, based on the shooting information of the captured image.

[0017] The image selection means 23 selects a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image occupied by the area of each class, and those contained in each of the candidate comparison images.

[0018] In the present disclosure, the candidate extraction means 22 extracts candidate comparison images from the information storage unit 25 based on the shooting information of the captured image. The image selection means 23 selects a comparison image from the candidate comparison images using the results of class classification. Images captured at the same location are considered to have the same class classification results. Therefore, by selecting a comparison image using the results of class classification, it is considered possible to select an image captured at the same location as the captured image as the comparison image. In the present disclosure, by extracting past images with shooting information similar to the shooting information of the captured image as candidate comparison images, it is possible to select an image appropriate for comparison with the captured image as the comparison image. Furthermore, by comparing the selected comparison image with the captured image, it is possible to find differences between the two images regardless of differences in appearance.

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. In addition, in the following drawings, the same or similar elements are designated by the same reference numerals, and duplicate explanations are omitted as necessary.

[0020] 2 shows an information processing system according to the first embodiment of the present disclosure. The information processing system 100 includes an information processing device 110 and a camera 210. In this embodiment, the information processing device 110 and the camera 210 are mounted on a mobile object. The mobile object is configured as a land vehicle such as an automobile, a motorcycle, a bus, a taxi, or a truck. The mobile object may be a train, a ship, an aircraft, or a mobile robot such as an AGV (Automated Guided Vehicle).

[0021] The camera 210 captures, for example, an image in the traveling direction of the moving object. The moving object may have multiple cameras 210 with different shooting directions. The information processing device 110 acquires images captured using the cameras 210 and performs image processing on the acquired images. The information processing system 100 corresponds to the information processing system 10 shown in FIG. 1. The information processing device 110 corresponds to the information processing device 20 shown in FIG. 1. The camera 210 corresponds to the camera 30 shown in FIG. 1.

[0022] The information processing device 110 includes an image acquisition unit 111, a class detection unit 112, a candidate extraction unit 113, an image selection unit 114, an image comparison unit 115, and an information storage unit 120. The information processing device 110 includes, for example, one or more memories and one or more processors. At least a part of the function of each unit in the information processing device 110 can be realized by the processor operating in accordance with a program read from the memory.

[0023] The image acquisition unit 111 acquires captured images from the camera 210. The image acquisition unit 111 acquires captured images from the camera 210 at predetermined time intervals, such as one second or ten seconds. The class detection unit 112 analyzes the captured images and classifies one or more objects included in the captured images into multiple classes. Examples of objects include immobile objects such as buildings, bridges, traffic lights, vending machines, and signs. The class detection unit 112, for example, divides the captured image into multiple segments and determines to which class each segment belongs. Instead of or in addition to segmentation, the class detection unit 112 may detect objects from the captured image and classify the regions where the objects are detected into classes corresponding to the objects. The class detection unit 112 corresponds to the class detection means 21 shown in FIG. 1.

[0024] The information storage unit 120 stores captured images taken by the camera 210. The captured images stored in the information storage unit 120 are images taken in the past and are also called past images. The information storage unit 120 also stores shooting information for captured images that are past images. The shooting information includes at least one of shooting date and time information and weather information at the time of shooting. Furthermore, the information storage unit 120 may store the results of class classification by the class detection unit 112. The results of class classification include, for example, at least one of the type of class, the size of the area of each class, and the proportion of the image that the area of each class occupies.

[0025] For example, the class detection unit 112 stores the captured image, its shooting position, and the result of class classification in the information storage unit 120. The class detection unit 112 may store, as the result of class classification, for example, at least one of the type of the classified class, the size of the area of each class, and the proportion of the area of each class in the image in the information storage unit 120. The class detection unit 112 also stores information related to the date and time and weather as shooting information in the information storage unit 120. The weather at the shooting position may be obtained from an external server not shown in FIG. 2, or may be obtained by analyzing the captured image.

[0026] It should be noted that the information storage unit 120 does not necessarily have to be included in the information processing device 110. The information storage unit 120 may be an external storage device, or may be cloud storage connected to the information processing device 110 via a network. The information storage unit 120 corresponds to the information storage unit 25 shown in FIG. 1.

[0027] The candidate extraction unit 113 extracts candidates for comparison images from the information storage unit 120 based on the shooting information of the captured image. For example, the candidate extraction unit 113 extracts one or more past images whose shooting information is similar to that of the captured image as candidates for comparison images. Here, similarity of shooting information between a captured image and a past image may mean, for example, that the images were taken in the same season, the same time of day, or the same weather. For example, the candidate extraction unit 113 may calculate the similarity between the captured image and the past image for each of the season, the time of day, and the weather, and extract past images whose similarity is equal to or greater than a predetermined value as candidates for comparison images.

[0028] The candidate extraction unit 113 may extract, as candidates for comparison images, one or more past images whose shooting information is similar to that of the captured image and whose shooting location information is similar to that of the captured image. Here, similar shooting locations between a captured image and a past image may mean that the distance between the shooting locations of the images is within a predetermined distance. The candidate extraction unit 113 acquires, for example, the location information of a moving object at the time the captured image was captured as the shooting location of the captured image. The location information is acquired, for example, using a Global Navigation Satellite System (GNSS). The candidate extraction unit 113 extracts, as candidates for comparison images, one or more past images that were captured at a location close to the shooting location of the captured image and have similar shooting information from among the past images stored in the information accumulation unit 120. In this case, the candidate extraction unit 113 may extract, for example, past images that are considered to have been captured at the same location as the captured image and under similar shooting conditions as candidates for comparison images. The candidate extraction unit 113 corresponds to the candidate extraction unit 22 shown in FIG. 1.

[0029] The image selection unit 114 selects a comparison image to be compared with the captured image from the comparison image candidates extracted by the candidate extraction unit 113. In selecting the comparison image, the image selection unit 114 compares the class classification result of the captured image with the class classification results of each of the comparison image candidates. For example, the image selection unit 114 acquires at least one of the class type, the size of the area of each class, and the area of each class as the class classification results for each of the comparison image candidates from the information storage unit 120. The image selection unit 114 selects a comparison image from the comparison image candidates based on the comparison results of the class classification results. For example, the image selection unit 114 selects, as the comparison image, a past image with a class classification result most similar to the class classification result of the captured image. When there are multiple images with the same class classification result, the image selection unit 114 may select the most recent image as the comparison image. Alternatively, the image selection unit 114 may select a predetermined number of images starting from the most recent as the comparison image.

[0030] The image selection unit 114, for example, compares the type of class contained in the captured image with the type of class contained in each of the candidate comparison images. The image selection unit 114 determines whether the same class is detected in the captured image and each of the candidate comparison images. The image selection unit 114 may exclude from the determination a class whose detection status changes due to factors other than the environment, such as the vehicle. The image selection unit 114 selects, for example, from the candidate comparison images, an image with the highest similarity or match with the detected class as the comparison image.

[0031] The image selection unit 114 may compare the size of the area of each class in the captured image with the size of the area of each class in each candidate comparison image. In this case, the image selection unit 114 may determine whether the same class is detected in the captured image and each candidate comparison image, and whether the size of the area of each class matches. For example, the image selection unit 114 may select, as the comparison image, an image in which the same class as the class detected in the captured image is detected, and the size of the area of each class matches the most.

[0032] Instead of or in addition to comparing the size of the area of each class, the image selection unit 114 may compare the proportion of the image occupied by the area of each class in the captured image with the proportion of the image occupied by the area of each class in each candidate comparison image. In this case, the image selection unit 114 may select, as the comparison image, an image in which the same class as the class detected in the captured image is detected and in which the proportion of the area of each class in the image is most similar to that in the captured image. The image selection unit 114 corresponds to the image selection means 23 shown in FIG. 1.

[0033] The image comparison unit 115 compares the captured image acquired by the image acquisition unit 111 with the comparison image selected by the image selection unit 114. The image comparison unit 115, for example, calculates the difference between the captured image and the comparison image. The image comparison unit 115 may calculate the difference between the captured image and the comparison image for each area of each class, for example. The image selected by the image selection unit 114 is an image captured at the same position as the captured image but on a different date and time than the captured image. By calculating the difference for each class, the image comparison unit 115 can detect a change in the shape of the class as a final difference. For example, the image comparison unit 115 can detect a broken traffic light as an abnormality. The image comparison unit 115 may display the captured image and the comparison image on a display screen so that they can be compared, allowing the user to perform image comparison. The image comparison unit 115 may also be called an image comparison means.

[0034] 3 shows an example of a captured image taken by camera 210. In this example, the captured image includes a road on which a mobile object travels, a building beside the road, and a vending machine as objects to be classified into classes. Class detection unit 112 classifies the objects included in the captured image into a road class, a building class, and a vending machine class. For each class, class detection unit 112 outputs the class type, the position, and the size of the object as the result of the class classification.

[0035] FIG. 4 shows an example of selecting a comparison image. Here, consider an example in which three past images are extracted as candidates for comparison images. In FIG. 4, classification result 300 is assumed to be the result of classifying the captured image shown in FIG. 3. Classification result 300 includes one class type each of building class 301, traffic light class 302, and vending machine class 303.

[0036] As shown in Figure 4 Ruku The class classification results 310-330 are the results of class classification of the candidate images to be compared. 310 The class classification result includes one building class 311 and one traffic light class 312. 320 includes one each of a building class 321, a traffic light class 322, and a vending machine class 323 as class types. Classification result 330 includes one each of a building class 331, a traffic light class 332, and a vending machine class 333 as class types.

[0037] Image selection unit 114 compares class classification result 300 with each of class classification results 310-330. When class classification result 300 is compared with class classification result 310, the types of classes included are different. When class classification result 300 is compared with class classification result 320, the types of classes are the same, but the sizes of the regions of each class are different. When class classification result 300 is compared with class classification result 330, the types of classes are the same and the sizes of the regions of each class are approximately the same. In this case, image selection unit 114 selects, from among the candidate comparison images, a past image from which class classification result 330 is obtained as the comparison image.

[0038] Next, the operation procedure will be explained. Fig. 5 shows the operation procedure in the information processing device. The operation procedure in the information processing device corresponds to the information processing method. The image acquisition unit 111 acquires a captured image from the camera 210 (step S1). The class detection unit 112 classifies objects included in the captured image acquired in step S1 into a plurality of classes (step S2). The candidate extraction unit 113 extracts candidates for comparison images from the information storage unit 120 based on the shooting information of the captured image (step S3). In step S3, the candidate extraction unit 113 extracts, for example, one or more past images whose shooting information matches the shooting information of the captured image and which were taken at a position close to the position where the captured image is thought to have been taken, as candidates for comparison images.

[0039] The image selection unit 114 determines whether the candidates for comparison images extracted in step S3 include a previous image taken at the same position as the captured image (step S4). In step S4, the image selection unit 114 determines whether the candidates for comparison images include a previous image taken at the same position as the captured image, based on the class classification result of the captured image and the class classification result of each of the candidate comparison images. If there is a previous image whose class classification result matches or is similar to the class classification result of the captured image, the image selection unit 114 determines that the candidates for comparison images include a previous image taken at the same position as the captured image.

[0040] If the image selection unit 114 determines that the candidates for comparison images include a previous image taken at the same position as the captured image, it selects the previous image as the comparison image (step S5). The image comparison unit 115 compares the captured image acquired in step S1 with the comparison image selected in step S6 (step S6). The image comparison unit 115 calculates, for example, the difference between the captured image and the comparison image for each class, and detects whether the shape has changed in each class.

[0041] If there is no previous image whose classification result is the same as or similar to the classification result of the captured image, the image selection unit 114 determines that the candidates for comparison images do not include previous images taken at the same position as the captured image. In this case, image comparison is not performed. The captured image acquired in step S1 and the classification result of step S2 are stored in the information accumulation unit 120 as the previous image and the classification result of the previous image, together with the shooting information and shooting position information.

[0042] In this embodiment, the candidate extraction unit 113 extracts candidates for comparison images from past images stored in the information accumulation unit 120, taking into consideration the date and time of shooting and the weather at the time of shooting. The image selection unit 114 selects a comparison image from the candidate comparison images. For example, the image selection unit 114 selects, as a comparison image, a past image that was taken at the same location as the captured image and under similar conditions to the captured image. In this way, the image selection unit 114 can select, as a comparison image, an image that is easy for the image comparison unit 115 to calculate a difference in time series, i.e., an image that is suitable for anomaly detection.

[0043] In this embodiment, the image selection unit 114 selects a comparison image from candidate comparison images using the results of class classification. If the comparison image were selected based on the traveling speed of the moving object or GNSS location information, these would contain measurement errors, which could result in a discrepancy between the shooting position of the captured image and the shooting position of the selected comparison image. If the shooting position of the captured image differs from the shooting position of the selected comparison image, in a use case where damage to a sign or the like is to be detected, the discrepancy in the shooting position could prevent accurate detection of anomalies. In this embodiment, the comparison image is selected using the results of class classification, so a previous image captured at the same location as the captured image can be selected as the comparison image. This allows the image comparison unit 115 to compare images captured at the same location, enabling accurate anomaly detection.

[0044] Next, a second embodiment of the present disclosure will be described. Fig. 6 shows an information processing system according to the second embodiment of the present disclosure. The information processing system 100a includes an information processing device 110 and a plurality of cameras 210. In the information processing system 100a, each camera 210 is mounted on a mobile object 200. The information processing device 110 is connected to the mobile object 200 via a network 150. The network 150 includes, for example, a network using a communication line standard such as LTE (Long Term Evolution). The network 150 may also include a wireless communication network such as WiFi (registered trademark) or a fifth-generation mobile communication system.

[0045] The configuration of the information processing device 110 according to this embodiment may be the same as the configuration of the information processing device 110 described in the first embodiment shown in FIG. 2. In this embodiment, the information processing device 110 is configured as, for example, a server device. In this embodiment, the image acquisition unit 111 acquires captured images from cameras 210 of multiple moving objects 200 via a network 150. The operation of the information processing device 110 according to this embodiment may be the same as the operation of the information processing device 110 according to the first embodiment, except that captured images of the cameras 210 are collected from multiple moving objects 200.

[0046] It is to be noted that the camera 210 may have a different viewpoint position and angle of view for each moving body 200. In other words, it is to be considered that the captured images of the camera 210 have individual differences for each moving body 200. In this embodiment, when comparing images, the image comparison unit 115 may correct the viewpoint position and angle of view of the captured image and the comparison image to predetermined reference viewpoint position and angle of view, respectively. Alternatively, the image acquisition unit 111 may correct the viewpoint position and angle of view of the captured images acquired from the camera 210 of each moving body 200 to predetermined reference viewpoint position and angle of view.

[0047] In this embodiment, the information processing device 110 acquires images captured by the camera 210 from multiple moving objects 200. In this case, it is possible to increase the number of images used for abnormality detection compared to when detecting an abnormality by comparing images captured by the camera 210 of a single moving object. Other advantages are similar to those described in the first embodiment.

[0048] In the first embodiment, an example has been described in which the information processing device 110 is mounted on a mobile body. However, the present disclosure is not limited to this. In the first embodiment, the information processing device 110 does not necessarily have to be mounted on a mobile body. For example, the information processing device 110 and the camera 210 may be connected via a network such as a wireless communication network, and the image acquisition unit 111 may acquire a captured image from the camera 210 via the network.

[0049] Next, the hardware configuration of the information processing device 110 will be described. Fig. 7 shows the hardware configuration of the information processing device 110. The information processing device 110 has a processor (CPU: Central Processing Unit) 501, a ROM (read only memory) 502, and a RAM (random access memory) 503. In the information processing device 110, the processor 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. Although not shown, the information processing device 110 may include other circuits such as peripheral circuits, communication circuits, and interface circuits.

[0050] The ROM 502 is a non-volatile storage device. For example, a semiconductor storage device with a relatively small capacity, such as a flash memory, is used for the ROM 502. The ROM 502 stores the programs executed by the processor 501.

[0051] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include RAM, ROM, flash memory, solid-state drive (SSD) or other memory technologies, compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0052] The RAM 503 is a volatile storage device. Various semiconductor memory devices such as a dynamic random access memory (DRAM) or a static random access memory (SRAM) are used for the RAM 503. The RAM 503 can be used as an internal buffer for temporarily storing data and the like.

[0053] The processor 501 loads a program stored in the ROM 502 into the RAM 503 and executes the program. When the CPU 501 executes the program, the functions of each unit in the information processing device 110 can be realized.

[0054] In each of the above embodiments, the information processing device 110 does not necessarily need to be configured as a single physical device. In the present disclosure, the information processing device 110 may be configured using multiple physically separated devices. For example, in FIG. 2, the information processing device 110 may be separated into a device having an image acquisition unit 111, a class detection unit 112, a candidate extraction unit 113, an image selection unit 114, and an information storage unit 120, and a device having an image comparison unit 115. Alternatively, the class detection unit 112, the candidate extraction unit 113, the image selection unit 114, and the image comparison unit 115 may each be configured as an independent device. The information processing device 110 may be configured using a device mounted on a mobile object and a device installed at a location away from the mobile object.

[0055] The above describes the embodiments of the present disclosure in detail, but the present disclosure is not limited to the above-described embodiments, and changes and modifications to the above-described embodiments that do not deviate from the spirit of the present disclosure are also included in the present disclosure.

[0056] For example, some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0057] [Appendix 1] a class detection means for analyzing an image captured by a camera mounted on a moving object and classifying objects included in the image into a plurality of classes; a candidate extraction means for extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; and and an image selection means for selecting a comparison image from the candidate comparison images based on at least one of the type of class included in the captured image, the size of the area of each class, and the proportion of the image that the area of each class occupies, and at least one of the type of class included in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that the area of each class occupies.

[0058] [Appendix 2] 2. The information processing device according to claim 1, wherein the candidate extraction means extracts one or more of the past images whose shooting information is similar to shooting information of the captured image as candidates for a comparison target image.

[0059] [Appendix 3] The information processing device described in Appendix 1 or 2, wherein the candidate extraction means extracts one or more past images whose shooting information is similar to the shooting information of the captured image and whose shooting location information is similar to the shooting location information of the captured image as candidates for comparison images.

[0060] [Appendix 4] the information storage unit further stores, for each of the past images, at least one of a class type, a size of an area of each class, and a proportion of the area of each class in the image; The information processing device described in any one of Appendix 1 to 3, wherein the image selection means acquires from the information storage unit at least one of the class type, the size of the area of each class, and the proportion of the image that the area of each class occupies for each of the candidate comparison images.

[0061] [Appendix 5] The information processing device according to any one of appendices 1 to 4, wherein the class detection means stores in the information storage unit the captured image, shooting information of the captured image, and at least one of the type of the classified class, the size of the area of each class, and the proportion of the image that the area of each class occupies.

[0062] [Appendix 6] 6. The information processing device according to any one of claims 1 to 5, wherein the class detection means detects an object from the captured image and classifies a region of the detected object into a class corresponding to the object.

[0063] [Appendix 7] 7. The information processing device according to claim 1, wherein the class detection means divides the captured image into a plurality of regions according to a class to which each region belongs.

[0064] [Appendix 8] The information processing device according to any one of claims 1 to 7, wherein the image selection means compares the class type contained in the captured image with the class type contained in each of the candidate comparison images, and selects, from the candidate comparison images, a past image that has the highest degree of match with the class type contained in the captured image as the comparison image.

[0065] [Appendix 9] The information processing device according to any one of appendices 1 to 8, wherein the image selection means compares the size of the area of each class in the captured image with the size of the area of each class in each of the candidate comparison images, and selects, from the candidate comparison images, a past image that has the highest degree of match with the size of the area of each class in the captured image as the comparison image.

[0066] [Appendix 10] The information processing device according to any one of appendices 1 to 9, wherein the image selection means compares the proportion of the image occupied by the area of each class in the captured image with the proportion of the image occupied by the area of each class in each of the candidate comparison images, and selects, from the candidate comparison images, a past image that has the highest degree of match with the proportion of the image occupied by the area of each class in the captured image.

[0067] [Appendix 11] 11. The information processing device according to any one of claims 1 to 10, further comprising a comparison unit that compares the captured image with the comparison image selected by the image selection unit.

[0068] [Appendix 12] 12. The information processing device according to claim 11, wherein the comparison means calculates a difference between the comparison image and the captured image.

[0069] [Appendix 13] An information processing system, one or more cameras mounted on a moving object; an information processing device that performs image processing on images captured using the one or more cameras, The information processing device includes: a class detection means for analyzing the captured image and classifying objects included in the captured image into a plurality of classes; a candidate extraction means for extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; and and an image selection means for selecting a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that the area of each class occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that the area of each class occupies.

[0070] [Appendix 14] The information processing system according to claim 13, wherein the information processing device acquires, via a network, a plurality of captured images from a plurality of moving bodies, the captured images being taken using cameras mounted on each moving body.

[0071] [Appendix 15] 15. The information processing system according to claim 13, wherein the candidate extraction means extracts one or more of the past images whose shooting information is similar to the shooting information of the captured image as candidates for comparison images.

[0072] [Appendix 16] The information processing system according to any one of appendices 13 to 15, wherein the candidate extraction means extracts one or more past images whose shooting information is similar to the shooting information of the captured image and whose shooting location information is similar to the shooting location information of the captured image as candidates for comparison images.

[0073] [Appendix 17] Analyzing a captured image taken using a camera mounted on a moving object, and classifying objects included in the captured image into a plurality of classes; extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; An information processing method comprising selecting a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that each class area occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that each class area occupies.

[0074] [Appendix 18] Analyzing a captured image taken using a camera mounted on a moving object, and classifying objects included in the captured image into a plurality of classes; extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; A computer-readable medium that stores a program for causing a computer to select a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that each class area occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that each class area occupies. [Explanation of symbols]

[0075] 10: Information Processing Systems 20: Information processing device 21: Class detection means 22: Candidate extraction means 23: Image selection means 25: Information storage unit 30: Camera 100: Information Processing Systems 110: Information processing device 111: Image acquisition unit 112: Class detection unit 113: Candidate extraction unit 114: Image selection section 115: Image comparison section 120: Information storage unit 150: Network 200: Mobile 210: Camera 501: Processor 502:ROM 503:RAM 504: Bus

Claims

1. a class detection means for analyzing an image captured by a camera mounted on a moving object and classifying objects included in the image into a plurality of classes; a candidate extraction means for extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; and and an image selection means for selecting a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that each class area occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that each class area occupies.

2. The information processing apparatus according to claim 1 , wherein the candidate extracting means extracts one or more of the past images whose photographing information is similar to that of the captured image as candidates for the comparison image.

3. 3. The information processing device according to claim 1, wherein the candidate extraction means extracts one or more of the past images whose shooting information is similar to the shooting information of the captured image and whose shooting location information is similar to the shooting location information of the captured image as candidates for comparison images.

4. the information storage unit further stores, for each of the past images, at least one of a class type, a size of an area of each class, and a proportion of the area of each class in the image; 3. The information processing device according to claim 1, wherein the image selection means acquires, from the information storage unit, at least one of the class type, the size of the area of each class, and the proportion of the area of each class in the image for each of the candidate comparison images.

5. 3. The information processing device according to claim 1, wherein the class detection means stores in the information storage unit at least one of the captured image, shooting information of the captured image, and the type of the classified class, the size of the area of each class, and the proportion of the image occupied by the area of each class.

6. The information processing apparatus according to claim 1 , wherein the class detection means detects an object from the captured image and classifies a region of the detected object into a class corresponding to the object.

7. 3. The information processing apparatus according to claim 1, further comprising a comparison unit that compares the captured image with the comparison image selected by the image selection unit.

8. An information processing system, one or more cameras mounted on a moving object; an information processing device that performs image processing on images captured using the one or more cameras, The information processing device includes: a class detection means for analyzing the captured image and classifying objects included in the captured image into a plurality of classes; a candidate extraction means for extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; and an image selection means for selecting a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that the area of each class occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that the area of each class occupies.

9. A computer comprising: Analyzing a captured image taken using a camera mounted on a moving object, and classifying objects included in the captured image into a plurality of classes; extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; An information processing method for selecting a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that each class area occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that each class area occupies.

10. Analyzing a captured image taken using a camera mounted on a moving object, and classifying objects included in the captured image into a plurality of classes; extracting candidates for comparison images to be compared with the captured image from an information storage unit that stores one or more past images taken in the past and photography information including at least one of photography date and time information and weather information at the time of photography of each of the one or more past images, based on the photography information of the captured image; A program for causing a computer to execute the following: selecting a comparison image from the candidate comparison images based on at least one of the type of class contained in the captured image, the size of the area of each class, and the proportion of the image that each class area occupies, and at least one of the type of class contained in each of the candidate comparison images, the size of the area of each class, and the proportion of the image that each class area occupies.

Images