Information processing device, information processing method, and program

Abstract

To provide an information processing device that can accurately detect images of the same scene from recorded video. [Solution] The information processing device 1 is an information processing device 1 for identifying the same object contained in different images, and comprises: a feature quantity calculation unit 162 that calculates a first local feature quantity indicating the local feature quantity of an object image TG containing the object T, and a second local feature quantity indicating the local feature quantity of a comparison image CG to be compared with the object image TG; an attention level calculation unit 166 that calculates the level of attention based on the first local feature quantity, the second local feature quantity, or both; and a similarity calculation unit 164 that compares the first local feature quantity and the second local feature quantity, weights the level of attention, and calculates the similarity between the object image TG and the comparison image CG.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, inspection work for road appurtenances such as signs and lighting fixtures is carried out in three stages of processes: "reconnaissance survey", "formal inspection", and "record preparation". Conventionally, in the "reconnaissance survey" process, personnel go to the site, check against the facility list to confirm the presence or absence of the facilities listed in the list, as well as the presence or absence of newly installed facilities not included in the list, and create an inspection implementation plan for the upcoming "formal inspection". In addition, preparation of a facility ledger for entering the inspection results in the "formal inspection" is also carried out, and specification information (type, management number, route name, up / down classification, address, distance marker value, latitude and longitude, location map, etc.) and the like are also entered in advance. As described above, not only in the "formal inspection" but also in the "reconnaissance survey" process, it is necessary to go to the site on foot to confirm the actual objects. Furthermore, since it is difficult to enter specification information from scratch into the facility ledger at the site, a duplicate work occurs where the information is entered into a field notebook or the like for outdoor work, and then transcribed into the facility ledger in subsequent indoor work. In addition, since no advance preparation as in the "formal inspection" has been made, it may be difficult to secure a parking space for the vehicle when moving to the vicinity of the site.

[0003] To address the above challenges, there is a technology that supports the creation of facility registers by analyzing camera footage from vehicles, such as drive recorders, to generate information on the type and location of facilities captured in the images (for example, Patent Document 1). Such field surveys using in-vehicle camera footage from drive recorders can overcome the difficulties that arise in on-site surveys. However, while facility classification information (signs, lighting, and information boards, etc.) can be identified from camera footage, management numbers may not be identifiable from camera footage alone. Management numbers may be written on stickers attached to the support pillars of road infrastructure facilities, but these are often too small to be seen from a moving vehicle, and may be posted in locations invisible from the roadway. Furthermore, location information listed in facility lists often lacks latitude and longitude information, and distance marker values ​​are often only approximate, making it difficult to distinguish between nearby facilities. Therefore, it is difficult to identify the management number of a facility captured in an image from location information. For these reasons, it is difficult to identify management numbers from facility lists and in-vehicle camera footage alone.

[0004] Camera footage and management numbers can be matched with panoramic photographs attached to past inspection records of the same facility. In this case, since individual road fixtures, especially streetlights, often have the same appearance, the matching will also include the background.

[0005] While there are many image matching methods, feature point matching methods using corner feature points, such as SHIFT and AKAZE, are widely available through libraries like OpenCV and can be used with high accuracy. Furthermore, Patent Document 2 describes an image matching technology specifically for in-vehicle cameras. This technology reduces the effects of ambient light changes by determining a special flow from panoramic images of the vehicle's side, enabling a clear understanding of the shapes of surrounding structures and other objects. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-197903 [Patent Document 2] Japanese Patent Publication No. 2016-143364 [Non-patent literature]

[0007] [Non-Patent Document 1] Takayuki Okaya, "Deep Learning," Revised 2nd Edition, Kodansha, Published September 1, 2022. [Overview of the project] [Problems that the invention aims to solve]

[0008] Patent Document 1 describes a machine learning model created using machine learning that enables the automatic detection of road infrastructure from in-vehicle camera footage. However, while this model can recognize "signs," it cannot determine "whether or not they are the same as a specific sign."

[0009] Patent Document 2 allows for matching elements that do not change much over time or with the seasons, such as building silhouettes, but it cannot handle cases where the silhouette changes significantly due to seasonal changes in the presence or absence of leaves, such as street trees, or due to regular pruning. Furthermore, trees have many characteristic points such as branch branching and leaf ends, and for the reasons mentioned above, they change significantly over time and with the seasons, making it difficult to apply feature point matching. Therefore, it has been difficult to automatically find road appurtenances in test images from video footage taken for reconnaissance.

[0010] This invention has been made in view of the above problems, and aims to provide an information processing device, an information processing method, and a program that can accurately detect images of the same scene from captured video. [Means for solving the problem]

[0011] (1) The present invention is an information processing device for identifying the same object contained in different images, comprising: a feature calculation unit that calculates a first local feature quantity indicating the local feature quantity of an object image containing the object, and a second local feature quantity indicating the local feature quantity of a comparison image to be compared with the object image; a focus calculation unit that calculates a focus based on the first local feature quantity, the second local feature quantity, or both; and a similarity calculation unit that compares the first local feature quantity and the second local feature quantity, weights the focus, and calculates a similarity between the object image and the comparison image.

[0012] (2) In the information processing device, the attention level calculation unit reduces the attention level for natural objects that change over time, such as the sky, clouds, and vegetation.

[0013] (3) In the information processing device, the attention calculation unit calculates the attention level such that the attention level of the feature quantity with high sparsity is high.

[0014] (4) In the information processing device, the first local feature is a multidimensional feature vector consisting of a combination of multiple numerical elements, and the attention calculation unit calculates the attention based on the maximum value of the multiple numerical elements.

[0015] (5) In the information processing device, the first local feature is a multidimensional feature vector consisting of a combination of multiple numerical elements, and the attention calculation unit calculates the attention score based on the number of items whose value is 0 among the multiple numerical elements.

[0016] (6) The information processing apparatus further comprises a division unit that divides the object image into a plurality of first image regions and the comparison image into a plurality of second image regions, wherein the feature amount calculation unit calculates the first local feature amount for each of the plurality of first image regions and calculates a second local feature amount that indicates a feature amount for each of the plurality of second image regions, and the similarity calculation unit compares the first local feature amount for each of the first image regions with the second local feature amount of the corresponding second image region and calculates the similarity between the object image and the comparison image.

[0017] (7) In the information processing apparatus, the attention level calculation unit identifies a similar second image region which is the image region with the closest feature quantities among a plurality of second image regions based on the first local feature quantity and the second local feature quantity, calculates the difference in coordinate change from the first image region to the similar second image region, and calculates the attention level of the first image region and / or the second image region, or either one thereof, according to the correlation coefficient between the difference and the difference in coordinate change similarly calculated in the surrounding region.

[0018] (8) In the information processing device, the first local feature and the second local feature are feature quantities of a feature point, the feature quantity includes a combination of multiple numerical values, the feature quantity calculation unit determines whether the amount of coordinate change from the first image region to the similar second image region at the feature point is greater than or equal to a threshold, and if it determines that the amount of coordinate change from the first image region to the similar second image region at the feature point is greater than or equal to a threshold, the feature point is excluded from the feature quantity. The similarity calculation unit calculates the similarity based on the variance of the coordinate changes of the feature points that were not excluded.

[0019] (9) An information processing method of the present invention is an information processing method for identifying the same object included in different images, the method comprising: a step of calculating a first local feature value indicating a local feature value of an object image including the object; a step of calculating a second local feature value indicating a local feature value of a comparison image to be compared with the object image; a step of calculating an attention level based on the first local feature value, the second local feature value, or both; and a step of comparing the first local feature value and the second local feature value, weighting the attention level, and calculating a similarity between the object image and the comparison image.

[0020] (10) A program of the present invention is a program for causing a computer to execute an information processing method for identifying the same object included in different images, the program causing the computer to execute: a step of calculating a first local feature value indicating a local feature value of an object image including the object; a step of calculating a second local feature value indicating a local feature value of a comparison image to be compared with the object image; a step of calculating an attention level based on the first local feature value, the second local feature value, or both; and a step of comparing the first local feature value and the second local feature value, weighting the attention level, and calculating a similarity between the object image and the comparison image. [Effects of the Invention]

[0021] According to the present invention, it is possible to accurately detect images of the same scene from a captured moving image. [Brief Description of the Drawings]

[0022] [Figure 1] This is a block diagram showing an example of a hardware configuration of a facility identification apparatus according to an embodiment of the present invention. [Figure 2] This is a block diagram showing an example of a functional configuration of the facility identification apparatus according to the embodiment of the present invention. [Figure 3] This is a diagram showing an example of a specification information sheet of a past-year inspection table. [Figure 4] This is a diagram showing a feature value calculation unit. [Figure 5A]This is a diagram showing the features map. [Figure 5B] This is a diagram showing the features map. [Figure 6A] This is a diagram showing an image of the object. [Figure 6B] This figure shows the results of calculating the level of attention for the object image using Equation 1. [Figure 7] This is a schematic diagram illustrating the structure of a neural network. [Figure 8] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 9] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 10] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 11] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 12] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 13] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 14] A flowchart shows an example of a facility identification method according to an embodiment of the present invention. [Figure 15] This is a diagram showing the feature map of the object image. [Figure 16] This figure shows the evaluation results of the comparative example. [Figure 17] This figure shows the evaluation results of the comparative example. [Figure 18] This figure shows the evaluation results for Example 1. [Figure 19] This figure shows the evaluation results for Example 1. [Figure 20] This figure shows the evaluation results for Example 2. [Figure 21] This figure shows the evaluation results for Example 2. [Figure 22] This figure shows the evaluation results for Example 6. [Figure 23]This figure shows the evaluation results for Example 6. [Figure 24] This figure shows the evaluation results for Example 7. [Modes for carrying out the invention]

[0023] The following describes a facility identification device as an information processing device according to one embodiment of the present invention. However, the present invention is not limited to the following embodiments. Furthermore, the figures referenced in the following description merely provide a schematic representation of the shape, size, and positional relationships to the extent that the content of the present invention can be understood. In other words, the present invention is not limited to the shapes, sizes, and positional relationships exemplified in the figures.

[0024] The facility identification device 1 according to an embodiment of the present invention will be described with reference to Figures 1 to 3. Figure 1 is a block diagram showing an example of the hardware configuration of the facility identification device 1 according to an embodiment of the present invention. Figure 2 is a block diagram showing an example of the functional configuration of the facility identification device 1 according to an embodiment of the present invention. Figure 3 is a diagram showing an example of the specifications information sheet 300 of the past year inspection checklist. Note that the facility identification device 1 corresponds to an example of an information processing device.

[0025] <Example of hardware configuration for facility identification device> As shown in Figure 1, the facility identification device 1 in this embodiment is a computer controlled by a program for performing facility identification processing, which will be described later, and comprises a control unit 11, a main memory unit 12, an auxiliary memory unit 13, an input unit 14, an output unit 15, and a communication unit 16.

[0026] The control unit 11 is composed of a processor that performs various calculations and control processes necessary for the operation of the facility identification device 1. The control unit 11 can also be referred to as a processor. The processors that make up the control unit 11 include, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), SoC (System on a Chip), DSP (Digital Signal Processor), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field-Programmable Gate Array), or a combination thereof. The control unit 11 may also be a combination of these processors with a hardware accelerator.

[0027] The main memory unit 12 stores programs such as firmware, system software, and application software, and also functions as a work area that is temporarily used for various processing tasks. The main memory unit 12 is composed of, for example, non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory).

[0028] The auxiliary storage unit 13 stores data such as input data used for facility identification processing and programs for executing facility identification processing, which will be described later. The auxiliary storage unit 13 is composed of semiconductor memory or the like. The auxiliary storage unit 13 may also be configured as a storage medium drive capable of reading portable optical storage media such as CD-ROMs.

[0029] The input unit 14 consists of various components such as a touch panel, keys, and a microphone, and accepts user input. The output unit 15 consists of a display equipped with the aforementioned touch panel to display images, a speaker to amplify sound, and a printer capable of printing, and outputs images, sound, and print results.

[0030] The communication unit 16 controls the communication between the facility identification device 1 and other external devices. The communication unit 16 may be equipped with, for example, a SIM (Subscriber Identity Module) card, network adapters and other network connection devices, and communication devices based on communication standards such as Wi-Fi (Wireless Fidelity) and NFC (Near Field Communication). Examples of external devices that communicate include in-vehicle camera systems that record video data of road facilities. The facility identification device 1 may be an independent computer or a cloud system in which functions are distributed and implemented on a network.

[0031] As shown in Figure 2, the facility identification device 1 is a device for identifying the same object contained in different images. More specifically, the facility identification device 1 is used, for example, to identify frames (scenes) that contain the same object as an image in which the object is contained. The facility identification device 1 is composed of, for example, a PC (personal computer). The facility identification device 1 includes, for example, one or more processors as a control device and one or more memory, hard disk drives, etc. as storage devices. The facility identification device 1 includes a test data receiving unit 110, a data extraction unit 120, a comparison data receiving unit 130, a coordinate processing unit 140, a storage unit 150, and an image comparison unit 160.

[0032] The test data receiving unit 110 receives test data. The test data is, for example, electronic data of a past inspection checklist including a specifications information sheet 300. As shown in Figure 3, the specifications information sheet 300 of the past inspection checklist is an inspection checklist for past years. In this embodiment, the past inspection checklist is a table in which inspection results related to road appurtenances have been entered. Note that road appurtenances are an example of the target object. Road appurtenances are, for example, guide signs installed on roads. Hereinafter, in this specification, "guide signs installed on roads" may be referred to as "guide signboards". The following explanation will use the case where the target object is a guide signboard as an example. The specifications information sheet 300 of the past inspection checklist has a text entry field 315 in which the specifications information of the facility to be inspected is entered in text, a map field 325, and a target object image field 335. The text entry field 315 shows the location and structural information of the facility. The map field 325 shows a map of the location where the guide signboard is installed. Object image field 335 shows object image TG. Object image TG includes the object. More specifically, object image TG is an image showing the entire road view, including the directional signs.

[0033] The data extraction unit 120 extracts the necessary data from the test data. In other words, the data extraction unit 120 extracts the necessary data from the specifications information sheet 300 of the past year inspection report. For example, the data extraction unit 120 outputs the latitude and longitude information of the location where the directional sign is installed to the coordinate processing unit 140. The data extraction unit 120 also outputs the specifications information and the object image TG. The specifications information includes, for example, the management number, latitude and longitude, address, manager, and previous inspection record. The data extraction unit 120 also outputs the object image TG to the image comparison unit 160.

[0034] The comparison data receiving unit 130 receives comparison data from the in-vehicle camera system 200. The in-vehicle camera system 200 is, for example, a drive recorder. The comparison data is, for example, video data of the road captured by the in-vehicle camera system 200 while the vehicle is in motion. The comparison data receiving unit 130 outputs the NMEA file and the video file to the coordinate processing unit 140 and the storage unit 150. Note that the comparison data receiving unit 130 is not limited to receiving comparison data from the in-vehicle camera system 200. For example, the comparison data receiving unit 130 may receive comparison data via a storage medium. The storage medium may be, for example, a removable media such as a USB memory stick.

[0035] The coordinate processing unit 140 roughly narrows down the range of the video to be played based on the latitude and longitude information. The coordinate processing unit 140 estimates the shooting direction of road fixtures. More specifically, it estimates the shooting direction of road fixtures from the road's center coordinates and the bias of the facility's latitude and longitude information. The road's center coordinates can be obtained from a website such as the vector tiles of the Geospatial Information Authority of Japan map. If the NMEA file input from the in-vehicle camera system 200 contains gyro data, the range of the video to be input to the image comparison unit 160 in the subsequent process can be efficiently narrowed down by searching for driving data that matches the shooting direction. The coordinate processing unit 140 calculates the bias of the facility's latitude and longitude information based on the latitude and longitude information obtained from the data extraction unit 120. If gyro data is unavailable, the driving direction (shooting direction) may be estimated from the time change of the latitude and longitude of the driving data.

[0036] The memory unit 150 stores NMEA files and video files.

[0037] The image comparison unit 160 performs comparison processing between the target image and the comparison image. The image comparison unit 160 includes a feature calculation unit 162, a similarity calculation unit 164, a focus calculation unit 166, and a division unit 168.

[0038] The feature calculation unit 162 calculates a first local feature. The first local feature represents the local feature of the object image TG. The object image TG includes the object T. The object T is, for example, a guide sign. The feature calculation unit 162 calculates a second local feature. The second local feature represents the local feature of the comparison image CG. The comparison image CG is an image to be compared with the object image TG. In this embodiment, the comparison image CG is an image included in the video data captured by the in-vehicle camera system 200.

[0039] The similarity calculation unit 164 compares the first local feature and the second local feature to calculate the similarity between the target image TG and the comparison image CG. The similarity calculation unit 164 calculates the similarity by applying the attention weight described later.

[0040] The attention calculation unit 166 calculates the attention level based on the first local feature, the second local feature, or both.

[0041] The division unit 168 divides the object image TG into multiple first image regions R. The division unit 168 also divides the comparison image CG into multiple second image regions R.

[0042] The feature calculation process of the feature calculation unit 162 will be explained with reference to Figures 4 to 5B. Figure 4 is a diagram of the feature calculation unit 162. Figure 5A is a diagram of the feature map M0. Figure 5B is a diagram of the feature map M1.

[0043] As shown in Figure 4, the feature calculation unit 162 includes a neural network 1622. The feature calculation unit 162 calculates features using the neural network 1622. In the example shown in Figure 4, we will describe the case where an object image TG, which is a 224x224 pixel RGB color image, is input as the input image. When a 224x224 pixel RGB color image is input to the neural network 1622, it divides the image into 7x7 regions and outputs 576 numerical values ​​to each region. Hereafter in this specification, the 7x7x576 data will be referred to as a "feature map," and the data consisting of 576 numerical values ​​in each region will be referred to as a "feature vector." Note that the image region division may be in a manner other than that described above.

[0044] Figure 5A shows the feature map M0. As shown in Figure 5A, the feature map M0 is arranged in a matrix of multiple regions R, with 7 regions each in the horizontal direction (i direction) (i=0 to 6) and 7 regions each in the vertical direction (j direction) (j=0 to 6). The neural network 1622 has a "convolutional neural network" structure and has multiple "convolutional layers 1662a" in the process from input to output. The convolutional layers extract features from the input image. map_out0 represents the feature map M0. map_out0[i,j], which is an element of map_out0, is the feature vector. In the example shown in Figure 5A, the features of each region R are represented as 576-dimensional feature vectors. In this way, the feature calculation unit 162 divides the object image TG into 7x7 regions R and outputs a feature map M0 containing features (feature vectors) for each divided region R. In this specification, the feature quantities for each region R of the object image TG may be referred to as first local features. Thus, the feature calculation unit 162 calculates first local features for each of the multiple first image regions R. The first local features are multidimensional feature vectors consisting of a combination of multiple numerical elements.

[0045] The feature calculation unit 162 outputs a feature map M1 containing feature quantities (feature vectors) for each divided region R of the comparison image CG, similar to the target image TG. Specifically, the feature calculation unit 162 divides the comparison image CG into 7x7 regions R and outputs a feature map M1 containing feature quantities (feature vectors) for each divided region R. In this specification, the feature quantities for each region R of the comparison image CG may be referred to as second local features. In this way, the feature calculation unit 162 calculates second local features for each of the multiple second image regions R. The second local features are multidimensional feature vectors consisting of a combination of multiple numerical elements.

[0046] The similarity calculation unit 164 compares the first local feature and the second local feature to calculate the similarity between the object image TG and the comparison image CG. In other words, the similarity calculation unit 164 calculates the similarity between the object image TG and the comparison image CG by comparing the feature map M0 and the feature map M1. In addition, the similarity calculation unit 164 calculates the similarity by weighting the degree of attention. Specifically, the similarity calculation unit 164 calculates the similarity by changing the weighting of the matching judgment between the object image TG and the comparison image CG depending on the objects captured in each region R. This weighting allows for efficient determination of the similarity between the object image TG and the comparison image CG. In other words, it is possible to identify the same object contained in different images.

[0047] The attention calculation unit 166 calculates the attention level based on the first local feature and the second local feature. As explained with reference to Figure 5A, the feature vector map_out0[i,j] for each 7x7 region R consists of 576 combinations of numbers. These individual numbers do not directly indicate what is depicted in that region. However, if a region R has the following features 1 and 2, the probability of it being a region of interest is high. In particular, if both features 1 and 2 are present, the probability of it being a region of interest is high.

[0048] Feature 1) Among the 576 numerical values ​​in map_out0[i,j], there is one that is large when compared with other regions R.

[0049] Feature 2) Of the 576 elements in map_out0[i,j], many are 0 or have small values ​​below the threshold (spaciousness).

[0050] The attention calculation unit 166 calculates the attention level using equation 1 for the region [i,j] as an example, as shown below.

[0051] Attention level [i,j] = α·|the maximum value of the elements in map_out0[i,j]| + β·(the number of zero elements in map_out0[i,j])

[0052] α and β are proportionality constants and can be changed according to the priority of conditions 1 and 2.

[0053] In this way, the attention calculation unit 166 calculates attention so that the attention of highly spars features is increased. The attention calculation unit 166 also calculates attention based on the maximum value of each numerical element of the multidimensional feature vector (feature 1). Furthermore, the attention calculation unit 166 calculates attention based on the number of items whose values ​​of multiple numerical elements are 0 (feature 2).

[0054] Referring to Figures 6A and 6B, the calculation of attention level by the attention level calculation unit 166 will be further explained. Figure 6A shows the object image TG. Figure 6B shows the result of calculating the attention level for the object image TG using Equation 1. In Figure 6B, the density of the dots in the dot hatching is changed according to the level of attention level. In Figure 6B, it is shown that the dots in the dot hatching become denser as the level of attention level increases.

[0055] As shown in Figure 6A, the object image TG includes object T, which in this case is a signpost. The object image TG further includes multiple street tree images 310, utility pole images 320, power line images 330, and car pillar images 340. As shown in Figure 6B, the street tree images 310 occupy a large portion of the region R in the object image TG. However, it can be seen that the regions R with high attention are those where artificial objects are located, such as object T (signpost), utility pole images 320, power line images 330, and car pillar images 340. In other words, regions R where natural objects are located, such as the street tree images 310, have low attention. Thus, the attention calculation unit 166 reduces the attention to the first local feature, the second local feature, or both, corresponding to the natural objects. Natural objects change over time. Natural objects include the sky, clouds, and vegetation. For example, trees like those that line streets have lush leaves in the summer, but they lose their leaves in the winter.

[0056] The attention-gathering calculation unit 166 may calculate the attention-gathering level using other calculation methods, provided that they reflect features 1 and 2. For example, the attention-gathering calculation unit 166 may calculate the attention-gathering level using the following formula (Formula 2).

[0057]

number

[0058] Note that in equation (2), α, β, and γ may be adjusted depending on the scene, but β < 0.

[0059] The similarity calculation unit 164 compares the first local feature and the second local feature to calculate the similarity between the object image TG and the comparison image CG. In other words, the similarity calculation unit 164 calculates the similarity between the object image TG and the comparison image CG by comparing the feature map M0 and the feature map M1. The feature vectors (map_out0[i,j], map_out1[i,j]) which are elements of each region R in the feature map M0 and the feature map M1, reflect the features of the objects captured within that region R. If similar objects are captured, the orientations of the feature vectors will also be similar. Therefore, the cosine similarity of the feature vectors can be used as an index to quantify the similarity of the captured objects. For example, if similar objects are captured in region [i0,j0] of the object image TG and region [i1,j1] of the comparison image CG, and a completely different object is captured in region [i2,j2] of the comparison image CG, the following relationship holds.

[0060] Cos similarity(map_out0[i0,j0],map_out1[i1,j1])>Cos similarity(map_out0[i0,j0],map_out1[i2,j2])

[0061] In other words, the more similar the region R located at the same position in the object image TG and the comparison image CG are, the larger the cos-similarity becomes. Using this relationship, the similarity calculation unit 164 calculates the overall similarity of the object image TG and the comparison image CG using the following equation (3). Although equation (3) shows an example using the attention level of the object image TG, the attention level of the comparison image CG may be used, or even the attention levels of both may be used.

[0062]

number

[0063] As described above, the level of attention can be calculated using the output of neural network 1622 and numerical calculations performed on that output. In this example, neural network 1622 and its subsequent processing are described separately, but it is also acceptable to perform all calculations on neural network 2622. Furthermore, the most desirable method for creating neural network 1622 is to perform machine learning to satisfy features 1 and 2 for artificial objects of interest, and conversely, to dissatisfy features 1 and 2 for natural objects. However, performing machine learning requires preparing a large amount of training data, and performing separate machine learning for each application would result in an excessively large development burden. Considering the above, neural network 1 in this embodiment does not undergo training for this application, but rather reuses a part of a neural network created for a different purpose.

[0064] Figure 7 is a schematic diagram showing the structure of Neural Network 0, which served as the basis for Neural Network 1622. Neural Network 0 is an artificial intelligence network that, when given an image as input, identifies what the object in the image is from a list of 1000 labels. The output of Neural Network 0 is a 1x1x1000 data structure, that is, an array of 1000 numbers. Each element in the array is assigned one of the 1000 labels, and the number represents the predicted probability that the object in the image is that label. This Neural Network 0 was trained using ImageNet1000 (https: / / image-net.org / index.php) as training data. In the example below, when an image of an African elephant is input, the output is a 70% probability that it is an "African elephant" and a 20% probability that it is an "Indian elephant".

[0065] Neural network 1622 is obtained by extracting the intermediate layer of neural network 0, which was trained using ImageNet1000 as training data. Since ImageNet1000 does not contain labels related to road appurtenances, this neural network is not suitable for this purpose. Nevertheless, by performing a process that extracts features 1 and 2, it is possible to determine the areas of interest from the test images. The reason why this neural network, which has not been trained on road appurtenances, possesses the properties of features 1 and 2 is thought to be that, by training with a large amount of training data labeled by humans, the network has built a human-like way of focusing on images that is independent of the type of object being detected.

[0066] Referring to Figure 8, a facility identification method according to an embodiment of the present invention will be described. Figure 8 is a flowchart of an example of a facility identification method according to an embodiment of the present invention. Facility identification is performed by executing the processes from step S102 to step S110 shown in Figure 8. In particular, the same object contained in different images is identified by executing the processes from step S102 to step S110 shown in Figure 8. Note that the facility identification method is an example of an information processing method.

[0067] In step S102, the feature calculation unit 162 calculates the first local feature. The first local feature represents the local feature of the object image TG that includes the object T. The process then proceeds to step S104.

[0068] In step S104, the feature calculation unit 162 calculates a second local feature. The second local feature represents the local features of the comparison image CG, which is compared with the target image TG. The process then proceeds to step S106.

[0069] In step S106, the attention calculation unit 166 calculates the attention level based on the first local feature. The process then proceeds to step S108.

[0070] In step S108, the similarity calculation unit 164 compares the first local feature and the second local feature, weights the degree of attention, and calculates the similarity between the object image TG and the comparison image CG. The process then proceeds to step S110.

[0071] In step S110, the similarity calculation unit 164 identifies the same object contained in different images based on the similarity calculated in step S108. The process then ends.

[0072] Referring to Figure 9, the facility identification method according to an embodiment of the present invention will be described in more detail. Figures 9 to 14 are flowcharts showing an example of the facility identification method according to an embodiment of the present invention. Facility identification is performed by executing the processes from step S202 to step S400 shown in Figure 9. Specifically, the same object contained in different images is identified by executing the processes from step S202 to step S400 shown in Figure 9. Note that the facility identification method is an example of an information processing method.

[0073] In step S202, the test data receiving unit 110 receives a file of past inspection reports (test data) including the specifications information sheet 300. The process then proceeds to step S204.

[0074] In step S204, the data extraction unit 120 aggregates specification information from the specification information sheet 300. The specification information includes, for example, the management number, latitude and longitude, address, administrator, and previous inspection record. The process then proceeds to step S206.

[0075] In step S206, the data extraction unit 120 outputs the specification information. The process then proceeds to step S208.

[0076] In step S208, the data extraction unit 120 extracts the latitude and longitude from the specification information and transmits it to the coordinate processing unit 140. The process then proceeds to step S210.

[0077] In step S210, the data extraction unit 120 extracts the full-view photograph, generates an object image TG, and outputs it to an external device. Furthermore, the data extraction unit 120 transmits the object image TG to the image comparison unit 160. The process then proceeds to step S212.

[0078] In step S212, the comparison data receiving unit 130 receives the NMEA file and the video file from the in-vehicle camera system 200 and stores them in the storage unit 150. The process then proceeds to step S214.

[0079] In step S214, the comparison data receiving unit 130 creates a list of NMEA files and video files from the in-vehicle camera system 200 and saves it to the storage unit 150. The process then proceeds to step S300.

[0080] In step S300, the coordinate processing unit 140 performs a playback range determination process. The playback range determination process will be described later with reference to Figure 10. The process then proceeds to step S400.

[0081] In step S400, the image comparison unit 160 compares the object image TG with the comparison image CG to identify the same object contained in the different images. The image comparison process will be described later with reference to Figures 11 to 14. The process then ends.

[0082] Referring to Figure 10, the playback range determination process according to an embodiment of the present invention will be described. Figure 10 is a flowchart showing an example of the playback range determination process according to an embodiment of the present invention. The playback range determination process is performed by executing the processes from step S302 to step S310 shown in Figure 10.

[0083] In step S302, the coordinate processing unit 140 receives the NMEA file list 1 and the video file list 1. The process then proceeds to step S304.

[0084] In step S304, the coordinate processing unit 140 receives the latitude and longitude information of the facility. The process then proceeds to step S306.

[0085] In step S306, the coordinate processing unit 140 reads the NMEA file from the file list from the storage unit 150. The process then proceeds to step S308.

[0086] In step S308, the coordinate processing unit 140 opens the NMEA file and creates a list of candidate times for when the vehicle traveled near the facility's latitude and longitude. The process then proceeds to step S310.

[0087] In step S310, a list of video files containing the times from the candidate time list is created, a list of frames to be played (video file list 2, candidate frame range) is created, and this is sent to the image comparison unit 160. The process then ends.

[0088] The image comparison process according to an embodiment of the present invention will be described with reference to Figures 11 to 14. Figures 11 to 14 are flowcharts of an example of the image comparison process according to an embodiment of the present invention. The image comparison process is performed by executing the processes from step S402 to step S416 shown in Figure 11, the processes from step S4062 to step S4066 shown in Figure 12, the processes from step S4141 to step S4149 shown in Figure 13, and the processes from step S4151 to step S4155 shown in Figure 14.

[0089] In step S402, the image comparison unit 160 receives the video file list 2 and the candidate frame range. The process then proceeds to step S404.

[0090] In step S404, the image comparison unit 160 receives the object image TG.

[0091] In step S406, the image comparison unit 160 executes the attention level calculation process from steps S4062 to S4066 shown in Figure 12. The process proceeds to step S4062 shown in Figure 12.

[0092] In step S4062, the feature calculation unit 162 adjusts the size of the object image TG. The process then proceeds to step S4064.

[0093] In step S4064, the feature calculation unit 162 inputs the object image TG to the neural network 1622. As a result, the feature calculation unit 162 calculates the first local features. The process then proceeds to step S4066.

[0094] In step S4066, the attention calculation unit 166 calculates the attention level from the first local feature quantities for each 7x7 region R. The process then proceeds to step S408, as shown in Figure 11.

[0095] In step S408, the image comparison unit 160 clears the variable i. In this example, 0 is stored in the variable i. The process then proceeds to step S410.

[0096] In step S410, the image comparison unit 160 determines whether the variable i is less than the number of videos in the list. If the image comparison unit 160 determines that the variable i is not less than the number of videos in the list (step S410: No), the process ends. In other words, it determines that the image comparison process has been completed for all video lists, and the process ends. If the image comparison unit 160 determines that the variable i is less than the number of videos in the list (step S410: Yes), the process proceeds to step S412.

[0097] In step S412, the image comparison unit 160 receives the i-th video file name, playback frame range, and target image. The process then proceeds to step S414.

[0098] In step S414, the image comparison unit 160 executes the matching image output processing steps S4141 to S4149 shown in Figure 13. The process proceeds to step S4141 shown in Figure 13.

[0099] In step S4141, the similarity calculation unit 164 receives the video file name, playback frame range, and target image TG. The process then proceeds to step S4142.

[0100] In step S4142, the similarity calculation unit 164 reads the video file from the storage unit 150. The process then proceeds to step S4143.

[0101] In step S4143, the similarity calculation unit 164 stores the minimum number of the playback frame range in the variable j. The process then proceeds to step S4144.

[0102] In step S4144, the similarity calculation unit 164 determines whether the variable j is smaller than the maximum number in the playback frame range. If the similarity calculation unit 164 determines that the variable j is not smaller than the maximum number in the playback frame range (step S4144: No), the process proceeds to step S4148. If the similarity calculation unit 164 determines that the variable j is smaller than the maximum number in the playback frame range (step S4144: Yes), the process proceeds to step S4145.

[0103] In step S4145, the similarity calculation unit 164 executes the similarity calculation process from steps S4151 to S4155 shown in Figure 14. The process proceeds to step S4151 shown in Figure 14.

[0104] In step S4151, the similarity calculation unit 164 extracts the j-th frame image from the video file and uses it as the comparison image CG. The process then proceeds to step S4152.

[0105] In step S4152, the feature calculation unit 162 adjusts the size of the comparison image CG. The process then proceeds to step S4153.

[0106] In step S4153, the feature calculation unit 162 inputs the comparison image CG to the neural network 1622. As a result, the feature calculation unit 162 calculates the second local feature. The process then proceeds to step S4154.

[0107] In step S4154, the attention calculation unit 166 finds the 7x7 region R of the comparison image CG that is most similar to the 7x7 region R of the object image TG. The process then proceeds to step S4155.

[0108] In step S4155, the similarity calculation unit 164 calculates a similarity evaluation value. Specifically, the similarity calculation unit 164 calculates the similarity evaluation value by performing the calculation Σ(similarity of the area × degree of interest). The process then proceeds to step S4146 shown in Figure 13.

[0109] In step S4146, the similarity calculation unit 164 updates the maximum value of the similarity evaluation value with the target image TG and the frame number at which the maximum value was obtained. The process then proceeds to step S4147.

[0110] In step S4147, the similarity calculation unit 164 increments the variable j by 1 and stores it. The process returns to step S4144. The similarity calculation unit 164 then repeats steps S4145 to S4147 until it determines in step S4144 that the variable j is not less than the maximum number in the playback frame range (step S4144: No). In other words, the similarity calculation unit 164 calculates the similarity for each frame until it has calculated the similarity for all frames in the playback frame range.

[0111] In step S4148, the similarity calculation unit 164 outputs the frame image with the highest similarity evaluation value to the target image as the matching image. The process then proceeds to step S4149.

[0112] In step S4149, the similarity calculation unit 164 outputs various information regarding the matching image. The process then proceeds to step S416, as shown in Figure 11.

[0113] In step S416, the image comparison unit 160 increments the variable i by 1 and stores the result. The process returns to step S410. The image comparison unit 160 then repeats steps S4110 to S416 until the similarity calculation unit 164 determines (step S410: No) that the variable j is not less than the number of videos in the video list. In other words, the image comparison unit 160 performs matching image processing until it has finished matching image processing for all videos in the video list. Once it has finished matching image processing for all videos in the video list, the process terminates.

[0114] As explained above with reference to Figures 1 to 14, in the information processing device (facility identification device 1), the similarity calculation unit 164 compares the first local feature quantity and the second local feature quantity, weights the degree of attention, and calculates the similarity between the object image TG and the comparison image CG. Therefore, even if the recorded video contains objects that change over time, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0115] Furthermore, the attention calculation unit 166 reduces the attention level for the first and second local features corresponding to natural objects. Natural objects change over time. Natural objects include the sky, clouds, and vegetation. Therefore, even if time-changing objects are captured in the recorded video, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0116] Furthermore, the attention calculation unit 166 calculates the attention level such that the attention level of the highly spars feature quantity is high. Therefore, even if the recorded video contains objects that change over time, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0117] Furthermore, the attention-grabbing calculation unit 166 calculates attention-grabbing based on the maximum value of multiple numerical elements. Therefore, even if objects that change over time are captured in the recorded video, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0118] Furthermore, the attention calculation unit 166 calculates attention based on the number of items whose values ​​are 0 among multiple numerical elements. Therefore, even if objects that change over time are captured in the recorded video, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0119] Furthermore, the similarity calculation unit 164 compares the first local feature of each first image region with the second local feature of the corresponding second image region, and calculates the similarity between the object image and the comparison image using the attention score. Therefore, even if the recorded video contains objects that change over time, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0120] The evaluation results for the overall image similarity will be explained below with reference to Figures 16 to 24. Figures 16 and 17 show the evaluation results for the comparative example. Figures 18 and 19 show the evaluation results for Example 1. Figures 20 and 21 show the evaluation results for Example 2. Figures 22 and 23 show the evaluation results for Example 6. Figure 24 shows the evaluation results for Example 7.

[0121] Figures 16, 18, and 20 show the object image TG and comparison image CG for Comparative Example, Example 1, and Example 2, respectively. Figure 22 shows the object image TG and comparison image CG for Example 6 and Example 7. In Figures 16, 18, 20, and 22, the two figures on the left show the object image TG. In Figures 16, 18, and 20, the two figures on the right are the comparison image CG of the most similar scenes. In Figure 22, the figure on the left shows the object image TG. In Figure 22, the figure on the right is the comparison image CG of the most similar scenes. The dashed line extending from point P2 in the upper left figure of Figures 16, 18, and 20 represents the coordinates of region R of the most similar comparison image CG in each region R. The dot hatching in the lower left and lower right figures of Figures 16, 18, and 20 indicates the cosine similarity of the feature vectors of the regions. The lower left and lower right figures show that the higher the cosine similarity, the denser the dots in the dot hatching. A higher similarity indicates that the scenes are more alike. For example, in Figure 17, the small peak at 136 seconds represents the scene with the highest similarity within the 5-minute video.

[0122] Figures 17, 19, 21, 23, and 24 show the evaluation results for Comparative Example, Example 1, Example 2, Example 6, and Example 7, respectively. In Figures 17, 19, 21, 23, and 24, the horizontal axis represents the length of the video (0 to 300 seconds, as it is 5 minutes long). In Figures 17, 19, 21, and 23, the vertical axis represents the similarity. In Figure 24, the vertical axis represents the standard deviation of the coordinate displacement.

[0123] After determining the level of attention for each region of the target image TG, the video file from the in-vehicle camera system is compared frame by frame to determine if it is the same scene. Image similarity is calculated as an indicator for this determination. The feature map, which is the output of neural network 1, can be used to determine image similarity. The feature map of the target image TG is map_out0, and the feature map of the comparison image CG is similarly processed and input into neural network 1 to obtain map_out1. The feature vectors (map_out0[i,j], map_out1[i,j]), which are elements of each region of the feature map, reflect the features of the objects captured within that region, and if similar objects are captured, the orientation of the feature vectors will also be similar. Therefore, the cosine similarity of the feature vectors can be used as an indicator to quantify the similarity of the captured objects. For example, if the region [i0,j0] of the object image TG and the region [i1,j1] of the comparison image contain similar objects, and the region [i2,j2] of the comparison image contains a completely different object, then the following relationship holds. Cos similarity(map_out0[i0,j0], map_out1[i1,j1]) > Cos similarity(map_out0[i0,j0], map_out1[i2,j2])

[0124] Using this relationship, the overall similarity of the object image TG and the comparison image CG can be expressed as follows.

[0125]

number

[0126] [i1,j1] is the coordinate of the region with the highest cosine similarity after searching the vicinity of [i0,j0], taking into account positional shifts due to differences in the mounting height and angle of the in-vehicle camera. In this way, the similarity of the entire image can be evaluated.

[0127] (Comparative example) In the comparative example, as shown in the lower left of Figure 16, the similarity evaluation includes areas R, such as the sky and street trees, which should not be the same across years and seasons, in the object image TG. In other words, images other than the object T are also included in the similarity evaluation. As a result, as shown in the lower right of Figure 16, the similarity is high even for areas R, such as the sky and street trees, which should not be the same across years and seasons. Therefore, as shown in Figure 17, the similarity peak P1 is not clearly obtained in the comparative example.

[0128] (Example 1) Therefore, in this invention, we introduce the concept of attention.

[0129]

number

[0130] As shown in the lower left of Figure 18, examining the density of the dot hatching reveals that the similarity between the target object T, the directional sign, and the utility pole on the opposite lane is high. This indicates that the matching is focused on the target object T, the directional sign, and the utility pole on the opposite lane. As a result, as shown in Figure 19, weighting the degree of focus on the cosine similarity makes the peak P1 at 136 seconds clearer.

[0131] Non-Patent Document 1, "7.2.4 Attention Mechanism for Convolutional Layers," introduces an attention mechanism using a squeeze-excitation network in a convolutional layer. In this squeeze-excitation network, an attention mechanism is described in which weights are generated from the output of the convolutional layer using a network consisting of global pooling and fully connected layers, and a sigmoid function. In such an attention mechanism, the difference between large and small elements of the feature vector becomes large due to the sigmoid function, and the small elements approach 0, so the attention effect using sparsity is greatly increased.

[0132] (Example 2) As shown in Figures 20 and 21, in Example 1, the similarity was evaluated for each individual 7x7 region, but by also considering the consistency of matching with adjacent regions, it is possible to more clearly identify identical shooting scenes.

[0133] Refer to Figure 15 to explain the correction of attention level. Figure 15 shows the feature map M0 of the object image TG.

[0134] In Figure 15, we will explain the correction of the level of attention for the region [i0=1,j0=3] in the object image TG as an example. The black arrow in [i0=1,j0=3] indicates that the feature vector of the comparison image that is most similar to the feature vector cos_out0[i0=1,j0=3] of the object image TG is cos_out1[i1=1,j1=2], which is one cell above. Similarly, the surrounding arrows indicate the coordinate relationship between each region and the regions that are similar to it. In this example, it can be seen that the tendency for regions approximately one cell above to be similar is common to the region [i0=1,j0=3] and its surrounding regions. In such cases, the level of attention for the region [i0=1,j0=3] is considered highly reliable, and its level of attention is further increased. The procedure for calculating the correction value, which is the magnification at that time, will be explained.

[0135] This example shows how to process the surrounding region [i0=1,j0=3] clockwise, starting from the 12 o'clock direction. The region at the 12 o'clock direction is in the same direction as the vector in the central coordinates, so the cosine similarity is set to 1 and the correction value is increased. The same applies to the directions from 2 o'clock to 6 o'clock, so 1 is added to each. At 7 o'clock, there is a 45° difference in direction, so cos(45°) ≈ 0.7 is used and 0.7 is added. Similarly, 0.7 is added to the directions from 9 o'clock to 10 o'clock. As a result, the correction value becomes 7.1. Then the region [i0=1,j0=3] is amplified by a factor of 7.1.

[0136] Next, let's explain an example of correcting the level of attention in the region [i0=5,j0=3]. Unlike the previous example, the vector directions to similar regions in region [i0=5,j0=3] are scattered. Therefore, the reliability of the level of attention in this region is interpreted as lower. As in the example above, the vector direction is opposite to that of the region at 12 o'clock, so we add cos(180°)=-1 to the correction value (subtract 1). Similarly, we add -1 to the direction at 2 o'clock. The region at 3 o'clock is in the same direction, so we add +1. The region at 5 o'clock is shifted by 90°, so we add cos(90°)=0. As a result of adding in this way, the correction value becomes -1. Since a negative level of attention is not needed, we set it to 0 in this case, and correct the original level of attention by multiplying it by 0.

[0137] In this way, the attention calculation unit 166 identifies the similar second image region, which is the image region with the closest features among multiple second image regions R, based on the first local feature quantity and the second local feature quantity. It then calculates the difference in coordinate changes from the first image region to the similar second image region, and calculates the attention level of the first and second image regions according to the correlation coefficient between the difference and the difference in coordinate changes with the surrounding region. Therefore, even if the coordinates of the object image TG and the comparison image CG are shifted, the same scene can be appropriately detected from the recorded video. As a result, images of the same scene can be detected with high accuracy from the recorded video.

[0138] In Example 1, the similarity was evaluated for each individual 7x7 region, but in Example 2, the consistency of the matching with adjacent regions is also taken into consideration.

[0139] As shown in Figure 20, the comparison image CG (right side) is shifted upward overall compared to the object image TG (left side) due to positional shifts caused by differences in the mounting height and angle of the in-vehicle camera. Therefore, the signpost in the comparison image CG is shifted upward compared to the signpost (object T) in the object image TG. Thus, even when the position (region R) of the comparison image CG is shifted relative to the object image TG, a high degree of similarity was shown in the region R where the signpost of object T exists, as shown in the comparison image CG (right side). On the other hand, a low degree of similarity was shown for regions R that cannot be the same across years and seasons, such as the sky and street trees. In other words, even when the position (region R) of the comparison image CG is shifted relative to the object image TG, it can be seen that matching focused on object T is performed. As a result, as shown in Figure 21, the peak P1 at 136 seconds became clearer. In other words, in Example 2, if the consistency of matching with adjacent regions is also taken into consideration, it is possible to more clearly identify identical shooting scenes.

[0140] (Example 3) In Example 1, neural network 0 only provided 1000 classifications of labels for the input image. However, if neural network 0 is used to output labels along with bounding boxes for the detected object, and if it has been trained on artificial objects such as road appurtenances and buildings, and natural objects such as street trees, it is possible to increase the attention given to the regions within the bounding boxes of artificial objects and decrease the attention given to the regions within the bounding boxes of natural objects such as street trees.

[0141] (Example 4) In the above examples, the target image TG and comparison image CG are divided into several regions, and attention processing is performed in each divided region. However, the method of the present invention is not limited to use by region division. Feature points also have local features of the image, and descriptors that store information about feature points can also be used as feature vectors, similar to the above examples. As for feature point detection methods, machine learning methods such as SuperPoint (https: / / arxiv.org / abs / 1712.07629) can be used. Furthermore, the attention-based method of this patent is not limited to machine learning methods, and feature point detection algorithms such as SHIFT and AKAZE can also be used.

[0142] (Example 5) In another embodiment of the present invention, suppose there are N feature points that have been matched between the object image TG and the comparison image CG, and the coordinates of the m-th (where m is 0 to N-1) are [x0(m), y0(m)], and the coordinates of the matching point in the comparison image CG that corresponds to it are [x1(m), y1(m)]. In this case, let vec01(m) be the vector quantity of the coordinate difference from the feature point of the object image TG to the matching point of the comparison image. vec01(m) = [Δx(m), Δy(m)] = [x1(m)-x0(m), y1(m)-y0(m)] In this case, the presence of natural objects such as street trees tends to result in many incorrect feature point matches. Furthermore, the correlation between feature points matched to natural objects like street trees and their nearby vec01 points tends to be low, while the correlation between vec01 points of artificial objects such as signs and buildings and their nearby vec01 points tends to be high. In other words, cosine similarity [vec01(natural object), vec01(neighboring feature points of the natural object)] < cosine similarity [vec01(artificial object), vec01(neighboring feature points of the artificial object)] Therefore, by using this relationship to weight each feature point, we can perform preferential feature point matching for artifacts.

[0143] (Example 6) Figure 22 shows a simpler implementation of Example 5. In Figure 22, feature point P3 represents a feature point in the object image TG. In Figure 22, the tip of the dashed line extending from feature point P3 indicates the position of feature point P3 in the corresponding comparison image CG. Since feature points of natural objects are often matched to completely unrelated locations, the attention level is set to 0 when the norm of vec01 (|vec01(m)|) is greater than or equal to the threshold. In other words, in Figure 22, feature points P3 with long dashed lines have an attention level of 0.

[0144] If we define the similarity evaluation value as (total number of feature point matches - number of matches with an attention level of 0) / total number of feature point matches, then even if natural objects are present, the evaluation value will decrease outside of the same scene, and a clear peak P1 can be obtained as shown in Figure 23.

[0145] Thus, the first and second local features are features of feature point P3. Each feature includes a combination of multiple numerical values. The feature calculation unit 162 determines whether the coordinate change from the first image region to the similar second image region at feature point P3 is greater than or equal to a threshold. If it determines that the coordinate change from the first image region to the similar second image region at feature point P3 is greater than or equal to a threshold, the feature point is excluded from the features. The similarity calculation unit 164 calculates the similarity based on the variance of the coordinate changes of the feature points that were not excluded. Therefore, images of the same scene can be detected with high accuracy from the captured video.

[0146] (Example 7) As shown in Figure 24, the features of vec01(m) limited to the highly significant feature point P3 tend to show less variation when they are in the same scene. Therefore, when evaluating the variance or standard deviation, it is found to be minimal in the same scene. In other words, as shown in Figure 24, the peak P1 at the lower limit of the standard deviation corresponds to the same scene. In this embodiment as well, a clear peak P1 was obtained. [Explanation of Symbols]

[0147] 1. Information Processing Device 162 Feature Calculation Unit 164 Similarity calculation unit 166 Attention Calculation Unit 168 Division

Claims

1. An information processing device for identifying the same object contained in different images, A feature calculation unit calculates a first local feature quantity that shows the local feature quantity of an object image including the object, and calculates a second local feature quantity that shows the local feature quantity of a comparison image compared with the object image. A unit for calculating the degree of attention based on the first local feature, the second local feature, or both; A similarity calculation unit compares the first local feature and the second local feature, applies the weighting of the degree of attention, and calculates the similarity between the object image and the comparison image. An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the attention-gathering calculation unit reduces the attention-gathering level for natural objects that change over time, such as the sky, clouds, and vegetation.

3. The information processing apparatus according to claim 1 or 2, wherein the attention calculation unit calculates the attention level such that the attention level of the feature quantity with high sparsity is high.

4. The first local feature is a multidimensional feature vector consisting of a combination of multiple numerical elements, The information processing apparatus according to claim 1 or 2, wherein the attention level calculation unit calculates the attention level based on the maximum value of the plurality of numerical elements.

5. The first local feature is a multidimensional feature vector consisting of a combination of multiple numerical elements, The information processing apparatus according to claim 1 or claim 2, wherein the attention level calculation unit calculates the attention level based on the number of items whose value among the plurality of numerical elements is 0.

6. The system further comprises a division unit that divides the object image into a plurality of first image regions and the comparison image into a plurality of second image regions, The feature calculation unit calculates the first local feature for each of the plurality of first image regions, and calculates a second local feature representing the feature for each of the plurality of second image regions. The information processing apparatus according to claim 1 or 2, wherein the similarity calculation unit compares the first local feature quantity with the second local feature quantity of the corresponding second image region for each of the first image regions, and calculates the similarity between the object image and the comparison image.

7. The information processing apparatus according to claim 6, wherein the attention level calculation unit identifies a similar second image region, which is the image region with the closest features among a plurality of second image regions, based on the first local feature quantity and the second local feature quantity, calculates the difference in coordinate change from the first image region to the similar second image region, and calculates the attention level of the first image region and / or the second image region, or either one of them, according to the correlation coefficient between the difference and the difference in coordinate change similarly calculated in the surrounding region.

8. The first local feature and the second local feature are feature quantities of feature points, The aforementioned feature includes a combination of multiple numerical values, The feature calculation unit determines whether the amount of coordinate change from the first image region to the similar second image region at the feature point is greater than or equal to a threshold, and if it determines that the amount of coordinate change from the first image region to the similar second image region at the feature point is greater than or equal to a threshold, it excludes the feature point from the feature calculation. The information processing apparatus according to claim 7, wherein the similarity calculation unit calculates the similarity based on the variance of the coordinate changes of the feature points that were not excluded.

9. An information processing method for identifying the same object contained in different images, A step of calculating a first local feature quantity that represents the local feature quantity of an object image including the aforementioned object, The steps include: calculating a second local feature quantity that represents the local feature quantity of a comparison image compared with the aforementioned object image; A step of calculating the degree of attention based on the first local feature, the second local feature, or both; The steps include comparing the first local feature and the second local feature, weighting the degree of attention, and calculating the similarity between the object image and the comparison image, An information processing method that encompasses this.

10. A program that performs an information processing method for identifying the same object contained in different images, On the computer, A step of calculating a first local feature quantity that represents the local feature quantity of an object image including the aforementioned object, The steps include: calculating a second local feature quantity that represents the local feature quantity of a comparison image compared with the aforementioned object image; A step of calculating the degree of attention based on the first local feature, the second local feature, or both; The steps include comparing the first local feature and the second local feature, weighting the degree of attention, and calculating the similarity between the object image and the comparison image, A program that executes something.

Images