Information processing program, information processing method and information processing unit

Abstract

To acquire a plurality of proper samples to be used for prototype-kNN.SOLUTION: An information processing unit 100 calculates, for each pair of two frames 111 succeeding with time, a first index value indicative of how much regions including objects, specified respectively from the two frames 111 of the pair, overlap. The information processing unit 100 calculates, for each pair of two frames 111 succeeding with time, a second index value indicative of how much predetermined kinds of feature quantities of two frames 111 of the pair overlap. The information processing unit 100 specifies the remaining frames 111 by removing a certain number of frames 111 before and after the pairs in which at least first index values are less than a first threshold or second indexes are less than a second threshold from the series of frames 111. The information processing unit 100 extracts frames 111 such that similarity degrees of predetermined kinds of feature quantities to other frames 111 among the remaining frames 111 meet predetermined conditions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Conventionally, there are machine learning models that recognize areas where objects appear within frames of video image data and calculate confidence scores that represent the likelihood that the object belongs to each of multiple classes. Here, there are patch attacks that degrade the accuracy of machine learning models by including specific patches within the frames. These specific patches are also called adversarial patches. Patch attacks reduce the confidence scores for classes to which the machine learning model correctly assigns the object, while increasing the confidence scores for classes to which the object incorrectly assigns the object. In response to this, there is a method called prototype-kNN (k-Nearest Neighbor) that detects patch attacks by preparing multiple samples of areas where objects appear.

[0003] Prior art includes, for example, a technique for excluding target images classified as containing structures other than overhead lines from processing. Another technique includes, for example, a technique for selecting a composite image as training data if the degree to which a first component image placed in the background is occluded by a second component image placed in the foreground is equal to or less than an upper limit. Another technique includes, for example, a technique for determining a media classification result using a first machine learning model and a technique for determining whether or not to share the media using a second machine learning model. Another technique includes, for example, a technique for determining a label corresponding to an input vector based on the similarity between the input vector and one or more training vectors. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-036054 [Patent Document 2] Japanese Patent Publication No. 2022-026456 [Patent Document 3] US Patent Application Publication No. 2018 / 0374105 [Patent Document 4] US Patent Application Publication No. 2020 / 0210888 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional techniques, it is difficult to obtain multiple appropriate samples to use in prototype-kNN. For example, when generating samples by extracting frames at regular time intervals from a series of frames taken of the same object while changing the shooting direction, many samples will have similar image features, which can degrade the performance of prototype-kNN.

[0006] In one aspect, the present invention aims to obtain a plurality of suitable samples to be used in a prototype-kNN. [Means for solving the problem]

[0007] According to one embodiment, an information processing program, an information processing method, and an information processing device are proposed that perform segmentation on each of a series of frames that show an object, thereby identifying an area in the frame in which the object is shown, determine for each pair of two chronologically consecutive frames in the series of frames whether a first index value representing the degree of overlap between the identified areas in each of the two frames in the pair is equal to or greater than a first threshold, determine for each pair of two chronologically consecutive frames in the series of frames whether a second index value representing the degree of similarity between a predetermined type of feature in each of the two frames in the pair is equal to or greater than a second threshold, identify remaining frames obtained by removing from the series of frames a certain number of frames before and after each chronologically consecutive pair, including pairs determined to have at least the first threshold or the second threshold, and extract frames from the identified remaining frames whose degree of similarity with other frames in the predetermined type of feature satisfies a predetermined condition. [Effects of the Invention]

[0008] According to one aspect, it is possible to obtain a plurality of suitable samples to be used in the prototype-kNN. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of an information processing method according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of an information processing system 200. As shown in FIG. [Figure 3] FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. As shown in FIG. [Figure 4] FIG. 4 is a block diagram showing an example of the functional configuration of the information processing device 100. As shown in FIG. [Figure 5] FIG. 5 is an explanatory diagram (part 1) showing an example of the operation of the information processing device 100. [Figure 6] FIG. 6 is an explanatory diagram (part 2) showing an example of the operation of the information processing device 100. [Figure 7] FIG. 7 is a flowchart (part 1) illustrating an example of the overall processing procedure. [Figure 8] FIG. 8 is a flowchart (part 2) illustrating an example of the overall processing procedure. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an information processing program, an information processing method, and an information processing device according to embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0011] (An example of an information processing method according to an embodiment) 1 is an explanatory diagram illustrating an example of an information processing method according to an embodiment. The information processing device 100 is a computer for acquiring a plurality of appropriate samples to be used in a prototype-kNN. The information processing device 100 is, for example, a server or a PC (Personal Computer).

[0012] Conventionally, there exists a machine learning model called an object detection model. The object detection model has the function of recognizing an area in a frame where an object is captured, identifying the type of object, and calculating a confidence score that indicates the likelihood of classifying the object into one of multiple classes. The confidence score is a value between 0 and 1. Examples of object detection models include YOLO and Faster RCNN (Region Based Convolutional Neural Networks). For information on YOLO, see, for example, Reference 1 below. For information on Faster RCNN, see, for example, Reference 2 below.

[0013] Reference 1: “YOLO: Real-Time Object Detection”, [online], [Retrieved June 20, 2023], Internet <URL: https: / / pjreddie.com / darknet / yolo / >

[0014] Reference 2: Ren, Shaoqing, et al. “Faster r-cnn: Towards real-time object detection with region proposal networks.” Advances in neural information processing systems 28 (2015).

[0015] Here, there is a patch attack that deteriorates the accuracy of an object detection model by projecting an adversarial patch into a frame. For example, an adversarial patch may be projected into a frame capturing an object by attaching an adversarial patch to the object. For example, an adversarial patch may be projected into a frame capturing an object by synthesizing an adversarial patch into the frame capturing an object. A patch attack reduces the confidence of the object detection model in the correct class that classifies the object, and increases the confidence of the object in the incorrect class that classifies the object. Therefore, a patch attack results in the object being incorrectly identified as belonging to a certain class. For more information on patch attacks, see, for example, References 3 and 4 below.

[0016] Reference 3: Brown, Tom B., et al. “Adversarial patch.” arXiv preprint arXiv:1712.09665 (2017).

[0017] Reference 4: Liu, Xin, et al. “Dpatch: An adversarial patch attack on object detectors.” arXiv preprint arXiv:1806.02299 (2018).

[0018] In response to this, there is a method called prototype-kNN, which detects patch attacks by preparing multiple samples of areas containing objects. The samples are called, for example, prototypes. Prototype-kNN uses the feature space of SIFT (Scale Invariant Feature Transform), for example. kNN is a method that determines the class into which a target frame is classified based on k samples that exist near the target frame in the SIFT feature space.

[0019] Here, it is preferable that the multiple samples used in prototype-kNN include many samples with different image features, but it is not preferable that many samples with similar image features are included. For example, there is a problem that the accuracy of detecting patch attacks using prototype-kNN tends to decrease as the number of samples with different image features decreases. For example, there is a problem that the processing time required to detect patch attacks using prototype-kNN tends to increase as the number of samples with similar image features increases.

[0020] Therefore, it is desirable to prepare a plurality of suitable samples to be used in prototype-kNN. However, it has been difficult to obtain a plurality of suitable samples to be used in prototype-kNN in the past.

[0021] For example, one possible approach is to extract frames at regular time intervals from a series of frames capturing the same object while changing the shooting direction, and then segment each of the extracted frames to select the area containing the object as a sample. However, due to the segmentation accuracy of this approach, it is possible that inappropriate areas, such as areas containing background other than the object or areas that do not contain the entire object, may be selected as samples for use in prototype-kNN. This can result in a decrease in the accuracy of patch attack detection using prototype-kNN. Another problem is that the number of samples with similar image features tends to increase, which increases the processing time required to detect patch attacks using prototype-kNN.

[0022] For example, Method 2 can be considered, in which an operator extracts frames from a series of frames, cuts out an area in which an object appears from each of the extracted frames, and uses the extracted frame as a sample. However, Method 2 has the problem of increasing the workload on the operator.

[0023] Therefore, in this embodiment, an information processing method capable of acquiring a plurality of appropriate samples to be used in prototype-kNN will be described.

[0024] 1, the information processing device 100 acquires a series of frames 111 showing an object. The information processing device 100 acquires the series of frames 111 included in video data 110, for example.

[0025] (1-1) The information processing device 100 performs segmentation on each of a series of frames 111 to identify an area in the frame 111 in which an object is captured.

[0026] (1-2) For each pair of two chronologically consecutive frames 111 among the series of frames 111, the information processing device 100 calculates a first index value that indicates the degree of overlap between regions identified from each of the pair of two frames 111. The first index value is, for example, an index value based on IoU (Intersection over Union). For each pair of two frames 111, the information processing device 100 determines whether the calculated first index value is equal to or greater than a first threshold value.

[0027] This allows the information processing device 100 to obtain a guideline for identifying regions that are inappropriate as samples to be used in the prototype-kNN. For example, the regions identified from each of the two frames 111 of a pair for which the first index value is determined to be less than the first threshold value are considered to have a relatively high probability of including background, and are therefore considered to be inappropriate as samples.

[0028] (1-3) For each pair of two chronologically consecutive frames 111 in the series of frames 111, the information processing device 100 calculates a second index value that indicates the degree of similarity between a predetermined type of feature amount in each of the two frames 111 of the pair. The second index value is, for example, an index value based on SIFT. For each pair of two frames 111, the information processing device 100 determines whether the calculated second index value is equal to or greater than a second threshold value.

[0029] This allows the information processing device 100 to obtain a guideline for identifying regions that are inappropriate as samples to be used in the prototype-kNN. For example, the regions identified from each of the two frames 111 of a pair for which the second index value is determined to be less than the second threshold value are considered to have a relatively high probability of including other objects, and are therefore considered to be inappropriate as samples.

[0030] (1-4) The information processing device 100 identifies a pair of frames 111 that are determined to be less than at least the first threshold or the second threshold. In the example of FIG. 1, the information processing device 100 specifically identifies a pair of frames 111-1 and 111-2. The information processing device 100 identifies the remaining frames 111 obtained by removing a certain number of frames 111 before and after the pair that is consecutive in time and includes the identified pair from the series of frames 111. The certain number is set in advance by the user, for example.

[0031] 1, the information processing device 100 specifically removes a plurality of frames 111, from frame 111-3 to frame 111-4, from a series of frames 111. Specifically, as a result of the removal, the information processing device 100 identifies a plurality of frames 111, from frame 111-5 to frame 111-6, and a plurality of frames 111, from frame 111-7 to frame 111-8, as remaining frames 111. In this way, the information processing device 100 can remove frames 111 that include areas that are not suitable as samples, and can leave frames 111 that include areas that are suitable as samples.

[0032] (1-5) The information processing device 100 extracts frames 111 from the identified remaining frames 111, the frames 111 whose degree of similarity of a predetermined type of feature quantity with other frames 111 satisfies a predetermined condition. The predetermined condition is, for example, that the degree of similarity of the predetermined type of feature quantity with other frames 111 is relatively small. This enables the information processing device 100 to thin out frames 111 with similar image features from frames 111 that include areas suitable as samples.

[0033] Therefore, the information processing device 100 can easily obtain a plurality of appropriate samples to be used in the prototype-kNN. Specifically, the information processing device 100 can generate appropriate samples to be used in the prototype-kNN based on the region identified from the extracted frame. The information processing device 100 can improve the accuracy of detecting patch attacks using the prototype-kNN. Furthermore, the information processing device 100 can reduce the processing time required to detect patch attacks using the prototype-kNN.

[0034] Here, the case where the functions of the information processing device 100 are realized by a single computer has been described, but this is not limiting. For example, the functions of the information processing device 100 may be realized by cooperation of multiple computers. For example, the functions of the information processing device 100 may be realized on the cloud.

[0035] Here, the case has been described in which the information processing device 100 calculates, for each pair of two chronologically consecutive frames 111, a first index value that indicates the degree of overlap between regions identified from each of the two frames 111 of the pair, but this is not limiting. For example, the information processing device 100 may receive a first index value from another computer that has a function for calculating a first index value.

[0036] Here, the case has been described in which the information processing device 100 calculates, for each pair of two chronologically consecutive frames 111, a second index value that indicates the degree of similarity between a predetermined type of feature amount in each of the two frames 111 of the pair, but this is not limiting. For example, the information processing device 100 may receive a second index value from another computer that has a function of calculating the second index value.

[0037] (An example of the information processing system 200) Next, an example of an information processing system 200 to which the information processing device 100 shown in FIG. 1 is applied will be described with reference to FIG.

[0038] 2 is an explanatory diagram showing an example of an information processing system 200. In FIG. 2, the information processing system 200 includes an information processing device 100 and one or more image capturing devices 201.

[0039] In the information processing system 200, the information processing device 100 and the image capturing device 201 are connected via a wired or wireless network 210. The network 210 is, for example, a local area network (LAN), a wide area network (WAN), or the Internet.

[0040] The information processing device 100 is a computer for realizing the prototype-kNN. The information processing device 100 receives a series of frames from the image capturing device 201. The information processing device 100 performs segmentation on each of the series of frames to identify an area in which an object is captured in the frame.

[0041] The information processing device 100 calculates, for each pair of two chronologically consecutive frames in a series of frames, a first index value that indicates the degree of overlap between regions identified from each of the two frames of the pair. The first index value is, for example, an index value based on IoU. The information processing device 100 determines, for each pair of two frames, whether the calculated first index value is equal to or greater than a first threshold.

[0042] The information processing device 100 calculates, for each pair of two consecutive frames in a series of frames, a second index value that indicates the degree of similarity between a predetermined type of feature amount in each of the two frames in the pair. The second index value is, for example, an index value based on SIFT. The information processing device 100 determines, for each pair of two frames, whether the calculated second index value is equal to or greater than a second threshold value.

[0043] The information processing device 100 identifies pairs of frames determined to have a value less than at least a first threshold or a second threshold from among the series of frames. The information processing device 100 identifies remaining frames obtained by removing a certain number of frames before and after the identified pair, which are consecutive in time series, from the series of frames. The information processing device 100 extracts frames from the identified remaining frames whose degree of similarity of a predetermined type of feature with other frames satisfies a predetermined condition.

[0044] The information processing device 100 uses the regions identified from each of the extracted frames as samples to be used for the prototype-kNN. The information processing device 100 realizes the prototype-kNN using the samples. The information processing device 100 is, for example, a server or a PC.

[0045] The image capturing device 201 is a computer that generates frames that capture an object. The image capturing device 201 includes, for example, multiple image capturing elements. The image capturing device 201 generates frames that capture a specific area using the multiple image capturing elements. If an object is present in the specific area, the image capturing device 201 will generate frames that capture the object. The image capturing device 201 transmits a series of consecutively generated frames to the information processing device 100. The image capturing device 201 is, for example, a digital camera. The image capturing device 201 is, for example, a fixed camera. The image capturing device 201 may be, for example, mobile. The image capturing device 201 is, for example, a surveillance camera.

[0046] (Application example of information processing system 200) Next, an application example of the information processing system 200 will be described. The information processing system 200 can be applied, for example, to supporting checkout operations at a cash register by recognizing products displayed in a frame. For example, a patch attack can be considered, in which a hostile patch is attached to a product to cause the product to be mistakenly recognized. In response to this, the information processing device 100 can implement a prototype-kNN that can accurately detect patch attacks. The information processing device 100 can easily avoid damage caused by mistakenly recognizing a product. The information processing device 100 can easily prevent a legitimate customer who is not an attacker from being mistakenly recognized as an attacker. As a result, the information processing device 100 can improve customer convenience and safety.

[0047] The information processing system 200 can be applied to, for example, searching for wanted or missing persons, or supporting surveillance work to detect intruders or suspicious individuals by recognizing people captured in a frame. For example, a patch attack can be considered, in which a hostile patch is attached to a person's face to cause the person to be mistakenly recognized. In response to this, the information processing device 100 can implement a prototype-kNN to accurately detect patch attacks. The information processing device 100 can easily avoid damage caused by mistakenly recognizing a person. The information processing device 100 can easily prevent a legitimate person, who is not a suspicious person, from being mistakenly recognized as a suspicious person.

[0048] (Example of hardware configuration of information processing device 100) Next, an example of the hardware configuration of the information processing device 100 will be described with reference to FIG.

[0049] Fig. 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. In Fig. 3, the information processing device 100 has a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. Furthermore, each component is connected to each other by a bus 300.

[0050] Here, CPU 301 is responsible for overall control of information processing device 100. Memory 302 includes, for example, a read-only memory (ROM), a random access memory (RAM), and a flash ROM. Specifically, for example, the flash ROM or ROM stores various programs, and RAM is used as a work area for CPU 301. The programs stored in memory 302 are loaded into CPU 301, causing CPU 301 to execute coded processes.

[0051] The network I / F 303 is connected to the network 210 via a communication line, and is connected to other computers via the network 210. The network I / F 303 manages the internal interface with the network 210 and controls the input and output of data from other computers. The network I / F 303 is, for example, a modem or a LAN adapter.

[0052] The recording medium I / F 304 controls reading and writing of data from and to the recording medium 305 under the control of the CPU 301. The recording medium I / F 304 is, for example, a disk drive, a solid state drive (SSD), or a universal serial bus (USB) port. The recording medium 305 is a non-volatile memory that stores data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, or a USB memory. The recording medium 305 may be detachable from the information processing device 100.

[0053] In addition to the components described above, the information processing device 100 may also include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, etc. The information processing device 100 may also include a plurality of recording medium I / Fs 304 and recording media 305. The information processing device 100 may also not include the recording medium I / Fs 304 and recording media 305.

[0054] (Example of functional configuration of information processing device 100) Next, an example of the functional configuration of the information processing device 100 will be described with reference to FIG.

[0055] 4 is a block diagram showing an example of the functional configuration of the information processing device 100. The information processing device 100 includes a storage unit 400, an acquisition unit 401, an identification unit 402, a first determination unit 403, a second determination unit 404, a removal unit 405, an extraction unit 406, and an output unit 407.

[0056] The storage unit 400 is realized by, for example, a storage area such as the memory 302 or the recording medium 305 shown in Fig. 3. In the following, a case where the storage unit 400 is included in the information processing device 100 will be described, but this is not limiting. For example, the storage unit 400 may be included in a device different from the information processing device 100, and the stored contents of the storage unit 400 may be accessible from the information processing device 100.

[0057] The acquiring unit 401 to the output unit 407 function as an example of a control unit. Specifically, the acquiring unit 401 to the output unit 407 realize their functions by causing the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 shown in Fig. 3, or by the network I / F 303. The processing results of each functional unit are stored in a storage area such as the memory 302 or the recording medium 305 shown in Fig. 3, for example.

[0058] The storage unit 400 stores various information that is referenced or updated in the processing of each functional unit. The storage unit 400 stores, for example, a series of frames. A frame is image data. A series of frames is, for example, a plurality of image data included in moving image data. A series of frames may be, for example, a plurality of image data captured in succession, rather than moving image data. A series of frames is acquired, for example, by an acquisition unit 401.

[0059] The storage unit 400 stores, for example, a region in a series of frames in which the target object is captured, in association with each of the frames. The region in which the target object is captured is identified, for example, by performing segmentation. The region in which the target object is captured is identified, for example, by the identification unit 402. The storage unit 400 may store, for example, a segmentation model that realizes the segmentation.

[0060] The storage unit 400 stores, for each pair of two consecutive frames in a series of frames, a first index value that indicates the degree of overlap between the regions in which the target object is captured in each of the two frames of the pair. The first index value is, for example, an index value based on IoU. The first index value is calculated, for example, by the first determination unit 403.

[0061] The storage unit 400 stores, for each pair of two consecutive frames in the series of frames, a second index value that indicates the degree of similarity between a predetermined type of feature amount in each of the two frames in the pair. The second index value is, for example, an index value based on SIFT. The second index value is calculated, for example, by the second determination unit 404.

[0062] The storage unit 400 stores a plurality of samples used in the prototype-kNN. The samples are, for example, image data. The samples are, for example, image data representing an area in which a target object is captured. The samples are, for example, generated by the extraction unit 406.

[0063] The acquisition unit 401 acquires various types of information used in processing by each functional unit. The acquisition unit 401 stores the acquired various types of information in the storage unit 400 or outputs it to each functional unit. The acquisition unit 401 may also output the various types of information stored in the storage unit 400 to each functional unit. The acquisition unit 401 acquires various types of information based on, for example, a user's operation input. The acquisition unit 401 may also receive various types of information from, for example, a device different from the information processing device 100.

[0064] The acquisition unit 401 acquires, for example, a series of frames. Specifically, the acquisition unit 401 receives the series of frames from another computer. The other computer is, for example, the image capturing device 201. Specifically, the acquisition unit 401 may acquire the series of frames by accepting input of the series of frames based on an operation input by a user.

[0065] The acquisition unit 401 may receive a start trigger that starts processing by any of the functional units. The start trigger may be, for example, a predetermined operational input by a user. The start trigger may be, for example, reception of predetermined information from another computer. The start trigger may be, for example, output of predetermined information by any of the functional units. Specifically, the acquisition unit 401 receives the acquisition of a series of frames as a start trigger that starts processing by the identification unit 402, the first determination unit 403, the second determination unit 404, the removal unit 405, and the extraction unit 406.

[0066] The identification unit 402 performs segmentation on each of a series of frames showing the target object, thereby identifying the area in the frames where the target object is shown. For example, the identification unit 402 applies a segmentation model to each of the series of frames to identify the area in the frames where the target object is shown.

[0067] Specifically, the identification unit 402 applies a segmentation model to each of a series of frames to identify regions in the frames in which one or more objects appear. Specifically, the identification unit 402 uses an IoU technique to identify regions in which the same target object appears among the one or more regions identified for each of the series of frames. This allows the identification unit 402 to identify regions that can be used as samples for prototype-kNN.

[0068] For each pair of two chronologically consecutive frames in the series of frames, the first determination unit 403 calculates a first index value that indicates the degree of overlap between the regions identified by the identification unit 402 from each of the two frames in the pair. For each pair of two chronologically consecutive frames in the series of frames, the first determination unit 403 determines whether the calculated first index value is equal to or greater than a first threshold. The first threshold is set in advance by the user, for example.

[0069] This allows the first determination unit 403 to obtain a guideline for determining whether the region identified by the identification unit 402 from each of the two frames of the pair is appropriate as a sample to be used for prototype-kNN. For example, the first determination unit 403 can determine that the region identified by the identification unit 402 from each of the two frames of the pair, for which the first index value is determined to be less than the first threshold, is relatively likely to contain noise such as background. Therefore, the first determination unit 403 can determine that the region identified by the identification unit 402 from each of the two frames of the pair, for which the first index value is determined to be less than the first threshold, is not appropriate as a sample to be used for prototype-kNN.

[0070] The second determination unit 404 calculates, for each pair of two chronologically consecutive frames in the series of frames, a second index value representing the degree of similarity between a predetermined type of feature amount in each of the two frames in the pair. For example, the second determination unit 404 may calculate, for each pair of two chronologically consecutive frames in the series of frames, a second index value representing the degree of similarity between a predetermined type of feature amount in a region identified by the identification unit 402 from each of the two frames in the pair. The second determination unit 404 determines, for each pair of two chronologically consecutive frames in the series of frames, whether the calculated second index value is equal to or greater than a second threshold value. The second threshold value is, for example, set in advance by a user.

[0071] This allows the second determination unit 404 to obtain a guideline for determining whether the region identified by the identification unit 402 from each of the two frames of the pair is appropriate as a sample to be used for prototype-kNN. For example, the second determination unit 404 can determine that the region identified by the identification unit 402 from each of the two frames of the pair, for which the second index value is determined to be less than the second threshold, is relatively likely to contain noise, such as an object different from the target object. Therefore, the second determination unit 404 can determine that the region identified by the identification unit 402 from each of the two frames of the pair, for which the second index value is determined to be less than the second threshold, is not appropriate as a sample to be used for prototype-kNN.

[0072] The removal unit 405 identifies a certain number of frames before and after a pair of frames that are consecutive in time sequence, including a pair determined to have a value less than the first threshold or less than the second threshold, from among the series of frames. The certain number is set in advance by, for example, a user. The certain number is, for example, 10. The certain number may also be, for example, 2. The certain number of frames may, for example, be only the pair.

[0073] For example, the removal unit 405 identifies a certain number of frames before and after a pair of frames determined to be less than the first threshold and equal to or greater than the second threshold from among a series of frames. For example, the removal unit 405 identifies a certain number of frames before and after a pair of frames determined to be greater than or equal to the first threshold and less than the second threshold from among a series of frames. For example, the removal unit 405 identifies a certain number of frames before and after a pair of frames determined to be less than the first threshold and less than the second threshold from among a series of frames.

[0074] The removal unit 405 identifies remaining frames obtained by removing the specified number of frames from the series of frames. In this way, the removal unit 405 can remove frames including regions that are not suitable as samples to be used for prototype-kNN from the series of frames, and leave frames including regions that are suitable as samples to be used for prototype-kNN.

[0075] The extraction unit 406 extracts frames from the identified remaining frames whose degree of similarity of a predetermined type of feature quantity with other frames satisfies a predetermined condition. The predetermined condition is, for example, that the degree of similarity of a predetermined type of feature quantity with other frames is small. The predetermined condition is, for example, that the sum of third index values ​​representing the degree of similarity with each of the other frames is less than a third threshold. The predetermined condition is, for example, that the remaining frames are a predetermined number of frames whose sum of third index values ​​representing the degree of similarity with each of the other frames is smallest. The predetermined number is, for example, set in advance by the user.

[0076] For example, for each of the identified remaining frames, the extraction unit 406 calculates the sum of index values ​​representing the degree of similarity of a predetermined type of feature between that frame and each of the other identified remaining frames. The extraction unit 406 then extracts, for example, a predetermined number of frames from the identified remaining frames, starting with those with the smallest calculated sum. This allows the extraction unit 406 to retain an appropriate number of frames that include regions suitable as samples to be used in prototype-kNN. The extraction unit 406 can thin out frames with similar image features from the remaining frames that include regions suitable as samples. Therefore, the extraction unit 406 can retain an appropriate number of frames to reduce the processing time required for detecting patch attacks using prototype-kNN.

[0077] For example, the extraction unit 406 may extract frames from the identified remaining frames whose calculated sum is less than a third threshold. This allows the extraction unit 406 to leave an appropriate number of frames that include regions to be used as samples for prototype-kNN. The extraction unit 406 can thin out frames with similar image features from the remaining frames that include regions appropriate for samples. This allows the extraction unit 406 to leave an appropriate number of frames so as to reduce the processing time required to detect patch attacks using prototype-kNN.

[0078] The extraction unit 406 forms a prototype-kNN based on the region identified by the identification unit 402 from each of the extracted frames. The extraction unit 406 adopts the region identified by the identification unit 402 from each of the extracted frames as a sample to be used for the prototype-kNN, and forms the prototype-kNN.

[0079] This enables the extraction unit 406 to detect patch attacks using prototype-kNN. The extraction unit 406 can reduce the processing time required to detect patch attacks using prototype-kNN. The extraction unit 406 can improve the accuracy of detecting patch attacks using prototype-kNN.

[0080] The output unit 407 outputs the processing result of at least one of the functional units. The output format is, for example, display on a display, printout to a printer, transmission to an external device via the network I / F 303, or storage in a storage area such as the memory 302 or the recording medium 305. In this way, the output unit 407 can notify the user of the processing result of at least one of the functional units, thereby improving the convenience of the information processing device 100.

[0081] The output unit 407 outputs, for example, the region identified by the identification unit 402 from each of the frames extracted by the extraction unit 406. Specifically, the output unit 407 outputs the region identified by the identification unit 402 from each of the frames extracted by the extraction unit 406 so that the region can be referenced by a user. Specifically, the output unit 407 may transmit the region identified by the identification unit 402 from each of the frames extracted by the extraction unit 406 to another computer. This allows the output unit 407 to form a prototype-kNN externally.

[0082] The output unit 407 outputs, for example, the prototype-kNN formed by the extraction unit 406. Specifically, the output unit 407 outputs the prototype-kNN formed by the extraction unit 406 so that it can be referenced by a user. Specifically, the output unit 407 may transmit the prototype-kNN formed by the extraction unit 406 to another computer. In this way, the output unit 407 can make the prototype-kNN formed by the extraction unit 406 available externally.

[0083] (An example of the operation of the information processing device 100) Next, an example of the operation of the information processing device 100 will be described with reference to FIGS.

[0084] 5 and 6 are explanatory diagrams showing an example of the operation of the information processing device 100. In Fig. 5, (5-1) a user photographs an object 500 from each of a plurality of different photographing directions while moving the image capturing device 201. The image capturing device 201 generates a frame group 510 in which the object 500 is captured from each of a plurality of different photographing directions based on an operational input by the user. The frame group 510 includes a plurality of frames 511 that are successively captured over time.

[0085] For convenience of explanation, it is assumed that an index is assigned to each of the multiple frames 511 in ascending order along the time series. The index is, for example, an integer equal to or greater than 1. Specifically, the index is 1, 2, ..., N, where N is the number of frames 511.

[0086] Here, a case has been described in which the user photographs the object 500 while moving the image capturing device 201, but the present invention is not limited to this. For example, the user may photograph the object 500 while moving or rotating the object 500. The image capturing device 201 transmits the generated frame group 510 to the information processing device 100. The information processing device 100 receives the frame group 510 from the image capturing device 201.

[0087] (5-2) The information processing device 100 performs segmentation on each frame 511 to identify an area in the frame 511 in which the object 500 appears. In the following description, the area identified by segmentation may be referred to as a “segmentation range.”

[0088] In the following description, the information processing device 100 extracts, from the frame group 510, a plurality of frames 511 whose segmentation range accuracy is equal to or higher than a certain level and whose image features are different from one another as frames 511 that include an area to be used as prototypes. For example, the segmentation range is considered to have high accuracy if it does not include noise other than the object 500. The noise may be the background or other objects different from the object 500.

[0089] (5-3) When the degree of change in a segmentation range in a short period of time in the frame group 510 is relatively large, the segmentation range is considered to have low accuracy and is not suitable as a prototype. In the following description, the information processing device 100 evaluates the degree of change in the segmentation range based on, for example, the IoU feature between the segmentation ranges and the similarity between the SIFT feature between the frames.

[0090] In response to this, the information processing device 100 calculates, for each pair of two chronologically consecutive frames 511 in the frame group 510, an IoU feature that indicates the degree of overlap between the segmentation ranges identified from each of the two frames 511 of the pair. The IoU feature is, for example, (size of the common part of the two segmentation ranges) / (size of the union of the two segmentation ranges). The size is, for example, an area. The IoU feature is, for example, a value between 0 and 1.

[0091] For each pair of two chronologically consecutive frames 511 in the frame group 510, the information processing device 100 calculates the similarity between the SIFT features in each of the two frames 511 of the pair. SIFT features are features that do not use color information. SIFT features are features that are unlikely to change with changes in the planar scaling and rotation of an object shown in the frame, or changes in lighting, etc. The similarity between SIFT features is, for example, cosine similarity. The similarity between SIFT features may also be calculated based on pairs of feature points based on Euclidean distance, for example. Specifically, the similarity between SIFT features is calculated as follows: (FA ×F B ) / (128×128). F A is the Σ i f A,i f A,i is the feature amount of the i-th feature point in one frame 511 of the pair. B is the Σ i f B,i f B,i is the feature amount of the i-th feature point in the other frame 511 of the pair.

[0092] (5-4) The information processing device 100 stores a first threshold for IoU features set in advance by the user. The information processing device 100 stores a second threshold for similarity set in advance by the user. The information processing device 100 prepares an empty list of indexes to be removed.

[0093] When the calculated IoU feature for a pair of the ith frame 511 and the (i+1)th frame 511 is less than a first threshold, the information processing device 100 registers indices ik to i+k in the list of indices to be removed. k is set in advance by a user, for example. Specifically, k may be set automatically depending on the frame rate. When the calculated similarity for a pair of the ith frame 511 and the (i+1)th frame 511 is less than a second threshold, the information processing device 100 registers indices ik to i+k in the list of indices to be removed. At this time, the information processing device 100 registers the indices in the list of indices to be removed without duplication, for example.

[0094] This allows the information processing device 100 to remove multiple frames 511 present in the vicinity of any pair including a segmentation range that is not suitable as a prototype from targets for acquiring prototypes. The information processing device 100 can remove multiple frames 511 present in the vicinity of any pair including a segmentation range that includes noise such as background from targets for acquiring prototypes, for example, based on the IoU feature. The information processing device 100 can remove multiple frames 511 present in the vicinity of any pair including a segmentation range that includes noise such as other objects, from targets for acquiring prototypes, for example, based on similarity. Next, we will move on to the description of FIG. 6.

[0095] In Fig. 6, (6-1) the information processing device 100 identifies the remaining frames 511 after removing frames 511 whose indexes are present in the list of indexes to be removed from the frame group 510. This allows the information processing device 100 to leave frames 511 that include appropriate segmentation ranges as prototypes from the frame group 510. A graph 600 in Fig. 6 represents a SIFT feature space. The graph 600 represents points corresponding to the remaining frames 511.

[0096] (6-2) The existence of prototypes with similar SIFT features increases the processing time required for the information processing device 100 to detect patch attacks using prototype-kNN. Furthermore, even if prototypes with similar SIFT features exist, it is difficult to improve the accuracy of detecting patch attacks using prototype-kNN.

[0097] Therefore, the information processing device 100 calculates the sum of the similarities of the SIFT features of each of the identified remaining frames 511 with each of the other frames 511 among the remaining frames 511. This allows the information processing device 100 to obtain a guideline for selecting frames 511 that are preferably to be retained as targets for acquiring prototypes. The information processing device 100 selects and extracts a predetermined number of frames 511 from the identified remaining frames 511 in descending order of the calculated sum.

[0098] As a result, the information processing device 100 can leave a predetermined number of frames 511 having diverse image features as targets for acquiring prototypes, as shown in graph 610. The information processing device 100 can remove frames 511 having similar image features. As a result, the information processing device 100 can leave a predetermined number of frames 511 that include an appropriate segmentation range as prototypes. Therefore, the information processing device 100 can reduce the processing time required to detect patch attacks using prototype-kNN.

[0099] (6-3) The information processing device 100 adopts the segmentation ranges identified from each of the extracted frames 511 as prototypes. The information processing device 100 forms a prototype-kNN using the adopted prototypes. The information processing device 100 provides the prototype-kNN to the user.

[0100] This enables the information processing device 100 to detect patch attacks using prototype-kNN. The information processing device 100 can reduce the processing time required to detect patch attacks using prototype-kNN. The information processing device 100 can improve the accuracy of detecting patch attacks using prototype-kNN.

[0101] In this way, the information processing device 100 can remove frames 511 that include segmentation ranges with relatively low accuracy based on the similarity between IoU features and SIFT features. Therefore, the information processing device 100 can remove frames 511 that include segmentation ranges in which the background is erroneously reflected. Furthermore, the information processing device 100 can remove frames 511 that include segmentation ranges in which other objects are erroneously reflected.

[0102] The information processing device 100 can comprehensively remove frames 511 that include segmentation ranges that are inappropriate as prototypes by using both IoU features and similarities between SIFT features. After removing frames 511 that include segmentation ranges that are inappropriate as prototypes, the information processing device 100 can thin out frames 511 that have similar image features. The information processing device 100 can appropriately reduce the number of prototypes, and can reduce the processing time required to detect patch attacks using prototype-kNN.

[0103] When frames 511 having similar image features are thinned out without removing frames 511 including segmentation ranges that are inappropriate as prototypes, frames 511 including segmentation ranges that are inappropriate as prototypes tend to remain. In contrast, the information processing device 100 can leave frames 511 including segmentation ranges that are appropriate as prototypes without leaving frames 511 including segmentation ranges that are inappropriate as prototypes.

[0104] For example, a conventional method may be considered in which frames are extracted at regular time intervals from a series of frames in which the same object is photographed while changing the photographing direction, and the segmentation ranges identified from each of the extracted frames are adopted as prototypes. In the conventional method, segmentation ranges that are not appropriate for prototypes may be adopted as prototypes.

[0105] In response to this, the information processing device 100 can prevent a segmentation range that is inappropriate as a prototype from being adopted as a prototype. For example, when a segmentation range becomes inappropriate due to light reflection at a specific angle of view, the information processing device 100 can prevent the segmentation range from being adopted as a prototype. Furthermore, the information processing device 100 can reduce the workload imposed on the user.

[0106] (Overall processing procedure) 7 and 8, an example of an overall processing procedure executed by the information processing device 100 will be described. The overall processing is realized by, for example, the CPU 301, storage areas such as the memory 302 and the recording medium 305, and the network I / F 303 shown in FIG.

[0107] 7 and 8 are flowcharts showing an example of an overall processing procedure. In Fig. 7, the information processing device 100 uses a segmentation model to identify the segmentation range of each of a series of frames of video data (step S701). The information processing device 100 sets i to 1 (step S702).

[0108] The information processing device 100 reads the i-th frame and the (i+1)-th frame from the series of frames (step S703). i The information processing device 100 sets the IoU feature between the segmentation ranges of the i-th frame and the (i+1)-th frame in Si The similarity between the SIFT feature amounts of the i-th frame and the (i+1)-th frame is set to (step S705).

[0109] The information processing device 100 is i < Threshold 1 and S i < Threshold 2 (step S706). i < Threshold 1 and S i <Threshold 2 (step S706: Yes), the information processing device 100 proceeds to the process of step S707. i ≧ threshold 1 and S i If the threshold value is greater than or equal to threshold 2 (step S706: No), the information processing device 100 proceeds to the process of step S708.

[0110] In step S707, the information processing device 100 registers the indexes of the frames from the ik-th frame to the i+k-th frame in the exclusion list (step S707). Then, the information processing device 100 proceeds to the process of step S708.

[0111] In step S708, the information processing device 100 determines whether i<the number of frames in the video data (step S708). If i<the number of frames in the video data (step S708: Yes), the information processing device 100 sets i to i+1 (step S709) and returns to the processing of step S703. On the other hand, if i≧the number of frames in the video data (step S708: No), the information processing device 100 proceeds to the processing of step S801 in FIG. 8.

[0112] 8, the information processing device 100 sets L to the remaining frames from a series of frames, excluding frames whose indexes are registered in the exclusion list (step S801). The information processing device 100 sets M to a specified number of frames randomly extracted from the remaining frames (step S802). For each combination of two frames in M, the information processing device 100 calculates the similarity between the SIFT features of the two frames in that combination (step S803).

[0113] The information processing device 100 determines whether L is empty (step S804). If L is empty (step S804: Yes), the information processing device 100 proceeds to the process of step S810. On the other hand, if L is not empty (step S804: No), the information processing device 100 proceeds to the process of step S805.

[0114] In step S805, the information processing device 100 randomly selects one frame from L and adds it to M (step S805). For each combination of the added frame and another frame in M, the information processing device 100 calculates the similarity between the SIFT features of the two frames in the combination (step S806). The information processing device 100 identifies the combination with the highest similarity (step S807).

[0115] The information processing device 100 calculates the sum of the similarities between each of the two frames in the identified combination and the other frames (step S808). The information processing device 100 excludes the frame in the identified combination with the larger calculated sum from M (step S809). Then, the information processing device 100 returns to the process of step S804.

[0116] In step S810, the information processing device 100 sets M to prototypes (step S810). Then, the information processing device 100 ends the entire process. This allows the information processing device 100 to set appropriate prototypes.

[0117] Here, the information processing device 100 may change the order of the processes of some of the steps in the flowcharts of Figures 7 and 8. For example, the order of the processes of steps S704 and S705 may be changed. Furthermore, the information processing device 100 may omit the processes of some of the steps in the flowcharts of Figures 7 and 8.

[0118] As described above, the information processing device 100 can identify the area in a series of frames in which an object is captured by performing segmentation on each of the frames. The information processing device 100 can determine, for each pair of two chronologically consecutive frames in the series of frames, whether a first index value representing the degree of overlap between the areas identified in each of the two frames in the pair is equal to or greater than a first threshold. The information processing device 100 can determine, for each pair of two chronologically consecutive frames in the series of frames, whether a second index value representing the degree of similarity between a predetermined type of feature in each of the two frames in the pair is equal to or greater than a second threshold. The information processing device 100 can identify the remaining frames obtained by removing a certain number of frames before and after a pair of chronologically consecutive frames, including a pair determined to have a feature size less than the first threshold or the second threshold, from the series of frames. The information processing device 100 can extract frames from the identified remaining frames whose degree of similarity between a predetermined type of feature and other frames satisfies a predetermined condition. This allows the information processing device 100 to extract frames that are appropriate as targets for acquiring prototypes. The information processing device 100 can make it easier to acquire prototypes.

[0119] According to the information processing device 100, a prototype-kNN can be formed based on a region identified from an extracted frame. This allows the information processing device 100 to use a prototype-kNN that uses appropriate prototypes.

[0120] The information processing device 100 can calculate, for each of the identified remaining frames, the sum of index values ​​representing the degree of similarity of a predetermined type of feature between that frame and each of the other frames among the identified remaining frames. The information processing device 100 can extract a predetermined number of frames from the identified remaining frames, starting from the frame with the smallest calculated sum. This allows the information processing device 100 to extract an appropriate number of frames as targets for obtaining a prototype.

[0121] The information processing device 100 can identify pairs determined to be less than at least the first threshold or the second threshold. The information processing device 100 can identify a certain number of frames, from a frame that is a specified number of frames before the identified pair to a frame that is a specified number of frames after the identified pair. The information processing device 100 can remove the specified number of frames from the series of frames. This allows the information processing device 100 to efficiently remove frames that are not appropriate as targets for obtaining prototypes.

[0122] According to the information processing device 100, a value based on IoU can be calculated as the first index value. This allows the information processing device 100 to use an index value that appropriately represents the degree of overlap between regions identified from each of two frames of a pair. The information processing device 100 can easily extract an appropriate frame as a target for acquiring a prototype.

[0123] According to the information processing device 100, a value based on SIFT can be calculated as the second index value. This allows the information processing device 100 to use an index value that appropriately represents the degree of similarity between predetermined types of feature amounts in each of the two frames of a pair. The information processing device 100 can easily extract an appropriate frame as a target for obtaining a prototype.

[0124] The information processing method described in this embodiment can be realized by executing a prepared program on a computer such as a PC or a workstation. The information processing program described in this embodiment is recorded on a computer-readable recording medium and executed by being read from the recording medium by the computer. The recording medium may be a hard disk, a flexible disk, a CD (Compact Disc)-ROM, an MO (Magneto Optical disc), a DVD (Digital Versatile Disc), or the like. The information processing program described in this embodiment may also be distributed via a network such as the Internet.

[0125] The following additional notes are provided regarding the above-described embodiment.

[0126] (Supplementary Note 1) By performing segmentation on each of a series of frames showing an object, a region in the frame in which the object is shown is identified; for each pair of two consecutive frames in the series of frames, determine whether a first index value representing a degree of overlap between the regions identified from each of the two frames of the pair is equal to or greater than a first threshold value; determining, for each pair of two successive frames in the series of frames, whether or not a second index value representing a degree of similarity between a predetermined type of feature amount in each of the two frames in the pair is equal to or greater than a second threshold value; Identifying residual frames obtained by removing a certain number of frames before and after a pair of frames that are consecutive in time, including a pair determined to have a frame rate less than the first threshold value or less than the second threshold value, from the series of frames; extracting frames from the identified remaining frames, the frames having a degree of similarity between the predetermined type of feature quantity and other frames that satisfies a predetermined condition; An information processing program that causes a computer to execute a process.

[0127] (Supplementary Note 2) A prototype-kNN (k-Nearest Neighbor) is formed based on the region identified from the extracted frame. 2. The information processing program according to claim 1, wherein the information processing program causes the computer to execute the processing.

[0128] (Supplementary Note 3) For each of the identified remaining frames, a sum of index values ​​representing a degree of similarity between the frame and each of the other frames among the identified remaining frames in terms of the predetermined type of feature is calculated. causing the computer to execute a process; The extraction process includes: 3. The information processing program according to claim 1, further comprising: extracting a predetermined number of frames from the identified remaining frames starting from the frame with the smallest calculated sum.

[0129] (Appendix 4) The process to be specified is: An information processing program as described in Appendix 1 or 2, characterized in that a certain number of frames are removed from the series of frames, from a frame that is a specified number of frames before a pair determined to be at least less than the first threshold or the second threshold to a frame that is a specified number of frames after the pair.

[0130] (Supplementary Note 5) The information processing program according to Supplementary Note 1 or 2, wherein the first index value is a value based on IoU (Intersection over Union).

[0131] (Supplementary Note 6) The information processing program according to Supplementary Note 1 or 2, wherein the second index value is a value based on SIFT (Scale Invariant Feature Transform).

[0132] (Supplementary Note 7) By performing segmentation on each of a series of frames showing an object, a region in the frame in which the object is shown is identified; for each pair of two consecutive frames in the series of frames, determine whether a first index value representing a degree of overlap between the regions identified from each of the two frames of the pair is equal to or greater than a first threshold value; determining, for each pair of two successive frames in the series of frames, whether or not a second index value representing a degree of similarity between a predetermined type of feature amount in each of the two frames in the pair is equal to or greater than a second threshold value; Identifying residual frames obtained by removing a certain number of frames before and after a pair of frames that are consecutive in time, including a pair determined to have a frame rate less than the first threshold value or less than the second threshold value, from the series of frames; extracting frames from the identified remaining frames, the frames having a degree of similarity between the predetermined type of feature quantity and other frames that satisfies a predetermined condition; An information processing method characterized in that the processing is executed by a computer.

[0133] (Appendix 8) For each of a series of frames showing an object, a segmentation is performed to identify an area in the frame in which the object is shown; for each pair of two consecutive frames in the series of frames, determine whether a first index value representing a degree of overlap between the regions identified from each of the two frames of the pair is equal to or greater than a first threshold value; determining, for each pair of two successive frames in the series of frames, whether or not a second index value representing a degree of similarity between a predetermined type of feature amount in each of the two frames in the pair is equal to or greater than a second threshold value; Identifying residual frames obtained by removing a certain number of frames before and after a pair of frames that are consecutive in time, including a pair determined to have a frame rate less than the first threshold value or less than the second threshold value, from the series of frames; extracting frames from the identified remaining frames, the frames having a degree of similarity between the predetermined type of feature quantity and other frames that satisfies a predetermined condition; An information processing device comprising a control unit. [Explanation of symbols]

[0134] 100 Information processing device 110 Video data 111,511 frames 200 Information Processing Systems 201 Imaging device 210 Network 300 Bus 301 CPU 302 memory 303 Network I / F 304 Recording Media I / F 305 Recording Media 400 Storage section 401 Acquisition Department 402 Specific section 403 1st Judgment Department 404 Second Judgment Section 405 Removal section 406 Extraction part 407 Output Section 500 objects 510 Frame Group 600,610 graphs

Claims

1. performing segmentation on each of a series of frames showing an object to identify an area of ​​the frame in which the object is shown; determining, for each pair of two consecutive frames in the series of frames, whether a first index value representing a degree of overlap between the regions identified from each of the two frames of the pair is equal to or greater than a first threshold value; determining, for each pair of two successive frames in the series of frames, whether a second index value representing a degree of similarity between a predetermined type of feature amount in each of the two frames in the pair is equal to or greater than a second threshold value; Identifying residual frames obtained by removing a certain number of frames before and after a pair of frames that are successive in time, including a pair determined to have a frame rate less than the first threshold value or less than the second threshold value, from the series of frames; extracting frames from the identified remaining frames, the frames having a degree of similarity between the predetermined type of feature quantity and other frames that satisfies a predetermined condition; An information processing program that causes a computer to execute a process.

2. A prototype-kNN (k-Nearest Neighbor) is formed based on the region identified from the extracted frame.

2. The information processing program according to claim 1, wherein the information processing program causes the computer to execute processing.

3. calculating, for each of the identified remaining frames, a sum of index values ​​representing a degree of similarity between the frame and each of the other frames among the identified remaining frames in terms of the predetermined type of feature amount; causing the computer to execute a process; The extraction process includes:

3. The information processing program according to claim 1, further comprising: extracting a predetermined number of frames from the identified remaining frames in order from the frame with the smallest calculated sum.

4. performing segmentation on each of a series of frames showing an object to identify an area of ​​the frame in which the object is shown; determining, for each pair of two consecutive frames in the series of frames, whether a first index value representing a degree of overlap between the regions identified from each of the two frames of the pair is equal to or greater than a first threshold value; determining, for each pair of two successive frames in the series of frames, whether a second index value representing a degree of similarity between a predetermined type of feature amount in each of the two frames in the pair is equal to or greater than a second threshold value; Identifying residual frames obtained by removing a certain number of frames before and after a pair of frames that are successive in time, including a pair determined to have a frame rate less than the first threshold value or less than the second threshold value, from the series of frames; extracting frames from the identified remaining frames, the frames having a degree of similarity between the predetermined type of feature quantity and other frames that satisfies a predetermined condition; An information processing method characterized in that the processing is executed by a computer.

5. performing segmentation on each of a series of frames showing an object to identify an area of ​​the frame in which the object is shown; determining, for each pair of two consecutive frames in the series of frames, whether a first index value representing a degree of overlap between the regions identified from each of the two frames of the pair is equal to or greater than a first threshold value; determining, for each pair of two successive frames in the series of frames, whether a second index value representing a degree of similarity between a predetermined type of feature amount in each of the two frames in the pair is equal to or greater than a second threshold value; Identifying residual frames obtained by removing a certain number of frames before and after a pair of frames that are successive in time, including a pair determined to have a frame rate less than the first threshold value or less than the second threshold value, from the series of frames; extracting frames from the identified remaining frames, the frames having a degree of similarity between the predetermined type of feature quantity and other frames that satisfies a predetermined condition; An information processing device comprising a control unit.

Images