Program, information processor, and information processing method

Abstract

To provide a program, an information processor, and an information processing method that shorten processing time until collating of an object is confirmed without reducing collating accuracy.SOLUTION: A program of the embodiment causes a computer to function as a detection unit, a collation unit, a voting unit, and a determination unit. The detection unit detects a tracking target area including a tracking target from a frame included in a moving image. The collation unit collates the tracking target using a collation dictionary that stores identification information for identifying a collation target, and acquires, from the collation dictionary, identification information that identifies a collation result of the tracking target in the frame. The voting unit acquires voting data by voting, for each of the tracking targets, on identification information that identifies the collation result acquired for each of the frames. The determination unit determines, on the basis of the voting data, whether or not to confirm the collation, and outputs identification information that identifies the confirmed collation result if the collation has been confirmed.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] As a method for improving the accuracy of matching persons included in a video, best shot selection has been conventionally known. When matching persons, it is better for the whole person to be clearly shown rather than the face or body being cut off, which improves the matching accuracy. Also, even if the whole person is shown, if it is too far away and the person is too small, the matching accuracy will decrease. For matching, it is desirable that the person is framed at just the right angle of view. Best shot selection is a method in which such a condition of just the right angle of view is defined in advance, and an image that satisfies the condition is used as the best shot for matching.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] However, in the prior art, it has been difficult to shorten the processing time until the collation of the object is determined without reducing the collation accuracy.

Means for Solving the Problems

[0006] The program of the embodiment causes a computer to function as a detection unit, a collation unit, a voting unit, and a determination unit. The detection unit detects a tracking target area including a tracking target from frames included in a video. The collation unit collates the tracking target using a collation dictionary that stores identification information for identifying a collation target, and acquires, from the collation dictionary, identification information for identifying a collation result of the tracking target in the frame. The voting unit obtains voting data by voting, for each tracking target, the identification information for identifying the collation result obtained for each frame. The determination unit determines whether or not to confirm the collation based on the voting data, and when the collation is confirmed, outputs the identification information for identifying the confirmed collation result.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of a program, an information processing apparatus, and an information processing method will be described in detail with reference to the accompanying drawings.

[0009] (First Embodiment) In the first embodiment, a case where an object ID for identifying an object appearing in a video captured by a camera or a video is collated will be described.

[0010] [Outline of Information Processing] FIG. 1 is a diagram showing the outline of the information processing of the first embodiment. The information processing apparatus of the first embodiment introduces a voting mechanism to achieve both an improvement in collation accuracy and a reduction in the time to collation confirmation.

[0011] As shown in FIG. 1, in the information processing of the first embodiment, first, an object is detected from an image of one frame included in a video.

[0012] Next, a target area including the detected object is cut out. The target area is represented by coordinate information indicating, for example, two vertices (for example, a set of the lower left vertex and the upper right vertex of a rectangle) that specify a rectangular area. Also, the detected object is identified by a tracking ID.

[0013] Next, the matching (estimation) of the object identified by the tracking ID is performed, and an object ID and an estimation score indicating the certainty of the estimation of the object ID are obtained. For example, the estimation score is represented by a numerical value between 0 and 1, and the larger the numerical value, the higher the certainty of the estimation.

[0014] Next, voting for the object ID and the estimation score obtained by the matching in the frame is performed for each tracking ID. For example, the voting is performed by adding the estimation scores. As shown in FIG. 1, the voting data is represented by a histogram of identification information (in the example of FIG. 1, the object ID) for identifying the matching result.

[0015] Next, a determination of the matching confirmation is made. If the matching is not confirmed, the above-described information processing is performed on the image of the next frame. If the matching is confirmed, the object ID with the highest cumulative estimation score is output.

[0016] Next, an example of the functional configuration of the information processing apparatus according to the first embodiment will be described.

[0017] [Example of Functional Configuration] FIG. 2 is a diagram showing an example of the functional configuration of the information processing apparatus 1 according to the first embodiment. The information processing apparatus 1 according to the first embodiment includes a detection unit 11, a cutout unit 12, a matching unit 13, a voting unit 14, and a determination unit 15. The information processing apparatus 1 according to the first embodiment also stores a dictionary 101 and voting data 102.

[0018] The detection unit 11 detects a tracking target area including a tracking target (an object in the first embodiment) from the frames included in the video. The cutout unit 12 cuts out the tracking target area from the frame.

[0019] The matching unit 13 matches the tracking target using a matching dictionary (dictionary 101 in the first embodiment) that stores identification information (object ID in the first embodiment) for identifying the matching target, and obtains identification information for identifying the matching result in the frame of the tracking target from the dictionary 101. Specifically, the matching unit 13 extracts the feature amount of the tracking target included in the tracking target area, and based on the similarity between the feature amount of the tracking target and the feature amount stored in the dictionary 101, matches the tracking target, thereby obtaining, from the dictionary 101, identification information for identifying the matching result in the frame of the tracking target.

[0020] The voting unit 14 obtains voting data 102 by voting, for each tracking target, the identification information for identifying the matching result obtained for each frame.

[0021] Specifically, when the voting data 102 is represented by a histogram, the voting unit 14 forms a histogram by adding an estimated score indicating the certainty of the matching result when voting, for each tracking target, the identification information for identifying the matching result obtained for each frame. Also, for example, the voting unit 14 forms a histogram by adding a predetermined value (for example, 1) when voting, for each tracking target, the identification information for identifying the matching result obtained for each frame.

[0022] The determination unit 15 determines whether or not to finalize the matching based on the voting data 102, and when the matching is finalized, outputs identification information (object ID in the first embodiment) for identifying the finalized matching result.

[0023] Specifically, when the voting data 102 is represented by a histogram, the determination unit 15 determines whether to finalize the collation based on at least one of the histogram frequency for each piece of identification information for identifying the collation result and the voting probability for each piece of identification information for identifying the collation result. Further, the determination unit 15 may change at least one of the threshold for determining the histogram frequency (for example, the cumulative estimation score) and the threshold for determining the voting probability according to the type of the collation target. For example, the threshold for determining the histogram frequency (for example, the cumulative estimation score) may be set smaller for an object with a lower collation difficulty.

[0024] The dictionary 101 stores an object ID and a feature vector (an example of a feature amount) indicating the features of the object identified by the object ID.

[0025] The voting data 102 is stored for each tracking ID. In the voting data 102 for each tracking ID, the cumulative estimation score for each object ID (voting destination label) for identifying the object estimated in each frame is stored.

[0026] FIG. 3 is a diagram showing an example of the object collation process of the first embodiment. The collation unit 13 extracts a feature vector for collation from the image region cut out by the cut-out unit 12 (feature extraction). The feature vector is, for example, a 512-dimensional vector.

[0027] As the feature extractor for performing feature extraction, for example, a neural network such as a convolutional neural network or a transformer is used. These feature extractors are trained by a method called metric learning. Metric learning is a method of learning a metric (such as distance and similarity) representing the relationship between data, and can be applied to image classification and image search. In metric learning, a metric is learned such that the feature amounts of data with similar meanings are close to each other, and the feature amounts of data with different meanings are far from each other. Methods such as Contrastive loss (Non-Patent Document 2), Triplet loss (Non-Patent Document 3), CosFace (Non-Patent Document 4), and ArcFace (Non-Patent Document 5) are used for metric learning.

[0028] The matching unit 13 matches the object ID by comparing the feature vector obtained from the image region with the feature vector of the object ID registered in the dictionary 101. For example, cosine similarity is used for the matching. Let the input feature vector be q and the feature vector stored in the dictionary be d. Then, the cosine similarity is defined as in the following formula (1).

[0029]

Equation

[0030] Here, · represents the inner product of vectors, and the norm in the denominator on the right side represents the L2 norm. The matching unit 13 calculates the cosine similarity with all the feature vectors in the dictionary 101, and sets the object ID associated with the feature vector having the highest similarity as the matched object ID. In the first embodiment, the cosine similarity obtained here is used as the estimation score.

[0031] [Example of Information Processing] FIG. 4 is a flowchart showing an example of the overall flow of the information processing according to the first embodiment. First, the detection unit 11 receives an input of an image for one frame from a moving image such as a video image previously captured by a camera (step S1).

[0032] Next, the detection unit 11 detects an object from the image by extracting a region including the object to be tracked from the image (step S2). For example, Faster R-CNN (Non-Patent Document 1) or the like is used for object detection. The detection unit 11 assigns a tracking ID for identifying the object to be tracked, and generates a tracking list including the tracking ID of the detected object.

[0033] Note that the detailed flow of step S2 will be described later with reference to FIG. 5.

[0034] Next, the detection unit 11 selects one tracking ID for identifying the object o to be processed from the tracking list (step S3). The subsequent processing is repeatedly performed for each tracking ID in the tracking list.

[0035] Next, the cutting-out unit 12 cuts out an image region including the object o selected in step S3 from the image (step S4). The shape of the image region may be arbitrary. For example, when it is a rectangle, the image region is specified by box coordinates (coordinates for specifying the position of the bounding box) for specifying the rectangle.

[0036] Next, the collation unit 13 obtains an object ID and an estimated score by collating the object o based on the dictionary 101 using the method described with reference to FIG. 3 above (step S5).

[0037] Next, the voting unit 14 performs voting on the object ID and the estimated score obtained in step S5 (step S6).

[0038] Note that the detailed flow of step S6 will be described later with reference to FIG. 6.

[0039] Next, the voting unit 14 determines whether the collation is finalized (step S7). When the collation is finalized (step S7, Yes), the voting unit 14 outputs the object ID for which the collation is finalized (step S8), and the process proceeds to step S9. When the collation is not finalized (step S7, No), step S8 is not executed, and the process proceeds to step S9.

[0040] Note that the detailed flow of step S7 will be described later with reference to FIG. 7.

[0041] Next, the detection unit 11 determines whether all the tracking IDs included in the tracking list have been processed (step S9). When not all the tracking IDs have been processed (step S9, No), the process returns to step S3, and one unprocessed tracking ID is selected from the tracking list. When all the tracking IDs have been processed (step S9, Yes), the process returns to step S1, and the input of the image for the next one frame is accepted.

[0042] Next, the detailed flow of step S2 described above will be described with reference to FIG. 5.

[0043] FIG. 5 is a flowchart showing an example of object detection and tracking processing according to the first embodiment. At the start of the processing, the detection unit 11 selects an image of one frame of the processing target and a tracking list obtained by the processing up to one frame before the current image.

[0044] The tracking list is a list of a tracking ID and box coordinates (bbox: Bounding Box) associated with the tracking ID, and indicates where the object being tracked exists. The format of the box coordinates is, for example, coordinates specifying a rectangle (x coordinate of the upper left corner, y coordinate of the upper left corner, x coordinate of the lower right corner, y coordinate of the lower right corner).

[0045] Note that the initial value of the tracking list is set to the object detected in the first frame included in the video to be processed.

[0046] In object detection and tracking processing, as shown in FIG. 5, a region corresponding to an object is extracted and tracked. In normal object detection, there are labels for various objects such as people and cars, but in the first embodiment, all labels are treated as one object without distinction (the presence or absence of some object is detected and tracked without detecting the type of object).

[0047] First, the detection unit 11 detects an object from the image and generates an object list including the detected object (step S21). When the detection unit 11 (object detection engine) finds an object, it returns box coordinates indicating a rectangle including the object. In the object detection in step S21, an object list in which zero or more detected box coordinates (bbox) are stored is generated.

[0048] Next, the detection unit 11 selects one object b to be processed from the object list generated in step S21 (step S22). Next, the detection unit 11 selects one object o to be processed from the tracking list (step S23).

[0049] For object tracking, a method is used to track the previous object with the largest bounding box overlap using Intersection over Union (IoU). That is, a tracking method is used in which the higher the overlap of the bounding boxes, the higher the likelihood that object o and object b are the same object.

[0050] Specifically, the detection unit 11 calculates the IoU (degree of overlap) between the bounding box of object o and the bounding box of object b (step S24).

[0051] If the IoU is greater than or equal to the threshold (step S25, Yes), the detection unit 11 updates the bounding box of object o in the tracking list with the bounding box of object b (step S26), and the process proceeds to step S29.

[0052] On the other hand, if the IoU is less than the threshold (step S25, No), the detection unit 11 determines whether all objects o included in the tracking list have been processed (step S27). If not all have been processed (step S27, No), the process returns to step S23, and one object o with an unprocessed tracking ID is selected from the tracking list.

[0053] If all have been processed (step S27, Yes), the detection unit 11 adds object b with a new tracking ID to the tracking list as a new object (step S28). Next, the detection unit 11 determines whether all objects b included in the object list have been processed (step S29). If not all have been processed (step S29, No), the process returns to step S22, and one unprocessed object b is selected from the object list.

[0054] If all have been processed (step S29, Yes), the processing for the image of the current frame ends.

[0055] Next, the detailed flow of step S6 described above will be described with reference to FIG. 6.

[0056] FIG. 6 is a flowchart showing an example of the voting process according to the first embodiment. At the start of the process, the voting unit 14 receives a tracking ID, an object ID, and a prediction score of the object ID.

[0057] First, the voting unit 14 determines whether the tracking ID is new (step S41). If the tracking ID is new (step S41, Yes), the voting unit 14 generates new voting data for the tracking ID (step S42).

[0058] If the tracking ID is not new (step S41, No), the voting unit 14 selects the voting data for the tracking ID and adds the estimated score to the bin of the object ID (step S43). Specifically, the voting unit 14 adds the estimated score to the bin of the object ID (the voting destination label included in the voting data) obtained in the matching process of step S5 described above.

[0059] In the first embodiment, the estimated score is added. However, instead of the estimated score, an arbitrary constant (for example, a predetermined value such as 1) may be added. When the arbitrary constant is 1, the cumulative estimated score for each object ID in the voting data indicates the number of votes.

[0060] Next, the detailed flow of step S7 will be described with reference to FIG. 7.

[0061] FIG. 7 is a flowchart showing an example of the determination process according to the first embodiment. At the start of the process, the voting unit 14 selects the voting data of the tracking ID to be determined.

[0062] First, the voting unit 14 calculates the total number of votes cast for the object identified by the tracking ID (step S51). Next, the voting unit 14 obtains the voting probability of each object ID by dividing each bin (each object ID) included in the voting data by the total number of votes (step S52). Next, the voting unit 14 identifies the bin (object ID) with the most votes and obtains the number of votes and the voting probability of that bin (step S53).

[0063] Next, the voting unit 14 determines whether the number of votes obtained in step S53 is equal to or greater than a threshold value (step S54). If the number of votes is less than the threshold value (step S54, No), the voting unit 14 sets the object matching as undetermined and ends the determination process.

[0064] If the number of votes is equal to or greater than the threshold value (step S54, Yes), the voting unit 14 determines whether the voting probability obtained in step S53 is equal to or greater than a threshold value (step S55). If the voting probability is less than the threshold value (step S55, No), the voting unit 14 sets the object matching as undetermined and ends the determination process.

[0065] If the voting probability is equal to or greater than the threshold value (step S55, Yes), the voting unit 14 determines the object matching as the object ID of the bin with the most votes and ends the determination process.

[0066] As described above in the flowchart of FIG. 7, the determination process is a process of determining whether to confirm an answer. When a moving image (for example, a camera image) is input and an answer indicating what the object included in the moving image is is determined, and a certain degree of confidence is obtained, the matching process is interrupted halfway and the answer is determined. In the example of FIG. 7, it is determined that a confidence level is obtained when the number of votes is equal to or greater than the threshold value and the voting probability is equal to or greater than the threshold value.

[0067] As described above, in the information processing apparatus 1 according to the first embodiment, the detection unit 11 detects a tracking target area including a tracking target from the frames included in the video. The matching unit 13 matches the tracking target using a matching dictionary (dictionary 101 in the first embodiment) that stores identification information (object ID in the first embodiment) for identifying a matching target, and obtains, from the matching dictionary, identification information for identifying the matching result in the frame of the tracking target. The voting unit 14 obtains voting data 102 by voting, for each tracking target, the identification information for identifying the matching result obtained for each frame. Then, the determination unit 15 determines whether to confirm the matching based on the voting data 102, and when the matching is confirmed, outputs the identification information (object ID in the first embodiment) for identifying the confirmed matching result.

[0068] According to the information processing apparatus 1 of the first embodiment, the processing time until the collation of the object (the object in the first embodiment) is determined can be shortened without reducing the collation accuracy. Specifically, in the information processing apparatus 1 of the first embodiment, by introducing a voting mechanism, both the collation accuracy and the determination time can be achieved.

[0069] [Verification of effects] When one collation result is obtained for each frame, a simple simulator is created in which one vote is cast for the answer based on the collation result, and the most votes are used as the final answer result. The effect of improving the accuracy by voting was confirmed by a simple simulation (Fig. 8).

[0070] Fig. 8 is a diagram showing the relationship between the number of votes and the collation accuracy in the first embodiment. In the example of Fig. 8, for each engine accuracy (correct answer rate of collation), the relationship between the number of frames and the final correct answer rate is plotted.

[0071] In the case of 2-frame voting, it is considered that the accuracy decreases because the votes are split and the decision-making power decreases, but it can be seen that the accuracy improves significantly with 3 frames or more. Of course, the higher the accuracy of the engine alone, the faster the final correct answer rate converges. As can be seen from this Fig. 8, even if the correct answer rate is only 50% per frame, it can be seen that a correct answer rate of nearly 90% can be obtained by voting 6 frames.

[0072] It is considered that even if the best shot is selected with a correct answer rate of 50% per frame, a correct answer rate of 90% is hardly obtained, but the correct answer rate can be surely increased by voting. In this way, even in a situation where the collation cannot be completed no matter how much time passes with the best shot, the collation can be achieved by introducing a voting mechanism.

[0073] (Modification example 1 of the first embodiment) Next, a first modification example of the first embodiment will be described. In the description of the first modification example, the same descriptions as those in the first embodiment will be omitted, and the differences from the first embodiment will be described. In the first modification example, the case where the tracking target is a face will be described.

[0074] [Example of functional configuration] FIG. 9 is a diagram showing an example of the functional configuration of the information processing apparatus 1-2 according to the first modification example of the first embodiment. The functional configuration of the first modification example is the same as that of the first embodiment. In the first modification example, the detection unit 11 detects a tracking target area including a face from the frames included in the video. The collation unit 13 collates the face of the tracking target using a dictionary 101 that stores identification information (face ID) for identifying the face to be collated, and acquires identification information for identifying the collation result in the frame from the dictionary 101. When the collation is confirmed, the determination unit 15 outputs the confirmed face ID.

[0075] (Second modification example of the first embodiment) Next, a second modification example of the first embodiment will be described. In the description of the second modification example, the same descriptions as those in the first embodiment will be omitted, and the differences from the first embodiment will be described. In the second modification example, the case where the tracking target is a person will be described.

[0076] [Example of functional configuration] FIG. 10 is a diagram showing an example of the functional configuration of the information processing apparatus 1-3 according to the second modification example of the first embodiment. The functional configuration of the second modification example is the same as that of the first embodiment. In the second modification example, the detection unit 11 detects a tracking target area including a person from the frames included in the video. The collation unit 13 collates the person of the tracking target using a dictionary 101 that stores identification information (person ID) for identifying the person to be collated, and acquires identification information for identifying the collation result in the frame from the dictionary 101. When the collation is confirmed, the determination unit 15 outputs the confirmed person ID.

[0077] (Third modification example of the first embodiment) Next, a third modification example of the first embodiment will be described. In the description of the third modification example, descriptions similar to those of the first embodiment will be omitted, and differences from the first embodiment will be described. In the third modification example, the case where the tracking target is a vehicle (for example, an automobile, etc.) will be described.

[0078] [Example of functional configuration] FIG. 11 is a diagram showing an example of the functional configuration of the information processing apparatus 1-4 according to the third modification example of the first embodiment. The functional configuration of the third modification example is the same as that of the first embodiment. In the third modification example, the detection unit 11 detects a tracking target area including a vehicle from the frames included in the video. The collation unit 13 collates the vehicle to be tracked using a dictionary 101 that stores identification information (vehicle ID) for identifying the vehicle to be collated, and acquires, from the dictionary 101, identification information for identifying the collation result in the frame. When the collation is confirmed, the determination unit 15 outputs the confirmed vehicle ID.

[0079] (Second embodiment) Next, the second embodiment will be described. In the description of the second embodiment, descriptions similar to those of the first embodiment will be omitted, and differences from the first embodiment will be described. In the second embodiment, the case where the tracking target is a VQA (Visual Question Answering) target included in the image of each frame will be described.

[0080] [Example of functional configuration] FIG. 12 is a diagram showing an example of the functional configuration of the information processing apparatus 1-5 according to the second embodiment. The information processing apparatus 1-5 of the first embodiment includes a detection unit 11, a cutting-out unit 12, a voting unit 14, a determination unit 15, and a VQA processing unit 16. Further, the information processing apparatus 1-5 of the second embodiment stores voting data 102. In the second embodiment, instead of the collation unit 13 and the dictionary 101, a VQA processing unit 16 is provided.

[0081] The VQA processing unit 16 performs VQA processing on the VQA target included in the image region cut out by the cut-out unit 12. VQA is a process of judging and answering the content from an image in response to an arbitrary question. The biggest feature of VQA is that the question is given as free-form natural language text. Thus, theoretically, it has a high degree of versatility that can handle all matters that can be expressed in text.

[0082] The voting unit 14 obtains voting data by voting, for each tracking target, the identification information that identifies the answer obtained for each frame.

[0083] The determination unit 15 determines whether or not to finalize the collation based on the voting data, and when the collation is finalized, outputs the identification information that identifies the finalized answer.

[0084] (Third Embodiment) Next, the third embodiment will be described. In the description of the third embodiment, the same descriptions as those in the first embodiment will be omitted, and the differences from the first embodiment will be described. In the third embodiment, a case where the information processing apparatus 1-6 further has a function of providing feedback on the processes of voting and confirmation determination to the user will be described.

[0085] [Example of Functional Configuration] FIG. 13 is a diagram showing an example of the functional configuration of the information processing apparatus 1-6 according to the third embodiment. The information processing apparatus 1-6 according to the third embodiment includes a detection unit 11, a cut-out unit 12, a collation unit 13, a voting unit 14, a determination unit 15, and a feedback unit 17. Also, the information processing apparatus 1-6 according to the first embodiment stores a dictionary 101 and voting data 102. In the information processing apparatus 1-6 according to the first embodiment, the feedback unit 17 is added.

[0086] The feedback unit 17 provides feedback on the processes of voting and confirmation determination to the user, for example, by presenting the user with display information that displays feedback information.

[0087] The feedback information includes, for example, a histogram representing voting data (e.g., the voting data shown in FIG. 1). Also, for example, the feedback information includes a progress bar indicating the current number of votes and the progress status until the collation is finalized.

[0088] According to the information processing apparatus 1-6 of the third embodiment, for example, a user can make adjustments such as shortening the time until the collation is finalized by adjusting the threshold value used for the collation based on the feedback information. This can solve problems such as taking too much time to finalize the collation or the collation not being finalized due to overemphasis on the collation accuracy.

[0089] Finally, an example of the hardware configuration of the information processing apparatus 1 (1-2 to 1-6) according to the first to third embodiments will be described.

[0090] [Example of Hardware Configuration] FIG. 14 is a diagram showing an example of the hardware configuration of the information processing apparatus 1 (1-2 to 1-6) according to the first to third embodiments. The information processing apparatus 1 includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0091] Note that the information processing apparatus 1 may not be provided with some of the above configurations. For example, when the information processing apparatus 1 can utilize the input function and display function of an external device, the information processing apparatus 1 may not be provided with the display device 204 and the input device 205.

[0092] The processor 201 executes a program read from the auxiliary storage device 203 into the main storage device 202. The main storage device 202 is a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary storage device 203 is an HDD (Hard Disk Drive), a memory card, or the like.

[0093] The display device 204 is, for example, a liquid crystal display or the like. The input device 205 is an interface for operating the information processing apparatus 1. Note that the display device 204 and the input device 205 may be realized by a touch panel or the like having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0094] For example, the program executed by the information processing apparatus 1 is a file in an installable format or an executable format, and is recorded on a computer-readable storage medium such as a memory card, a hard disk, a CD-RW, a CD-ROM, a CD-R, a DVD-RAM, and a DVD-R, and is provided as a computer program product.

[0095] Also, for example, the program executed by the information processing apparatus 1 may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network.

[0096] Also, for example, the program executed by the information processing apparatus 1 may be provided via a network such as the Internet without being downloaded. Specifically, for example, it may be configured by an ASP (Application Service Provider) type cloud service.

[0097] Also, for example, the program of the information processing apparatus 1 may be configured to be provided by being pre-embedded in a ROM or the like.

[0098] The program executed by the information processing apparatus 1 has a module configuration including functions that can also be realized by the program among the above-described functional configurations. Each of these functions, as actual hardware, is such that the processor 201 reads the program from the storage medium and executes it, whereby each of the above functional blocks is loaded onto the main storage device 202. That is, each of the above functional blocks is generated on the main storage device 202.

[0099] Part or all of each of the functions described above may be implemented by hardware such as an IC (Integrated Circuit) instead of by software.

[0100] Also, each function may be implemented using a plurality of processors 201. In this case, each processor 201 may implement one of the functions or two or more of the functions.

[0101] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention and are included in the invention described in the claims and the equivalent scope thereof.

Description of Reference Numerals

[0102] 1 Information processing apparatus 11 Detection unit 12 Extraction unit 13 Verification unit 14 Voting unit 15 Judgment unit 16 VQA processing unit 17 Feedback unit 201 Processor 202 Main memory device 203 Auxiliary storage device 204 Display device 205 Input device 206 Communication device 210 Bus

Claims

1. A computer, a detection unit that detects a tracking target area including a tracking target from frames included in a video; a collation unit that collates the tracking target using a collation dictionary that stores identification information for identifying a collation target, and obtains identification information for identifying a collation result of the tracking target in the frame from the collation dictionary; a voting unit that obtains voting data by voting, for each tracking target, identification information for identifying the collation result obtained for each frame; a determination unit that determines whether to confirm the collation based on the voting data, and outputs, when the collation is confirmed, identification information for identifying the confirmed collation result; A program for causing the computer to function as such.

2. The voting data is represented by a histogram of identification information for identifying the collation result. The program according to claim 1.

3. When the voting unit votes, for each tracking target, identification information for identifying the collation result obtained for each frame, the voting unit forms the histogram by adding an estimation score indicating the certainty of the collation result. The program according to claim 2.

4. When the voting unit votes, for each tracking target, identification information for identifying the collation result obtained for each frame, the voting unit forms the histogram by adding a predetermined value. The program according to claim 2.

5. The determination unit determines whether to confirm the collation based on at least one of the frequency of the histogram for each piece of identification information for identifying the collation result and the voting probability for each piece of identification information for identifying the collation result. The program according to claim 2.

6. The determination unit changes at least one of a threshold for determining the frequency and a threshold for determining the voting probability according to the type of the collation target. The program according to claim 5.

7. The detection unit generates a tracking list including the detected tracking target, and updates the tracking list each time a new frame is input. The program according to any one of claims 1 to 6.

8. The collation dictionary stores the identification information for identifying the collation target in association with the feature amount of the collation target. The matching unit extracts feature amounts of a tracking target included in the tracking target area, and acquires identification information for identifying a matching result in the frame of the tracking target from the matching dictionary by matching the tracking target based on a similarity between the feature amounts of the tracking target and the feature amounts of the matching target. The program according to any one of claims 1 to 6.

9. The tracking target includes at least one of a face, a person, an object, and a Visual Question Answering (VQA) target. The program according to any one of claims 1 to 6.

10. A detection unit that detects a tracking target area including a tracking target from frames included in a video; A matching unit that matches the tracking target using a matching dictionary that stores identification information for identifying a matching target, and acquires identification information for identifying a matching result in the frame of the tracking target from the matching dictionary; A voting unit that obtains voting data by voting, for each tracking target, identification information for identifying the matching result obtained for each frame; A determination unit that determines whether or not to confirm the matching based on the voting data, and outputs, when the matching is confirmed, identification information for identifying the confirmed matching result; An information processing apparatus comprising the above.

11. A step in which an information processing apparatus detects a tracking target area including a tracking target from frames included in a video; A step in which the information processing apparatus matches the tracking target using a matching dictionary that stores identification information for identifying a matching target, and acquires identification information for identifying a matching result in the frame of the tracking target from the matching dictionary; A step in which the information processing apparatus obtains voting data by voting, for each tracking target, identification information for identifying the matching result obtained for each frame; A step in which the information processing apparatus determines whether or not to confirm the matching based on the voting data, and outputs, when the matching is confirmed, identification information for identifying the confirmed matching result; An information processing method including the above.

Images