Information processing device, information processing method, and program

Abstract

To enable highly efficient collation with less errors.SOLUTION: An information processing device provided herein is configured to detect a target object in an image and perform collation processing for the target object on the basis of similarity between a feature quantity of the detected target object and a feature quantity for collation. The information processing device selects a candidate object blocking the target object to correct a collation processing result according to a result of the selection.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for performing matching of target objects using images. [Background technology]

[0002] In recent years, devices have been proposed that identify a target person by detecting a person region from an image captured by a surveillance camera or the like, extracting features from the person region, and comparing the extracted features with features of previously registered people. Information identifying the target person is used, for example, to detect suspicious behavior in a store or analyze customer traffic patterns. When a person is clearly and continuously captured in a captured video, the person region can be easily detected, and feature comparison can be performed with high accuracy. However, in cases where a person is temporarily occluded by another person or the like and then reappears, the feature extracted from the person region that appears after the occlusion may change from the feature before the occlusion. In such cases, the person before and after the occlusion may be determined to be a different person.

[0003] On the other hand, Patent Document 1 discloses a method for monitoring the periphery of an object region when a tracked object overlaps and is occluded by another object region, and waiting for the object to reappear. Patent Document 2 discloses a method for associating an object involved in occlusion with an object, and performing feature matching with the occluded person when the occlusion is removed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-191354 [Patent Document 2] US Patent Application Publication No. 2009 / 0059007 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology disclosed in Patent Document 1, in a situation where an object that occludes a target object moves, the number of locations where the occluded target object may reappear increases without limit, which may result in an enormous processing load and reduced processing efficiency.Furthermore, with the technology disclosed in Patent Document 2, when an object appears behind an object that occludes a target object, there is a risk of incorrectly comparing whether the object that has appeared is the target object or a newly appeared different object.

[0006] Therefore, an object of the present invention is to enable highly efficient matching with fewer errors. [Means for solving the problem]

[0007] The information processing device of the present invention includes a detection means for detecting a target object from an image, a matching means for performing a matching process for the target object based on a similarity between a feature amount of the detected target object and a feature amount for matching, a selection means for selecting a candidate object that occludes the target object, and a correction means for correcting a result of the matching process by the matching means based on a result of the selection by the selection means. The correction means corrects the result of the matching process when an object different from the selected candidate object appears in the vicinity of the selected candidate object after the target object is no longer detected. It is characterized by: [Effects of the Invention]

[0008] According to the present invention, highly efficient matching with fewer errors is possible. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating a data flow of the information processing device. [Figure 3] FIG. 2 is an explanatory diagram of the positional relationship in the first embodiment. [Figure 4] 4 is a flowchart of information processing according to the first embodiment. [Figure 5] 10 is a flowchart of an occlusion determination process. [Figure 6]FIG. 4 is an explanatory diagram of an occlusion relationship and a feature matching score according to the first embodiment. [Figure 7] 10A and 10B are explanatory diagrams of positional relationships and score correction maps according to the second embodiment. [Figure 8] 10 is a flowchart of a feature amount matching score correction process according to the second embodiment. [Figure 9] FIG. 10 is an explanatory diagram of an occlusion relationship and a feature matching score according to the second embodiment. [Figure 10] 13 is an explanatory diagram of a positional relationship and a score correction map according to the third embodiment. FIG. [Figure 11] FIG. 10 is an explanatory diagram of a shielding relationship in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various situations (usage situation, usage environment, etc.). Furthermore, a configuration may be achieved by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same components will be described with the same reference symbols.

[0011] In this embodiment, an information processing device is exemplified, which detects a target object from a captured image, extracts features from the detected target object, and compares and collates the extracted features with features for matching, which will be described later, to identify the target object. In this embodiment, the target object detected in the captured image is, for example, a person, and all people appearing in the captured image are identified and each person is tracked individually. In this embodiment, information on the identification result of identifying each person to be tracked can be used, for example, to detect suspicious behavior in a store or analyze customer movement lines.

[0012] First Embodiment FIG. 1 is a diagram showing an example of the hardware configuration of an information processing device according to this embodiment. CPU 100 is a central processing unit that performs calculations and logical decisions for various processes by executing a control program stored in ROM (Read Only Memory) 110. ROM 110 also stores an information processing program according to this embodiment, and CPU 100 executes the information processing program to configure each of the functional units: detection unit 111, collation unit 112, determination unit 113, and correction unit 114. Details of the operation of each of these functional units will be described later.

[0013] The RAM (Random Access Memory) 120 is used as a temporary storage area such as the main memory or work area of ​​the CPU 100. The RAM 120 temporarily stores various data, expands programs read from the ROM 110, and makes them executable by the CPU 100. The HDD 130 is a hard disk drive that can store image data and other programs used in the information processing according to this embodiment. An external storage device that functions similarly to the HDD may be used instead of the HDD 130. In this case, the external storage device may be realized, for example, by media (recording media) and an external storage drive for realizing access to the media. Examples of media include a flexible disk (FD), CD-ROM, DVD, USB memory, MO, and flash memory. The external storage device may also be a server device connected via a network.

[0014] The input unit 140 is configured with a keyboard, touch panel, etc., and accepts input from a user. The display unit 150 is configured with a liquid crystal display, etc., and can display various data such as image data and information processing results to a user. The information processing device of this embodiment can communicate with other devices via the communication unit 160. The information processing device may receive instructions from a user from other devices via the communication unit 160, or may output information processing results to other devices. The information processing device of this embodiment is also assumed to acquire captured images from an imaging device (such as a surveillance camera) via the communication unit 160.

[0015] FIG. 2 is a diagram showing a data flow illustrating the flow of data processing in each functional unit of the information processing apparatus of this embodiment. The image acquisition unit 200 acquires, via the communication unit 160, chronologically ordered images captured at regular intervals by an imaging device (camera) (not shown) having a solid-state imaging element such as a CMOS sensor or a CCD sensor. The captured images acquired by the image acquisition unit 200 are then input to the detection unit 111. In this embodiment, the images acquired by the image acquisition unit 200 are assumed to be moving images captured by a surveillance camera or the like installed in a store or the like. Note that in this embodiment, moving image frames are used as an example of chronologically ordered images captured at regular intervals. However, the image acquisition unit 200 may also acquire images obtained by thinning consecutive frames of a moving image at regular intervals, for example. Furthermore, the image acquisition unit 200 may not only acquire chronologically ordered captured images from an imaging device, but also acquire chronologically ordered images stored in an external device such as a server device or external memory. The information processing device of this embodiment may also include an imaging device.

[0016] The detection unit 111 is an image detector that detects a person, which is a detection target object, from a captured image. The detection unit 111 detects an image area of ​​a person from images input in chronological order. In this embodiment, the detection unit 111 detects an image area of ​​a person from an input image using a detection model constructed in advance using, for example, a convolutional neural network (CNN). The detection model is constructed by learning using a large amount of training data consisting of pairs of images containing a person and ground truth images indicating the position of the person in the image. The detection unit 111 inputs a frame to the detection model, extracts an image area of ​​the person from the frame, and outputs the image area to the matching unit 112. In this embodiment, the image area of ​​a person detected from an image is detected as a detection frame area of ​​a predetermined shape, such as a rectangle, surrounding the person. However, for simplicity, the following description will be appropriately referred to as just a "person."

[0017] The matching unit 112 performs a matching process to determine whether or not the person detected by the detection unit 111 is the same person as a person who appeared in a previously captured image. In this embodiment, the matching unit 112 extracts image features (hereinafter referred to as features) from the person detected by the detection unit 111. Then, the matching unit 112 compares the extracted features with features for matching stored in the database 201, thereby performing a matching process to determine whether or not the person detected by the detection unit 111 is the same person as a person who appeared in a previously captured image. Specific examples of features of a person will be described later.

[0018] The database 201 is a storage area for storing matching features for each person used by the matching unit 112 in the matching process. In this embodiment, the matching features stored in the database 201 are matched features for each person resulting from the matching process performed on past frames. The past frames for which the database 201 stores features may be at least one frame prior to the current frame, but may also be frames several frames prior or all past frames. The matching unit 112 calculates the similarity between the features of the person detected from the current frame and the matching features stored in the database 201, and performs a matching process to determine whether the person in the current frame is the same as any of the matched people based on the similarity. Note that, for example, if the person to be matched is known in advance, such as a registered member, the matching features stored in the database 201 may be features prepared in advance for the known registered person.

[0019] Here, when the target object to be detected from an image is a person, the feature is information that quantifies information that can be used to individually identify a person, such as the face, head, upper body, lower body, clothing, gait, and body shape of the person in the image, and serves as an index for person identification. The matching unit 112 calculates a feature matching score that quantifies the similarity between the feature of the person detected from the current frame and the feature for matching for each person in the database 201. For feature comparison, for example, a method of calculating the distance between feature data (such as cosine similarity or Euclidean distance) can be used. That is, the matching unit 112 calculates the distance between the feature extracted from the person in the current frame and the feature for matching for each person in the database 201, and obtains a numerical value (feature matching score value) that indicates that the closer the calculated distance, the higher the similarity. If there is a feature for which the feature matching score is equal to or greater than a predetermined score threshold among the features for matching in database 201, matching unit 112 obtains a matching result that the person in the current frame is the same person as the person in the feature for matching. In this way, matching unit 112 performs matching processing using the features of people in the previous and current frames for each frame in chronological order, and outputs the matching result for each frame to determination unit 113.

[0020] The determination unit 113 performs an occlusion determination process to determine whether a person is occluded based on the matching result by the matching unit 112. If a person who existed in the previous frame is no longer detected in the current frame, the determination unit 113 determines that the person may be occluded by another object based on the matching result for the previous frame and the matching result for the current frame. In this embodiment, a person who existed in the previous frame but is no longer detected in the current frame is referred to as an "undetected person." In this embodiment, candidate objects that may be occluding the undetected person include people other than the undetected person and objects other than people, and hereinafter, candidate objects that may be occluding the undetected person are referred to as "occluding candidate objects."

[0021] The determination unit 113 predicts the position of a person who was detected in the previous frame but is no longer detected in the current frame based on the position of the person in the previous frame or the change in the position of the person over the past few frames, and determines whether an occluding candidate object is present at the predicted position. Furthermore, if an occluding candidate object is present at the predicted position, the determination unit 113 selects the occluding candidate object as occluding the undetected person, i.e., confirms the occluding candidate object. In this way, the determination unit 113 calculates the position of the undetected person in the current frame by a prediction process, and if an occluding candidate object is present at the predicted position, determines that the undetected person is occluded by the occluding candidate object. The determination unit 113 then retains information about the occluding candidate object occluding the undetected person. In other words, information about the undetected person is retained in association with information about the occluding candidate object occluding the undetected person. Note that in the first embodiment, it is assumed that the occluding candidate object occluding the undetected person is a different person from the undetected person, and hereinafter, the different person will be referred to as an “occluding candidate object” as appropriate.

[0022] The correction unit 114 corrects the feature amount matching score for the undetected person, taking into consideration the possibility that the undetected person may reappear later, and reflects the corrected score in the database 201. First, the correction unit 114 sets a monitoring area based on the position information on the image of the occluding candidate object occluding the undetected person, i.e., the position information of the occluding person in the first embodiment. The monitoring area is set in the image based on the position information of the occluding person, and is, for example, a rectangular area having a predetermined size that is large enough to surround the occluding person and to include the undetected person if the person reappears later. In other words, the monitoring area is an area for detecting a person appearing near the occluding person when the undetected person is occluded by the occluding person. Furthermore, the size and set position of the monitoring area in the image are changed (updated) in accordance with changes in the size and position of the occluding person in the image.

[0023] When a person other than the occluding person appears in the monitoring area, the correction unit 114 determines that the appearing person is likely to be an undetected person who was obscured by the occluding person, and corrects the feature matching score of the appearing person so that the appearing person can be more easily matched in the matching process. The corrected feature matching score is sent to the matching unit 112 via the database 201, and the matching unit 112 performs the matching process for the appearing person using the corrected feature matching score. That is, the matching unit 112 at this time uses the corrected feature matching score to correct the matching result of the appearing person, thereby making it possible to prevent erroneous matching in which an undetected person who reappears is mistaken for a different, new person. Note that the correction process for the feature matching score in this embodiment will be described in detail later. In the information processing device of this embodiment, tracking of a person is performed based on the result of the person matching process as described above, and information on the tracking result is displayed on the display unit 150.

[0024] FIG. 3 is a diagram used to explain the positional relationship of people and the monitoring area when people are matched as described above in the information processing apparatus of the first embodiment. In FIG. 3, images 300, 310, and 320 are images to be processed by the information processing device of this embodiment. Image 300 is a frame image taken at time t1, image 310 at time t2, and image 320 at time t3. The frames at times t1, t2, and t3 are assumed to be in chronological order, but may also be frames obtained after thinning out at regular intervals. These images contain two people, person A and person B, who are moving in the direction indicated by the solid arrows in each image. The detection unit 111 detects person areas 301 and 302 as image areas of people from image 300, person area 312 from image 310, and person areas 322 and 323 from image 320. However, because these person areas 301, 302, 312, 322, and 323 have not yet been subjected to matching processing by the matching unit 112, it is not yet clear whether they belong to person A or person B. In order to identify whether it is person A or person B, it is necessary to compare and match the feature amounts of person areas 301, 302, 312, 322, and 323 using the feature amounts of person A and person B that have been previously matched and stored in database 201. Note that in this embodiment, it is assumed that person A and person B have appeared in the image before time t1, and that the feature amounts of person A and person B have already been stored in database 201.

[0025] Here, image 310 at time t2 shows a state in which person A and person B overlap, with person B being the occluding person and person A being occluded and therefore an undetected person. In the example of image 310, detection unit 111 detects only person B and obtains person area 312, while person A is not detected. Therefore, matching unit 112 identifies only person B through matching processing. Also in this case, determination unit 113 determines person A as an undetected person and person B as an occluding person who is occluding person A. In this case, correction unit 114 sets monitoring area 311 based on the position information of person area 312 of person B.

[0026] After that, as shown in image 320 at time t3, assume that a person other than person B (person area 323) appears in monitoring area 321. In this case, correction unit 114 corrects the feature amount matching score of person area 322 of the appearing person so that the person is more likely to be matched as person A in the matching process.

[0027] 4 is a flowchart showing the flow of information processing in the information processing device of this embodiment. In the following flowcharts, the symbol S represents a processing step. First, in S401, the image acquisition unit 200 acquires images to be processed in chronological order from a camera or an external device such as a server device or an external memory.

[0028] Next, in S402, the detection unit 111 acquires one frame to be processed from the image acquisition unit 200, and further in S403 performs processing to detect a person from that frame using the detection model that has been trained in advance as described above.

[0029] Next, in S404, the matching unit 112 performs the above-described matching process using the feature amount of the image area of ​​the person (person area) detected in S403 and the feature amount for matching stored in the database 201. Next, in S405, the determination unit 113 determines whether the person is occluded based on the result of the comparison in S404.

[0030] FIG. 5 is a detailed flowchart of the occlusion determination process performed by the determination unit 113 in S405. First, in S501, the determination unit 113 determines whether there is a person who was detected in the previous frame but not detected in the current frame. That is, the determination unit 113 determines whether there is an undetected person who was detected in the previous frame but not detected in the current frame, based on the matching result in the previous frame and the matching result in the current frame. Then, in S502, if the determination unit 113 determines that there is an undetected person in S501, the process proceeds to S503, and if it determines that there is no undetected person, the process proceeds to S507.

[0031] If it is determined that there is an undetected person and the process proceeds to S503, the determination unit 113 selects one person from the undetected people, and then the process proceeds to S504. Note that if there is only one undetected person, the determination unit 113 selects that one undetected person in S503, and the process proceeds to S504.

[0032] In S504, the determination unit 113 determines whether the undetected person is occluded. As described above, the determination unit 113 predicts the position of the undetected person in the current frame based on the position of the person in the previous frame or the change in the position of the person in the past few frames. If an occluding candidate object is present at the predicted position in the current frame, the determination unit 113 determines that the undetected person is occluded by the occluding candidate object. Note that whether or not the undetected person is occluded may be determined using information other than the position information of the person in the past frames, such as information about features or motion vectors. If the determination unit 113 determines that the undetected person is occluded in S504, the process proceeds to S505. On the other hand, if the determination unit 113 determines that the undetected person is not occluded, the process proceeds to S506.

[0033] In S505, the determination unit 113 selects an occluding candidate object that is thought to be occluding the undetected person. In the first embodiment, a person is assumed to be the occluding candidate object that occludes the undetected person, and therefore the person is determined to be the occluding person.

[0034] Next, in S506, the determination unit 113 determines whether or not the above-described series of processes for all undetected persons has been completed, and if completed, the process proceeds to S507, and if not completed, the process returns to S503.

[0035] When the process proceeds to S507, the determination unit 113 performs a process of narrowing down the occlusion candidate objects based on predetermined conditions, and the details of the process of narrowing down the occlusion candidate objects based on predetermined conditions in S507 will be described later in the third embodiment. After the process of S507, the determination unit 113 ends the process of the flowchart in FIG. 5.

[0036] When the occlusion determination process in S405 of FIG. 4 (flowchart of FIG. 5) is completed, in S406, the correction unit 114 performs a process of correcting the matching result, that is, a process of correcting the feature amount matching score. Figures 6(a) to 6(c) are figures used to explain the occlusion relationship between person A and person B at each time in the example images 300, 310, and 320 of Figure 3 described above, example feature matching scores, and example corrections to the feature matching scores.

[0037] FIG. 6(a) shows tables representing the occlusion relationship between an occluding person and an occluded person (undetected person). Table 601 shows the occlusion relationship for image 300 at time t1, table 602 shows the occlusion relationship for image 310 at time t2, and table 603 shows the occlusion relationship for image 320 at time t3. In these tables 601 to 603, the vertical elements represent occluding people and the horizontal elements represent occluded people (undetected people), with a circle (◯) indicating that a person is occluded by the occluding person and a cross (×) indicating that the person is not occluded. In the example of image 310 at time t2 in FIG. 3, person A is occluded by person B, so as shown in table 602, person B is the occluding person and person A is the occluded person (undetected person).

[0038] 6(b) is a diagram showing a table representing feature matching scores calculated from the matching features of person A and person B stored in database 201 and the features of person regions 301, 302, 312, 322, and 323 detected from the image. Table 611 in FIG. 6(b) shows an example of feature matching scores calculated in an example where the occlusion relationship between person A and person B is as shown in table 601 in FIG. 6(a), as in image 300 in FIG. 3. Table 612 shows an example of feature matching scores calculated in an example where the occlusion relationship between person A and person B is as shown in table 602, as in image 310 in FIG. 3. Table 613 shows an example of feature matching scores calculated in an example where the occlusion relationship between person A and person B is as shown in table 603, as in image 320 in FIG. 3.

[0039] Here, it is assumed that the predetermined score threshold for the feature matching score when determining whether or not two people are the same person is 0.6, for example. In other words, if the feature matching score is 0.6 or higher, it is determined by the matching process that the two people are the same person. In the case of table 611 corresponding to image 300 in FIG. 3 , the feature matching score calculated from the matching features of person region 301 and person A is 0.6, so person region 301 is determined to be person A. Similarly, the feature matching score calculated from the matching features of person region 302 and person B is 0.7, so person region 302 is determined to be person B. In the case of table 612 corresponding to image 310 in FIG. 3 , person region 312, with a feature matching score of 0.7, is determined to be person B, but person A is an undetected person and does not obtain a feature matching score of 0.6 or higher.

[0040] 3, person region 323, which has a feature matching score of 0.7, is determined to be person B, but person region 322, which has a feature matching score of 0.4, is not determined to be person A. In other words, while the feature matching score of person region 301 and person A is 0.6 in table 611 at time t1, the feature matching score of person region 322 is 0.4 in table 613 at time t3, and so it is not determined to be person A.

[0041] Therefore, in this embodiment, when person A becomes an undetected person as in image 310 at time 2, correction unit 114 sets monitoring area 311 based on the position and size of person region 312 of person B, who is occluding person A. Then, when a person other than person B appears in monitoring area 321 in image 320 of FIG. 3 at time 3, the correction unit 114 corrects the feature matching score of person region 322 of the person so that the person is more likely to be matched as person A, since the person is likely to be an undetected person. FIG. 6(c) is a diagram showing table 623 after the feature matching scores shown in table 613 of FIG. 6(b) have been corrected. That is, compared to table 613 before correction, table 623 shows that the feature matching score of person region 322 has been corrected from 0.4 to 0.6. As a result, in a matching process using the corrected feature matching score, person region 322 is matched as person A.

[0042] In table 623, a positive correction is made to increase the score value of the feature amount matching score so that person area 322 appearing in monitoring area 321 can be more easily matched as person A. In this embodiment, an example is given in which the score value is increased by 0.2 as the amount of positive correction, but the amount is not limited to 0.2 and may be a value greater than 0.2 or, conversely, a value smaller than 0.2.

[0043] Furthermore, in the example of table 613 before correction and table 623 after correction, the feature matching score of person region 322 is 0.4 before correction and 0.6 after correction, and the predetermined score threshold for matching whether or not the person is the same is 0.6. In addition to this example, the score threshold for matching whether or not the person is the same may be, for example, 0.7 as a first threshold before feature matching score correction, and 0.5 as a second threshold after feature matching score correction. That is, the predetermined score threshold may be changed before and after feature matching score correction so that people with a first threshold of 0.7 or higher are matched as the same person before feature matching score correction, but people with a second threshold of 0.5 or higher are matched as the same person after feature matching score correction. In this example, correction unit 114 has a first threshold and a second threshold, and corrects the feature matching score when the score value of a person region appearing in the monitoring area is equal to or lower than the first threshold. Then, in the matching process, if the corrected score value for a person area that appears in the monitoring area is equal to or greater than the second threshold, the person is determined to be an undetected person.

[0044] After S406 in Fig. 4, the processing of the information processing device proceeds to S407. When proceeding to S407, the correction unit 114 stores in the database 201 the feature amounts in the current frame of the person determined to be the same person as the person whose feature amounts are stored in the database 201. That is, the feature amounts in the database 201 are updated. As a result, in the next frame, the updated feature amounts are used in the matching process, thereby improving the accuracy of the next matching process and the tracking accuracy. Note that, although the information processing device of this embodiment stores feature amounts for each frame, this is not necessarily required.

[0045] Then, in S408, the information processing device checks whether the above-mentioned series of processes has been completed for all frames, and if so, ends the processing of the flowchart in Figure 4, and if not, returns to S402 and repeats the processing from S402 onwards.

[0046] According to the first embodiment, even if a person is occluded by an occluding person, it is possible to suppress erroneous matching of the appearing person by correcting the feature amount matching score of the person who appears in the set monitoring area. Furthermore, in the case of the first embodiment, for example, since the feature amount matching score is merely corrected rather than forcibly associating the person with the person, it is also possible to determine that if a person different from the undetected person appears in the monitoring area, that person is not the same person as the undetected person.

[0047] <Second embodiment> Next, a second embodiment will be described. In the first embodiment, an example in which two people appear in an image has been described, but in the second embodiment, an example in which three or more people appear will be described. In the first embodiment, when a certain person is occluded and becomes an undetected person, if a new person different from the undetected person appears in the image and the new person (hereinafter referred to as a newly appeared person) resembles the undetected person, the newly appeared person may be erroneously matched with the undetected person.

[0048] Therefore, in the second embodiment, an example will be described in which a newly appearing person is prevented from being erroneously matched as an undetected person, and a feature matching score is corrected in preparation for the case in which an undetected person reappears. When correcting the feature matching score, the information processing device of the second embodiment uses a score correction map that quantifies an expected value (i.e., the likelihood) representing the probability of where an occluded and undetected person will subsequently appear in an image. Note that the hardware configuration of the second embodiment is the same as that of FIG. 1, and the data flow of each functional unit is also the same as that of FIG. 2, so illustration and description thereof will be omitted. Only the processing and configuration that are different from those of the first embodiment will be described below.

[0049] In the second embodiment, the score correction map is a map in which a captured image is divided into a positive correction area, a negative correction area, and an uncorrected area that is neither positive nor negative, and a different correction score is assigned to each area. The positive correction area is set as an area where the expected value for the reappearance of an occluded undetected person is high (high possibility). The negative correction area is set as an area where the expected value for the reappearance of an occluded undetected person is low (low possibility). The uncorrected area is set as an area where the expected value for the reappearance of an occluded undetected person cannot be said to be high (high possibility) nor low (low possibility).

[0050] For example, suppose that person A and person B move across the image with person B occluding person A. In the second embodiment, as in the first embodiment, a monitoring area is set based on the person area of ​​person B, and monitoring is performed to see if person A reappears in that monitoring area. Here, if person A is occluded and remains an undetected person, and person C with similar features to person A appears outside the monitoring area, person C may be mistakenly matched as person A. However, in this case, person A, who remains an undetected person, is considered to be behind person B, and therefore it is considered unlikely that person A will appear outside the monitoring area.

[0051] For this reason, the correction unit 114 of the second embodiment generates a score correction map for the captured image, dividing the image into a positive correction area in which the feature matching score of an undetected person is positively corrected, a negative correction area in which the feature matching score is negatively corrected, and an uncorrected area in which the feature matching score is not corrected. Note that a positive correction is a correction that increases the score value of the feature matching score, and a negative correction is a correction that decreases the score value. In this embodiment, the monitoring area is a positive correction area because there is a high possibility that person A, an undetected person, will reappear. Areas that are not considered to be on the movement trajectory of person B are negative correction areas because there is a low possibility that person A will reappear. Other areas are uncorrected areas. In this way, in the second embodiment, the score correction amount is appropriately set for each area of ​​the score correction map, and the feature matching score of person A, an undetected person, is appropriately corrected for each location in the captured image where people appear, thereby making it possible to reduce erroneous matching.

[0052] In this embodiment, the score correction amount for each area of ​​the score correction map can be determined as follows: Note that the method for determining the score correction amount for each area is merely an example. A method for determining the score correction amount for correcting the feature matching score of person A, who has become undetected due to being occluded, can be, for example, a method of calculating it from the past feature matching scores of person A. For example, a value obtained by adding a constant to the average feature matching score of person A during the period from when person A appeared until when person A became undetected due to being occluded, or by multiplying this average by a constant, can be determined as the score correction amount for the positive correction area. Alternatively, a value obtained by adding a constant to the average feature matching score of person A or by multiplying this average by a constant can be determined as the score correction amount for the negative correction area. Furthermore, if there is a large variation in person A's past feature matching scores, the score correction amount for the positive correction area or the score correction amount for the negative correction area can also be determined based on the amount of variation. For example, a value obtained by adding a constant to or multiplying the amount of variation in the past feature matching scores by a constant can be determined as the score correction amount for the positive correction area, or a value obtained by adding a constant to or multiplying the amount of variation by a constant can be determined as the score correction amount for the negative correction area. In this way, by using the past feature amount matching score of person A who has become an undetected person, it is possible to determine the score correction value according to the matching accuracy for person A.

[0053] Another method for determining the score correction amount for correcting the feature matching score of person A who has become an undetected person is, for example, a method of calculating the score correction amount based on information about past mismatches. For example, if a past mismatch occurs in which an undetected person is mismatched as another newly appearing person, the score correction amount for the positive correction area can be determined using the average value of the feature matching scores for that person at that time. For example, if a past mismatch occurs in which a newly appearing person is mismatched as an undetected person who has reappeared, the score correction amount for the negative correction area can be determined using the average value of the feature matching scores for that person at that time. In this way, by appropriately setting the score correction amounts for the positive correction area and the negative correction area according to past mismatch information, it is possible to prevent mismatches in which an undetected person is mismatched as another newly appearing person or a newly appearing person is mismatched as a reappearing undetected person.

[0054] FIG. 7(a) is a diagram showing an example in the second embodiment in which three or more persons, namely persons A, B, C, and D, appear in an image, and, for example, person A is obscured by person B and becomes an undetected person, and person D appears as another newly appearing person.

[0055] Images 700, 710, 720, and 730 in FIG. 7(a) are images to be processed in this embodiment. Image 700 is an example of a frame taken at time t1, image 710 at time t2, image 720 at time t3, and image 730 at time t4. The frames at times t1, t2, t3, and t4 are assumed to be in chronological order, but may also be frames thinned out at regular intervals. Here, it is assumed that Person A, Person B, and Person D are moving in the direction indicated by the solid arrows in each image. Furthermore, it is assumed that Person A and Person B moved along a flow line 725, indicated by a dotted line in the image, between times t2 and t3. In other words, flow line 725 represents the trajectories of Person A and Person B's movements between times t2 and t3.

[0056] In the second embodiment, it is assumed that the detection unit 111 detects person regions 701, 702, and 703 from image 700. It is also assumed that the detection unit 111 detects person regions 711 and 712 from image 710, person regions 721, 722, and 723 from image 720, and person regions 731, 732, 733, and 734 from image 730. However, because these person regions 701, 702, 703, 711, 712, 721, 722, 723, 731, 732, 733, and 734 have not yet been subjected to matching processing by the matching unit 112, it has not yet been identified whether they belong to persons A, B, C, or D. In order to identify whether a person is person A, B, C, or D, it is necessary to use previously matched feature amounts stored in database 201 and match the feature amounts of each person region. 7, it is assumed that persons A, B, and C appeared in the image before time t1, that the feature amounts of persons A, B, and C have already been stored in database 201, and that person D appeared for the first time at time t3. Furthermore, monitoring areas 713, 724, and 735 are set based on the person area of ​​person B, and are areas for monitoring the reappearance of person A, who has become an undetected person.

[0057] In image 710 at time t2, person A and person B overlap, with person B being the occluding person and person A being the undetected person (occluded person). In the example of image 710, detection unit 111 detects only person B and obtains person area 711, while person A is not detected. Therefore, matching unit 112 identifies only person B through matching processing, and determination unit 113 determines person A as an undetected person and further determines person B as an occluding person who is occluding person A. Therefore, correction unit 114 sets monitoring area 713 based on person area 711 of person B. Furthermore, in image 720 at time t3, the position of person area 721 changes due to the movement of person B, and the position of monitoring area 724 also changes accordingly. Then, as shown in image 730 at time t4, person A, who was occluded by person B, reappears in monitoring area 735.

[0058] In the second embodiment, similarly to the first embodiment, person A is occluded by person B and becomes an undetected person, and therefore monitoring areas 713, 724, and 735 are set. Then, when a person other than person B appears in the monitoring area, the feature amount matching score is corrected during feature matching for that person so that the person is more likely to be matched as person A. However, in the second embodiment, together with setting the monitoring area, a score correction map is generated for correcting the feature amount matching score of person A, who has become an undetected person.

[0059] Fig. 7(b) is a diagram showing an example of a score correction map set for a captured image according to the setting of a monitoring area. In the example of Fig. 7(b), in order to make it easier to understand the relationship between the captured image and the score correction map, the person areas (persons A to D) detected from the image are also shown. In Figure 7(b), score correction map 714 is a map generated corresponding to image 710 at time t2, score correction map 726 is a map generated corresponding to image 720 at time t3, and score correction map 736 is a map generated corresponding to image 730 at time t4. In these score correction maps 714, 726, and 736, the hatched areas represent positive correction areas where an occluded and undetected person is likely to appear. The positive correction areas correspond to the monitoring area. The dotted areas represent negative correction areas where an undetected person is unlikely to appear, and the blank areas represent uncorrected areas.

[0060] In the second embodiment, the correction unit 114 corrects the feature matching score for an undetected person based on the score correction maps 714, 726, and 736 generated at each time. That is, in the second embodiment, the feature matching score of an undetected person is corrected depending on whether the location is likely to be a location where the undetected person will reappear, thereby enabling more accurate matching. Note that the score correction amount for the positive correction area and the score correction amount for the negative correction area are calculated based on past feature matching scores, as described in the first embodiment. In the first embodiment, the correction amount for the feature matching score was 0.2, which is a positive correction. However, in the second embodiment, the score correction amount for the positive correction area is +0.2, and the score correction amount for the negative correction area is -0.2.

[0061] FIG. 8 is a flowchart of the feature matching score correction process and score correction map generation process performed in S406 of the flowchart in FIG. 3 described above in the second embodiment. First, in S801, the correction unit 114 determines whether or not a person who is a candidate for occlusion is present in the image, and if it is determined that a person who is a candidate for occlusion is present, the process proceeds to S802, whereas if it is determined that a person who is a candidate for occlusion is not present, the process proceeds to S807. In the image 700 at time t1 in Fig. 7(a), there is no undetected person, so it is determined that there is no person who is a candidate for occlusion, and the process proceeds to S807, but from time t2 onwards, person B is occluding person A, so the process proceeds to S802.

[0062] In S802, the correction unit 114 selects one of the occluding persons who are occluding candidate objects, and proceeds to S803. In the example of Fig. 7(a), since the only occluding person is person B, the person region 702 of person B is selected in S802. Next, in S803, the correction unit 114 sets a monitoring area based on the person area of ​​person B, who is the obscuring person. In the second embodiment, as in the first embodiment, the processing for setting the monitoring area includes processing for updating the size and position of the monitoring area in accordance with changes in the size and position of person B. That is, the sizes and positions of the monitoring areas 713, 724, and 735 are updated in accordance with the size and position of the detection frame area when the detection unit 111 detected person B appearing in the image at times t2, t3, and t4.

[0063] In the next step S804, the correction unit 114 updates the movement line information of the occlusion candidate object. This movement line corresponds to the movement line 725 shown in the image 720 of FIG. 7, and is found by calculating the trajectory of person B's movement from past frames. In the example of FIG. 7(b), it is calculated that person B moved along the trajectory of movement line 725 from the frames between time t2 and time t3. In this embodiment, the movement line is calculated using position information of the detection results of past frames, but the movement line may also be calculated using information such as the movement speed of the occlusion candidate object.

[0064] Next, in S805, the correction unit 114 generates and updates a score correction map for person A, who has become an undetected person. In updating the score correction map, it is determined that person A is highly likely to reappear in a monitoring area corresponding to person B, and that person A is unlikely to reappear in an area to which person B has not moved. Note that, since it cannot be said with certainty that person A is highly likely to reappear in an area on person B's movement path 725 that person B has already passed through, the area is treated as a non-correction area and the feature matching score is not corrected. Furthermore, in this embodiment, the score correction map is generated using the monitoring area based on the person area of ​​person B and the movement path 725, but a map generation method using the shooting environment or person information stored in the database 201 may also be used.

[0065] Next, in S806, the correction unit 114 checks whether a series of processes has been completed for all person regions, and if completed, proceeds to S807, and if not completed, returns to S802.

[0066] Proceeding to S807, if a new person appears in the image, the correction unit 114 detects which area of ​​the score correction map the new person's appearance position is in and corrects the feature amount matching score of the new person based on the score correction amount set for that area. For example, if person D appears in image 720 at time t3, person region 723 of person D appears in a negative correction area where person A, an undetected person, is unlikely to reappear. Therefore, the feature amount matching score of person region 723 is negatively corrected. Therefore, in this case, person region 723 of person D is matched as a person different from person A. In contrast, in image 730 at time t4, if person region 731 other than person B appears in the monitoring area 735, person region 731 appears in a positive / negative correction area where person A, an undetected person, is likely to reappear. Therefore, the feature amount matching score of person region 731 is positively corrected. Therefore, in this case, person region 731 is matched as person A. After S807, the information processing apparatus ends the processing of the flowchart in Fig. 8 and proceeds to the processing of S407 in Fig. 4. The processing from S407 onwards is the same as that described above, and therefore a description thereof will be omitted.

[0067] 9(a) to 9(c) are diagrams used to explain the positional relationships of persons A to D at each time in the example images 700, 710, 720, and 730 in FIG. 7(a), example feature matching scores, and example corrections to the feature matching scores.

[0068] FIG. 9(a) is a diagram showing a table showing the occlusion relationship between an occluding person and an occluded person (undetected person). Table 901 shows the occlusion relationship in the case of image 700 at time t1. Similarly, table 902 shows the occlusion relationship in the case of image 710 at time t2, table 903 shows the occlusion relationship in the case of image 720 at time t3, and table 904 shows the occlusion relationship in the case of image 730 at time t4. In these tables 901 to 904, as in FIG. 6(a), the vertical elements indicate occluding people and the horizontal elements indicate non-occluding people (undetected people), with a circle (◯) indicating a case where a person is occluded by the occluding person and a cross (×) indicating a case where a person is not occluded. In the case of FIG. 7(a), person A is occluded by person B in image 710 at time t2 and image 720 at time t3, so person B is the occluding person and person A is the occluded person (undetected person).

[0069] 9(b) is a diagram showing a table representing feature matching scores calculated from the matching features of persons A to D in database 201 and the features of person regions 701, 702, 703, 711, 712, 721, 722, 723, 731, 732, 733, and 734. Table 911 in FIG. 9(b) shows an example of feature matching scores for image 700 in FIG. 7, and similarly, tables 912 to 914 show examples of feature matching scores for images 710 to 730, respectively. Also in the second embodiment, the predetermined score threshold for determining whether or not two people are the same person using the feature matching scores is assumed to be 0.6. For example, in table 911 corresponding to image 700 in FIG. 7(a), person region 701 is determined to be person A, person region 702 is determined to be person B, and person region 703 is determined to be person C. On the other hand, in the table 912 corresponding to the image 710, the person area 711 is determined to be person B, and the person area 712 is determined to be person C, but person A is determined to be an undetected person.

[0070] Fig. 9(c) is a diagram showing a table indicating the feature matching scores after the feature matching scores calculated for persons A to D stored in database 201 have been corrected in accordance with the score correction map. Table 923 in Fig. 9(c) shows the feature matching scores after table 913 in Fig. 9(b) has been corrected in accordance with the score correction map, and similarly, table 924 in Fig. 9(c) shows the feature matching scores after table 914 in Fig. 9(b) has been corrected.

[0071] In table 913 corresponding to image 720, person region 721 is determined to be person B and person region 722 is determined to be person C. However, person region 723, which should be determined to be person D, is erroneously determined to be person A because its feature matching score is 0.6. In contrast, in the second embodiment, a score correction map is generated (updated) based on monitored area 724 and traffic line 725, and the feature matching score is corrected using this score correction map as shown in table 923 in FIG. 9( c). That is, compared to table 913 before correction, table 923 has the feature matching score of person region 723 corrected from 0.6 to 0.4. Therefore, in the matching process, the corrected feature matching score is used, and person region 723 is not determined to be person A. As a result, a matching result is obtained for person region 723 that it is the newly appeared person D. As a result, the features of the newly appeared person D are added to database 201.

[0072] Furthermore, in the case of table 614 corresponding to image 730, person region 732 is determined to be person B, person region 733 is determined to be person C, and person region 734 is determined to be person D. On the other hand, person region 731, which should be determined to be person A, has a feature matching score of 0.5 and is therefore not determined to be person A. In contrast, in the case of the second embodiment, a score correction map is generated (updated) based on monitoring region 735, and the feature matching score is corrected as shown in table 924 in FIG. 9( c). That is, when comparing table 924 with table 914 before correction, the feature matching score of person region 731 has been corrected from 0.5 to 0.7. Therefore, in the matching process, the corrected feature matching score is used and person region 723 is determined to be person A.

[0073] Note that in the second embodiment as well, the correction amount of the feature matching score is not limited to 0.2, and may be a value greater than or less than 0.2. Also in the second embodiment, as in the first embodiment, the score threshold for matching whether or not the persons are the same may be set to, for example, 0.7 as the first threshold before the feature matching score correction, and may be set to, for example, 0.5 as the second threshold after the feature matching score correction.

[0074] According to the second embodiment, even if three or more people appear in an image and a person occludes the image, it is possible to suppress erroneous matching when an undetected person reappears or a new person appears by using a score correction map that takes into account the possibility that an undetected person may reappear.

[0075] <Third embodiment> Next, a third embodiment will be described. In the first and second embodiments, a case in which an occlusion candidate object does not pass other people or objects has been described as an example. However, in actual images acquired from a surveillance camera installed in a store or the like, a tracking target person may pass multiple people while being occluded and becoming an undetected person. In this case, it becomes difficult to determine which person is occluding the tracking target person. In the third embodiment, when the tracking target person moves while being occluded, occlusion candidate objects are selected and narrowed down from people and objects that may occlude the tracking target person. In the third embodiment, by selecting and narrowing down occlusion candidate objects, it is possible to reduce erroneous matching when an undetected person reappears. Note that the hardware configuration of the third embodiment is the same as that of FIG. 1, and the data flow is also the same as that of FIG. 2, so illustration and description thereof will be omitted. Below, only the processing and configuration different from those of the first and second embodiments will be described.

[0076] Fig. 10(a) is a diagram showing an example in the third embodiment in which four or more people, people A to D, appear in an image, and person A moves while being occluded by person B and becoming an undetected person. Furthermore, in the example of Fig. 10(a), it is assumed that there is another object (referred to as occluding object E) other than the people, which may be a candidate occluding object that has the potential to occlude person A.

[0077] Images 1000, 1010, 1020, and 1030 in Figure 10(a) are the images to be processed, and are assumed to be example images of frames at times t1, t2, t3, and t4. Note that the frames at times t1, t2, t3, and t4 are assumed to be in chronological order, but they may also be frames that have been thinned out at regular intervals. Furthermore, persons A and B are assumed to be moving in the direction of the solid arrows drawn in each image.

[0078] In the third embodiment, it is assumed that the detection unit 111 detects person regions 1001, 1002, 1003, and 1004 from the image 1000. It is also assumed that the detection unit 111 detects person regions 1011, 1012, and 1013 from the image 1010, person regions 1021, 1022, and 1024 from the image 1020, and person regions 1031, 1032, 1033, and 1035 from the image 1030. However, because these detected person regions have not yet been subjected to the matching process by the matching unit 112, it is not yet determined whether they belong to persons A, B, C, or D. In the example of FIG. 10 , it is assumed that persons A, B, C, and D appeared in the image before time t1 and that the feature quantities of persons A, B, C, and D have already been stored in the database 201. In the third embodiment, the detection unit 111 also has a function of detecting objects other than people from the image. That is, the detection unit 111 detects, as an object region, an image region of an object (occluding object E) that may occlude a person, as an object region other than a person. In Fig. 10(a), object regions 1005, 1014, 1023, and 1033 are regions detected as the image region of occluding object E from images 1000, 1010, 1020, and 1030, respectively.

[0079] Moreover, monitoring areas 1015, 1027, and 1036 are areas that are set based on the person area of ​​person B, and are used to monitor the reappearance of person A, who has become an undetected person. Here, when person A is occluded by person B and has become an undetected person, and when the moving person B passes by, for example, an object whose distance to the person area of ​​person B is equal to or less than a predetermined distance threshold may also be a candidate occluding object that may be able to occlude person A. Also, when person A is occluded by person B and has become an undetected person and has become an undetected person, and when the moving person B passes by, for example, an object that overlaps with the person area of ​​person B by more than a predetermined overlap ratio threshold may also be a candidate occluding object that may be able to occlude person A.

[0080] For this reason, in the third embodiment, an object whose distance to the person region of person B is equal to or less than the distance threshold, or an object whose overlap with the person region of person B is equal to or more than the overlap ratio threshold, is added as an occlusion candidate. That is, in the third embodiment, other people or objects that may occlude person A, including person B who is occluding person A, are set as occlusion candidates, and a monitoring area is set for each of these occlusion candidates. Note that an object may be added as an occlusion candidate if it satisfies either the condition that it is equal to or less than the distance threshold with person B or the condition that it overlaps with person B by equal to or more than the overlap ratio threshold, or it may be added as an occlusion candidate if it satisfies both conditions. However, instead of setting all objects whose distance to person B is equal to or less than the distance threshold or the object whose overlap with person B is equal to or more than the overlap ratio threshold as occlusion candidates, only objects that are large enough to occlude person A are set as occlusion candidates and a monitoring area is set. The reason for setting the monitoring area in this way, including person B and other people or objects that may be blocking person A as potential blocking objects, is to deal with the possibility that person A may hide behind these potential blocking objects when passing by person B, and the object that person A is blocking may change.

[0081] Image 1020 in FIG. 10( a) shows an example in which person C and occluding object E have become occluding candidates for person A due to, for example, passing by person B, who is occluding person A. Therefore, a monitoring area 1025 is set based on person area 1021 of person C, and similarly, a monitoring area 1026 is set based on object area 1023 of occluding object E. On the other hand, person D is a small person, such as a child, and the size of the person area of ​​person D indicates that the person is not large enough to occlude person A. Therefore, person D is not included in the occluding candidate objects, and a monitoring area is not set for person D. That is, in this embodiment, the size of the person area is used as a predetermined condition for the occluding candidate narrowing-down process in S507 of FIG. 5. If the size of the person area is not large enough to occlude person A, the narrowing-down process is performed so that the person area is not added to the occluding candidate objects. In other words, if the size of the person area is clearly smaller than that of person A, it is not added to the occluding candidate objects. It should be noted that a small area size here also includes a state in which the person is crouching or bending over.

[0082] FIG. 11 is a diagram used to explain the occlusion relationships between persons A to D and occluding object E at each time in the examples of images 1000, 1010, 1020, and 1030 in FIG. 10(a). Table 1101 shows the occlusion relationships for image 1000 at time t1. Similarly, table 1102 shows the occlusion relationships for image 1010 at time t2, table 1103 shows the occlusion relationships for image 1020 at time t3, and table 1104 shows the occlusion relationships for image 1030 at time t4. In these tables 1101 to 1104, the vertical elements indicate occluding persons or objects, and the horizontal elements indicate occluded persons (undetected persons). Cases where there is a possibility of occlusion by a person or object are indicated by a circle (◯), and cases where there is no occlusion are indicated by a cross (×). Table 1102 corresponding to image 1010 at time t2 shows that person B is the occluding person, and person A is the occluded person (undetected person). In the table 1102 corresponding to the image 1020 at time t3, the people or objects that are occluding or may occlude person A are person B, person C, and occluding object E.

[0083] 10A shows an image 1030 at time t4 in which the head of person A, which had been occluded by person B, appears in the monitoring area 1036 as person area 1034. At this time, according to table 1103 in FIG. 11, the occluding candidates that may be occluding person A are person C, occluding object E, and person B. However, if person area 1034 appears behind person area 1035, it can be determined that person A is the only person occluded by person B. Therefore, the correction unit 114 can determine that the person that appeared in the monitoring area 1036 is person A. In the third embodiment, the determination unit 113 determines, as a condition for the occlusion candidate narrowing down process in S507 in FIG. 5, whether person A is the only person occluded by person B. If only person A is present, person C and occluding object E are deleted from the occluding candidates, excluding person B. That is, when there are multiple occluding candidates such as person B, person C, and occluding object E that may be occluding an undetected person, the determination unit 113 deletes person C and occluding object E from the occluding candidates, excluding person B, based on the condition that a person has appeared behind person B. Then, the determination unit 113 reflects the deletion of person C and occluding object E from the occluding candidates in the occlusion relationship as shown in table 1104 of FIG. 11. As a result, table 1104 indicates that person B is the occluding person and person A is the undetected person (occluded person). Note that in image 1030 of FIG. 10(a), information such as the person's face, arms, or legs may be detected behind person B.

[0084] As described above, in the third embodiment, a monitoring area is set for an occluding candidate object that satisfies at least one of the conditions that the distance to person B occluding the undetected person is equal to or less than the distance threshold and the overlapping condition is equal to or greater than the overlapping ratio, and that is large enough to occlude person A. Furthermore, in the third embodiment, when there are multiple occluding candidate objects that may be occluding the undetected person, occluding candidates other than a certain occluding candidate object are deleted from the occluding candidate objects based on the condition that a person has appeared from behind that occluding candidate object.

[0085] In the third embodiment, as in the second embodiment, the correction unit 114 generates a score correction map divided into a positive correction area where positive correction is performed, a negative correction area where negative correction is performed, and an uncorrected area where no correction is performed. Then, the correction unit 114 uses the score correction map generated at each time to correct the feature matching score to accommodate the reappearance of an undetected person.

[0086] FIG. 10B shows an example of a score correction map set for a captured image in accordance with the setting of the monitoring area as described above. In FIG. 10B, a score correction map 1016 is a map generated for an image 1010 at time t2, and a score correction map 1028 is a map generated for an image 1020 at time t3. A score correction map 1037 is a map generated for an image 1030 at time t4. In the score correction map, the hatched areas represent positive correction areas where an occluded and undetected person is likely to appear, the dotted areas represent negative correction areas where an undetected person is unlikely to appear, and the blank areas represent uncorrected areas. In the third embodiment, multiple monitoring areas may be set, and therefore multiple positive correction areas may also be present, as shown in the score correction map 1028. That is, in the score correction map 1028, three positive correction areas are set corresponding to the person area 1021, the object area 1023, and the person area 1027 in the image 1020, respectively.

[0087] According to the third embodiment, even when a person to be tracked moves while being occluded and passes other people or objects, it is possible to reduce the amount of processing by sequentially narrowing down the occluding candidate objects while suppressing erroneous matching.

[0088] In this embodiment, examples of predetermined conditions for selecting and narrowing down objects that may occlude a target object as occluding objects include the size of the object, the appearance of the target object from behind a candidate object, etc. Furthermore, other conditions such as those exemplified below may be used in appropriate combination or added.

[0089] That is, in this embodiment, it is possible to select and narrow down objects that may occlude a target object as occluding objects based on the following conditions (1) to (8). (1) Any other object whose distance to a candidate object selected as occluding a target object is less than a predetermined distance threshold is added as a candidate object that may occlude the target object. (2) Any other object whose overlap rate with the candidate object selected as occluding the target object is equal to or greater than a predetermined overlap rate threshold is added as a candidate object that may occlude the target object. (3) Small objects that cannot occlude the target object are excluded from the candidate objects. (4) If the time elapsed since the target object began to be obscured exceeds a predetermined threshold, the object that may obscure the target object is excluded from the candidate objects. (5) If an object moves at a speed greater than a predetermined speed threshold, the object is excluded from the candidate objects. (6) If, while an object is moving, no other object is detected in its vicinity for a predetermined undetected time threshold or longer, the object is excluded from the list of candidate objects. (7) If the difference in velocity vector between an object and another object approaching the object is equal to or greater than a predetermined difference threshold, the object is not added to the candidate objects. (8) When there are multiple candidate objects, if another object appears behind one of the candidate objects, the remaining candidate objects except for the candidate object when the other object appeared are deleted from the candidate objects.

[0090] <Other embodiments> In each of the above-described embodiments, examples have been described in which people are detected and tracked, but the objects to be detected and tracked are not limited to people, and may also be cars, bicycles, motorcycles, etc. Furthermore, the detection target is the entire body of a person (human body), but the information processing device of this embodiment can also be applied to cases where a person is tracked by detecting human body parts such as the face and hands. In the above-described embodiment, an example was described in which a person or the like is detected and matched using CNN, but the present invention is not limited to this, and matching using a histogram of an image region or template matching may also be used. Furthermore, when updating the score correction map, the score correction map may be optimized by providing a valid time for the path of movement of the occlusion candidate object.

[0091] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0092] 100: CPU, 110: ROM, 111: detection unit, 112: matching unit, 113: determination unit, 114: correction unit, 201: database

Claims

1. a detection means for detecting a target object from an image; a matching unit that performs a matching process on the target object based on the degree of similarity between the feature amount of the detected target object and the feature amount for matching; a selection means for selecting a candidate object that occludes the target object; a correction means for correcting a result of the matching process by the matching means based on a result of the selection by the selection means; and The information processing device is characterized in that the correction means corrects the result of the matching process when an object different from the selected candidate object appears in the vicinity of the selected candidate object after the target object is no longer detected.

2. The information processing device according to claim 1, characterized in that, when an object different from the selected candidate object appears in the vicinity of the selected candidate object after the target object is no longer detected, the correction means corrects the result of the matching process so that the appeared object is matched as the target object.

3. The information processing device according to claim 2, characterized in that the correction means sets a predetermined monitoring area for detecting the appearance of an object different from the selected candidate object, and when an object different from the candidate object appears in the monitoring area, corrects the result of the matching process so that the appeared object is matched as the target object.

4. 4. The information processing apparatus according to claim 3, wherein the correction means sets the predetermined monitoring area based on at least the position of the selected candidate object in the image.

5. 5. The information processing apparatus according to claim 4, wherein the correction means updates the position of the predetermined monitoring area in response to a change in the position of the selected candidate object within the image.

6. 6. The information processing apparatus according to claim 4, wherein the correction means sets the size of the predetermined monitoring area in accordance with the size of the selected candidate object.

7. The information processing device according to any one of claims 2 to 6, characterized in that the correction means has a first threshold and a second threshold, and when an object different from the selected candidate object appears near the candidate object and the similarity between the features of the appeared object and the features for matching is equal to or less than the first threshold, performs the correction of the result of the matching process, and when the similarity becomes equal to or greater than the second threshold after the correction, determines as a matching result that the appeared object is an object corresponding to the similarity for matching.

8. 8. The information processing device according to claim 1, wherein, when the target object is no longer detected, the selection means selects the candidate object that occludes the target object based on a past position of the target object.

9. 9. The information processing apparatus according to claim 1, wherein the selection means selects a candidate object that has a possibility of occluding the target object based on a predetermined condition.

10. the predetermined condition is a distance to the selected candidate object, 10. The information processing apparatus according to claim 9, wherein the selection means adds another object whose distance from the selected candidate object is equal to or less than a predetermined distance threshold as a candidate object that may occlude the target object.

11. the predetermined condition is an overlap rate with the selected candidate object, The information processing device according to claim 9 or 10, characterized in that the selection means adds other objects whose overlapping ratio with the selected candidate object is equal to or greater than a predetermined overlapping ratio threshold as candidate objects that may occlude the target object.

12. the predetermined condition is the size of the object, 12. The information processing device according to claim 9, wherein the selection means, when a size of the object is smaller than a size of the target object, excludes the object from candidate objects that may occlude the target object.

13. the predetermined condition is an elapsed time since the target object was occluded, 13. The information processing device according to claim 9, wherein the selection means excludes from the candidate objects any object that may occlude the target object if the elapsed time since the target object was occluded exceeds a predetermined elapsed time threshold.

14. the predetermined condition is the speed at which the object moves, 14. The information processing device according to claim 9, wherein the selection means, when the object is moving at a speed equal to or greater than a predetermined speed threshold, excludes the object from candidate objects that may occlude the target object.

15. the predetermined condition is a time period during which another object has not been detected in the vicinity of the moving object; The information processing device according to any one of claims 9 to 14, characterized in that the selection means excludes a moving object from candidate objects that may occlude the target object if no other object is detected in the vicinity of the moving object for a predetermined undetected time threshold or longer.

16. the predetermined condition is a difference in velocity vectors between objects, The information processing device according to any one of claims 9 to 15, characterized in that the selection means excludes an object from candidate objects that may occlude the target object if a difference between the velocity vector of the object and the velocity vector of another object approaching the object is equal to or greater than a predetermined difference threshold.

17. When there are a plurality of candidate objects, the predetermined condition is whether or not an object other than the candidate object appears behind any of the candidate objects; 17. The information processing device according to claim 9, wherein, when an object other than the candidate object appears behind the candidate object, the selection means deletes the remaining candidate objects from the candidate objects except for the candidate object behind which the other object appeared.

18. 18. The information processing device according to claim 1, wherein the correction means generates a correction map in which the image is divided into areas according to an expected value for the target object to appear on the image, and sets a correction amount when correcting the result of the matching process to a different value for each area of ​​the correction map.

19. 19. The information processing apparatus according to claim 18, wherein the correction map has at least a positive correction area where a positive correction is performed in which the amount of correction is large, and a negative correction area where a negative correction is performed in which the amount of correction is small.

20. 20. The information processing apparatus according to claim 19, wherein the correction map further includes a no-correction area in which the result of the matching process is not corrected.

21. 21. The information processing apparatus according to claim 20, wherein the correction means sets the positive correction area as an area in the vicinity of the selected candidate object, and sets the negative correction area and the non-correction area based on a movement line of the selected candidate object.

22. 22. The information processing apparatus according to claim 18, wherein the correction means determines the amount of correction based on a past result of the matching process on the detected target object.

23. 23. The information processing apparatus according to claim 22, wherein the correction means determines the amount of correction based on an average value of values ​​representing the results of the past collation processes.

24. 23. The information processing apparatus according to claim 22, wherein the correction means determines the amount of correction based on the amount of variation in the results of the past matching processes.

25. 23. The information processing apparatus according to claim 22, wherein the correction means determines the amount of correction based on information about mismatches in the results of the past matching processes.

26. a detection step of detecting a target object from the image; a matching step of performing a matching process for the target object based on the similarity between the feature amount of the detected target object and the feature amount for matching; a selection step of selecting a candidate object that occludes the target object; a correction step of correcting the result of the matching process performed in the matching step based on the result of the selection performed in the selection step; and The information processing method is characterized in that the correction process corrects the results of the matching process when an object different from the selected candidate object appears in the vicinity of the selected candidate object after the target object is no longer detected.

27. A program for causing a computer to function as the information processing device according to any one of claims 1 to 25.

Images