Information processing apparatus, information processing method, and program

Abstract

To allow stable continuation of tracking even when an object having a similar posture or similar appearance characteristics approaches.SOLUTION: An information processing apparatus according to the present invention that solves the above-mentioned problem is an information processing apparatus that detects at least one or more objects from an image, and has: estimation means that, based on a learned model that has learned image characteristics indicating the shielding relationship between a shielding object and a shielded object, estimates, for each of the objects detected from the image, the shielding relationship with the other object detected from the image; and specification means that, based on the shielding relationship estimated by the estimation means, specifies, for each of the objects detected from the image, the correspondence relationship with an object detected in an image picked up at a time different from that of the image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for tracking a subject. [Background technology]

[0002] There are several techniques for tracking a specific subject in an image, such as those that use brightness or color information, template matching, etc. In recent years, technology that uses Deep Neural Networks (DNN) has been attracting attention as a highly accurate tracking technique.

[0003] Non-Patent Document 1 is one of the methods for tracking a specific subject in an image. An image showing the target to be tracked and an image serving as a search range are input to a Convolutional Neural Network (hereinafter abbreviated as CNN) with the same weights. The position of the target to be tracked in the image of the search range is specified by calculating the cross-correlation between the respective feature amounts obtained from the CNN. While such a tracking method can accurately identify the position of the target to be tracked, it is prone to failing to track the wrong target when an object similar to the target to be tracked overlaps on the screen.

[0004] To avoid this, there are methods, such as that described in Patent Document 1, which create a histogram from the color features and depth information of the area of ​​the detected object and examine the changes in the histogram to determine whether the object is occluded or not. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent Application No. 10185877(B2) [Non-patent literature]

[0006] [Non-Patent Document 1] Bertinetto et al.,Fully-Convolutional Siamese Networks for Object Tracking,arXiv 2016 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the method shown in Patent Document 1 has a problem that when objects with similar postures or similar external features overlap on the screen, it is difficult to determine the difference in the histograms of features such as color and texture. For example, in group sports, multiple people in a small area often have the same clothes and postures, and different people may fail to be tracked as the same person. The present invention has been made in consideration of such problems, and aims to continue stable tracking even when objects with similar external features and postures are close to each other. [Means for solving the problem]

[0008] The information processing device of the present invention that solves the above problem is an information processing device that detects at least one object from an image, and has an estimation means that estimates, for each object detected from the image, an occlusion relationship with other objects detected from the image based on a trained model that has learned image features that indicate an occlusion relationship between an occluding object and an occluded object, and an identification means that identifies, for each object detected from the image, a correspondence relationship with an object detected in an image captured at a time different from that of the image, based on the occlusion relationship estimated by the estimation means. Effect of the Invention

[0009] According to the present invention, stable tracking can be continued even when objects with similar external features and poses are in close proximity. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of object detection; [Diagram 2] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device; [Diagram 3] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 4] 1 is a flowchart showing a processing procedure executed by an information processing device; [Diagram 5] FIG. 13 is a diagram showing an example of a result of processing by an information processing device; [Figure 6] FIG. 13 is a diagram showing an example of a result of processing by an information processing device; [Figure 7] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 8] Example of deriving occlusion information [Figure 9] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 10] FIG. 13 is a diagram showing an example of a result of processing by an information processing device; [Figure 11] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 12] FIG. 13 is a diagram showing an example of a learning process of an information processing device; [Figure 13] FIG. 13 is a diagram showing an example of a result of processing by an information processing device; [Figure 14] FIG. 13 is a diagram showing an example of a result of processing by an information processing device; [Figure 15] FIG. 13 is a diagram showing an example of a result of processing by an information processing device; [Figure 16] FIG. 13 is a diagram showing an example of a learning process of an information processing device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Embodiment 1> The information processing device according to the embodiment will be described with reference to the drawings. Note that the same reference numerals in the drawings perform the same operation, and therefore the description will be omitted. Also, the components shown in the embodiment are merely examples, and are not intended to limit the scope of the present invention to those only.

[0012] In this embodiment, a function for detecting and tracking a person from a video or continuously captured still image frames will be described. The scope of application is not limited to the category of the object to be detected and tracked, but in this embodiment 1, the target is limited to a person. In this embodiment, a person is detected for each image that is consecutive in time, and a person is associated with a person between consecutive images to track the person. In particular, in this embodiment, a sporting event or the like is assumed to be filmed, and the people's clothing, movement direction, etc. are similar, and they are close to each other and cross each other frequently. In such a case, if people who have similar external characteristics, such as the position of the person in each image or the color of their clothing, are simply associated with each other, erroneous association is likely to occur. Such a failure is called erroneous matching here.

[0013] In this embodiment, we focus on occlusion information output by a trained model that has learned the occlusion pattern when objects overlap when viewed from the photographer. By using the occlusion relationship output by the trained model in addition to identifying the correspondence between objects, we can reduce mistakes in matching a person in the foreground with a person in the background, and improve tracking accuracy.

[0014] FIG. 1 is a diagram that shows this diagrammatically. Images 2100, 2120, and 2140 in FIG. 1(A) show three still frames of a video showing two playing cards of the same design crossing over each other on a table from above (arranged from left to right in chronological order). It is not possible to determine how the cards moved in each image by observing only images 2100, 2120, and 2140. On the other hand, FIG. 1(B) is an example of the same scene captured at a higher frame rate than (A). In other words, it is a group of images captured at a shorter time interval. By observing the images in FIG. 1(B) in chronological order, it is possible to associate which card moved to where in the next image. Overall, it is relatively easy to estimate that the card 2201 on the left side passed over the card 2202 on the right side and then moved to the right side. As shown in images 2210 and 2230, by observing the appearance that occurs transiently at the moment when objects cross, it is possible to determine which card passes in front and which is in the back in image 2220. This determination does not necessarily require special sensors such as 2.5-dimensional depth images or information with high generation costs such as optical flow. It can be solved as a pattern recognition problem between the occlusion relationship and appearance features, which is what appearance is likely to occur when objects overlap in front and back. This is also true for scenes such as the intersection of people shown in Figures 1(C) and 1(D). In these Figures 1(C) and 1(D), the appearance of the people's clothes, posture, etc., and the direction of movement are assumed to be the same. In such a case, if the observation is made while paying attention to the appearance state before and after the objects cross, it is determined that person 2401 is in the front and person 2402 is in the back in image 2420 of Figure 1(D). If this occlusion relationship is maintained in subsequent images, when the person 2401 occludes the person 2402 once, the person 2401 in the foreground is tracked, and the occlusion relationship and image features of the person 2401 in the background are maintained. Then, when the occlusion is removed, it is possible to continue tracking the person 2401 and detect the person 2402 in the background again. The above is an explanation showing an outline of the principle of this embodiment. Detailed processing will be described later.

[0015] FIG. 2 is a hardware configuration diagram of the information processing device 1 that tracks a tracking target by image recognition in this embodiment. The CPU H101 executes a control program stored in the ROM H102 to control the entire device. The RAM H103 temporarily stores various data from each component. In addition, the program is expanded to make the CPU H101 executable. The storage unit H104 stores data to be processed in this embodiment, and stores data to be tracked. As a medium for the storage unit H104, a HDD, a flash memory, various optical media, etc. can be used. The input unit H105 is composed of a keyboard, a touch panel, a dial, etc., and accepts input from a user, and is used when setting a tracking target, etc. The display unit H106 is composed of a liquid crystal display, etc., and displays the subject and the tracking result to the user. In addition, this device can communicate with other devices such as a shooting device via the communication unit H107.

[0016] FIG. 3 is a block diagram showing an example of a functional configuration of an information processing device. In FIG. 3, the processes executed in the CPU H101 are shown as functional blocks. The information processing device 1 has an image acquisition unit 201, an object detection unit 202, an occlusion information generation unit 203, an extraction unit 204, and a matching unit 205, and is connected to an external storage unit 206. The storage unit 206 may be inside the information processing device 1. The functions of each unit will be briefly described. The image acquisition unit 201 acquires images of a video or continuous still images in which a specific object (a person in this embodiment) is captured by an imaging device. The object detection unit 202 detects image features indicating a predetermined object set in advance from the images acquired by the image acquisition unit 201. For example, the area of ​​a person in an image is detected based on a learned model that has previously learned image features indicating a human body (head and moving body) using images of people in various postures. The occlusion information generating unit 203 estimates occlusion information indicating an occlusion relationship between each object detected from an image and other objects detected from the image based on a trained model that has learned image features indicating an occlusion relationship between an occluding object and an occluded object. The occlusion information is a likelihood (occlusion / occlusion score) that indicates the possibility that an object of interest is occluded by another object. For example, if an object is highly likely to be occluded by another object, the occlusion / occlusion score is made closer to 1. If an object is highly likely to occlude another object, the occlusion / occlusion score is made closer to 0. The occlusion / occlusion score indicating such an occlusion relationship is estimated using the trained model. The extracting unit 204 stores the occlusion information for each object detected in an image in the storage unit 206. The matching unit 205 matches objects detected between a plurality of images. That is, based on the occlusion information, a correspondence relationship between each object detected from an image and an object detected in an image captured at a different time from the image is identified. By correctly associating objects detected from images captured at different times, the objects can be tracked. In addition, by assuming that the occlusion relationship between the objects is maintained for a certain period of time, the objects can be associated with each other using the occlusion relationship. The storage unit 206 stores the occlusion score of each detected object. The details of the processing of each functional unit will be described using the flowchart of FIG. 4.

[0017] Fig. 4 is a flowchart showing the process flow of this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted. However, the information processing device does not necessarily have to perform all the processes described in this flowchart. The process shown in the flowchart of Fig. 4 is executed by the CPU H101 of Fig. 2, which is a computer, in accordance with a computer program stored in the storage unit H104.

[0018] In S301, the information processing device 1 starts a loop process that is repeated for each video frame. In S302, the image acquisition unit 201 sequentially acquires image frames of a video or continuous still images capturing a person. Subsequent processes from S301 to S311 are performed sequentially for each image. Note that the image acquisition unit 201 may acquire images captured by an imaging device connected to the information processing device, or may acquire images stored in the storage unit H104. Video frames 3100, 3110, 3120, 3130, and 3140 in FIG. 5(A) are examples of acquired image frames.

[0019] Next, in S303, the object detection unit 20 detects at least one or more predetermined objects from the acquired image based on the image features of the predetermined objects (here, a person). A known technique for detecting objects from within an image is the method by Liu (Liu, SSD: Single Shot Multibox Detector. In: ECCV2016). The result of detecting candidate objects from within an image is shown in FIG. 5(A). The rectangular frames 3101, 3102, 3103, 3111, 3112, 3113, 3121, 3122, 3131, 3132, 3141, and 3142 in FIG. 5(A) are bounding boxes (hereinafter referred to as BB) indicating the detected object regions.

[0020] In S304, the occlusion map generating unit 203 generates an occlusion map showing occlusion information on whether each region is occluded (occlusion relationship) for each image. The occlusion map generating unit 203 estimates the visible region (occluded object region) of the occluded object for each image. Here, it is determined whether each person overlaps with another person, and if so, whether the person is in the back or in the front, and the result is output as an occlusion state score (likelihood) for each region. This is a type of recognition task of semantic region segmentation, and can be realized using a known method such as the method of Chen et al. (Chen, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, 2016).

[0021] FIG. 6(A) shows a schematic diagram for explaining the generation process of the occlusion map and an example of the result. The neural network 402 is a neural network that judges the occlusion state of each pixel of the input image from the input image. When an RGB image 401 is input, the neural network 402 outputs the result of estimating whether or not a person is present in the image and whether or not the person is occluded as an occlusion map 404. The map outputs an occlusion score of 0 for an area estimated to be an unoccluded person or a non-person area, and 1 for an area of ​​an occluded person. The darker the area in the occlusion map 404, the higher the occlusion score. In other words, the black area indicates that it is estimated to be an area of ​​an occluded person. The neural network 402 has previously been trained so that it can output such an output for the input image (learning will be described later). The occlusion map 404 shown in the figure shows an example of an ideal output state as an estimation result.

[0022] In addition to the RGB image 401, a derivative form may be considered in which a 2.5-dimensional depth image 405 is separately acquired using a dedicated sensor or the like. The depth image 405 is linked with the 3-channel RGB image 401 to obtain 4-channel information, which is learned and recognized as an image input instead of an RGB image. This makes it possible to improve the accuracy of information on occluded areas.

[0023] Next, in S305, the information processing device 1 executes a loop process from S306 to S307 for each object detected in S303. In S305 to S308, the extraction unit 204 extracts information indicating the occlusion relationship for each detected object from the generated occlusion map, and stores the information in the storage unit 206. In S306, the extraction unit 204 extracts information indicating the occlusion relationship for each detected object from the occlusion map. Specifically, the occlusion scores of the occlusion map 404 in the person detection frame 407 in FIG. 6(A) are averaged. The closer the occlusion score is to 1, the higher the possibility that the object is occluded, and the closer the occlusion score is to 0, the higher the possibility that the object is not occluded. Note that, assuming that the position of the detection frame is shifted and that the occlusion map 404 contains noise, the weighted average is acquired with emphasis on the center of the frame as shown in FIG. 6(B). The calculation 412 in the figure means the element product (Hadamard product) for each partial region of each image. Map 413 has a peak in the center and is a two-dimensional Gaussian function whose image block sum is 1 (the vertical and horizontal sizes are transformed to fit the person detection frame). An example of the obtained results is shown in FIG. 6(A) with the symbol occ as occlusion score values ​​409 and 410. The detection frame on the left is determined to have a high occlusion score because the person is at the back, and the detection frame on the right is determined to have a low occlusion score because the person is at the front. Examples of the results of processing the above-mentioned process on the input image in FIG. 5 are shown in FIG. 5(B) (occlusion map) and FIG. 5(C) (estimated occlusion score for each frame). It shows that the occlusion score corresponding to person 3101 located at the back is relatively higher than that of person 3102 between the start and end of the intersection.

[0024] In S307, the storage unit 206 stores the occlusion score of each detected object acquired by the extraction unit 204. At the same time, information on the position and size of each detected object, and feature quantities related to the appearance of the object, such as color and texture histograms, are also stored. Here, these multiple types of quantities are collectively referred to as the feature quantities of the detected object. Note that, in addition to this, intermediate layer information of a neural network may be used as the feature quantities related to the appearance. (For example, "Hariharan, et.al, Hypercolumns for Object Segmentation and Fine-grained Localization, inCVPR2015").

[0025] In S308, the information processing device 1 ends a loop that is repeated for each image and a loop that is repeated for a person detected in each image. This loop ends when occlusion information is acquired for each image for all people detected in that image. Next, in S309 to S310, the matching unit 205 matches objects between previous and next images (however, in the case of the first video frame, this is not performed because there is no previous frame). First, in S309, the matching unit 205 acquires the occlusion score, the position size, and the appearance feature, which are the feature amounts of past objects stored in the storage unit 206. Next, in S310, the matching unit 205 matches objects detected in past video frames with objects detected in the frame currently being processed.

[0026] A detailed process flow of the matching unit 205 in S310 is shown in Fig. 7. In S501, the matching unit 205 first creates pairs of all combinations between the detected objects in the current frame and the objects detected in the previous frame. If n people and m people are detected in the previous and next frames, respectively, a total of n x m pairs are generated. Next, in S502, the matching unit 205 calculates the similarity between all object pairs. As the similarity, an index based on the difference in the feature amounts between the detected objects can be used. As an example, the similarity between the previously detected object c1 and the currently detected object c2 is calculated as follows: (Formula 1) L(c1,c2)=-W1||BB1-BB2||-W2||f1-f2||-W3||occ1-occ2||Here, BB is a vector that combines the four variables (center coordinate value x, center coordinate value y, width, height) of each object, and f indicates the characteristics of each object. ||x|| is the L of x. p norm. occ is the occlusion score of each object. W1, W2, and W3 are balance coefficients equal to or greater than 0, which are set by empirical or machine learning adjustment. Here, the variation of each feature may be statistically obtained in advance and each feature may be normalized. Even when objects intersect, by tracking a person who is occluding another object, when the occluded person is confirmed again in the image, the value of the third term in Equation 1 becomes small and the similarity is calculated to be low when matching with the person who occluded the other object immediately before. In other words, this process reduces the possibility of matching people with different occlusion relationships, and erroneous matching of tracking can be suppressed.

[0027] Next, in S503, the matching unit 205 performs matching (matching) to identify a correspondence between objects based on the similarity between the past object and the current object. There are several matching methods. For example, there is a method of matching candidates with higher similarity first, a method using the Hungarian algorithm, etc. Here, the former is used.

[0028] In S503, if the matching unit 205 has not yet completed matching for all objects in the current frame, it will proceed to S506 to match objects as being the same person, starting with the pair with the greatest similarity. Objects of pairs for which matching has been completed are removed from the matching candidates. During the above process, if the magnitude of the greatest similarity among the pairs remaining at that point falls below a predetermined threshold, this means that there are no more similar object pairs remaining. In that case, no further matching is attempted (S505), and matching ends.

[0029] The above steps S301 to S311 are performed for each video frame. As a result, as shown in the example of the result in Fig. 5(D), people are detected from the video, and a series of tracking results showing where each object has moved to are obtained (the same people between frames are marked with symbols A, B, and C to show the tracking results).

[0030] <Modification> In this embodiment, the distance of each index such as the occlusion score and the appearance is weighted and summed based on the difference as the similarity of matching between object pairs. Here, for example, KL divergence can be used. It is also possible to perform metric learning to obtain a more accurate distance index. In addition, instead of using a single similarity only once, a rule-based method or a stepwise judgment method can be used in which the similarity is first judged based on the appearance feature, and if the condition is satisfied, it is judged based on the similarity of the occlusion score. Furthermore, it is also possible to use a known classifier method such as a neural network or a support vector machine to learn and classify the explanatory variable as a feature amount and the objective variable as the result of whether or not the object is the same, and to judge the matching based on this value. As described above, the correspondence between objects between frames is not limited to a specific form.

[0031] As yet another variation, in order to stabilize the estimated value of the occlusion score, a moving average of past scores may be used as shown in the following formula. (Formula 2) occ EMA (t) = (1-α) × occ EMA (t-1) + α×occ (t) The above formula is called the exponential moving average value, and OCC EMA (t) is the exponential moving average of the occlusion score at time t, occ (t)is the occlusion score at time t, and α is a coefficient between 0 and 1. For objects that have been successfully tracked in multiple past frames, the exponential moving average is calculated using the above formula, and when comparing similarities, the exponential moving average occlusion score is used instead of the original occlusion score. This makes it possible to perform matching based on the average occlusion score for multiple frames when gradual overlap occurs over multiple frames during intersection, resulting in more stable correspondence between objects.

[0032] As yet another derived form, a form in which matching is performed using not only the similarity between previous and next frames, but also the feature values ​​and positions of multiple past frames n steps back is also considered. By using this method, even if a frame occurs in which an object is occluded and cannot be tracked, tracking becomes possible again if the occlusion is removed in a subsequent frame. In this form, for example, the average value of the feature values ​​of the object is calculated going back to n frames, and the similarity with the object detected in the current frame is calculated based on this. Alternatively, the similarity between the object in the past n frames and the object in the current frame is calculated, and the object with the highest average value of the obtained n similarities is associated with the object. It is also possible to make a judgment in both directions using not only the past but also the results of future frames n steps. This form is inferior in real-time processing because the results are not known until future frames are processed, but it is more accurate than the method of looking only at the past.

[0033] Another possible derivative is a form to deal with detection failures. In object detection and tracking, temporary failures in the object detection stage can occur due to reasons such as the object's posture changing to a special shape. If such a failure to detect occurs, when matching between frames, an object that existed in the previous frame is determined to not match in the current frame. Then, tracking will be interrupted. In order to prevent such failures, the following can be devised. That is, if an unmatched person occurs in matching, the information is stored in a list and added to the candidates for matching when matching the next frame (if the object is still unmatched after a certain time has passed, it is determined that the object itself no longer exists and is removed from the list. Here, this is called timeout processing).

[0034] Various methods of matching objects across video frames in this manner are possible, and the method is not limited to a specific form.

[0035] <Variations in the form of occlusion information and learning methods> In this embodiment, the visible area (occluded object area) of the occluded object is estimated as the occlusion map. Various derived forms are also conceivable. An example is shown in FIG. 8. Here, as shown in FIG. 8(B), it is acceptable to estimate other than the visible area of ​​the object on the back side as in image 801. For example, the entire area of ​​the object on the back side is estimated as in image 802. Also, as in image 803, the area of ​​the occluding object on the front side is estimated (an object that does not overlap with other objects as in area 440 in the figure is also estimated as the front side area. However, as another form, it is conceivable not to include such a single object in the front side area). Also, instead of estimating the foreground area as in images 801 to 803, it is conceivable to estimate the position of the center or center of gravity of the object as in image 804. In the case of image 804, a large positive value is estimated in the area near the center of the occluded object, and a small negative value is estimated in the area near the center of the occluded object (the central area of ​​the object here can be estimated as a Gaussian function area as shown in the figure).

[0036] Here, a learning method of information on the occlusion state will be described with reference to FIG. 6(D). A neural network 402 shown in the above-mentioned Chen et al. method etc. outputs an occlusion score map 403 of an occluded object for an input image, an RGB image 401. An example of the result of 403 is shown in 430. The CHEN et al. method etc. is a method for estimating the foreground region of a specific category object, but here, a teacher value 431 of occlusion information is given, and the neural network 402 is trained so that a map similar to the teacher value 431 can be obtained by estimation. Specifically, the output result map 403 is compared with the teacher value 431, and a loss value calculation 432 is performed by a known method such as cross entropy or square error. The weight parameters of the neural network 402 are adjusted by the error backpropagation method or the like so that the loss value gradually decreases (this process is the same as the Chen et al. method, so details are omitted). It is necessary to give a sufficient amount of input image and teacher value. Since creating training values ​​for overlapping object regions is costly, it is possible to create learning data using CG or image synthesis, which cuts out and overlaps object images.

[0037] Furthermore, in this embodiment, an index called an occlusion score is obtained within the frame of the object obtained above. Various examples of the form of obtaining occlusion information are shown in FIG. 8(C). FIG. 8(C1) shows the form of this embodiment. In addition, various methods are conceivable, such as (C2) obtaining the difference value between the occlusion score on the back side and the occlusion score on the front side within the object frame, and (C3) referring to only one point of the score at the center of the object. In addition, when obtaining within a frame, a method is conceivable in which an area overlapping with another object frame is omitted from the obtaining because it is unclear which object's area it is.

[0038] Furthermore, a method of simultaneously performing the above-mentioned "estimation of the occlusion state" and "acquisition of the occlusion score of each object" is also conceivable. As an example, there is a method called "anchor", which is a well-known method used in the Liu method and the like. Since this method obtains a set of candidate frames for objects, it is conceivable to estimate the occlusion score of each candidate frame as to whether it is an occluding object or an occluded object and associate them using this (the details of this form will be shown in the third embodiment, so the explanation will be omitted here).

[0039] Furthermore, the occlusion information in a plurality of forms as described above may be acquired and used as a multidimensional feature related to occlusion for subsequent object matching. Alternatively, the proportion of the occluded area of ​​the object may be estimated by machine learning from the multidimensional feature related to occlusion. In this case, the multidimensional feature related to occlusion may be used as an explanatory variable, the proportion of the occluded area of ​​the object may be used as a target variable, and regression estimation may be performed using a known technique such as logistic regression.

[0040] <Embodiment 2> In this embodiment, people are detected and tracked in the same way as in the first embodiment. The hardware configuration is the same as in FIG. 2 of the first embodiment. FIG. 9(A) is a block diagram showing an example of the functional configuration in this embodiment. An occlusion state determination unit 301 is newly added to the configuration of the first embodiment. In the first embodiment, the frame of a person may not be able to detect a person during tracking due to overlapping of the people. For example, as shown in video frame 4120 in FIG. 10(A), when the overlapping area between people is large, it is often the case that the person in the back cannot be detected. In such a case, the occlusion state determination unit 301 determines that a person is present but is in an occluded state.

[0041] Unlike non-detection caused by temporary failure of the object detection unit as described in the first embodiment, when a group of people are moving in the same direction at the same speed, the non-detection state continues for a long time. Furthermore, the position on the screen where the object has left the occluded state may be far from the position where the occluded state began. For this reason, it is desirable to proactively determine that the object is in an occluded state and perform processing according to the estimated state to increase the success rate of tracking.

[0042] The overall processing flow of this embodiment is the same as that of FIG. 4 of the first embodiment, but the details of the processing of S310 are different as follows. Here, only the processing of S310 that differs from that of the first embodiment will be described. A detailed flow of the processing of S310 performed by the occlusion state determination unit 301 will be described with reference to FIG. 11. First, in S601, objects are associated with each other in the current frame and the previous frame as in the previous embodiment. If there is an object in the previous frame that could not be associated with each other in S602, there is a possibility that the object has entered an occluded state. Therefore, in S603, it is checked whether the occlusion score of the object up to that point is equal to or higher than a threshold value. This is because if the frame rate of the video frame is sufficiently high, the occlusion score often becomes high before and after the object becomes undetected due to occlusion. Furthermore, in S604, it is checked whether the number of detections of the object in the current frame has decreased in the surrounding area of ​​the object, and if the above two results are true, it is estimated that the object has entered an occluded state and is stored in the list of occluded states (S605). For the objects stored in the list of occluded states, the feature amount and position at the time of the previous detection are also stored. This improves the chances of tracking when the object is unobstructed and redetected.

[0043] S606 to S610 are processes for judging whether an object in an occluded state has reappeared. If there is an object in the current frame that was not associated in S603, there is a possibility that it has escaped from the occluded state and can be detected again. Therefore, in S607, it is checked whether the occluded score of the object is equal to or higher than a threshold. Furthermore, in S608, it is checked whether the number of detections of the object in the current frame has increased in the peripheral area of ​​the object. If both results are true and the object has a similarity to any of the objects stored in the occluded state list that is higher than a predetermined threshold (S608), it is estimated that the object has escaped from the occluded state and reappeared. At that time, the associated object is removed from the occluded state list (S609). For the object removed from the occluded state list, the feature amount and position detected from the current input image are obtained.

[0044] Here, some ideas for the matching process could be, for example, reducing the distance-based similarity penalty when matching with an occluded person when matching objects between frames, lengthening the timeout period for waiting for a reappearance, setting a lower threshold for matching with occluded objects than for matching between normal objects, etc.

[0045] Furthermore, although the explanation here assumes two people overlapping, overlaps can occur between three or more people. In this case, if it is determined that a person has entered an occluded state, they are added to the list of occluded states, and when they reappear, they are associated with the previous frame and removed from the list of occluded states each time. This makes it possible to handle cases of three or more people to a certain extent.

[0046] <Embodiment 3> In this embodiment, a form of tracking a single object designated by a user will be described. Here, the tracking target is not limited to a specific category such as a human body, but a form of tracking an unspecified object designated by a user will be dealt with. For example, it may be an animal such as a dog, or a vehicle such as a car.

[0047] The functional block diagram is shown in FIG. 9(B). A new tracking object designation unit 302 has been added to the previous configuration. The functions of the tracking object designation unit 302 and the object detection unit 202 can be easily realized by using the method of Non-Patent Document 1. The tracking object designation unit 302 is a functional unit in which the user designates the frame position of the tracking target object in the video frame. This initializes the features of the object to be tracked. The object detection unit 202 identifies the image area in each video that is most similar to the target object. An example of the identification result is shown in FIG. 12(A). The frame 5111 on the video frame 5110 in FIG. 12(A) is the frame of the tracking object designated by the user. In the video frame 5120, this object moves to the right side in the screen, and is detected as a frame 5121 by the object detection unit 202. The method of Non-Patent Document 1 is an excellent object tracking method, but erroneous switching easily occurs between similar objects. Therefore, in this embodiment, as in the previous embodiments, information regarding the occlusion state of the tracking object is estimated, and it is determined whether or not an erroneous switch has occurred.

[0048] For this purpose, a neural network 6300 as shown in FIG. 13B is used as the occlusion information generating unit 203. This is a classifier that, when an image 6301 of a detected tracking object (here, the aspect ratio is normalized to a square image for ease of processing) is input, looks at the image pattern and outputs a classification result 6302 of whether it is occluded (Yes) or not (No). The presence or absence of occlusion is defined as the percentage or more of the object's area that is occluded. The values ​​of these two classes are given as teacher values ​​to train the neural network 6300. This technology is a widely known method similar to normal image classification tasks, so details will be omitted. Also, if the teacher value (target variable) is given as the percentage of the occluded area rather than the binary value of occlusion or not and regression learning is performed, the occlusion percentage can be estimated as in the estimation result 6303. In this regression learning, a square error or the like is used as a loss value given during learning.

[0049] 14A and 14B show the results of estimating the occlusion degree of a candidate object to be tracked by the occlusion information generating unit 203. In response to the object detection result shown in Fig. 14A, the estimated occlified area by the object detection unit 202 is shown in Fig. 14B with the symbol occ. In the figure, the fluctuation range of the occlusion score is smaller than a predetermined value (e.g., a value of 0.3), and it can be determined that tracking has not failed (here, the success or failure of tracking may be determined using not only the occlusion score but also the similarity of the feature amounts of position and appearance as used in the first embodiment).

[0050] On the other hand, in Fig. 14(C), object 7201 passes behind object 7202 from video frame 7220 to 7230, and as a result, the object detection unit 202 erroneously estimates the position of the object in video frame 7230 as frame 7231. In this case, the occlusion score fluctuates significantly from 0.4 to 0.0 as shown in Fig. 14(D), so it can be determined that erroneous tracking has occurred due to the intersection. If it is found that erroneous tracking has occurred, it is possible to take measures such as stopping detection at that point or correcting it at a later stage.

[0051] This concludes the description of this embodiment.

[0052] It is preferable that the occlusion information generating unit 203 learns so as to be able to estimate the occlusion states of various objects so as to be able to determine the occlusion state of unspecified objects, as shown in 5110 to 5150 in FIG.

[0053] As another derivative form, in Fig. 13(B), an image 6301 cut by an object frame is shown as an input image. However, since it is important to observe not only the object but also its surroundings in order to determine whether or not it is in an occluded state, it is also possible to input a wider range as an input image (in that case, a similar range is also cut out and input during estimation).

[0054] As another derived form, as shown in FIG. 13(C), a form in which object detection and occlusion degree estimation are performed simultaneously using candidate frames called anchors, as in the Liu method described above, is conceivable. The anchor frames are a set of candidate frames with multiple sizes and aspect ratios as shown in FIG. 13(D) (three types of anchor frames are illustrated here). The anchor frames are placed in each block region in the image, as shown in the result image 6450 in FIG. 13(C). When an image is input, the neural network 6400 generates an occlusion score map 6430 indicating whether or not the object is present in each anchor of each block region. The occlusion score map 6430 is a set of three maps corresponding to the three types of anchor frames. An example of the estimation result is shown in FIG. 13(C) 6450 (the above method is widely known, so please refer to the Liu method described above for details).

[0055] Here, as a derived form of this embodiment, an object occlusion score map 6440 is generated at the same time as estimating whether an object exists. This is a map that estimates what the occlusion ratio would be for each anchor frame if an object exists there. This map also consists of three sheets corresponding to the number of anchor types (during learning, a teacher value of the occlusion score can be given to each anchor frame in each block of the image to learn the neural network 6400). An example of the result is shown in FIG. 13(C) 6460. An example of the final integration of the two estimated maps is shown as an example of integration result 6470.

[0056] The above explanation is an example of object detection, but since the method in Non-Patent Document 1 is also an anchor candidate frame-based method, it is possible to configure a derivative form that tracks an object while simultaneously estimating its occlusion score.

[0057] <Embodiment 4> In this embodiment, a form in which a single object designated by a user is tracked will be described. The diagram of the functional blocks is the same as that of the third embodiment, and is shown in FIG. 9(B). In the previous embodiments, when comparing similarities, object matching between frames is performed on a rule basis, such as by comparing features between the immediately preceding and following frames, or by comparing using n frames before and after. In this embodiment, this part is replaced with machine learning to perform more accurate matching.

[0058] A recurrent neural network is a technology that can process time-series data to perform identification and classification, and a representative method is a long short term memory network (hereinafter referred to as LSTM), which is known from the method of Byeon et al. (Byeon et al.,Scene labeling with LSTM recurrent neural networks, CVPR 2015). This method can determine changes in the features of objects over time and associate objects with each other. A schematic diagram of the configuration and result example of this embodiment is shown in FIG. 15. Here, one object 9102 is specified as the tracking target and is tracked by the method of Bertinetto et al. (a false switch occurs in the video frame at t=2). LSTM units 9501 to 9504 in FIG. 15(C) receive features 9401 to 9404 of the object being tracked at each time, determine whether tracking is successful or unsuccessful, and output them as outputs 9701 to 9704. Here, multiple LSTM units are drawn for illustration, but this does not show the existence of multiple units, but the state of the same unit at each time. The LSTM unit at each time sends a recurrent input 9802 to the LSTM unit at the next time. The LSTM unit changes its internal state as necessary based on the recurrent input 9802, which contains the features of the object at that time and past information up to that point. This makes it possible to determine whether the current tracking is successful or not, taking into account how the object pattern has changed over time.

[0059] The feature quantities input to the LSTM unit can be the features of the neural network described in the third embodiment. For example, the input value to the final layer 6320 of the neural network 6310 that determines the occlusion score of the object in FIG. 13(B) is used. Here, the input value (multidimensional vector) is used instead of the output value (single-valued scalar) of the previous layer. This is because by using the same multidimensional features as those used to determine the occlusion state, various types of information regarding occlusion can be incorporated into the LSTM. This is expected to enable various occlusion patterns to be determined.

[0060] When the LSTM unit is trained, whether tracking at each moment is successful or unsuccessful is given as a teaching value, and each weight parameter of the LSTM is adjusted. As another form, as shown in Fig. 16(D), if the teaching value is given as to whether the object is in an occluded state or not, instead of the success or failure of tracking, it is possible to train the unit to determine whether it is in an occluded state or not.

[0061] As another embodiment, similar to the derived embodiment described in the third embodiment, a tracked object may be detected based on an anchor frame, and feature amount 6420 of neural network 6400 that simultaneously determines the object's position and occlusion score in Fig. 13(C) may be used as 9401. With this embodiment, it is possible to simultaneously and quickly determine the tracking and detection of an object and the occlusion score of the object.

[0062] <Embodiment 5> In this embodiment, a form in which a single object designated by a user is tracked will be described. The basic functional configuration is the same as in embodiment 1. In this embodiment, relative perspective information between objects is used as occlusion information of the object.

[0063] FIG. 16(A1) shows an example. Here, an RGB image 801 is prepared as a learning image. Furthermore, a distance image 833 corresponding to the RGB image 801 is obtained by a laser range finder device, stereo measurement, or the like. The distance image 833 is a grayscale representation of the absolute value of the distance from the camera, and the whiter the color, the closer the object is. In this embodiment, the RGB image 801 is used as an input image, and the distance image is used as a teacher value 831 to learn the weights of the neural network 402. However, obtaining an output result 830 that is exactly the same as the distance image 831 as an absolute value is a relatively difficult problem in terms of pattern recognition, and the occlusion information used in this embodiment does not need to be so highly accurate. Therefore, in this embodiment, learning is performed to estimate the relative perspective relationship between nearby objects.

[0064] For example, as shown in output result 830 in the figure, distance estimation results 8301 and 8302 for persons 8011 and 8012 are incorrect as absolute values. Estimation results 8301 and 8303 for person 8013, who is distant from person 8011, also have incorrect perspective relationships. However, if we limit the perspective order relationship to two nearby persons 8011 and 8012, the results are correct. In this way, the system learns to be able to correctly estimate the <perspective order relationship> between <local objects>, and these are collected and used as object occlusion information.

[0065] The above is realized by applying the following device to the calculation of loss value during learning. FIG. 16(A2) shows a teacher value area 831a obtained by enlarging the area around the symbol * on the teacher value 831 in FIG. 16(A1). The corresponding output result area 830a is also shown. Here, attention is paid to each pixel i and pixel j on the area 831a, and the loss of the pixel pair is calculated based on whether the perspective relationship is correct or not. Here, the perspective relationship between pixel i and pixel j on the area 830a matches the teacher value, so no loss occurs. On the other hand, if the estimated result is as in area 830b, the perspective relationship is incorrect, so a loss is recorded. This judgment is performed for all pixel pairs within a specified distance. Finally, the total loss value is calculated by dividing the number of pairs with incorrect perspective relationships by the total number of pairs. The neural network 802 that has been trained in this way finishes learning, and the output result 834 of the estimated distance is shown in FIG. 16(B).

[0066] Next, the relative distance output results 834 are aggregated to obtain the occlusion likelihood of the object. Here, the results are aggregated for each detection frame using the person detection frames 8351 and 8352 that were detected separately. The distance values ​​within each frame are averaged, and d ave 1 and d ave 2. Next, the distance values ​​are compared between adjacent object frames, normalized, and converted into a score value occ of the occlusion likelihood. For example, the conversion is performed using the following formula: (Formula 3) occ i =Sigmoid(Log(d ave i / d ave j ))=1 / (1+d ave j / dave i ),occ j =1 / (1+d ave i / d ave j ),however Sigmoid(x)=1 / (1+exp(-x)). where i and j are two adjacent detection object frames with overlapping parts. If three or more objects overlap, the occlusion score occ is calculated by the above formula. i and the maximum value among them may be used as the occlusion score of the object.

[0067] The above is the processing content from relative distance estimation to tallying up the occlusion scores. The tracking processing using the occlusion scores is the same as in the first embodiment, so it will be omitted here.

[0068] The following are some of the derivative learning ideas that can be considered. (1) Losses are tallied only for pairs where the difference between the teacher distance values ​​is equal to or greater than a certain threshold Θ. This allows learning to be robust to noise when observing distance images. (2) (1) is performed, and a margin area is set. For example, losses are generated not only when the perspective relationship of the pair is correct or not, but also when the perspective relationship is correct and the values ​​are not relatively far apart by more than a certain threshold Θ. (3) Losses are also given as errors when the output value for a pixel pair where the difference between the teacher distance values ​​is less than the threshold Θ is greater than the threshold Θ. This suppresses noisy output.

[0069] As described above, there are various possible forms, but any form that can learn distance relatively and locally is acceptable, and is not limited to one form. The present invention can also be realized by executing the following process. That is, software (program) that realizes the functions of the above-mentioned embodiments is supplied to a system or device via a data communication network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium. [Explanation of symbols]

[0070] 1. Information processing device 201 Image Acquisition Unit 202 Object detection unit 203 Shielding information generation unit 204 Feature Acquisition Unit 205 Mapping section 206 Memory section

Claims

1. An information processing device that detects at least one object from an image, an estimation means for estimating, for each object detected from the image, occlusion information indicating the occlusion relationship between the occluding object and another object detected from the image, based on a trained model that has learned image features indicating the occlusion relationship between an occluding object and an occluded object; and an identification means for identifying, for each object detected from the image, a correspondence relationship with an object detected in an image captured at a different time from the image based on at least the occlusion information.

2. The information processing device according to claim 1 , wherein the estimation means estimates the occlusion information indicating a partial area of ​​an object that is occluded for each object detected from the image based on the trained model that estimates an occluded area of ​​the object in the input image.

3. The information processing device according to claim 1 or 2, characterized in that the identification means identifies a correspondence between the image and an object detected in an image captured at a different time from the image, based on the image features of the object and the occlusion information estimated by the estimation means.

4. the estimation means estimates the occlusion information, which is a likelihood indicating that each object detected from the image is occluded by another object; 4. The information processing apparatus according to claim 1, further comprising a storage unit for storing the likelihood in association with an image feature of the object.

5. The information processing device according to claim 4 , wherein the occlusion information indicates, for each region of the image, a larger likelihood in a region of the object that is occluded and a smaller likelihood in other regions.

6. The information processing device according to any one of claims 1 to 5, characterized in that the identification means identifies, for each object detected from the image, a correspondence relationship between the object detected from an image captured at a different time from the image based on the position in the image, the image features detected from the image, and the occlusion relationship in the image.

7. The method further includes acquiring an area for each object from the image, The information processing apparatus according to claim 1 , wherein the estimation means estimates the occlusion information indicating the presence or absence of an occluded object for each of the object regions acquired by the acquisition means.

8. a determination means for determining whether or not an object is occluded based on a correspondence between each object detected from the image identified by the identification means and an object detected in an image captured at a different time from the image; and 8. The information processing apparatus according to claim 1, further comprising: a storage unit for storing the object that is obstructed.

9. the determining means determines, as a first object, an object that does not correspond to any object detected in an image captured prior to the image captured, and 9. The information processing apparatus according to claim 8, wherein said storage means stores information indicating that said first object was occluded at the time when said image was captured.

10. the determining means determines, among the objects detected from the image, an object that does not correspond to an object detected in an image captured prior to the image as a second object; The information processing device according to claim 9, characterized in that the storage means stores information indicating that the first object was not occluded at the time the image was captured when a similarity between the second object detected from the image and the first object determined by the storage means to be occluded in an image captured prior to the image is greater than a predetermined threshold value.

11. If two objects are detected in a first image and one object is detected in a second image captured after the first image, The estimation means estimates the occlusion information indicating that an object detected in the second image occludes another object, The information processing device according to any one of claims 1 to 10, characterized in that the identification means is configured to identify, based on the occlusion information estimated by the estimation means, an object detected in the first image that is occluding other objects, as being the same object as the object detected in the second image.

12. When two objects are detected from a third image captured after the second image, the estimation means estimates the occlusion information indicating that an object detected from the third image, which is associated with an image feature of the object detected from the second image, is occluding another object; and The information processing device according to claim 11, characterized in that the identification means identifies, for an object detected from the first image that is occluded by another object, an object detected from the third image that is different from an object associated with an image feature of an object detected from the second image, as being the same object as the object occluded in the second image.

13. The information processing device according to claim 1 , wherein the trained model is a neural network.

14. The information processing device according to any one of claims 1 to 13, characterized in that the trained model is a model that has learned image features indicating an occlusion relationship of the object based on a plurality of images captured at a shorter time interval.

15. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 14.

16. An information processing method for detecting at least one object from an image, comprising: an estimation step of estimating, for each object detected from the image, occlusion information indicating an occlusion relationship between the occluding object and another object detected from the image, based on a trained model that has learned image features indicating an occlusion relationship between an occluding object and an occluded object; and an identification step of identifying, for each object detected from the image, a correspondence relationship with an object detected in an image captured at a different time from the image based on at least the occlusion information.

Images