Information processing device, information processing method, and program

Abstract

In order to reduce the load of object tracking processing, an information processing device according to the present invention comprises: an image acquisition means for acquiring a target image including a plurality of objects; an object-image specification means for detecting an object included in the target image and specifying, in the target image, an object image that is an image of a region surrounding the object; an aggregation object determination means for determining an object to be included in an aggregation group on the basis of information pertaining to the object image; an aggregation-image generation means for generating an aggregation image from the object image of the object included in the aggregation group; a feature-amount calculation means for calculating a feature amount of the aggregation image; an object tracking means for performing, on the basis of the feature amount of the aggregation image, tracking processing for the object included in the aggregation group in the target image; and an output means for outputting the result of the object tracking processing.

Description

Information processing device, information processing method, and recording medium

[0001] The present invention relates to processing image information.

[0002] Processing using moving images includes object detection, object re-identification, object tracking, etc. Patent Literature 1 discloses a technique for verifying whether time-series images of a moving object are of the same moving object.

[0003] JP 2013-210845 A

[0004] Object tracking and other processes are processes that impose a high load. Therefore, it is desirable to reduce the load of these processes. Patent Document 1 discloses a technology that uses an average value of feature amounts to suppress a decrease in accuracy due to variations in the posture and orientation of a moving object and variations in lighting. However, Patent Document 1 does not relate to reducing the load of object tracking and other processes.

[0005] An object of the present invention is to provide an information processing device or the like that reduces the load of object tracking processing.

[0006] An information processing device in one form of the present invention includes an image acquisition means for acquiring a target image including a plurality of objects, an object image identification means for detecting objects included in the target image and identifying an object image that is an image of an area surrounding the object in the target image, an aggregate object determination means for determining objects to be included in an aggregate group based on information related to the object image, an aggregate image generation means for generating an aggregate image from the object images of objects included in the aggregate group, a feature calculation means for calculating features of the aggregate image, an object tracking means for performing a tracking process of objects included in the aggregate group in the target image based on the features of the aggregate image, and an output means for outputting the results of the object tracking process.

[0007] An information processing method in one form of the present invention acquires a target image including a plurality of objects, detects the objects included in the target image, identifies object images in the target image that are images of areas surrounding the objects, determines objects to be included in an aggregation group based on information about the object images, generates an aggregate image from the object images of the objects included in the aggregation group, calculates features of the aggregate image, performs a tracking process of the objects included in the aggregation group in the target image based on the features of the aggregate image, and outputs the results of the object tracking process.

[0008] A recording medium in one embodiment of the present invention records a program that causes a computer to execute the following processes: acquiring a target image including multiple objects; detecting objects included in the target image and identifying object images in the target image, which are images of areas surrounding the objects; determining objects to be included in an aggregation group based on information about the object images; generating an aggregate image from object images of objects included in the aggregation group; calculating features of the aggregate image; performing tracking of objects included in the aggregation group in the target image based on the features of the aggregate image; and outputting the results of the object tracking process.

[0009] According to the present invention, it is possible to achieve the effect of reducing the load of object tracking processing.

[0010] 1 is a block diagram showing an example of the configuration of an information processing device according to a first embodiment; FIG. 2 is a diagram showing an example of an aggregated image; FIG. 3 is a flow diagram showing an example of an operation of determining objects to be included in an aggregated group; FIG. 4 is a flow diagram showing an example of an operation of generating an aggregated image; FIG. 5 is a flow diagram showing an example of an object tracking process in an information processing device; FIG. 6 is a flow diagram showing an example of an object tracking process using an object image; FIG. 7 is a diagram showing an example of the configuration of an information processing device including an information storage unit; FIG. 8 is a diagram showing an example of information related to an object image; FIG. 9 is a diagram showing an example of information of objects included in an aggregated group; FIG. 10 is a diagram showing an example of information related to tracking of objects included in an aggregated group; FIG. 11 is a block diagram showing an example of a hardware configuration; FIG. 12 is a block diagram showing an example of the configuration of an information processing system including an information processing device.

[0011] The object detection task detects the position and class (type) of an object to be detected, such as a person, a car, or luggage, present in an image, and generates a list of pairs of positions and classes. The position is, for example, the coordinates of the four vertices of a rectangular area (BB: Bounding Box) in which the object is captured. In the following description, BB is also used as an example of an object position. For example, the object detection task applies an image including objects to a trained model that detects objects from images, and outputs a pair of BB and class for each object included in the image. The object detection task may also output an evaluation result of the object detection result along with the BB and class. The evaluation result is, for example, the confidence or score of the object detection.

[0012] The object tracking task involves tracking an object that moves over time in multiple frames captured over time, such as a video. For example, the object tracking task involves calculating object features from images of the object in multiple frames, determining whether the objects are the same based on the similarity of the features in the multiple frames, and tracking the object based on the determination result. The task of determining whether objects in multiple frames are the same object is also called a re-identification task.

[0013] An object tracking task requires high throughput. For example, when tracking an object in a video captured by a surveillance camera, the object tracking task must process the video in accordance with the frame rate of the surveillance camera to prevent the object from being overlooked. Generally, reducing the processing load leads to improved throughput. Therefore, it is desirable to reduce the load of the object tracking process and improve the throughput of the object tracking task. As described below, the information processing device of each embodiment of the present invention achieves a reduction in the processing load. Hereinafter, the embodiments of the present invention will be described with reference to the drawings. However, the embodiments are not limited to the configurations shown in the drawings and described below.

[0014] 1 is a block diagram showing an example of the configuration of an information processing device 10 according to the first embodiment. The information processing device 10 includes an image acquisition unit 110, an object image identification unit 120, an aggregate object determination unit 130, an aggregate image generation unit 140, a feature amount calculation unit 150, an object tracking unit 160, and an output unit 170. The number of components and the connection relationships in FIG. 1 are merely examples.

[0015] The image acquisition unit 110 acquires a target image including multiple objects to be tracked. For example, the image acquisition unit 110 acquires a video from an imaging device such as a surveillance camera as the target image. The image acquisition unit 110 may acquire multiple target images from multiple imaging devices. In this case, at least some of the target images may have different specifications, such as the number of pixels, aspect ratio, size, and frame rate. Alternatively, if the information processing device 10 includes multiple image acquisition units 110, at least some of the image acquisition units 110 may acquire images with different numbers of pixels, aspect ratios, sizes, and frame rates.

[0016] The object image identification unit 120 detects an object included in the target image and identifies an image of an area surrounding the object in the target image. Hereinafter, the image of the area surrounding the object will be referred to as an "object image." For example, when an image of a BB is used as the object image, the object image identification unit 120 detects the position and class of the object included in the target image. Then, the object image identification unit 120 identifies the position and size of the BB of each object in the target image. For example, the object image identification unit 120 applies the acquired target image to a trained model trained using training data including the target image, the BB of the object image, the object class, and a score, to identify the BB of each object included in the target image, determine the class, and calculate a score.

[0017] The aggregated object determination unit 130 determines objects to be included in an aggregated group based on information about the object image. An aggregated group is a collection of objects that generates an aggregated image, which will be described later. For example, an aggregated group is a collection that includes at least some of the objects being tracked. Note that objects included in an aggregated group may also be called "members." The aggregated object determination unit 130 may store information about objects that have been determined to be included in an aggregated group in a storage unit (not shown).

[0018] The information about the object images is, for example, stability, which will be described below, but is not limited to this. For example, the aggregated object determination unit 130 determines the objects to be included in the aggregated group based on the stability of the object images identified in the target images. The stability is the degree of stability of the positions of the object images corresponding to each object. Specifically, the stability takes a larger value the smaller the temporal change in the positions in the multiple target images.

[0019] The reason for using the temporal stability of position in determining aggregation groups is as follows. Generally, when using images of objects whose positions change little over time, the success rate of object tracking determination is higher than when using images of objects whose positions change greatly. For example, an object that moves very little is easier to track than an object that moves at high speed. As will be explained later, an aggregated image is generated using object images of objects included in an aggregation group and is used for object tracking processing. Therefore, an aggregated image generated using object images whose positions are highly stable over time is expected to have a higher success rate of object tracking than an aggregated image generated using object images whose positions are less stable.

[0020] The aggregated object determination unit 130 may calculate the stability by combining the following values ​​in addition to the temporal change in position.

[0021] (1) The larger the size of the object image, the larger the value.

[0022] (2) A value that increases as the overlap with images of other objects decreases. Note that IoU (Intersection of Union) is an index that indicates the degree of image overlap. IoU is one of the evaluation indices in object detection, and is an index that indicates the degree to which two regions overlap. Specifically, IoU is the ratio of the intersection of two regions to the union of the two regions. Therefore, for example, the aggregated object determination unit 130 may use "a value obtained by subtracting the IoU with other objects from 1" as a value that increases as the overlap with other objects decreases.

[0023] (3) A value that increases as the object is further away from other objects. For example, the aggregated object determination unit 130 may use the distance from other objects as a value that increases as the object is further away from other objects.

[0024] (4) Brightness or contrast.

[0025] (5) A value that increases as the position approaches the center of the target image and decreases as the position approaches the edge. However, the aggregated object determination unit 130 may determine objects to be included in an aggregated group using other criteria, such as reducing the load of the object tracking process, in addition to the stability and the above values.

[0026] The aggregated object determination unit 130 may set a limit on the number of objects to be included in an aggregated group. As will be described later, aggregated images are generated so as to reduce the load of calculating feature quantities. Therefore, the greater the number of object images used to generate an aggregated image, i.e., the greater the number of objects to be included in an aggregated group, the lower the processing load. However, the greater the number of object images to be aggregated, the higher the likelihood of object tracking determination failure. Therefore, the aggregated object determination unit 130 may statically or dynamically set an upper limit on the number of objects to be included in an aggregated group. For example, the aggregated object determination unit 130 may use a predetermined upper limit number, or may set the upper limit number using a stability distribution, etc. The aggregated object determination unit 130 may use multiple aggregated groups. In this case, the aggregated object determination unit 130 may determine the objects to be included in each aggregated group so as to satisfy the above upper limit number.

[0027] The aggregated image generation unit 140 generates an aggregated image from object images of objects included in an aggregated group. Specifically, the aggregated image generation unit 140 generates an aggregated image such that the load for calculating feature amounts is lower than the total load for calculating feature amounts of object images used to generate the aggregated image. Alternatively, the aggregated image generation unit 140 may generate an aggregated image such that the load for object tracking processing is lower. In other words, the aggregated image is an image in which at least one of the load for calculating feature amounts and the load for object tracking processing is lower than the total of at least one of the load for calculating feature amounts of object images used to generate the aggregated image and the load for object tracking processing.

[0028] For example, the load of feature calculation is generally proportional to the amount of data. Therefore, the aggregated image generation unit 140 may generate an aggregated image with a data amount smaller than the total data amount of the target images. The data amount may be, for example, the number of pixels, the number of images, the number of channels (color channels or selection channels), the data size, or a combination thereof, but is not limited to these. The aggregated image generation unit 140 may reduce either the height or width of the image to reduce the number of pixels or the data size. However, depending on the characteristics of the computing unit that calculates the feature, or the combination of the feature calculation method and the data used to calculate the feature, the calculation speed may improve even if the amount of data increases, i.e., the load of feature calculation may decrease. In such cases, the aggregated image generation unit 140 may generate an aggregated image with a data amount larger than the total data amount of the target images.

[0029] The aggregate image generating unit 140 generates an aggregate image using, for example, the methods described below or a combination thereof, but is not limited to these, and the aggregate image may be generated using other methods.

[0030] (1) Averaging: For example, the aggregated image generating unit 140 generates an aggregated image by averaging the pixel values ​​of all object images at each pixel position. If the object images have different shapes, the aggregated image generating unit 140 may resize the images to match the shapes and then average them. During resizing, the aggregated image generating unit 140 may pad the images with a black or white background to maintain the aspect ratio. The aggregated image generating unit 140 may perform the averaging process by shifting the positions of at least some of the object images. Alternatively, the aggregated image generating unit 140 may perform the averaging process using a weighted average in which some object images are weighted more heavily. For example, the aggregated image generating unit 140 may perform the averaging process using weights proportional to the size, i.e., by weighting larger object images.

[0031] (2) Tile: For example, the aggregate image generating unit 140 reduces the object image and generates an image in which the reduced object image is arranged in a tiled pattern as the aggregate image.

[0032] (3) Channel integration: For example, the aggregated image generating unit 140 generates an image in which some channel information is combined as an aggregated image.

[0033] FIG. 2 is a diagram showing an example of an aggregated image. The four images at the top of FIG. 2 are object images cut out from a target image. The image on the left side of the bottom of FIG. 2 is an aggregated image obtained when averaging is used. The image on the right side of the bottom of FIG. 2 is an aggregated image obtained when tiling is used. The object image at the top and the aggregated image at the bottom are images of the same size. Therefore, the data amount (e.g., the number of pixels) of each of the two aggregated images at the bottom of FIG. 2 is one-fourth the total number of pixels of the four object images. Furthermore, the number of images is reduced from four to one.

[0034] The aggregate image generating unit 140 may perform image processing on the object images before generating the aggregate image. The image processing may include, but is not limited to, enlargement, reduction, partial cropping, geometric transformation, pixel value correction, left-right flipping, up-down flipping, translation, rotation, monochrome conversion, aspect ratio change, or a combination thereof. The aggregate image generating unit 140 may apply different image processing to each object image.

[0035] The aggregate image generation unit 140 may extract an image including both object regions such as BB and non-object regions as the target image, or may extract an image from which the non-object regions have been removed. The object region and the non-object region are also referred to as the foreground and background, respectively. For example, as the background-removed image, the aggregate image generation unit 140 may generate an image by subtracting the pixel values ​​of the background image from the pixel values ​​of the object image. In this case, the aggregate image generation unit 140 may use a background image acquired in advance, or may generate a background image using the target image. For example, the aggregate image generation unit 140 may use a target image in which no object is detected as the background image. Alternatively, the aggregate image generation unit 140 may generate a background image by excluding regions in which an object is detected from the target image. The aggregate image generation unit 140 may generate a background image by averaging multiple generated background images. In this case, the aggregate image generation unit 140 may use a weighted average in which the weight of the background image generated from the older target image is reduced.

[0036] The aggregate image generating unit 140 may generate a single aggregate image or multiple aggregate images as the aggregate image. For example, if there are multiple aggregate groups, the aggregate image generating unit 140 may generate a single aggregate image for each aggregate group, or may generate multiple aggregate images for at least some of the aggregate groups.

[0037] The aggregate image generating unit 140 may store the generated aggregate image, the object image, and the target image used for generation in a storage unit (not shown) in association with each other. For example, the aggregate image generating unit 140 may store the aggregate image, the target image, the position of the object image in the target image, and the position of the object image in the aggregate image in association with each other.

[0038] The feature calculation unit 150 calculates feature amounts of the aggregate image. The feature amounts are, for example, color histograms or SIFT (Scale-Invariant Feature Transform) feature amounts, but are not limited to these. For example, the feature calculation unit 150 calculates feature amounts of the aggregate image using a trained model trained using training data including the aggregate image and its feature amounts. Note that the trained model used by the feature calculation unit 150 calculates feature amounts that are more similar to images of the same object, for example, but is not limited to this, and other feature amounts may be calculated. The feature calculation unit 150 may store the calculated feature amounts in a storage unit (not shown) in association with the aggregate image.

[0039] The object tracking unit 160 performs a tracking process of an object included in an aggregate group based on the feature amounts of the aggregate image. For example, similar to a general object tracking process, the object tracking unit 160 tracks an object included in an aggregate group based on the feature amounts of an aggregate image generated from a target image for which the object is to be tracked and the feature amounts of an aggregate image generated from past target images. Specifically, for example, the object tracking unit 160 calculates the feature amounts by applying the aggregate image of the target image and aggregate images of past target images to a trained model trained using training data including the feature amounts of multiple aggregate images and information about the object to be tracked. In the following description, the aggregate image generated from past target images will also be simply referred to as a "past aggregate image."

[0040] The object tracking unit 160 then calculates the similarity between the feature amounts of the aggregated image to be tracked and the feature amounts of the past aggregated images. If the calculated similarity is equal to or greater than a threshold, the object tracking unit 160 determines that the objects included in the aggregated groups are the same. On the other hand, if the similarity is less than the threshold, the object tracking unit 160 determines that at least some of the objects included in the aggregated groups are not the same. In this way, the object tracking unit 160 uses the feature amounts of the aggregated images to determine whether the objects included in the aggregated groups are the same. If it is determined that the objects included in the aggregated groups are the same, the object tracking unit 160 determines that the objects included in the aggregated groups have moved to the positions of the object images of the objects identified by the object image identification unit 120 in the target image. For example, the object tracking unit 160 determines that the objects have moved from the positions of the object images in the past target images from which the past aggregated images were generated to the positions of the object images in the target images to be tracked.

[0041] Here, the load will be explained. Assume that the number of objects (number of members) included in the aggregation group is N. Assume also that the size of the object image and the size of the aggregated image are the same. When performing object tracking processing using object images of each object, the feature amount calculation unit 150 calculates feature amounts for N object images. Then, the object tracking unit 160 performs object tracking processing N times using the N feature amounts. On the other hand, when an aggregated image is used, the feature amount calculation unit 150 calculates feature amounts for one aggregated image. Then, the object tracking unit 160 performs object tracking processing once using one feature amount. In this way, the information processing device 10 can perform feature amount calculation processing and object tracking processing for N objects included in the aggregation group in a single process, thereby reducing the processing load. As a result, for example, the information processing device 10 can improve the throughput of the object tracking processing.

[0042] The output unit 170 outputs the result of the object tracking process by the object tracking unit 160. For example, the output unit 170 outputs the object tracking result to a system that determines the movement of a person using the target image. The output unit 170 may output at least a part of the target image, the object image, the BB, class, and score of the object image, etc., along with the tracking result.

[0043] In the description so far, the information processing device 10 uses a trained model that identifies an area surrounding an object and a trained model that calculates feature amounts. However, the information processing device 10 may also use a trained model that performs both the process of identifying an area surrounding an object and the process of calculating feature amounts. In other words, the information processing device 10 may include a configuration that realizes the functions of both the feature amount calculation unit 150 and the object tracking unit 160.

[0044] The information processing device 10 may perform object tracking using an object image in addition to object tracking using an aggregated image. For example, if a target image includes an object not included in an aggregated group, the feature calculation unit 150 and the object tracking unit 160 may operate using the object image to track the object. Alternatively, if object tracking using an aggregated image fails, the feature calculation unit 150 and the object tracking unit 160 may operate using the object image. In these cases, the feature calculation unit 150 calculates feature amounts of the object image. Then, the object tracking unit 160 tracks the object using the calculated feature amounts and the position of the object image in the target image. Note that the object tracking unit 160 may perform object tracking using the IoU of the object's current BB and past BB. For example, the object tracking unit 160 may perform object tracking on an object to be tracked, for which the IoU between the current BB and past BB is equal to or greater than a threshold.

[0045] [Explanation of Operation] FIG. 3 is a flow diagram showing an example of the operation of determining objects to be included in an aggregation group. The aggregated object determination unit 130 repeats the following operation for objects detected by the object image identification unit 120. The aggregated object determination unit 130 calculates the stability of each object (step S201). For example, the aggregated object determination unit 130 calculates the amount of change in the temporal position of the object image as the stability of the object. Then, the aggregated object determination unit 130 determines objects (members) to be included in the aggregation group based on the stability (step 202). For example, the aggregated object determination unit 130 determines objects (members) to be included in the aggregation group if the stability is equal to or greater than a threshold. Note that if information about objects to be included in the aggregation group has already been saved, the aggregated object determination unit 130 updates the saved information.

[0046] 4 is a flow diagram showing an example of an operation for generating an aggregate image. When generating multiple aggregate images, the aggregate image generation unit 140 repeats the following operation for each aggregate image. The aggregate image generation unit 140 cuts out an object image of an object included in an aggregate group from a target image (step S211). For example, the aggregate image generation unit 140 cuts out an image of object BB. Then, the aggregate image generation unit 140 generates an aggregate image from the object image (step S212). For example, the aggregate image generation unit 140 generates an aggregate image by averaging the object images. Alternatively, the aggregate image generation unit 140 generates an aggregate image by arranging reduced-sized object images in a tiled pattern.

[0047] FIG. 5 is a flow diagram illustrating an example of object tracking processing in the information processing device 10. When there are multiple aggregated groups, the object tracking unit 160 repeats the following operation for each aggregated group. The object tracking unit 160 calculates the similarity between the feature amount of the aggregated image and the feature amount of a past aggregated image (step S221). The object tracking unit 160 determines whether the similarity is equal to or greater than a threshold (step S222). If the similarity is equal to or greater than the threshold (Yes in step S222), the object tracking unit 160 performs object tracking processing using the aggregated image (step S223). That is, the object tracking unit 160 performs tracking processing for each object included in the aggregated group. When performing tracking processing for each object included in the aggregated group, the object tracking unit 160 may manage the processing of the objects to be tracked using a list of objects included in the aggregated group. For example, the object tracking unit 160 creates a list of objects included in the aggregated group. Then, the object tracking unit 160 marks the objects for which object tracking processing has been performed as having been completed in the list. The object tracking unit 160 may continue to perform the tracking process for unmarked objects until all objects have been marked. If the similarity is less than the threshold (No in step S222), the object tracking unit 160 performs the object tracking process using the object image described next (step S224).

[0048] 6 is a flow diagram showing an example of an object tracking process using an object image. For example, the information processing device 10 performs object tracking process using an object image for objects that are not included in an aggregated group and for objects for which tracking process could not be performed using an aggregated image. In other words, if all objects can be tracked using an aggregated image, the information processing device 10 may omit this operation.

[0049] The information processing device 10 repeats the following operations for each object to be tracked using the object image. The feature calculation unit 150 calculates the feature of the object image (step S231). The object tracking unit 160 performs object tracking processing using the calculated feature of the object image (step S232). For example, the object tracking unit 160 determines, as a moved object, an object for which the similarity between the feature of the object image in the target image and the feature of the object image in a past target image is equal to or greater than a threshold. Note that if there are two or more similarities equal to or greater than the threshold, the object tracking unit 160 may select the object with the greatest similarity, or may select an object according to a predetermined rule, such as selecting an object using IoU, and then perform the object tracking processing.

[0050] As described above, the information processing device 10 includes an image acquisition unit 110, an object image identification unit 120, an aggregated object determination unit 130, an aggregated image generation unit 140, a feature calculation unit 150, an object tracking unit 160, and an output unit 170. The image acquisition unit 110 acquires a target image including multiple objects. The object image identification unit 120 detects objects included in the target image and identifies an object image, which is an image of an area surrounding the object, in the target image. The aggregated object determination unit 130 determines objects to be included in an aggregated group based on information about the object image. The aggregated image generation unit 140 generates an aggregated image from the object images of the objects included in the aggregated group. The feature calculation unit 150 calculates feature amounts of the aggregated image. The object tracking unit 160 performs tracking processing of objects included in the aggregated group in the target image based on the feature amounts of the aggregated image. The output unit 170 outputs the results of the object tracking processing.

[0051] The aggregated image generation unit 140 generates an aggregated image in which the load for calculating feature quantities is lower than the load for calculating feature quantities of all object images included in the aggregated group. For example, the aggregated image generation unit 140 generates an aggregated image in which the number of pixels is lower than the total number of pixels of all object images included in the aggregated group, as an aggregated image in which the load for calculating feature quantities is lower. Therefore, the information processing device 10 uses the aggregated image to reduce the load of the tracking process. As a result, the information processing device 10 can improve the processing speed and throughput of the object tracking process. Alternatively, the information processing device 10 can reduce the cost or power consumption of the object tracking process. Alternatively, the information processing device 10 can reduce the size of the hardware required for the object tracking process.

[0052] The information processing device 10 may perform object tracking processing for at least some objects using object images. For example, if the similarity of the feature amounts of the aggregated image is less than a threshold, the information processing device 10 cannot track the object using the feature amounts of the aggregated image. Therefore, in such a case, the information processing device 10 may track the object using the object image. In detail, the information processing device 10 may operate as follows. First, the feature amount calculation unit 150 calculates the feature amounts of the object image. Then, the object tracking unit 160 tracks the object based on the feature amounts of the object image. If the failure rate of the object tracking processing using the aggregated image is less than a threshold, the information processing device 10 may perform the object tracking processing using the object image. Note that, when using a trained model, the feature amount calculation unit 150 may use different trained models for the aggregated image and the object image. For example, the aggregated image includes more objects than the object image. Therefore, the feature amount calculation unit 150 may use a trained model with higher accuracy as the trained model used for the aggregated image than the trained model used for the object image. In this way, the feature calculation unit 150 may calculate optimal feature amounts for the aggregate image and the object image using models suitable for each. As a result, the object tracking unit 160 can appropriately perform object tracking processing for each of the aggregate image and the object image. Note that the object tracking unit 160 may use different trained models for the aggregate image and the object image.

[0053] Alternatively, the information processing device 10 may perform object tracking processing using object images for all objects at the start of operation, and start object tracking processing using aggregate images when the following conditions are satisfied:

[0054] (1) When the load on the information processing device 10 exceeds a threshold.

[0055] (2) When the number of objects contained in the target image exceeds a threshold.

[0056] (3) When the number of objects to be tracked exceeds a threshold.

[0057] (4) When the number of highly stable objects exceeds a threshold.

[0058] [Variation] After determining the objects to be included in the aggregated group, the information processing device 10 may continue to use the aggregated group. For example, the information processing device 10 may operate as follows. First, the information processing device 10 performs operations including the aggregated object determination unit 130 on a first target image. That is, the information processing device 10 determines the objects to be included in the aggregated group using the first target image. Thereafter, the information processing device 10 repeats the operations of the image acquisition unit 110, the object image identification unit 120, the aggregated image generation unit 140, the feature calculation unit 150, the object tracking unit 160, and the output unit 170 on the target image using the aggregated group determined by the aggregated object determination unit 130. In this way, after operating the aggregated object determination unit 130 once, the information processing device 10 may repeatedly perform object tracking processing using the aggregated image using the determined aggregated group. In this case, the information processing device 10 can reduce the processing load of the aggregated object determination unit 130, such as calculating stability.

[0059] However, objects move. Therefore, the objects included in the target image change. Therefore, the information processing device 10 may operate the aggregated object determination unit 130 periodically or when a predetermined condition occurs to update the objects to be included in the aggregated group. For example, the information processing device 10 may operate the aggregated object determination unit 130 to update the objects to be included in the aggregated group when the similarity of the feature amounts of the aggregated image falls below a threshold. Even in this case, the information processing device 10 can respond to the movement of objects while reducing the processing load of the aggregated object determination unit 130 most of the time. An example of the object update operation will be described below.

[0060] (1) Object Removal The aggregated object determination unit 130 may update the objects to be included in the aggregated group based on at least one of the object detection results of the object image identification unit 120, the object image identification results of the object image identification unit 120, and the object tracking results of the object tracking unit 160. For example, the aggregated object determination unit 130 may remove from the aggregated group an object that the object image identification unit 120 previously detected but is no longer able to detect. Note that, if there are no more objects to be included in the aggregated group, the information processing device 10 may stop the object tracking process using the aggregated image and perform the object tracking process using the object image. However, if the success rate of the object tracking process using the aggregated image falls below a threshold, the information processing device 10 may stop the object tracking process using the aggregated image and perform the object tracking process using the object image. Note that, if the success rate of the object tracking process using the aggregated image falls below a threshold, the information processing device 10 may change the trained model being used. In this case, the information processing device 10 may also change the threshold, etc. In this way, based on the object tracking results in the object tracking unit 160, the information processing device 10 may change the trained model or threshold value being used.

[0061] (2) Addition of Object When the object image identification unit 120 detects a new object that has not been detected previously in the target image, the information processing device 10 may operate, for example, as follows. When the object image identification unit 120 detects a new object, the aggregated object determination unit 130 may determine whether to include the new object in an aggregated group based on the detection result of the object image identification unit 120. For example, the aggregated object determination unit 130 may determine whether to include the new object in an aggregated group based on the stability of the new object detected by the object image identification unit 120. Specifically, for example, if the stability of the new object is equal to or greater than a threshold, the aggregated object determination unit 130 determines to add the object to the objects to be included in the aggregated group. On the other hand, if the stability of the new object is less than the threshold, the aggregated object determination unit 130 determines not to include the object in the aggregated group.

[0062] Some objects may temporarily move behind other objects, or may move out of the camera's field of view. In these cases, the information processing device 10 may temporarily operate using an aggregated group that does not include the object, rather than removing the object from the aggregated group. Hereinafter, an aggregated group in which some objects have been removed from the aggregated group is referred to as a "second aggregated group." For example, if the object image identification unit 120 can no longer detect some objects, the aggregated object determination unit 130 determines a second aggregated group in which the objects that the object image identification unit 120 no longer detects are removed from the objects to be included in the aggregated group. Then, the aggregated image generation unit 140 generates an aggregated image using object images of the objects included in the second aggregated group. Hereinafter, an aggregated image generated using object images of the objects included in the second aggregated group is referred to as a "second aggregated image." The feature calculation unit 150 calculates the feature values ​​of the second aggregated image. Then, the object tracking unit 160 performs tracking processing based on the feature values ​​of the second aggregated image. In other words, if the object image identification unit 120 cannot detect some objects, the aggregated object determination unit 130 determines a second aggregated group in which the objects that the object image identification unit 120 cannot detect are removed from the objects included in the aggregated group. The aggregated image generation unit 140, the feature amount calculation unit 150, and the object tracking unit 160 may then operate using the second aggregated group. The information processing device 10 may operate using the second aggregated group until the object image identification unit 120 can detect an object in the aggregated group. In this way, the information processing device 10 can also use the second aggregated group to handle cases where an object cannot be temporarily detected.

[0063] Alternatively, the stability changes when the moving speed of the object changes. Therefore, the information processing device 10 may determine the second aggregated group based on the stability calculated by the object image identification unit 120. For example, the aggregated object determination unit 130 determines, from among the objects to be included in the aggregated group, some objects with high stability to be included in the second aggregated group. In other words, the aggregated object determination unit 130 determines the objects to be included in the second aggregated group by removing some objects with low stability. Then, for example, if the object tracking unit 160 is unable to track at least some objects using the feature amounts of the aggregated image, the aggregated image generation unit 140, the feature amount calculation unit 150, and the object tracking unit 160 may operate using the second aggregated image. For example, if the object tracking unit 160 is unable to track at least some objects using the aggregated image, the aggregated object determination unit 130 determines, based on the object image, objects to be included in the second aggregated group from which some of the objects included in the aggregated group have been removed. For example, the aggregated object determination unit 130 determines the objects to be included in the second aggregated group based on the stability of the object image. The aggregated image generating unit 140, the feature amount calculating unit 150, and the object tracking unit 160 may operate using the second aggregated group. In this way, the information processing device 10 can also handle cases where the stability of the object changes by using the second aggregated group.

[0064] The information processing device 10 may include an information storage unit 180 that stores images such as object images and information related to the images. Each component of the information processing device 10 may store information in the information storage unit 180 and acquire information from the information storage unit 180 as necessary. FIG. 7 illustrates an example of the configuration of the information processing device 10 including the information storage unit 180. For example, the information storage unit 180 stores target images and object images. For example, if an object is hidden behind another object, the object image identification unit 120 cannot identify an object image for that object. Therefore, the aggregate image generation unit 140 may generate an aggregate image using a stored past object image instead of an object image that the object image identification unit 120 could not identify. In this way, using the information storage unit 180, the information processing device 10 can generate an aggregate image even if an object is hidden behind another object.

[0065] The information storage unit 180 may store information related to an image. FIG. 8 is a diagram showing an example of information related to an object image stored by the information storage unit 180. In FIG. 8, the information storage unit 180 stores an object's entry ID (identifier), a tracking ID, a camera ID, a shooting time, the object's BB, a score, and object feature amounts. The entry ID is an identifier assigned by the object image identification unit 120 to an object detected in a target image. The object image identification unit 120 assigns entry IDs to detected objects in the order in which they are detected in the target image. The tracking ID is an identifier assigned by the object tracking unit 160 to an object to be tracked, and the same tracking ID is set for entry IDs determined to be identical. The camera ID is the identifier of the camera that captured the target image containing the object. The BB is the coordinates of the four vertices of a rectangle (top left, top right, bottom right, bottom left). The score is a score for the object detection process. The feature amounts are feature amounts calculated from the object image. The information storage unit 180 may store the above information as a history. The information storage unit 180 may store information related to images using a database.

[0066] The information storage unit 180 may store information about objects included in an aggregation group. FIG. 9 is a diagram showing an example of information about objects included in an aggregation group. In FIG. 9, the information storage unit 180 stores an aggregation group ID, a camera ID, and an object ID as information about two aggregation groups. The aggregation group ID is an identifier of the aggregation group. The camera ID is an identifier of the camera that captured the target image that includes the aggregation group. The object ID is an identifier of the object included in the aggregation group. The information storage unit 180 may store a tracking ID as the object ID. The information storage unit 180 may store information about objects included in an aggregation group using a database.

[0067] The information storage unit 180 may store information related to tracking of objects included in an aggregation group. FIG. 10 is a diagram showing an example of information related to tracking of objects included in an aggregation group. In FIG. 10 , the information storage unit 180 stores an entry ID of an aggregated image, an aggregation group ID, a tracking ID of the aggregation group, a detection time of the aggregated image, feature amounts of the aggregated image, and an entry ID of an object. The entry ID of an aggregated image is an identifier of the aggregated image. The aggregation group ID is an identifier of the aggregation group that generated the aggregated image. The aggregate tracking ID is an identifier of the aggregation group in the object tracking process. The detection time is the detection time of the aggregated image, and is the shooting time of the target image from which the aggregated image was generated. The entry ID of an object is an identifier of an object included in the aggregation group corresponding to the aggregated image. The information storage unit 180 may store information related to tracking of objects included in an aggregation group using a database.

[0068] The information processing device 10 may determine objects to be included in an aggregation group based on stored past object images. For example, the information processing device 10 may operate as follows. First, the aggregated object determination unit 130 determines objects to be included in each of multiple aggregation group candidates from objects identified in past target images. For example, the aggregated object determination unit 130 may set all object combinations containing two or more objects as aggregation group candidates. For example, in the case of three objects, the aggregated object determination unit 130 may operate as follows. Note that, hereinafter, the three objects will be referred to as ID1, ID2, and ID3, respectively, using their identifiers. In this case, the aggregated object determination unit 130 may determine group 1 [ID1, ID2], group 2 [ID2, ID3], group 3 [ID3, ID1], and group 4 [ID1, ID2, ID3] as aggregation groups. Alternatively, the aggregated object determination unit 130 may obtain aggregation group candidates from a user or the like.

[0069] The aggregated image generating unit 140 generates an aggregated image from the object images of the objects included in the aggregated group candidates stored in the information storing unit 180. For example, if the aggregated group candidates are groups 1 to 4 described above, the aggregated image generating unit 140 generates aggregated images of groups 1 to 4. Furthermore, the aggregated image generating unit 140 may generate multiple aggregated images as aggregated images for each aggregated group candidate. The aggregated image generating unit 140 may generate an aggregated image for each aggregated group candidate from, for example, all or some of the past target images. Then, the feature calculating unit 150 calculates the feature amounts of each aggregated image of the aggregated group candidate.

[0070] The object tracking unit 160 performs an object tracking process using the feature amounts. For example, for each combination of two aggregate images of an aggregate group candidate, the object tracking unit 160 determines whether the objects included in the aggregate group candidate are the same based on the feature amounts of the aggregate images. Hereinafter, the determination of whether the objects are the same will be simply referred to as an "identity determination." For example, for each combination of aggregate images, the object tracking unit 160 determines that the objects in the aggregate images are the same if the similarity of the feature amounts is equal to or greater than a first threshold, and determines that the objects are not the same if the similarity is less than the first threshold. The object tracking unit 160 may perform identity determination for all combinations of generated aggregate images, or may perform identity determination for some combinations.

[0071] The object tracking unit 160 then calculates a self-relevance rate for each aggregation group candidate. The self-relevance rate is the number of combinations determined to be identical among the aggregated image groups of the aggregation group candidate divided by the total number of combinations. The number of combinations determined to be identical is the number of true positives. For example, a self-relevance rate of 1.0 for a certain aggregation group candidate means that all identity determinations using the aggregated images of that aggregation group candidate were successful. In this case, if the image acquisition unit 110 acquires target images containing object images of that aggregation group candidate, all object tracking processes using the aggregated images of that aggregation group candidate are expected to be successful. In other words, aggregation group candidates with a high self-relevance rate are expected to be more likely to succeed in object tracking processes using aggregated images. Therefore, the aggregate object determination unit 130 determines the aggregation group candidate with the highest self-relevance rate as the aggregation group. Note that if there is no aggregation group candidate whose self-relevance rate is equal to or greater than the second threshold, the information processing device 10 may operate without using aggregation groups. In this way, the aggregated object determination unit 130 may determine an aggregated group from the aggregated group candidates based on the result of the object tracking process by the object tracking unit 160, for example, the result of identity determination for an aggregated image generated based on past object images. The aggregated object determination unit 130 may use other indices in addition to the self-relevance rate.

[0072] For example, the aggregated object determination unit 130 may determine aggregated groups using an error rate such as a confusion rate in addition to the self-relevance rate. The confusion rate is the ratio of different aggregated groups determined to be the same aggregated group as a result of identity determination using aggregated images. For example, the confusion rate is a value obtained by dividing the number of combinations of different aggregated group candidates determined to be the same among all combinations of aggregated images of the aggregated group candidate by the total number of combinations. The lower the confusion rate, the better the determination. Note that determining different aggregated groups as the same aggregated group is an example of an error in identity determination. Furthermore, the number of combinations of different aggregated groups determined to be the same is the number of incorrect answers (false positives). Then, the aggregated object determination unit 130 may determine aggregated group candidates with a confusion rate lower than a fourth threshold as aggregated groups from among multiple aggregated group candidates with a self-relevance rate higher than a third threshold. In addition, the aggregated object determination unit 130 may determine aggregated groups from aggregated group candidates using a confusion rate instead of a self-matching rate, or may determine aggregated groups from aggregated group candidates by combining the confusion rate with other indicators.

[0073] The aggregated object determination unit 130 may determine the similarity threshold used in step S222 so that the confusion rates for some or all of the aggregated images are within a predetermined range. Alternatively, the aggregated object determination unit 130 may not include, in any aggregated image, an object that is frequently included in an aggregated group candidate with a low self-relevance rate or an object that is frequently included in a combination of aggregated group candidates with a high confusion rate. In this case, the aggregated object determination unit 130 may determine whether to include an object in an aggregated group using, for example, at least one of the total number of appearances of each object in aggregated group candidates with a low self-relevance rate and the total number of appearances of each object in aggregated group candidates with a high confusion rate.

[0074] [Hardware Configuration] The configuration of the information processing device 10 will be described with reference to the drawings. The information processing device 10 may be configured with one or more hardware circuits. Alternatively, the information processing device 10 may be realized as a computer device including a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), and a network interface. In this case, the information processing device 10 operates the hardware in combination with software. In this way, the information processing device 10 may realize its functions as a combination of hardware and software. FIG. 11 is a block diagram showing an example of the hardware configuration of the information processing device 10. The information processing device 10 includes a CPU 610, a ROM 620, a RAM 630, a storage device 640, and a network interface 650.

[0075] The CPU 610 reads a program from at least one of the ROM 620 and the storage device 640. Based on the read program, the CPU 610 controls the RAM 630, the storage device 640, and the network interface 650. The CPU 610 controls these components to realize the functions of the image acquisition unit 110, the object image identification unit 120, the aggregate object determination unit 130, the aggregate image generation unit 140, the feature amount calculation unit 150, the object tracking unit 160, and the output unit 170.

[0076] The CPU 610 may read, using a recording medium reading device (not shown), a program contained in a recording medium 690 that stores the program in a computer-readable manner. Alternatively, the CPU 610 may receive a program from an external device (not shown) via the network interface 650, store the program in the RAM 630 or the storage device 640, and operate based on the stored program.

[0077] The ROM 620 stores programs and fixed data executed by the CPU 610. The ROM 620 is, for example, a programmable ROM (P-ROM) or a flash ROM. The RAM 630 temporarily stores programs and data executed by the CPU 610. The RAM 630 is, for example, a dynamic RAM (D-RAM). The storage device 640 stores data and programs that the information processing device 10 stores long-term. The storage device 640 operates as the information storage unit 180. The storage device 640 may also operate as a temporary storage device for the CPU 610. The storage device 640 is, for example, a hard disk drive or a solid state drive (SSD).

[0078] The ROM 620 and the storage device 640 are non-volatile (non-transitory) recording media. On the other hand, the RAM 630 is a volatile (transitory) recording media. The CPU 610 can operate based on programs stored in the ROM 620, the storage device 640, and the RAM 630. In other words, the CPU 610 can operate using either a non-volatile recording medium or a volatile recording medium. When realizing each function, the CPU 610 may use at least one of the RAM 630 and the storage device 640 as a temporary storage medium for programs and data.

[0079] The network interface 650 exchanges information with an external device (not shown) via a network. The network interface 650, for example, acquires an image of a target from an external camera. Alternatively, the network interface 650 outputs the results of the object tracking process to an external device. The network interface 650 is, for example, a local area network (LAN) card or a LAN hub. The network interface 650 is not limited to wired communication and may also communicate wirelessly.

[0080] [System] An example of a system using the information processing device 10 will be described as an explanation of the information processing device 10. FIG. 12 is a block diagram showing an example of the configuration of an information processing system 40 including the information processing device 10. The information processing system 40 includes the information processing device 10, a data acquisition device 20, and a display device 30. The information processing system 40 may include multiple devices as each of the devices. For example, the information processing system 40 may include multiple data acquisition devices 20. The data acquisition device 20 outputs a target image to the information processing device 10. The data acquisition device 20 is, for example, a surveillance camera. The information processing device 10 operates as described above. That is, the information processing device 10 performs object tracking processing using the target image acquired from the data acquisition device 20. The information processing device 10 outputs the object tracking result to the display device 30. The display device 30 displays the acquired object tracking result. For example, the display device 30 displays a frame indicating the tracked object superimposed on the target image. The display device 30 is, for example, a liquid crystal display or an organic electroluminescence display.

[0081] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0082] (Supplementary Note 1) An information processing device including: an image acquisition means for acquiring a target image including a plurality of objects; an object image identification means for detecting objects included in the target image and identifying an object image that is an image of an area surrounding the object in the target image; an aggregate object determination means for determining objects to be included in an aggregation group based on information related to the object images; an aggregate image generation means for generating an aggregate image from object images of objects included in the aggregation group; a feature calculation means for calculating a feature of the aggregate image; an object tracking means for performing a tracking process of objects included in the aggregation group in the target image based on the feature of the aggregate image; and an output means for outputting a result of the object tracking process.

[0083] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the aggregate image generation means generates an aggregate image in which at least one of a load for calculating feature amounts and a load for object tracking processing is lower than at least one of a total load for calculating feature amounts of object images used in generating the aggregate image and a total load for object tracking processing.

[0084] (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the aggregate image generating means generates an aggregate image having a smaller amount of data than a total amount of data of the object images used to generate the aggregate image.

[0085] (Supplementary Note 4) An information processing device according to any one of Supplementary Notes 1 to 3, wherein the operations of the image acquisition means, object image identification means, aggregate image generation means, feature calculation means, object tracking means, and output means are repeated for a plurality of target images using the aggregated group determined by the aggregated object determination means.

[0086] (Supplementary Note 5) The information processing device described in any one of Supplementary Notes 1 to 4, wherein the aggregated object determination means changes the objects included in the aggregated group based on at least one of the identification of the object image by the object image identification means, the object image identified by the object image identification means, and the result of object tracking by the object tracking means.

[0087] (Supplementary Note 6) The information processing device according to Supplementary Note 5, wherein the aggregated object determination means deletes from the aggregated group an object that is no longer detected by the object image identification means.

[0088] (Supplementary Note 7) In the information processing device described in Supplementary Note 5 or 6, when the object image identification means detects a new object that has not been detected before, the aggregated object determination means determines whether or not to include the new object in an aggregated group based on information about the object image corresponding to the new object.

[0089] (Supplementary Note 8) The information processing device according to any one of Supplementary Notes 1 to 7, wherein, when the object tracking means cannot track the object using the feature amount of the aggregate image, the feature amount calculation means calculates the feature amount of the object image, and the object tracking means tracks the object based on the feature amount of the object image.

[0090] (Supplementary Note 9) The information processing device according to Supplementary Note 8, wherein the feature calculation means uses a trained model to calculate the feature, and different trained models are used to calculate the feature of the aggregate image and to calculate the feature of the object image.

[0091] (Supplementary Note 10) The information processing device according to Supplementary Note 9, wherein the feature amount calculation means changes a trained model used for calculating the feature amount based on a tracking result of the object by the object tracking means.

[0092] (Supplementary Note 11) When the object image identification means cannot detect some objects, the aggregated object determination means determines a second aggregated group by removing some of the objects that the object image identification means cannot detect from the objects included in the aggregated group, and the aggregated image generation means, feature calculation means, and object tracking means operate using the second aggregated group. (Supplementary Note 11) An information processing device described in any one of Supplementary Notes 1 to 10.

[0093] (Supplementary Note 12) The information processing device according to Supplementary Note 11, wherein when the object tracking means cannot track at least some of the objects using the aggregated image, the aggregated object determining means determines a second aggregated group based on the object image.

[0094] (Supplementary Note 13) The information processing device according to any one of Supplementary Notes 1 to 12, wherein the aggregated object determination means determines objects to be included in an aggregated group based on stability of the object image.

[0095] (Supplementary Note 14) An information processing device according to any one of Supplementary Notes 1 to 13, further comprising a storage means for storing object images, and if the objects detected by the object image identification means do not include an object included in the aggregated group, the aggregated image generation means generates an aggregated image using the object images stored in the storage means.

[0096] (Supplementary Note 15) An information processing device according to Supplementary Note 14, wherein the aggregate object determination means determines objects to be included in a plurality of aggregation group candidates; the aggregate image generation means generates an aggregate image from object images of the objects included in the aggregation group candidates stored in the storage means; the feature calculation means calculates features of the aggregate images of the aggregation group candidates; the object tracking means performs a tracking process of the objects included in the aggregation group candidates based on the features of the aggregate images of the aggregation group candidates; and the aggregate object determination means determines an aggregation group from the aggregation group candidates based on a result of the tracking process of the objects in the aggregation group candidates.

[0097] (Supplementary Note 16) An information processing method comprising: acquiring a target image including a plurality of objects; detecting the objects included in the target image; identifying object images in the target image, which are images of areas surrounding the objects; determining objects to be included in an aggregation group based on information about the object images; generating an aggregate image from the object images of the objects included in the aggregation group; calculating feature amounts of the aggregate image; performing a tracking process on the objects included in the aggregation group in the target image based on the feature amounts of the aggregate image; and outputting results of the object tracking process.

[0098] (Supplementary Note 17) A recording medium that records a program that causes a computer to execute the following processes: a process of acquiring a target image that includes multiple objects; a process of detecting objects included in the target image and identifying object images that are images of areas surrounding the objects in the target image; a process of determining objects to be included in an aggregation group based on information about the object images; a process of generating an aggregate image from the object images of objects included in the aggregation group; a process of calculating features of the aggregate image; a process of performing tracking processing of objects included in the aggregation group in the target image based on the features of the aggregate image; and a process of outputting results of the object tracking processing.

[0099] REFERENCE SIGNS LIST 10 Information processing device 20 Data acquisition device 30 Display device 40 Information processing system 110 Image acquisition unit 120 Object image identification unit 130 Aggregated object determination unit 140 Aggregated image generation unit 150 Feature amount calculation unit 160 Object tracking unit 170 Output unit 180 Information storage unit 600 Computer device 610 CPU 611 Arithmetic unit 620 ROM 630 RAM 640 Storage device 650 Network interface 690 Recording medium

Claims

1. image acquisition means for acquiring a target image including a plurality of objects; an object image specifying means for detecting an object included in the target image and specifying an object image in the target image, the object image being an image of a region surrounding the object; an aggregated object determination means for determining the objects to be included in an aggregated group based on information about the object images; an aggregate image generating means for generating an aggregate image from the object images of the objects included in the aggregate group; a feature amount calculation means for calculating a feature amount of the aggregate image; an object tracking unit that performs a tracking process on the object included in the aggregated group in the target image based on the feature amount of the aggregated image; an output means for outputting the result of the object tracking process; An information processing device comprising:

2. The aggregate image generating means generates the aggregate image in which at least one of a load for calculating the feature amounts and a load for object tracking processing is lower than at least one of a total load for calculating the feature amounts of the object images used in generating the aggregate image and a total load for object tracking processing. The information processing device according to claim 1 .

3. The aggregate image generating means generates the aggregate image having a data amount smaller than the total data amount of the object images used to generate the aggregate image. The information processing device according to claim 2 .

4. Using the aggregated group determined by the aggregated object determination means, the operations of the image acquisition means, the object image identification means, the aggregated image generation means, the feature amount calculation means, the object tracking means, and the output means are repeated for the plurality of target images. The information processing device according to claim 1 .

5. The aggregated object determination means changes the objects included in the aggregated group based on at least one of the identification of the object image by the object image identification means, the object image identified by the object image identification means, and the result of tracking the object by the object tracking means.

5. The information processing device according to claim 1.

6. When the object tracking means cannot track the object using the feature amount of the aggregate image, the feature amount calculation means calculates the feature amount of the object image; The object tracking means tracks the object based on the feature amount of the object image. The information processing device according to claim 1 .

7. When the object image identification means cannot detect some of the objects, the aggregated object determination means determines a second aggregated group by deleting some of the objects that the object image identification means cannot detect from the objects included in the aggregated group, The aggregate image generating means, the feature amount calculating means, and the object tracking means operate using a second aggregate group. The information processing device according to claim 1 .

8. Further comprising a storage means for storing the object image; If the object detected by the object image specifying means does not include the object included in the aggregated group, the aggregated image generating means generates the aggregated image using the object images stored in the storage means. The information processing device according to claim 1 .

9. Acquire a target image containing multiple objects; Detecting an object included in the target image, and identifying an object image in the target image, which is an image of a region surrounding the object; determining the objects to include in an aggregation group based on information about the object images; generating an aggregated image from the object images of the objects included in the aggregated group; Calculating a feature amount of the aggregate image; performing a tracking process of the object included in the aggregated group in the target image based on the feature amount of the aggregated image; Output the results of the object tracking process Information processing methods.

10. acquiring a target image including a plurality of objects; A process of detecting an object included in the target image and identifying an object image in the target image, the object image being an image of a region surrounding the object; determining, based on information about the object images, the objects to be included in an aggregation group; generating an aggregated image from the object images of the objects included in the aggregated group; A process of calculating a feature amount of the aggregate image; a process of performing a tracking process of the object included in the aggregated group in the target image based on the feature amount of the aggregated image; A process to output the results of the object tracking process. A program that causes a computer to execute the following.