Recognition Device

Abstract

To properly recognize a plurality of targets even when the targets overlap each other.SOLUTION: A recognition apparatus that recognizes a target with respect to image data includes: detection means of detecting a plurality of targets with respect to image data and outputting a plurality of detection objects that is based on the detected plurality of targets; feature extraction means of extracting respective feature quantities from the plurality of detection objects output from the detection means; filter means of outputting, for each of the detection objects, a filtered feature, which is a feature quantity obtained by filtering the feature quantity, based on a first mask for current time for each detection object predicted at previous time; and mask prediction means of predicting the first mask for next time for each of the detection objects.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a recognition device, a learning device, a recognition method for a recognition device, a learning method for a learning device, and a program. [Background technology]

[0002] Conventionally, a surveillance camera is known that, when an image of an object that is not a tracking target is included in a search range, extracts an image of the tracking target by masking the feature amount of the object image (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-16134 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to enable a plurality of objects to be appropriately recognized even when the objects overlap each other. [Means for solving the problem]

[0005] A recognition device according to one aspect of the present invention comprises: picture The apparatus includes a detection means for detecting a plurality of objects in image data and outputting a plurality of detected objects based on the detected plurality of objects, a feature extraction means for extracting a feature from each of the detected objects output by the detection means, a filter means for outputting, for each detected object, a filter feature which is a feature obtained by filtering the feature based on a first mask at the current time for each detected object predicted at the previous time, and a mask prediction means for predicting a first mask at the next time for each detected object. [Effects of the Invention]

[0006] According to the present invention, even when multiple objects overlap each other, the multiple objects can be appropriately recognized. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating a moving image and a tracking target. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a tracking device. [Figure 3] 10 is a flowchart illustrating an example of processing performed by a tracking device. [Figure 4] FIG. 1 is a diagram illustrating an example of a configuration of a neural network. [Figure 5] FIG. 10 is a diagram illustrating an example of a cost matrix. [Figure 6] FIG. 1 is a diagram illustrating an example of a filter operation and an RNN configuration. [Figure 7] FIG. 2 illustrates an example of a functional configuration of a learning device. [Figure 8] FIG. 2 is a diagram illustrating an example of the functional configuration of a learning data creation device and a learning unit. [Figure 9] 10 is a flowchart illustrating an example of processing by a learning device. [Figure 10] FIG. 1 is a diagram illustrating an example of a CNN. [Figure 11] FIG. 1 is a diagram illustrating an example of an RNN. [Figure 12] FIG. 10 is a diagram illustrating an example of learning data used in online learning. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings.

[0009] (First embodiment) In the first embodiment, a tracking device that uses a two-stage method (Tracking-by-detection paradigm) to track multiple tracking targets in close proximity will be described. The tracking device can track multiple targets captured as a video. Tracking combines two tasks: detection processing for each frame of the video, and ID assignment, which associates the detection results of the current frame with the tracking results up to the previous frame (hereinafter referred to as tracks). ID assignment is performed by associating the target positions with the proximity of feature values.

[0010] The two-stage method uses an independent detector and feature extractor for tracking. In the two-stage method, the tracker first performs detection on video frames using a detector, obtaining the resulting bounding box (BBox) of the object to be tracked. Next, the tracker inputs each BBox into a feature extractor to obtain the features of each BBox. Next, the tracker calculates the similarity between the features of the BBox and the features of the track as an allocation cost. For example, the tracker predicts the position and calculates the similarity from the predicted position of the track and the detected position. Next, the tracker assigns an ID based on the calculated allocation cost. Because the two-stage method performs tracking based on detection, the two-stage method framework is also called the tracking-by-detection paradigm.

[0011] A major technical issue in tracking is ID switching. ID switching is an error related to the ID (identifier) ​​assigned to a tracking target. An ID switching error (called ID transfer) occurs when an ID that has already been assigned to another tracking target is mistakenly assigned to the target being tracked. This embodiment aims to reduce ID transfer.

[0012] In the two-stage method, the tracking device crops (cuts out) the image using the BBox obtained by the detector, inputs the cropped image into a feature extractor, and acquires features for person identification. If multiple people are close to each other in the image, the BBox will contain multiple people, and the acquired features will become muddied. As a result, the tracking device may fail to match IDs, resulting in ID transfer.

[0013] In this embodiment, to solve the above-mentioned problem, the tracking device predicts the mask of the tracking target based on temporal information and filters the features in the BBox in a two-stage method. This allows the tracking device to prevent feature confusion even when multiple targets are close to each other, and to perform tracking with reduced occurrence of ID transfer.

[0014] FIG. 1 is a diagram illustrating a moving image and a tracking target tracked by a tracking device according to the first embodiment. The moving image is made up of a plurality of temporally consecutive frames (images) 101, 102, and 103. Frame 101 is a frame at time t=1. Frame 102 is a frame at time t=2. Frame 103 is a frame at time t=3. Assume that the tracking target is a person. Three people 104a, 104b, and 104c exist in each of frames 101 to 103. The tracking device detects people 104a, 104b, and 104c in each of frames 101 to 103, and obtains bounding boxes (hereinafter referred to as BBoxes) 105a, 105b, and 105c for people 104a, 104b, and 104c. BBox 105a is a circumscribing rectangle for person 104a. BBox 105b is a circumscribing rectangle for person 104b. BBox 105c is a circumscribing rectangle for person 104c. The tracking device assigns an ID (identifier) ​​to each of BBoxes 105a, 105b, and 105c to identify people 104a, 104b, and 104c. In frames 101 to 103, a group of BBoxes assigned the same ID is called a track. Hereinafter, this ID may be simply called an ID, or may be called a track ID.

[0015] For example, in frame 103 at time t=3, three people 104a, 104b, and 104c are close to each other. People 104b and 104c are partially obscured by people 104a and 104b in the foreground. Person 104b is located in front of person 104c. The area of ​​person 104b within BBox 105c is larger than the area of ​​person 104c within BBox 105c.

[0016] The tracking device calculates the similarity between the feature amount of the person acquired from the BBox 105c in frame 103 and the feature amounts of the people 104a, 104b, and 104c in frame 102. As a result, the feature amount of the person acquired from the BBox 105c in frame 103 will have a low similarity to the feature amount of the person 104c in frame 102, and a high similarity to the feature amount of the person 104b in frame 102. Similarly, there are cases where the feature amount of the person acquired from the BBox 105b in frame 103 will have a low similarity to the feature amount of the person 104b in frame 102, and a high similarity to the feature amount of the person 104c in frame 102.

[0017] The tracking device assigns IDs to associate people acquired from each BBox in the current frame 103 with tracks up to the previous frame based on the proximity of the features. As a result, the tracking device may assign the ID of person 104b to the person in BBox 105c in frame 103, and the ID of person 104c to the person in BBox 105b in frame 103, resulting in incorrect assignment. This incorrect assignment error is called ID transfer. This is just one example, but the tracking device of this embodiment can prevent ID transfer caused by such confusion of features due to occlusion.

[0018] 2(a) is a diagram showing an example of the functional configuration of a tracking device 200 according to the first embodiment during runtime processing. The tracking device 200 is an example of a recognition device. The tracking device 200 includes an image data acquisition unit 201, a detection unit 202, a feature extraction unit 203, a filter unit 204, a cost calculation unit 205, an ID assignment unit 206, a mask prediction unit 207, a track update unit 208, a dictionary storage unit 209, and an internal storage unit 210. Details of each of these functional configuration units will be described with reference to FIG. 3(a) and the like.

[0019] 3(a) is a flowchart showing an example of the runtime processing of the tracking device 200 according to this embodiment. Using this flowchart, the runtime processing of the tracking device 200 will be described.

[0020] In step S301, the tracking device 200 sets a dictionary stored in the dictionary storage unit 209 (FIG. 2(a)). This dictionary is a dictionary of CNN or RNN used in the detection unit 202 (FIG. 2(a)), feature extraction unit 203 (FIG. 2(a)), filter unit 204 (FIG. 2(a)), and mask prediction unit 207 (FIG. 2(a)). CNN is a convolutional neural network. RNN is a recurrent neural network. The dictionary is a collection of parameters that collectively include the weights and biases of each layer of a neural network. The dictionary is created in the learning process, which will be described later. The dictionary used in the runtime process is a learned dictionary learned in the learning process.

[0021] Loop L301 is a loop for times t=1 to T. Tracking device 200 performs processing for each frame of a moving image. The first loop is loop processing at time t=1. Time t=T is the time of the end frame of the moving image. Here, a moving image of a predetermined length is assumed, but tracking device 200 can also process a moving image of an undetermined length. In that case, tracking device 200 can end loop L301 according to another appropriate end condition.

[0022] In step S302, the image data acquisition unit 201 (Figure 2(a)) acquires image data of a moving image. This moving image may be a moving image file stored in data storage, streaming data distributed over a network, or moving image captured by a camera. Regardless of the format of the moving image, the image data acquisition unit 201 acquires a frame (image) in the moving image corresponding to time t in loop L301 using an appropriate method. For example, in the case of Figure 1, when time t = 1, the image data acquisition unit 201 acquires image data of frame 101.

[0023] In step S303, the detection unit 202 (FIG. 2(a)) detects a tracking target from the image data acquired in step S302. If the tracking target is a person, the detection unit 202 is a human body detection unit that detects a human body in the image data. The detection unit 202 is a detection unit that uses CNN. The parameters (dictionary) of the CNN are stored in the dictionary storage unit 209 (FIG. 2(a)). This dictionary has already been set in the detection unit 202 in step S301.

[0024] The detection unit 202 detects a human body in the image data and outputs a bounding rectangle (BBox) of the detected human body to the feature extraction unit 203. The BBox is a bounding rectangle image used to extract features for assigning a track ID. To extract features for identifying a person from the BBox, it is desirable that the positions of human body parts such as the head and torso within the BBox image are stable. Therefore, the detection unit 202 outputs a BBox that includes the feet or top of the head of the human body even if they are occluded.

[0025] In steps S304, S305 and S306, the tracking device 200 uses the neural network shown in Fig. 4. Before describing steps S304, S305 and S306, Fig. 4 will first be described.

[0026] Frame 401 in Figure 4 represents a frame at time t. In this example, the detection unit 202 detects three human bodies in frame 401 and outputs BBoxes 402 of the detected human bodies to the feature extraction unit 203. The feature extraction unit 203 crops the image of frame 401 using the BBoxes 402 and resizes it to the specified BBox size. The feature extraction unit 203 outputs the image in the BBox 402 to a CNN 403 for feature extraction. The CNN 403 has three Conv units. Each Conv unit has a convolutional layer, an activation layer, a pooling layer, and, in some cases, skip connections. Features can be extracted between Conv units. Here, pre-filtered features (p-feat) 404 and middle features (m-feat) 405 can be extracted between Conv units. The middle features 405 are input to the RNN 406. The RNN 406 has a function of predicting a human body region mask 407 for the next time t+1. Details of this RNN 406 will be described later. The CNN 403 has a filter unit 408. The filter unit 408 filters the pre-filter feature 404 with a human body region mask 409 for the next time t predicted at the previous time t-1. Details of this filter unit 408 will be described later. The CNN 403 finally outputs a filtered feature (f-feat; filtered feature) 410 for ID assignment.

[0027] In step S304 of FIG. 3(a), the feature extraction unit 203 (FIG. 2(a)) extracts pre-filter features 404 and intermediate features 405 from the image of BBox 402 from the previous step S303. In step S304, the feature extraction unit 203 first crops the frame 401 of the input image using BBox 402, which is the output of the previous step S303, resizes it to a predetermined size, and obtains the number of cropped images of the same size to be detected. For the sake of explanation, it is assumed here that the feature extraction unit 203 resizes it to [width, height]=[32, 64] pixels.

[0028] Next, the feature extraction unit 203 extracts pre-filter features 404 and intermediate features 405 from the cropped image using a CNN 403. The parameters (dictionary) of the CNN 403 are learned in advance, stored in the dictionary storage unit 209 (FIG. 2(a)), and have been set in step S301.

[0029] The intermediate features 405 and the pre-filter features 404 are hierarchical features with spatial information. For example, the intermediate features 405 have dimensions of [[16,32,64],[8,16,128]]. Here, the inner brackets are [width, height, channel]. The intermediate features 405 have features with two spatial resolutions. Similarly, the pre-filter features 404 have dimensions of [[16,32,64],[8,16,128],[4,8,512]].

[0030] In step S305, the filter unit 204 (Fig. 2(a)) calculates the ID allocation cost using the cost calculation unit 205 (Fig. 2(a)). The ID allocation cost is calculated for each combination of tracks up to the previous time and human bodies detected at the current time, and the calculation results are output as a matrix. As mentioned above, a track refers to a group of BBoxes with the same ID, but in this case, the track also has information necessary for cost calculation, which will be described later. In cost calculation, information necessary for calculation regarding tracks is saved in the internal storage unit 210 (Fig. 2(a)) and is read as appropriate in step S305.

[0031] FIG. 5(a) shows an example of a cost matrix when there are four tracks Track1 to Track4 up to the previous time point, and three detected objects Det1 to Det3 are detected at the current time point. Matrix 501 is a cost matrix. The rows of matrix 501 correspond to tracks Track1 to Track4. The columns of matrix 501 correspond to detected objects Det1 to Det3 detected by detection unit 202. The detected objects Det1 to Det3 are, for example, BBoxes 105a, 105b, and 105c. The smaller the element value (cost) of matrix 501, the higher the similarity, and the larger the element value, the lower the similarity. Each of tracks Track1 to Track4 has a filter feature (f-feat) 503 calculated at the previous time point and a human body region mask (mask) 504. Here, human body region mask 504 is a human body region mask for the current time point predicted at the previous time point. The human body region mask for the next time to be predicted at the current time will be described in step S307. Each of the detected objects Det1 to Det3 has a pre-filter feature (p-feat) 502. A cost 505 is an allocation cost calculated from the first track Track1 and the first detected object Det1.

[0032] In this calculation, first, the filter unit 204 (FIG. 2(a)) filters the feature amount. Here, the filtering corresponds to calculating the filter feature 503 of the detected object Det1 using the human body region mask 504 of the track Track1 and the pre-filter feature 502 of the detected object Det1. The filter feature 503 is calculated by the filter unit 408 in the neural network of FIG. 4.

[0033] FIG. 6(a) is a detailed diagram of the filter unit 408. As described above, the sizes of the pre-filter features (p-feat) 605 are [[16,32,64], [8,16,128], [4,8,512]]. f1, f2, and f3 in FIG. 6(a) are the three spatial resolution features of the pre-filter features 605 separated from each other. f1 is the spatial resolution feature of [16,32,64]. f2 is the spatial resolution feature of [8,16,128]. f3 is the spatial resolution feature of [4,8,512].

[0034] The human body region mask 606 is a one-channel mask with the same size as the resolution of the input image, and has a size of [32, 64, 1]. The multiplier 608 performs element-by-element multiplication. The resizer 810 resizes in the spatial direction and copies in the channel direction to align the size to the target of element-by-element multiplication.

[0035] For example, for f1, the resizing unit 810 resizes the human body region mask 606 from [32,64,1] to [16,32,1] and then copies it in the channel direction to make it [16,32,64]. Next, the multiplication unit 608 multiplies the resized human body region mask by the feature f1 element by element. As a result of this process, the feature is taken from the human body region at the current time point predicted at the previous time point, and the feature from the background is filtered out. The GAP unit performs GAP (Global Average Pooling) on ​​the filtered tensor. As a result, f1 becomes a 64-dimensional feature.

[0036] The concatenation unit 609 concatenates the features f1 to f3 after the gap. In this embodiment, the resulting feature is 704-dimensional. The FC is a fully connected layer. Here, the FC has a 704-dimensional input and a 256-dimensional output. The weights and biases of the FC are stored in the dictionary storage unit 209 (FIG. 2(a)) and are set in step S301. Finally, the FC outputs a filter feature (f-feat) 607. The filter feature 607 has a 256-dimensional feature.

[0037] The cost calculation unit 205 (FIG. 2(a)) calculates the similarity between the filter feature 607 obtained by the above calculation and the filter feature already possessed by the track, and converts it into a cost. Here, the cost calculation unit 205 calculates cosine similarity as the similarity. Cosine similarity takes on real values ​​ranging from −1 to +1, with +1 indicating the most similar. Generally, an evaluation value where the larger the value, the more positive the meaning, is called a score, and the inverse is called a cost. To make a small value an evaluation value indicating high similarity, i.e., cost, the cost calculation unit 205 multiplies the cosine similarity by −1. Furthermore, depending on the algorithm for the assignment problem used in the next step S306, it is desirable for the cost to be 0 or greater. Therefore, here, the cost calculation unit 205 adds 1 to the cost as a bias. The above calculation is performed as many times as there are combinations of detected objects Det1 to Det3 and tracks Track1 to Track4, thereby calculating the cost matrix 501.

[0038] The explanation here is based on the assumption that a track exists in the previous frame. There may be cases where a track does not exist in the previous frame, such as immediately after tracking begins. In such cases, a cost matrix with a finite size cannot be calculated, so this process is omitted.

[0039] In step S306 of Fig. 3(a), the ID assignment unit 206 (Fig. 2(a)) assigns track IDs to the detected objects Det1 to Det3. The ID assignment unit 206 applies an assignment problem algorithm, such as the Hungarian algorithm or the greedy algorithm, to the cost matrix 501 created in the previous step S305.

[0040] If there are many detected objects Det1 to Det3 for tracks Track1 to Track4, or if the allocation cost of the optimal allocation is high and the allocation result is unreliable, some detected objects will not be assigned a track ID. Also, a track ID will not be assigned to a detected object if no track exists up to the previous frame. A new ID will be assigned to such a detected object to which a track ID cannot be assigned. In this way, a track ID will be assigned to the detected object.

[0041] In step S307 of FIG. 3(a), the mask prediction unit 207 (FIG. 2(a)) predicts a human body region mask for the next time. The mask prediction unit 207 performs the prediction using the RNN 406. The RNN 406 is a neural network that has a recursive input of the previous state, but what has temporal identity is the track. Therefore, in this case, the previous state of the RNN 406 is held by the track.

[0042] Now, as a result of associating track IDs with detected objects executed in the previous step S306, the detected objects and tracks are associated with each other. Therefore, the mask prediction unit 207 predicts the human body region mask 407 for the next time point using the intermediate features (m-feat) 405 of the detected objects and the previous state of the RNN 406 of the tracks.

[0043] No track is associated with the detected object to which a new ID was assigned in the previous step S306. Therefore, in step S307, the mask prediction unit 207 predicts the human body region mask 407 for the next time point using an initial value in the previous state of the RNN 406. The initial value may be set to zero, as is the initial value of a general RNN 406.

[0044] The dictionary of the RNN 406 is stored in the dictionary storage unit 209 (FIG. 2(a)) and set in step S301. The previous state of the RNN 406 is held in the internal storage unit 210 (FIG. 2(a)) in FIG. 2 and is read at the beginning of step S307.

[0045] FIG. 6(b) is a diagram illustrating the details of the RNN 406. The RNN 601 in FIG. 6(b) corresponds to the RNN 406 in FIG. 4. The RNN 601 includes a fully connected layer (FC), a recurrent layer (represented here as a GRU; Gated-Recurent-unit), a convolutional layer (Conv), a sigmoid function (Sigmoid), and so on. Each layer is accompanied by an activation layer, etc. In addition, tensor size conversion processes (reshape processes and resize processes) are performed between layers as appropriate. These processes are omitted in FIG. 6(b). The GRU uses the state from the previous time point. As described above, the state from the previous time point is read from the internal storage unit 210 (FIG. 2(a)).

[0046] The intermediate features (m-feat) 602 are input to the RNN 601, and as mentioned above, have a size of [[16,32,64],[8,16,128]]. When input to the first FC of the RNN 601, the intermediate features 602 are flattened. That is, the input size of the intermediate features 602 is

[49152] . The second FC of the RNN 601, the convolution layer, reshape layer, and resize layer, convert the features to the appropriate size. The sigmoid receives as input a one-channel tensor of the same size as the input image. That is, its size is [32,64,1]. Finally, the RNN 601 outputs the human body region mask 603 for the next time (t+1, where t is the current time). The size of the human body region mask 603 is [32,64,1].

[0047] In step S308 of FIG. 3(a), the track update unit 208 (FIG. 2(a)) updates the track. As a result of step S306, the filter feature (f-feat) 410 of the detected object calculated by the filter unit 204 (FIG. 2(a)) is currently associated with the track. The track update unit 208 associates the filter feature 410 with the human body region mask 407 predicted in step S307 and the current state of the RNN 406 with the track ID and stores them in the internal storage unit 210. That is, in step S308, the filter feature 410 associated with the track and the states of the human body region mask 407 and RNN 406 are updated.

[0048] In addition, for detected objects to which new IDs have been assigned, the filter feature (f-feat) has not been calculated. In this case, the track update unit 208 calculates the filter feature (f-feat) using a human body region mask (mask) in which all elements are 1, and updates it as a new filter feature (f-feat) for the track.

[0049] The above is the process during runtime. Next, the process during learning will be described. During learning in this embodiment, the learning device 700 in FIG. 7(a) and the learning device 710 in FIG. 7(b) are used. In addition, to provide a detailed description of the data used by the learning device 710, an example of a learning data creation device for the learning device 710 will also be disclosed.

[0050] 7(a) is a diagram illustrating an example of the functional configuration of a learning device 700. The learning device 700 includes a learning data acquisition unit 701, a feature extraction unit 702, a loss calculation unit 703, a parameter update unit 704, a ReID learning data storage unit 705, and a dictionary storage unit 706. Details of each of these functional configuration units will be described later.

[0051] 7(b) is a diagram illustrating an example of the functional configuration of a learning device 710. The learning device 710 includes a learning data acquisition unit 711, a mask prediction unit 712, a loss calculation unit 713, a parameter update unit 714, a mask GT data storage unit 715, an intermediate feature data storage unit 716, and a dictionary storage unit 717. Details of each of these functional configuration units will be described later.

[0052] 8(a) is a diagram showing an example of the functional configuration of a training data creation device 800. The training data creation device 800 includes a training data acquisition unit 801, a feature extraction unit 802, a segmentation unit 803, an ID assignment unit 804, and an annotation unit 805. The training data creation device 800 also includes a dictionary storage unit 806, a tracking data storage unit 807, an intermediate feature data storage unit 808, and a mask GT data storage unit 809. Details of each of these functional components will be described later.

[0053] 9(a) is a flowchart showing the overall flow of the learning process. In step S901, the learning device 700 executes a subroutine for learning the CNN 403 used in the feature extraction unit 203 (FIG. 2(a)) during runtime processing. Details of step S901 will be described later with reference to FIG. 9(b).

[0054] Then, in step S902, the training data creation device 800 executes a subroutine to create data for training the mask prediction unit 207 (FIG. 2(a)) using the trained feature extraction unit 203. Details of step S902 will be described later with reference to FIG. 9(c).

[0055] Finally, in step S903, the learning device 710 executes a subroutine to train the RNN 406 used in the mask prediction unit 207 (FIG. 2(a)) during runtime processing. Details of step S903 will be described later with reference to FIG. 9(d).

[0056] FIG. 9(b) is a flowchart showing details of step S901 in FIG. 9(a), in which the feature extraction unit 203 is trained. In step S911, the learning device 700 performs settings related to the learning of the CNN 403 used in the feature extraction unit 203. When learning from scratch, the learning device 700 sets initial values ​​for the parameters of each layer of the CNN 403. When additional learning is performed on learned parameters, the learning device 700 sets the learned parameters. In addition, the learning device 700 sets hyperparameters related to learning. In this embodiment, the learning device 700 performs learning using the stochastic gradient descent method. Therefore, the learning device 700 sets the mini-batch size, learning coefficient, solver parameters for the stochastic gradient descent method, and the like as hyperparameters.

[0057] Loop L911 is a loop for n=1 to N related to the iteration of the stochastic gradient descent method. In this case, N may be a preset value, or the loop may be terminated when the loss drops below a certain level.

[0058] In step S912, the learning data acquisition unit 701 (FIG. 7(a)) acquires ReID learning data stored in the ReID learning data storage unit 705 (FIG. 7(a)). The ReID learning data storage unit 705 stores pairs of person cut-out images for learning and their corresponding person IDs. The same person ID is assigned to images of the same person. Even when multiple data sets are used as sources, the IDs are reassigned so that the same ID is assigned to the same person.

[0059] In step S912, the training data acquisition unit 701 acquires the image and ID pairs for the mini-batch size, and performs preprocessing on the images, such as random clipping, color conversion, and other augmentation processes, and normalization. The size of the images supplied to the following steps is [width, height] = [32, 64], the same as in runtime processing.

[0060] In step S913, the feature extraction unit 702 (FIG. 7(a)) extracts person ID features. FIG. 10(a) shows a CNN 1001 used in the processing at the time of learning. In step S913, the feature extraction unit 702 drives the CNN 1001. Unlike the CNN 403 in FIG. 4 used in the processing at runtime, the CNN 1001 does not use the filter unit 408, but uses a GAP 1012 and an FC 1013. This FC 1013 has the same configuration as the FC in FIG. 6(a). Furthermore, the Conv 1011 of the CNN 1001 has the same configuration as the Conv of the CNN 403 in FIG. 4. Furthermore, the GAP 1012 performs global average pooling for each layer of features with different spatial resolutions, and then concatenates the results. In other words, the input to the FC 1013 is the same as the input from the filter processing unit 604 in FIG. 6(a) to the FC, which is a 704-dimensional vector. The output of FC1013 is 256-dimensional. The 256-dimensional features of the output of FC1013 using only GAP1012 (without filtering) are hereafter referred to as GAP features (g-feat).

[0061] 4, the CNN 1001 can acquire features from the intermediate layer. That is, the CNN 1001 can acquire pre-filter features (p-feat) 1002 and intermediate features (m-feat) 1003 from the intermediate layer.

[0062] Here, the case where the CNN 1001 uses the GAP 1012 without using the filter in Fig. 4 has been described. However, if the ReID training data has the ground-truth (GT) of the region mask in addition to the person ID, the CNN 1001 may use the filter. In that case, the feature extraction unit 702 uses the GT of the region mask, as with the CNN 403 used in the runtime processing, to execute the filter processing unit 604 in Fig. 6(a), and then executes the FC processing.

[0063] In step S914, the loss calculation unit 703 (FIG. 7(a)) calculates the loss. In this embodiment, the learning device 700 performs person ID learning as a classification problem. The loss calculation unit 703 multiplies the GAP feature calculated in the previous step S913 by a Softmax function using Softmax 1014, and calculates the cross-entropy loss between the person ID, which is the correct label. However, this embodiment is not limited to learning person feature extraction as a classification problem. Contrastive loss, triplet loss, etc. may also be used.

[0064] In step S915, the parameter update unit 704 (FIG. 7(a)) updates the parameters of the CNN 1001. Here, the parameter update unit 704 uses the cross-entropy loss calculated in the previous step S914 to calculate the amount of update for the parameters of each layer of the CNN 1001 by backpropagation, and updates the parameters.

[0065] Here, among the updated parameters, three Conv 1011 of CNN 1001 are used in the feature extraction unit 203 (FIG. 2(a)). Also, FC 1013 of CNN 1001 is used to calculate filter features in the filter unit 204 (FIG. 2(a)), i.e., is used in FC in FIG. 6(a). The parameter update unit 704 saves the updated parameters in the dictionary storage unit 706 (FIG. 7(a)).

[0066] The learning device 700 repeats the above loop processing for n=1 to N. The above is the processing related to learning of the CNN 403 used in the feature extraction unit 203 (FIG. 2(a)).

[0067] FIG. 9(c) is a flowchart showing details of step S902 in FIG. 9(a), and shows the creation of data used in learning by the mask prediction unit 207 (FIG. 2(a)).

[0068] In step S921, the training data acquisition unit 801 (FIG. 8(a)) acquires training data. The training data is tracking data and is stored in the tracking data storage unit 807 (FIG. 8(a)). The tracking data is stored by associating a video with a BBox and a track ID of the tracking target present in each frame of the video. The training data acquisition unit 801 sequentially acquires multiple videos, and further acquires the video one frame at a time. Furthermore, the training data acquisition unit 801 associates and acquires the BBox and track ID of the tracking target present in that frame. Then, the training data acquisition unit 801 performs preprocessing of data to be supplied to the feature extraction unit 802 (FIG. 8(a)) and the segmentation unit 803 (FIG. 8(a)), which will be used in later steps. As preprocessing of the data to be supplied to the feature extraction unit 802, the training data acquisition unit 801 crops the image (frame) using a BBox and resizes it to a predetermined size. The size is [32, 64], as before. The learning data acquisition unit 801 associates the cropped image with the video number, frame number, and track ID, and supplies the image to the feature extraction unit 802. The learning data acquisition unit 801 also associates the image (frame) with information on the BBox, video number, frame number, and track ID, and supplies the image to the segmentation unit 803.

[0069] In step S922, the feature extraction unit 802 (FIG. 8(a)) extracts features from the cropped image. The feature extraction unit 802 uses the CNN 1001 that has already been trained in the feature learning step S901 in the training process of FIG. 9(a). The dictionary of the trained CNN 1001 is stored in the dictionary storage unit 806 (FIG. 8(a)) and is read by the feature extraction unit 802. The feature extraction unit 802 acquires intermediate features (m-feat) 1003 from this CNN 1001. The feature extraction unit 802 stores the acquired intermediate features 1003 in the intermediate feature data storage unit 808 (FIG. 8(a)) in association with information such as the video number, frame number, and track ID.

[0070] In step S923, the segmentation unit 803 (FIG. 8(a)) performs instance segmentation on the image. A pre-trained model is used for instance segmentation. The model parameters are stored in the dictionary storage unit 806 (FIG. 8(a)) and are read by the segmentation unit 803. When an image is input to the instance segmentation module, it outputs region masks of instances of multiple categories including the person to be tracked, and also outputs a BBox corresponding to each instance. The region masks and BBoxes are associated with each other.

[0071] In step S924, the ID assignment unit 804 (FIG. 8(a)) assigns a track ID to the region mask of the instance. The ID assignment unit 804 creates a cost matrix from the BBox and track ID supplied from the learning data acquisition unit 801 (FIG. 8(a)) and the BBox output by the segmentation unit 803 (FIG. 8(a)), and assigns a track ID by solving the assignment problem. The cost is the Euclidean distance of the four-dimensional coordinate values ​​of the BBox (e.g., center x, y, width, height). In this way, the ID assignment unit 804 assigns a track ID to the region mask of the instance.

[0072] In step S925, the annotation unit 805 (FIG. 8(a)) corrects the segmentation results of the segmentation unit 803 and the track ID assignment results of the ID assignment unit 804. In this process, the annotator visually checks the image and its corresponding region mask and track ID using a GUI tool, and if it determines that correction is necessary, manually operates the GUI tool to make the correction. There are many possible GUI tools, so no specific example is given, but any tool that can perform the above operations should be used. Furthermore, if the instance segmentation and ID assignment are sufficiently accurate, step S925 may be omitted. The annotation unit 805 associates the region mask and track ID processed above with the video number and frame number, and stores them in the mask GT data storage unit 809 (FIG. 8(a)). The intermediate features stored in the intermediate feature data storage unit 808 (Figure 8(a)) and the region masks stored in the mask GT data storage unit 809 (Figure 8(a)) are both stored in association with the video number, frame number, and track ID.

[0073] This concludes the description of how data used for learning in the mask prediction unit 207 (FIG. 2(a)) is created.

[0074] FIG. 9(d) is a flowchart showing details of step S903 in FIG. 9(a), illustrating the learning of the mask prediction unit 207 (FIG. 2(a)). In step S931, the learning device 710 (FIG. 7(b)) performs settings related to the learning of the RNN 406 used in the mask prediction unit 207. This process is similar to step S911 in FIG. 9(b), and therefore a detailed description thereof will be omitted.

[0075] Loop L931 is a loop related to the stochastic gradient descent method. This loop L931 is similar to loop L911 in FIG. 9(b), so a detailed description of this loop L931 will be omitted.

[0076] In step S932, the training data acquisition unit 711 (FIG. 7(b)) acquires training data. The RNN 406 is trained using a method called Back Probagation Through Time (BPTT). In this method, the training device 710 sequentially inputs data to the RNN 406 and sequentially acquires the output of the RNN 406. The training device 710 calculates the loss for each sequential output and propagates the error back through the sequence to update the parameters. Therefore, in step S932, the training data acquisition unit 711 extracts a temporally continuous data sequence with the same track ID as one sample of data and supplies it to the subsequent mask prediction unit 712. The data sequence supplied here is the intermediate feature (m-feat) stored in the intermediate feature data storage unit 716 (FIG. 7(b)).

[0077] In step S933, the mask prediction unit 712 (FIG. 7(b)) predicts a person region mask for the next time. The RNN 406 has the same configuration as the RNN 601 illustrated in FIG. 6(b). The mask prediction unit 712 extracts the input data sequence (intermediate feature (m-feat) sequence) created in the previous step S932 for one time period and inputs it sequentially to the RNN 601. The RNN 601 outputs a region mask corresponding to the input data for each time period. Since the RNN 601 receives multiple pieces of data that are continuous in time, it outputs multiple region masks corresponding to the respective input data.

[0078] In step S934, the loss calculation unit 713 (FIG. 7(b)) calculates the loss. First, the loss calculation unit 713 obtains the region mask of the next time from the mask GT data storage unit 715 as the ground-truth (GT) of the region mask for the input data at each time. Next, the loss calculation unit 713 calculates the cross-entropy between pairs of the region mask for each time and the GT of the region mask for the next time. Then, the loss calculation unit 713 determines the sum of the cross-entropies of each of multiple consecutive pairs as the loss for one sample.

[0079] In step S935, the parameter update unit 714 (FIG. 7(b)) updates the parameters of the RNN 601. The parameter update unit 714 applies BPTT to the loss calculated in the previous step S934, calculates the amount of update for the parameters of each layer of the RNN 601, and updates the parameters. As a result, when intermediate features of a temporally continuous intermediate feature sequence are input to the RNN 601 sequentially in time as tracks, the RNN 601 learns to output region masks corresponding to the time immediately following the input time. The parameter update unit 714 saves the learned parameters of the RNN 601 in the dictionary storage unit 717 (FIG. 7(b)).

[0080] The learning device 710 repeats the above loop processing for n=1 to N. The above is the details of the processing during learning. Through these processing steps, the parameters of the CNN and RNN used at runtime are learned.

[0081] As described above, according to this embodiment, when creating features for ID assignment, tracking device 200 filters intermediate features 405 of CNN 403 with spatial resolution using human body region mask 409, and is able to extract only features corresponding to the human body region. Therefore, when people are close to each other, tracking device 200 can prevent confusion of features between the people in close proximity and reduce ID transfer during tracking.

[0082] While this embodiment has been described with reference to an example of the tracking device 200 applied to tracking, it can also be used, for example, in a recognition device for a crowd counting task. In this case, the recognition device uses the current human body region mask predicted in the previous state to filter the pre-filter features acquired using the current frame as input, and performs person counting. That is, among the functional components shown in FIG. 2( a), the recognition device uses the image data acquisition unit 201, the detection unit 202, the feature extraction unit 203, the filter unit 204, the cost calculation unit 205, and the mask prediction unit 207. Of these, the cost calculation unit 205 only performs filtering processing and person counting. This allows the recognition device to achieve more accurate person counting by reducing the influence of the background region.

[0083] As described above, tracking device 200 is an example of a recognition device that recognizes objects in image data. A recognition method of the recognition device will be described below. Detection unit 202 detects multiple objects in image data and outputs multiple detected objects Det1 to Det3 based on the multiple detected objects. The objects are, for example, people 104a, 104b, and 104c. Detected objects Det1 to Det3 are, for example, BBoxes 105a, 105b, and 105c, which are circumscribing rectangles of the above objects.

[0084] The feature extraction unit 203 extracts pre-filter features (feature amounts) 404 and intermediate features (feature amounts) 405 from each of the multiple detected objects Det1 to Det3 output by the detection unit 202. The filter unit 204 outputs, for each detected object, filtered features 410, which are features obtained by filtering the pre-filter features 404, based on a target area mask 409 for the current time for each detected object predicted at the previous time. The feature extraction unit 203 and the filter unit 204 use a convolutional neural network (CNN) 403.

[0085] The mask prediction unit 207 predicts an object region mask 407 at the next time for each detected object based on the intermediate features 405 extracted by the feature extraction unit 203 and the state of the mask prediction unit 207. The object region mask 407 is a mask of the region in the BBox where the object exists. The state of the mask prediction unit 207 is the state of a recurrent neural network (RNN) 406 that has a recursive input of the previous state. The ID assignment unit 206 assigns an identifier (ID) to each of the multiple detected objects based on the filter features 410 for each detected object.

[0086] A group of detected objects assigned the same identifier at each time is called Track 1 to Track 4. The cost calculation unit 205 calculates the cost for each track and detected object based on the similarity between the filter features of the multiple tracks at the previous time and the filter features of each detected object at the current time. The ID assignment unit 206 assigns an identifier to each of the multiple detected objects based on the cost.

[0087] The track update unit 208 associates and stores, for each track, the filter feature 410 at the current time output by the filter unit 204, the target region mask 409 at the next time predicted by the mask prediction unit 207, and the assigned identifier. In this way, the track update unit 208 updates the track information.

[0088] The learning device 700 in FIG. 7(a) and the learning device 710 in FIG. 7(b) are learning devices for training the tracking device 200 in FIG. 2(a). The learning method of the learning device 710 will be described below. The mask prediction unit 712 of the learning device 710 corresponds to the mask prediction unit 207 in FIG. 2(a) and inputs temporally continuous feature quantities of the same track to predict the target region mask 407 at the next time. The loss calculation unit 713 calculates the loss based on the target region mask 407 predicted by the mask prediction unit 712 and the correct value of the target region mask 407. The parameter update unit 714 updates the parameters of the mask prediction unit 712 based on the loss calculated by the loss calculation unit 713.

[0089] As described above, according to this embodiment, the recognition device can prevent confusion of features and reduce the occurrence of ID transfer even when multiple targets are close to each other. The tracking device 200 can reduce identifier assignment errors (ID transfer) when tracking multiple targets in a video.

[0090] (Second embodiment) In the first embodiment, a case has been described in which all of the multiple people present in a frame are filtered using a human body region mask. However, it is rare for people present in a frame to always overlap each other, and depending on the number of people appearing, there may be cases in which people barely overlap each other. In such cases, ID transfer rarely occurs even without using filtering.

[0091] As explained in step S305 (FIG. 3(a)), the filtering calculation involves the calculation of the fully connected layer (see FIG. 6(a)) and the element-by-element product of the tensor of intermediate features and the human body region mask, as many times as the number of combinations of detected objects and tracks. Therefore, in the first embodiment, the tracking device 200 requires a larger amount of calculation than when no filtering is performed. Therefore, in the second embodiment, the tracking device 200 uses filtering only when the overlap between people is equal to or greater than a certain amount, and does not use filtering when the overlap is less than the certain amount.

[0092] Frame 102 at t=2 in FIG. 1 is an important state in the second embodiment, and will be described below. In frame 102, person 104a in BBox 105a is not occluded by other people, and person 104b in BBox 105b and person 104c in BBox 105c overlap. BBox 105b partially includes the area of ​​person 104c in BBox 105c, and BBox 105c partially includes the area of ​​person 104b in BBox 105b. Therefore, the feature amount extracted from BBox 105b is contaminated by the feature amounts derived from person 104b and person 104c. Similarly, the feature amount extracted from BBox 105c is contaminated by the feature amounts derived from person 104b and person 104c. On the other hand, since person 104a in BBox 105a is not occluded, it is believed that the feature amount extracted from BBox 105a is sufficient to identify the track ID. Therefore, for people who are occluded by a certain amount or more, such as BBox 105b and BBox 105c, feature filtering is performed as in the first embodiment. For people who are not occluded or who are occluded by less than a certain amount, such as BBox 105a, filtering is not performed, and feature extraction is performed from the entire area within BBox 105a.

[0093] Next, a specific functional configuration and processing of the second embodiment will be described. The functional configuration and flowchart of the processing at runtime of the tracking device 200 described in the second embodiment are the same as those of the first embodiment. That is, the functional configuration of the processing at runtime of the tracking device 200 is as shown in FIG. 2(a). The flowchart of the processing at runtime of the tracking device 200 is as shown in FIG. 3(a). Furthermore, since many of the specific processing flows of the second embodiment are the same as those of the first embodiment, detailed explanations will be omitted except for those that are different.

[0094] 10(b) is a diagram showing an example of a CNN 1004 used in the second embodiment. The CNN 1004 is used in place of the CNN 403 in FIG. 4. The CNN 1004 has three Convs 1021, a filter 1022, an FC 1023, a GAP (Global Average Pooling) 1024, and an FC 1025. The CNN 1004 can extract intermediate features (m-feat) 405 and pre-filter features (p-feat), and inputs a human body region mask 409.

[0095] The CNN 1004 branches into two paths at the position of the pre-filter feature 404: one that passes through the filter 1022 and the FC 1023, and the other that passes through the GAP 1024 and the FC 1025. The FC 1023 outputs the filter feature (f-feat) 1005. The FC 1025 outputs the GAP feature (g-feat) 1006. Details of the calculation of the filter 1022 are the same as those described in step S305 (FIG. 3(a)) of the first embodiment.

[0096] Step S304 in FIG. 3(a) of the second embodiment differs from that of the first embodiment and will be described below. In step S304, the feature extraction unit 203 (FIG. 2(a)) performs feature extraction using the CNN 1004 in FIG. 10(b) to obtain three features: intermediate features (m-feat) 405, pre-filter features (p-feat) 404, and gap features (g-feat) 1006. These three pieces of information can be calculated even when the detected object has not yet been associated with a track. The gap feature 1006 is a feature that is not filtered.

[0097] Step S305 in FIG. 3(a) of the second embodiment differs from that of the first embodiment and will be described in detail below. In step S305, the filter unit 204 (FIG. 2(a)) uses the cost calculation unit 205 (FIG. 2(a)) to calculate an ID assignment cost for each combination of tracks up to the previous time and detected objects at the current time. At this time, the cost calculation unit 205 calculates the overlap between human bodies and determines whether the overlap is less than or equal to a threshold value. The overlap may be calculated using, for example, IoU (Intersection Of Union). A reference value for this overlap is set as a threshold value.

[0098] 1, there is no overlap of human bodies in BBox 105a, but the overlap of human bodies in BBox 105b and BBox 105c is equal to or greater than the threshold. Also, assume that there are four tracks for the previous time.

[0099] Fig. 5(b) is a diagram showing an example of a cost matrix 506 for tracks Track1 to Track4 and detected objects Det1 to Det3 in this case. BBox 105a corresponds to detected object Det1, BBox 105b corresponds to detected object Det2, and BBox 105c corresponds to detected object Det3. As shown in Fig. 5(b), in this embodiment, tracks Track1 to Track4 always have a GAP feature (g-feat).

[0100] Here, since the overlap between the detected object Det1 and the human bodies is less than the threshold, the cost calculation unit 205 calculates a GAP feature (g-feat) 1006, which is a feature that is not filtered, using the GAP 1024 and FC 1025 in FIG. 10(b). Then, the cost calculation unit 205 calculates a cost using the GAP feature (g-feat) 1006. In this case, the cost calculation unit 205 calculates a cosine similarity between the GAP feature (g-feat) 1006 and the GAP feature (g-feat) of the track, and converts it into a cost (see step S305 in the first embodiment). The cost calculation unit 205 does not perform the calculations of the filter 1022 and FC 1023 in FIG. 10(b), and therefore the cost calculation unit 205 requires a small amount of calculation. Here, the cost calculation unit 205 calculates four costs 507, 508, 509, and 510, which are combinations of the detected object Det1 and the tracks Track1 to Track4.

[0101] Since the overlap between the detected objects Det2 and Det3 is equal to or greater than the threshold, the cost calculation unit 205 calculates the filter feature (f-feat) 1005 using the filter 1022 and FC 1023 in FIG. 10(b) as in the first embodiment. Then, as in the first embodiment, the cost calculation unit 205 calculates the cosine similarity between the filter feature (f-feat) 1005 and the filter feature (f-feat) of the track, and converts it into a cost. Here, the cost calculation unit 205 calculates costs other than costs 507, 508, 509, and 510 in the cost matrix 506.

[0102] In step S308 of FIG. 3(a), in addition to the processing of step S308 in the first embodiment, the track update unit 208 (FIG. 2(a)) calculates a filter feature (f-feat) for detected objects in which the overlap between human bodies assigned track IDs is below a threshold.

[0103] For example, suppose that the detected object Det1 is assigned to the track Track1 as a result of ID assignment in the cost matrix 506 in Fig. 5(b). At this point, the filter feature (f-feat) for the detected object Det1 has not been calculated, so in step S308, the track update unit 208 calculates the filter feature (f-feat).

[0104] In this embodiment, the information associated with the track ID includes the filter feature (f-feat), the predicted human body region mask (mask), the current state of the RNN, and the GAP feature (g-feat). That is, in step S308, the track update unit 208 updates the filter feature (f-feat), the human body region mask (mask), the current state of the RNN, and the GAP feature (g-feat) corresponding to the track.

[0105] The functional configuration and flowchart of the learning process in this embodiment are the same as those in the first embodiment. That is, the functional configuration of the learning process is as shown in the learning device 700 in FIG. 7(a), the learning device 710 in FIG. 7(b), and the learning data creation device 800 in FIG. 8(a). The flowchart of the learning process is as shown in FIGS. 9(a) to 9(d). Furthermore, since the specific processing flow of the learning process is the same as that in the first embodiment, a detailed description will be omitted.

[0106] As described above, according to this embodiment, when the overlap between human bodies within a BBox is less than a threshold, the tracking device 200 does not filter the feature amount, and assigns IDs using the GAP feature (g-feat) acquired from the entire BBox. As a result, when the overlap between human bodies is less than a threshold, the tracking device 200 can calculate the assignment cost relatively lightly, thereby reducing the overall processing load of tracking.

[0107] The tracking device 200 is an example of a recognition device. The detection unit 202 detects multiple objects in image data and outputs multiple detected objects Det1 to Det3 based on the detected multiple objects. The GAP 1024 and FC 1025 are examples of an averaging unit. If the degree of overlap of multiple objects in the detected objects is less than a threshold, the GAP 1024 and FC 1025 output a GAP feature (average feature) 1006 obtained by spatially averaging the pre-filter features 404 extracted by the feature extraction unit 203. The filter unit 204 uses the filter 1022 and FC 1023 to output a filter feature 1005 if the degree of overlap of multiple objects in the detected objects is equal to or greater than a threshold.

[0108] If the degree of overlap of multiple objects in the detected objects is less than a threshold, the cost calculation unit 205 calculates the cost for each track and detected object based on the similarity between the GAP features of the multiple tracks at the previous time and the GAP features of each detected object at the current time.

[0109] In addition, if the degree of overlap of multiple objects in the detected objects is equal to or greater than a threshold, the cost calculation unit 205 calculates the cost for each track and detected object based on the similarity between the filter features of the multiple tracks at the previous time and the filter features of each detected object at the current time.

[0110] If the degree of overlap of multiple detected objects is equal to or greater than a threshold, the track update unit 208 associates and stores the current filter feature 1005, the object region mask 407 for the next time, and the assigned identifier for each track. In this way, the track update unit 208 updates the track information. The current filter feature 1005 is output by the filter unit 204. The object region mask 407 for the next time is predicted by the mask prediction unit 207. The identifier is assigned by the ID assignment unit 206.

[0111] Furthermore, if the degree of overlap of multiple objects in the detected objects is less than a threshold, the track update unit 208 acquires a filter feature 1005, which is a feature obtained by filtering the pre-filter feature 404 extracted by the feature extraction unit 203. In this case, the track update unit 208 associates and stores, for each track, the acquired filter feature 1005 at the current time, the gap feature 1006 at the current time, the object area mask 407 at the next time, and the assigned identifier. In this way, the track update unit 208 updates the track information. The gap feature 1006 at the current time is output by the filter 1022 and the FC 1023. The object area mask 407 at the next time is predicted by the mask prediction unit 207. The identifier is assigned by the ID assignment unit 206.

[0112] As described above, when the degree of overlap between multiple objects is less than a threshold, the recognition device can calculate the cost relatively lightly, thereby reducing the processing load.

[0113] (Third embodiment) In the first and second embodiments, a data set of human body region masks is created in advance, and the RNN 406 is trained to predict the human body region mask for the next time. At runtime, the tracking device 200 uses the human body region mask predicted by the trained RNN 406 to filter the features, thereby preventing the features from becoming cloudy due to overlapping of people and reducing ID transfer during tracking.

[0114] In the third embodiment, a case will be described in which a mask for distinguishing between people is learned online from data acquired in real time in an actual operation scene, and the mask is used to filter features. As a result, even when there is a large overlap between people, important information for distinguishing between people can be extracted, and ID transfer during tracking can be reduced.

[0115] 2(b) is a diagram showing an example of the functional configuration of a tracking device 220 according to the third embodiment during runtime processing. The tracking device 220 includes an image data acquisition unit 221, a detection unit 222, a feature extraction unit 223, a filter unit 224, a cost calculation unit 225, an ID assignment unit 226, a mask prediction unit 227, and a track update unit 228. The tracking device 220 also includes a learning data creation unit 229, a learning unit 230, a dictionary storage unit 231, an internal storage unit 232, and an intermediate feature data storage unit 233. Details of each of these functional components will be described with respect to those that differ from the first embodiment. Descriptions of those that do not differ will be omitted as appropriate.

[0116] The image data acquisition unit 221 corresponds to the image data acquisition unit 201 in FIG. 2(a). The detection unit 222 corresponds to the detection unit 202 in FIG. 2(a). The feature extraction unit 223 corresponds to the feature extraction unit 203 in FIG. 2(a). The filter unit 224 corresponds to the filter unit 204 in FIG. 2(a). The cost calculation unit 225 corresponds to the cost calculation unit 205 in FIG. 2(a). The ID assignment unit 226 corresponds to the ID assignment unit 206 in FIG. 2(a). The mask prediction unit 227 corresponds to the mask prediction unit 207 in FIG. 2(a). The track update unit 228 corresponds to the track update unit 208 in FIG. 2(a). The dictionary storage unit 231 corresponds to the dictionary storage unit 209 in FIG. 2(a). The internal storage unit 232 corresponds to the internal storage unit 210 in FIG. 2(a).

[0117] 3(b) is a flowchart showing an example of runtime processing of the tracking device 220 according to the third embodiment. Differences between the third embodiment and the first embodiment will be mainly described, and descriptions of other differences will be omitted as appropriate.

[0118] Step S311, loop L311, steps S312, S313, and S314 in Fig. 3(b) are similar to step S301, loop L301, and steps S302, S303, and S304 in Fig. 3(a), and therefore their explanations are omitted. Also, steps S316 and S318 in Fig. 3(b) are similar to steps S306 and S308 in Fig. 3(a), and therefore their explanations are omitted.

[0119] In step S315 of FIG. 3(b), the filter unit 224 (FIG. 2(b)) and the cost calculation unit 225 (FIG. 2(b)) calculate the allocation costs of the track up to the previous time and the detected object at the current time.

[0120] In the first embodiment, the filter unit 408 in Fig. 4 uses the human body region mask 409 for the current time predicted at the previous time to filter the features. In this embodiment, the filter unit 224 filters the features using not only the human body region mask 409 but also a mask that focuses on important positions for identifying the tracking target. The latter mask is called an attention mask.

[0121] The human body region mask and the attention mask are both masks of the same size that take real values ​​between 0 and 1. When filtering, the filter unit 224 takes the product or average of each element of both masks and combines them into a mask of the same size as the original. The filter unit 224 performs the same calculation as in the first embodiment (see FIG. 6(a)) on the combined mask to filter the pre-filter feature (p-feat) 605. The filter unit 224 can choose whether to use the product or average of each element as appropriate, and the effect will not change substantially depending on the selection.

[0122] In step S317 of FIG. 3(b), the mask prediction unit 227 (FIG. 2(b)) predicts a human body region mask and an attention mask. The RNN used by the mask prediction unit 227 is the RNN 1101 of FIG. 11. This RNN 1101 has a configuration that outputs both a human body region mask 1103 and an attention mask 1104. More specifically, the Conv of the RNN 601 of FIG. 6(b) is a convolutional layer with two-channel output. The input to the RNN 1101 is the intermediate feature (m-feat) 1102, similar to that of the first embodiment.

[0123] In step S319 of FIG. 3(b), the training data creation unit 229 (FIG. 2(b)) creates data for online training of an attention mask. An attention mask is a mask that focuses on important locations for distinguishing people appearing in the same scene. In online tracking, feature quantities at multiple times for a track with a certain ID are all feature quantities for the same ID, while feature quantities for another track occurring simultaneously are feature quantities for a different ID. The training data creation unit 229 associates the time and track ID handled in step S318 with the feature quantities and stores them as training data for online training. In step S319, the training data creation unit 229 associates the time and track ID with intermediate features (m-feat) and pre-filter features (p-feat) and stores them in the intermediate feature data storage unit 233 (FIG. 2(b)).

[0124] At branch B311 in FIG. 3(b), the tracking device 220 determines whether to perform attention mask learning. This determination is made based on the amount of data in the intermediate feature data storage unit 233 (FIG. 2(b)). Here, when about 10 samples of data are stored in the intermediate feature data storage unit 233, the tracking device 220 determines to perform learning (i.e., branches to Yes), and the process proceeds to step S320.

[0125] 3(b), the learning unit 230 (FIG. 2(b)) executes learning of the attention mask. After that, the tracking device 220 repeats the above-mentioned loop process of t=1 to T.

[0126] FIG. 8(b) is a diagram illustrating an example of the functional configuration of the learning unit 230. The learning unit 230 includes a learning data acquisition unit 811, a mask prediction unit 812, a loss calculation unit 813, a parameter update unit 814, an intermediate feature data storage unit 816, and a dictionary storage unit 817. The learning data acquisition unit 811 corresponds to the learning data acquisition unit 711 in FIG. 7(b). The mask prediction unit 812 corresponds to the mask prediction unit 712 in FIG. 7(b). The loss calculation unit 813 corresponds to the loss calculation unit 713 in FIG. 7(b). The parameter update unit 814 corresponds to the parameter update unit 714 in FIG. 7(b). The intermediate feature data storage unit 816 corresponds to the intermediate feature data storage unit 716 in FIG. 7(b). The dictionary storage unit 817 corresponds to the dictionary storage unit 717 in FIG. 7(b).

[0127] Details of the process of step S320 are the same as the process of the flowchart in Fig. 9(d). That is, the process of step S320 is the same as the process of step S903 in Fig. 9(a). The process of learning unit 230 will be described with reference to Fig. 9(d).

[0128] Step S931 and loop L931 in FIG. 9(d) are the same as the processes described in the first embodiment, and therefore detailed description thereof will be omitted.

[0129] In step S932 of FIG. 9(d), the training data acquisition unit 811 (FIG. 8(b)) acquires training data from the intermediate feature data storage unit 816 (FIG. 8(b)). The training data acquisition unit 811 acquires intermediate features (m-feat) 1102 and pre-filter features (p-feat) 1105 as input information to the RNN 1101. The input to the RNN 1101 is data for multiple time periods that are consecutive in time. Here, the pre-filter features (p-feat) 1105 are data one time period ahead of the intermediate features (m-feat) 1102. In other words, if the intermediate features (m-feat) 1102 are data for five time periods, t=1, 2, 3, 4, and 5, the pre-filter features (p-feat) 1105 are data for five time periods one time period in the future, t=2, 3, 4, 5, and 6. The learning data acquisition unit 811 creates this time series data for a plurality of tracks and supplies it to the next step S933.

[0130] In step S933 of FIG. 9(d), the mask prediction unit 812 (FIG. 8(b)) predicts an attention mask 1104. The mask prediction unit 812 has the network of FIG. 11. Here, the intermediate feature (m-feat) 1102 is an intermediate feature input to the RNN 1101. The RNN 1101 is an RNN that predicts the attention mask 1104. The attention mask 1104 is a predicted attention mask. The human body region mask 1103 is a predicted human body region mask for the next time point, and is the same as the human body region mask (mask) 603 of FIG. 6(b). That is, the convolution layer of the RNN 1101 (Conv of the RNN 601 of FIG. 6(b)) has two channels of output, the first channel being the human body region mask 1103 described in the first embodiment, and the second channel being the attention mask 1104.

[0131] In step S933, the mask prediction unit 812 further extracts a filter feature (f-feat) 1108 to learn an attention mask for distinguishing between people. To extract the filter feature (f-feat) 1108, the mask prediction unit 812 inputs a pre-filter feature (p-feat) 1105 at the next time point for the intermediate feature (m-feat) 1102 to a filter (Filter) 1106. The filter 1106 inputs a human body region mask 1103, an attention mask 1104, and the pre-filter feature (p-feat) 1105. The filter 1106 and FC 1107 perform the same calculation as in FIG. 6(a) of the first embodiment and output the filter feature (f-feat) 1108.

[0132] Through the above calculations, time series data of the filter feature (f-feat) 1108 one time period in the future is obtained for the time series data of the input intermediate feature (m-feat) 1102. The mask prediction unit 812 executes this inference process for multiple tracks.

[0133] In step S934 of FIG. 9(d), the loss calculation unit 813 (FIG. 8(b)) calculates the loss.

[0134] 12 is a diagram showing an example of a table 1201 indicating the relationship between time t of the filter feature (f-feat) 1108 output in the previous step S933 and multiple tracks Track 1 to Track 4. In table 1201, the columns indicate time t, and the rows indicate tracks Track 1 to Track 4. In table 1201, area 1205 indicates matrix elements with no data, and areas other than area 1205 indicate matrix elements with data.

[0135] The triplet loss is used for the loss to learn to distinguish between people. The triplet loss is designed to minimize the distance between samples in the same category and maximize the distance between samples in different categories. If the distance between samples in the same category is dp and the distance between samples in different categories is dn, the loss is expressed by the following equation. Here, α is a hyperparameter that is set separately as a margin. Loss=max(dp-dn+α,0)

[0136] In step S934, the loss calculation unit 813 calculates the loss for tracks Track1 to Track4 and times t=1 to 10 in order. For example, the loss calculation unit 813 selects track Track3 and sample 1202 at time t=4 in FIG. 12. In this case, the loss calculation unit 813 samples the distance dp from sample 1203 of track Track3 at time t=3, and samples the distance dn from sample 1204 of another track at time t=4. The loss calculation unit 813 performs this for each time and each track, and the sum of the respective losses is the loss of the sample sequence for one mini-batch.

[0137] In step S935 of FIG. 9(d), the parameter update unit 814 (FIG. 8(b)) updates the parameters of the RNN 1101. The parameter update unit 814 does not need to learn parameters for all layers of the RNN 1101, but learns only the parameters corresponding to the output of the second channel of the Conv of the RNN 601 in FIG. 6(b). In this case, the parameter update unit 814 applies BPTT to the loss calculated in the previous step S934, calculates the amount of parameter update, and updates the parameters. As a result, the RNN 1101 is trained to output an attention mask 1104 in which important parts for distinguishing between people have values ​​close to 1. The parameter update unit 814 saves the trained parameters of the RNN 1101 in the dictionary storage unit 817 (FIG. 8(b)). The trained parameters of the RNN 1101 are rewritten in step S935 and used in step S317 (FIG. 3(b)) of the subsequent runtime processing.

[0138] The learning unit 230 repeats the above loop processing for n = 1 to N. The above is the details of the processing in step S320 (FIG. 3(b)) of learning the attention mask 1104. The above is also the details of the processing at runtime of the tracking device 220 in FIG. 3(b).

[0139] As described above, according to this embodiment, the tracking device 220 uses the learning unit 230 to learn the attention mask 1104 online for distinguishing between people from data acquired in real time in actual operational scenes. The tracking device 220 filters the features using the human body region mask 1103 and the attention mask 1104. This allows the tracking device 220 to extract important information for distinguishing between people even when there is a large overlap between people, thereby reducing ID transfer during tracking.

[0140] The tracking device 220 is an example of a recognition device. The mask prediction unit 227 predicts an object region mask 1103 and an attention mask 1104 of an area important for identifying the object. The filter unit 224 outputs, for each detected object, filter features 1108 obtained by filtering pre-filter features 1105 based on the object region mask 1103 and attention mask 1104 at the current time for each detected object predicted at the previous time. The pre-filter features 1105 are extracted by the feature extraction unit 223. Specifically, the filter unit 224 outputs the filter features 1108 based on a mask obtained by integrating the object region mask 1103 and the attention mask 1104. The learning unit 230 learns parameters for the mask prediction unit 227 to predict the attention mask 1104 based on the intermediate features 405 extracted by the feature extraction unit 223.

[0141] As described above, according to this embodiment, the recognition device can use the attention mask 1104 to extract important information for distinguishing between multiple objects, even when there is a large degree of overlap between the multiple objects, thereby reducing ID transfer.

[0142] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0143] It should be noted that the above-described embodiments merely illustrate specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0144] 200 Tracking device, 201 Image data acquisition unit, 202 Detection unit, 203 Feature extraction unit, 204 Filter unit, 205 Cost calculation unit, 206 ID assignment unit, 207 Mask prediction unit, 208 Track update unit, 209 Dictionary storage unit, 210 Internal storage unit

Claims

1. A detection means for detecting a plurality of objects in image data and outputting a plurality of detected objects based on the detected plurality of objects; a feature extraction means for extracting feature amounts from each of the plurality of detected objects output by the detection means; a filter means for filtering the feature amount based on a first mask at the current time for each detected object predicted at a previous time, and outputting a filter feature for each detected object; a mask prediction means for predicting a first mask at a next time for each of the detected objects; A recognition device comprising:

2. 2. The recognition device according to claim 1, further comprising an assigning unit that assigns an identifier to each of the plurality of detected objects based on the filter feature of each detected object.

3. A group of detected objects assigned the same identifier at each time is a track. The method further comprises a cost calculation means for calculating a cost for each track and each detected object based on a similarity between a filter feature at a previous time of the plurality of tracks and a filter feature at a current time of each detected object, 3. The recognition device according to claim 2, wherein the assigning means assigns an identifier to each of the plurality of detected objects based on the cost.

4. 4. The recognition device according to claim 3, further comprising an updating means for updating track information by storing, for each track, the filter feature at the current time output by the filtering means, the first mask at the next time predicted by the mask prediction means, and the identifier assigned by the assigning means in association with each other.

5. The recognition device according to any one of claims 1 to 4, characterized in that the mask prediction means predicts a first mask at a next time for each detected object based on the feature extracted by the feature extraction means and a state of the mask prediction means.

6. the feature extraction means extracts a first feature amount and a second feature amount from each of the plurality of detected objects; the mask prediction means predicts a first mask at a next time for each of the detected objects based on the first feature extracted by the feature extraction means and a state of the mask prediction means; 6. The recognition device according to claim 5, wherein the filter means outputs, for each detected object, a filter feature which is a feature obtained by filtering the second feature extracted by the feature extraction means based on a first mask at the current time for each detected object.

7. 7. The recognition device according to claim 5, wherein the state of said mask prediction means is the state of a first neural network.

8. 8. The recognition device of claim 7, wherein the first neural network is a recurrent neural network (RNN).

9. 9. The recognition device according to claim 1, wherein the feature extraction means and the filter means use a second neural network.

10. 10. The recognition device of claim 9, wherein the second neural network is a convolutional neural network (CNN).

11. 11. The recognition device according to claim 1, wherein the detected object is a circumscribing rectangle of the target.

12. 12. The recognition device according to claim 11, wherein the first mask is a mask of an area in the circumscribing rectangle where the object exists.

13. the detection device further comprises an averaging means for outputting an average feature obtained by spatially averaging the feature amounts extracted by the feature extraction means when the degree of overlap of a plurality of objects in the detected objects is less than a threshold value, the filter means outputs the filter feature when a degree of overlap of a plurality of objects in the detected object is equal to or greater than a threshold; The cost calculation means If the degree of overlap of the plurality of objects in the detected objects is less than a threshold, calculate a cost for each track and each detected object based on the similarity between the average feature of the plurality of tracks at the previous time and the average feature of each detected object at the current time; The recognition device according to claim 3, characterized in that, when the degree of overlap of multiple objects in the detected objects is equal to or greater than a threshold, a cost for each track and detected object is calculated based on the similarity between the filter features of the multiple tracks at a previous time and the filter features of each detected object at a current time.

14. further comprising an update means for updating track information by storing, for each track, the filter feature at the current time output by the filter means, the first mask at the next time predicted by the mask prediction means, and the identifier assigned by the assignment means, in association with each other, when the degree of overlap of a plurality of objects in the detected objects is equal to or greater than a threshold value; 14. The recognition device according to claim 13, wherein, when the degree of overlap of multiple objects in the detected object is less than a threshold, the updating means acquires filter features, which are features obtained by filtering the features extracted by the feature extraction means, and updates track information by storing, for each track, the acquired filter feature at the current time, the average feature at the current time output by the averaging means, a first mask at the next time predicted by the mask prediction means, and an identifier assigned by the assignment means in association with each other.

15. the mask prediction means predicts the first mask and a second mask of an area important for identifying the object; 13. The recognition device according to claim 12, wherein the filter means outputs, for each detected object, a filter feature which is a feature obtained by filtering the feature extracted by the feature extraction means based on a first mask and a second mask at the current time for each detected object predicted at a previous time.

16. 16. The recognition device according to claim 15, wherein the filter means outputs the filter feature based on a mask obtained by combining the first mask and the second mask.

17. 17. The recognition device according to claim 15, further comprising a learning means for learning parameters for predicting the second mask by the mask prediction means based on the feature extracted by the feature extraction means.

18. A training device for training the recognition device according to claim 1, comprising: a loss calculation means for calculating a loss based on a first mask predicted by the mask prediction means and a correct value of the first mask; an updating means for updating parameters of the mask prediction means based on the loss calculated by the loss calculation means; and A learning device characterized in that temporally continuous feature quantities of the same track are input to the mask prediction means, and the first mask at the next time is predicted.

19. A detection step in which a detection unit detects a plurality of objects in image data and outputs a plurality of detected objects based on the detected plurality of objects; a feature extraction step in which a feature extraction unit extracts feature amounts from each of the plurality of detected objects output in the detection step; a filtering step in which the filter unit outputs, for each detected object, a filter feature which is a feature obtained by filtering the feature based on a first mask at the current time for each detected object predicted at the previous time; a mask prediction step in which a mask prediction unit predicts a first mask at a next time for each of the detected objects; A recognition method for a recognition device, comprising:

20. A method for training a learning device for training the recognition device according to claim 1, comprising: a mask prediction step in which the mask prediction means according to claim 1 predicts the first mask at a next time from time-sequential feature quantities of the same track; a loss calculation step in which a loss calculation unit calculates a loss based on a first mask predicted by the mask prediction step and a correct value of the first mask; an updating step in which a parameter updating unit updates parameters of the mask prediction means based on the loss calculated in the loss calculation step; A learning method for a learning device, comprising:

21. A program for causing a computer to function as each means of the recognition device according to any one of claims 1 to 17.

22. A program for causing a computer to function as each of the means of the learning device according to claim 18.

Images