Learning device, inference device, learning method, inference method, and program

Abstract

To provide a behavior recognition technique in which a processing cost is reduced with a simple architecture.SOLUTION: A learning device that performs learning for behavior recognition, includes: a convolutional neural network that inputs image frames constituting a video sequence, and outputs a motion feature map, a re-identified feature map, a size feature map, and a position feature map; a feature selection unit that inputs the motion feature map, the re-identified feature map, and the size feature map, and outputs a motion feature, a re-identified feature, and a size feature; a relation modeling unit that inputs the motion feature and the re-identified feature, and outputs a feature in which an interaction among the features is taken into consideration; a categorizing unit that outputs a group behavior categorization result and a motion categorization result on the basis of the feature output by the relation modeling unit; and a model parameter updating unit that updates a model parameter so that a difference between the output and correct data is minimized.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] This invention relates to a technology that detects and tracks objects captured in input video, identifies the actions taken by each tracked target, and, when multiple objects are captured in the video, identifies collective behavior, which is the behavior formed by a group of such objects. [Background technology]

[0002] An example of the technology for identifying group behavior as described above is shown in Figure 1. In the example shown in Figure 1, people captured in the input video are detected and tracked, and the actions (individual actions) taken by each tracked subject are identified, and when multiple objects are captured, group behavior formed by the group is identified.

[0003] Figure 1 shows an example in which the actions of the first and second tracking results are identified as "Moving" and the third action as "Waiting", and the action taken by the group formed by them is also identified as "Moving". In the following, the above problem will be referred to as "action recognition".

[0004] If the above behavior recognition is realized, it will be possible to automatically recognize the behavior of individuals captured on video. This can be applied, for example, to monitoring abnormal behavior from camera footage installed around the city. At the same time, it will be possible to automatically recognize not only individual behavior, but also the behavior of groups consisting of multiple subjects. This will enable the recognition of set plays consisting of cooperation between multiple players in sports, and the detection of abnormal behavior of groups captured on city cameras, expanding the scope of analytical applications of sports and surveillance footage.

[0005] From the above, it is clear that behavior recognition has extremely high industrial applicability. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.

Non-licensed Document 2

Non-licensed Document 4

Non-licensed Document 5

Non-licensed Document 6

Non-licensed Document 7

[0007] When the above-mentioned behavior recognition is performed using conventional technology, there are problems in that the overall architecture becomes complex and redundant, processing takes a long time, and behavior recognition performance is low.

[0008] The present invention has been made in consideration of the above-mentioned points, and has an object to provide a behavior recognition technology that reduces processing costs by using a simple architecture. [Means for solving the problem]

[0009] According to the disclosed technology, there is provided a learning device that performs learning for behavior recognition, comprising: a convolutional neural network that inputs each image frame constituting a video sequence and outputs a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit that receives the motion feature map, the re-identification feature map, and the size feature map, and outputs motion features, re-identification features, and size features; a relational modeling unit that receives the operation features and the re-identified features and outputs features in which interactions between the features are taken into consideration; a first classification unit that outputs a group behavior classification result based on the features output from the relational modeling unit; a second classification unit that outputs an action classification result based on the features output from the relational modeling unit; a model parameter update unit that updates model parameters of the convolutional neural network, the feature selection unit, the relational modeling unit, the first classification unit, and the second classification unit so that errors between the position feature map, the size feature, the re-identification feature, the group behavior classification result, and the action classification result and ground truth data are minimized; A learning device is provided, comprising: Effect of the Invention

[0010] According to the disclosed technology, it is possible to provide a behavior recognition technology that uses a simple architecture and reduces processing costs. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining behavior recognition. [Diagram 2] FIG. 1 is a diagram showing a method assumed from known technology and a technology according to the present invention. [Diagram 3] FIG. 1 illustrates a convolutional neural network. [Figure 4] FIG. 13 is a diagram showing learning data per video sequence. [Diagram 5] FIG. 1 illustrates a learning device according to a first embodiment. [Figure 6] FIG. 1 illustrates an inference device according to a first embodiment. [Figure 7] FIG. 11 is a diagram illustrating a learning device according to a second embodiment. [Figure 8] FIG. 11 is a diagram illustrating an inference device according to a second embodiment. [Figure 9] FIG. 11 is a diagram illustrating a learning device according to a third embodiment. [Figure 10] FIG. 13 is a diagram illustrating an inference device according to a third embodiment. [Figure 11] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. Below, first, the problem will be described in more detail, and then the learning device and inference device according to the present embodiment will be described.

[0013] (About the assignment) The behavior recognition described in the Background Art section is composed of multiple subtasks. Specifically, it is necessary to detect the target object in each frame of the video, match (track) the detection results between frames, identify the motion of each detection or tracking result, and identify the collective behavior of the detection / tracking results as a whole.

[0014] An example of a processing configuration for performing behavior recognition using publicly known technology is shown on the left side of Figure 2. Note that the configuration shown on the left side of Figure 2 itself is not publicly known. As shown on the left side of Figure 2, in order to perform behavior recognition using publicly known technology, results output by multiple methods for independently solving the above subtasks are combined. A specific example is as follows.

[0015] First, a target object is detected from each video frame using the method disclosed in Non-Patent Document 1. Next, a tracking result is output from the obtained detection result using the method disclosed in Non-Patent Document 2. In parallel, the motion of each detection result and the behavior of the group are identified using the method disclosed in Non-Patent Document 3. Finally, the motion identification results of each detection result are matched with the tracking result, and the motion of each tracking result is output.

[0016] The above method has three major problems. The first problem is that the overall architecture is complex and the overall computational cost is high. It is clear that the overall architecture is excessively redundant, considering that the techniques disclosed in Non-Patent Documents 1-3, which are known techniques for solving each of the subtasks listed above, contain a common convolutional neural network (CNN) structure.

[0017] The second challenge is that although the above-mentioned subtasks are interrelated, when action recognition is performed by simply combining independent methods, it is not possible to explicitly consider the interactions between tasks. For example, in tracking and action recognition, it is highly likely that an object (i.e., the tracking result) will continue to perform the same action over a short time interval, and conversely, the identity of the target object is highly likely to be useful information when determining the action. However, when simply combining independent methods, it is not possible to consider these interactions, and as a result, it is not possible to improve the performance of the overall action recognition.

[0018] As a third issue, since the subtasks are interrelated, the degradation of the performance of one subtask strongly affects the performance of the others. The most prominent example is the impact of the object detection subtask on the others. Object detection often fails when the entire image of the target is hidden, i.e., when occlusion occurs. However, in known techniques of subtasks that use the results of object detection as input, the model is trained without considering the possibility of these detection failures, so that when an incomplete detection result is input, it may be significantly affected.

[0019] In summary, methods that combine known technologies as subtasks have the problem that the overall architecture is complex and redundant, taking a long time to process, and the performance of behavior recognition is low because the interactions between the subtasks are not taken into account.

[0020] (Outline of the embodiment) The technology for solving the above problems will be described below. First, five features of the technology will be described.

[0021] <First feature> The first feature is the use of a convolutional neural network (CNN) that outputs multiple feature representations related to subtasks constituting action recognition in one shot from frames constituting an input video. Specifically, as described later with reference to Fig. 3, this CNN is characterized by being composed of a backbone 1 that extracts a feature map from an input frame, a position feature branch 2 that outputs a position feature map indicating the point positions of a target object from the feature map, a size feature branch 3 that outputs a size feature map indicating the size of the target object at each position, a re-identification branch 4 that outputs a re-identification feature map for re-identifying the same object between different frames from the feature map, and a motion feature branch 5 that outputs a motion feature map for recognizing individual motions and group actions.

[0022] By sharing the configuration of the feature extractors required for object detection, re-identification-based tracking, and motion / action classification, the architecture can be simplified and processing costs can be reduced, as shown on the right side of Figure 2.

[0023] <Second feature> The second feature is the relational modeling unit 14, which will be described later with reference to Fig. 5 etc. In the process by the relational modeling unit 14, the motion features and re-identified features extracted from the input video sequence are input, and the motion features are transformed into ones that take into account the interactions between the features while using the re-identified features as auxiliary information. This makes it possible to take into account information about the identity of the target object when determining the motion, as mentioned in the second problem, and as a result, it is possible to improve the performance of motion classification and group behavior classification.

[0024] The process in the relational modeling unit 14 can also be applied to converting the re-identified features into motion features as auxiliary information (described later in Figs. 7 and 8). Furthermore, the conversion by the process in the relational modeling unit 14 can also be applied simultaneously to motion features and re-identified features (described later in Figs. 9 and 10).

[0025] This makes it possible to utilize information about the consistency of behavior over short time intervals, as mentioned in the second challenge, in transforming re-identification features, thereby improving tracking performance and ultimately the overall performance of action recognition.

[0026] <Third feature> The third feature is the processing of the feature selection unit 12, which will be described later with reference to Fig. 5 etc. When learning the parts of the entire model, particularly those related to tracking, action classification, and group behavior classification based on re-identification, the feature selection unit 12 selects a portion of all correct answer target objects based on the degree of occlusion of each object, and simulates an incomplete object detection result.

[0027] By training a model using this method, it becomes possible to robustly identify individual actions and population behaviors, even if the object detection results fail to detect some objects.

[0028] <Fourth feature> The fourth feature is that, in the process of the model parameter update unit 17 described later with reference to Fig. 5 etc., all parameters of the model in this embodiment determined by learning are updated so as to minimize an error function consisting of an error function related to the position feature map, an error function related to the size feature, an error function related to the re-identification feature, an error function related to the action classification result, and an error function related to the group behavior classification result. By using this in combination with the first, second, and third features, it becomes possible to learn the model while taking into account the relationship between the subtasks, and as a result, it is possible to improve the performance of behavior recognition.

[0029] <Fifth feature> The fifth feature is the motion classifier 16, which will be described later with reference to Fig. 6 etc. In the processing of the motion classifier 16, the motion of each detection result is classified in consideration of the consistency of the individual captured by the detection result, thereby making it possible to improve the performance of motion classification.

[0030] <Effects of the embodiment> The technology according to the embodiment having the above five features allows for accurate behavior recognition at low processing cost. Note that it is not essential to use all of the five features to obtain such an effect. Such an effect can be obtained even with only some of the five features.

[0031] A more specific example of the device configuration and its operation will be described below with reference to Examples 1 to 3. It is assumed that the functions in each of the examples described below are implemented using a neural network model. However, the use of a neural network is just one example, and a machine learning method other than a neural network may be used. Furthermore, a neural network may be mixed with a method other than a neural network.

[0032] Example 1 First, a description will be given of the first embodiment. Fig. 5 shows the configuration of a learning device 100 according to the first embodiment, and Fig. 6 shows the configuration of an inference device 200 according to the first embodiment.

[0033] Learning device 100 learns a model given learning data. Inference device 200 performs inference on input video data using the model obtained by learning device 100, i.e., detects target objects appearing in each frame constituting the video data, tracks the detected objects, identifies the motion of each tracking result, and identifies the behavior of a group of tracking results.

[0034] Note that by adding a peak detection unit 18, a post-detection processing unit 19, and a tracking unit 20 to learning device 100, learning device 100 may be used as inference device 200. Furthermore, by adding a model parameter update unit 17 to inference device 200, learning and inference may be performed by inference device 200 alone.

[0035] Note that a model is a set of all parameters other than those manually set when performing learning and inference.

[0036] <About the learning data> An example of the training data used in training is shown in Fig. 4. Fig. 4 shows the training data for one video sequence. As shown in Fig. 4, a video sequence and its corresponding correct answer data are taken as unit elements. The number of unit elements included in the training data may be any number equal to or greater than 1. A video sequence is T image frames arranged in chronological order. T is arbitrary and may be different for each sequence.

[0037] The correct answer data corresponding to one sequence consists of a detection correct label, a tracking correct label, an action correct label, and a collective behavior correct label, as shown in Figure 4. The detection correct label is a label related to the position of the target object captured in each frame of the video sequence, and each can be defined, for example, as a rectangle that completely surrounds the target. The tracking correct label is an ID assigned to each detection correct label, and detection correct labels that capture the same individual are assigned the same and unique ID. The action correct label is the label of the action assigned to each tracking ID.

[0038] In the example of Fig. 4, the action label "Moving" is assigned to id1 and id2, and the action label "Waiting" is assigned to id3. Note that action labels may be assigned to each tracked object as shown in Fig. 4, or may be assigned to each correct detected label. Finally, a collective action label is assigned to each video sequence, and in the example of Fig. 4, the label "Moving" is assigned.

[0039] <Learning device configuration and processing flow> As shown in FIG. 5, the learning device 100 includes a convolutional neural network 11, a feature selection unit 12, a pooling unit 13, a relational modeling unit 14, a classification unit 15, a classification unit 16, and a model parameter update unit 17. There is also a database 30 that stores learning data. Note that the configuration shown in FIG. 5 is an example. A certain functional unit may include another functional unit. For example, the classification unit 15 may include the pooling unit 13. The details of each unit will be described later. The flow of processing will be described below with reference to FIG. 5.

[0040] First, each image frame constituting a video sequence of the training data is input to the convolutional neural network 11, which outputs a position feature map, a size feature map, a re-identification feature map, and a motion feature map.

[0041] For the size feature maps, re-identification feature maps, and motion feature maps corresponding to all image frames constituting the video sequence, only features at positions that correspond to correct position data generated from the learning data and are determined to be unaffected by occlusion are selected by the feature selection unit 12, and the size features, re-identification features, and motion features are output.

[0042] The motion features are input to the relational modeling unit 14, where feature conversion is performed taking into account the relationships and interactions between objects. Here, the re-identified features are used as auxiliary information for the conversion of the motion features in the relational modeling unit 14. The obtained motion features are input to the pooling unit 13, where pooling is performed to output group behavior features.

[0043] The motion features and group behavior features are input to the classification units 16 and 15, respectively, which output motion classification results and group behavior classification results. That is, the motion of the target object corresponding to each correct position selected by the feature selection unit 12 and the group behavior corresponding to the video sequence are classified into one of the predetermined motion categories and group behavior categories.

[0044] The position feature map, size features, re-identification features, motion classification results, and group behavior classification results output by the processing up to this point are input to the model parameter update unit 17 together with the ground truth data (e.g., Figure 4), and the model parameters are updated so that the error between the current model output and the ground truth is minimized.

[0045] <Configuration of the inference device 200 and processing flow> FIG. 6 shows the configuration of the inference device 200 in the first embodiment. As shown in FIG. 6, the inference device 200 in the first embodiment includes a convolutional neural network 11, a feature selection unit 12, a pooling unit 13, a relational modeling unit 14, a collective behavior classification unit 15, an action classification unit 16, a peak detection unit 18, a post-detection processing unit 19, and a tracking unit 20. Note that the configuration shown in FIG. 6 is an example. A certain functional unit may include another functional unit. For example, the collective classification unit 15 may include the pooling unit 13. The details of each unit will be described later. The flow of processing will be described below with reference to FIG. 6.

[0046] Each image frame constituting the input video sequence to be inferred is input to the convolutional neural network 11, which outputs a position feature map, a size feature map, a re-identification feature map, and a motion feature map. In the inference process, unlike the learning process, the point positions of the target object are detected as peak positions of the position feature map by processing by the peak detection unit 18. These point positions, the size feature map, the re-identification feature map, and the motion feature map are input to the feature selection unit 12, and a feature set corresponding to each point position for all image frames is output as the size feature, the re-identification feature, and the motion feature.

[0047] The point position data and size features are output as detection results after processing in post-detection processing unit 19. Tracking process is performed by tracking unit 20 using the re-identified features and detection results as inputs, and the tracking results are output. As in the learning process, the movement features are converted in relational modeling unit 14 using the re-identified features as auxiliary information. The movement features output from relational modeling unit 14 are input to pooling unit 13, and group behavior features are output from pooling unit 13.

[0048] The motion features and tracking results are input to the motion classification unit 16, which outputs the motion labels of each tracking result so that the motion labels in each tracking result are consistent. The group behavior features are input to the group behavior classification unit 15, which outputs the group behavior classification results.

[0049] <Details of each part of the learning device 100> Each unit of the learning device 100 shown in FIG. 5 will be described in detail below.

[0050] <Convolutional Neural Network 11 of Learning Device 100> An example of the configuration of the convolutional neural network (CNN) 11 is shown in Fig. 3. The convolutional neural network receives a video sequence as input, and outputs a position feature map, a size feature map, a re-identification feature map, and a motion feature map corresponding to each image frame that constitutes the video sequence.

[0051] The position feature map is a feature map that gives a high score at positions in the input image frame where a target object is present, the size feature map is a feature map that outputs the size of the object captured at each position in the input image frame, the re-identification feature map is a feature map that outputs features for matching objects captured at each position in the input image frame between different frames, and the motion feature map is a feature map that outputs features for identifying the motion and collective behavior of objects captured at each position in the input image frame.

[0052] Such a CNN can be realized by connecting convolutional processing layers in parallel as branches that take the backbone output of a known encoder-decoder CNN as input and output each of the above feature maps. Any technology can be used for the encoder-decoder CNN, for example, the technology described in Non-Patent Documents 4 and 5. The method of defining the convolutional processing layer is also arbitrary. An output feature map of the desired channel size can be obtained by applying a nonlinear processing layer, such as ReLU, after a convolutional processing layer with a 3×3 filter size, and then connecting a convolutional processing layer with a 1×1 filter size.

[0053] In the following example, an image frame of size H×W×3 in a sequence is input, and a position feature map of H´×W´×1, a size feature map of H´×W´×2, and a feature map of H´×W´×d reid Re-identification feature map, H´×W´×d act Assume that the motion feature map of d reid and d act are optional parameters, for example both can be set to 128.

[0054] <Feature selection unit 12 of learning device 100> The feature selection unit 12 inputs the size feature map, re-identification feature map, and motion feature map from the outputs of the convolutional neural network 11, extracts features corresponding to the object point positions calculated from the ground truth data, and outputs the size features, re-identification features, and motion features.

[0055] The method of calculating the object point position calculated from the ground truth data is arbitrary. If the object position is defined as a rectangle, the center coordinates of the rectangle are calculated, and the object point position is calculated according to the scale ratio between the input image frame size and the output feature map size.

[0056] In addition, in the learning process, the feature selection unit 12 may select only object points that are not affected by occlusion from among object points calculated from the ground-truth data. Any method may be used to select those that are not affected by occlusion, and the following method may be used, for example. First, the overlapping degree of the ground-truth object positions in each image frame is calculated in a brute-force manner.

[0057] Next, the correct objects are arranged in order of how close they are to the camera. Finally, only if the overlapping degree with the objects closer to the camera is below a certain threshold, the object is determined to be a target object for feature selection.

[0058] Here, the overlap is calculated using Intersection-over-Union (IoU), and the coordinate position of the bottom of the rectangle is used as the criterion for sorting the objects in order of the foreground. The threshold value may be a constant that is manually determined in advance, or may be set randomly for each trial.

[0059] Now, let the total number of position data selected from all image frames in the video sequence be N seq Then, from all size feature maps, N seq × 2 size features are extracted, and N seq ×d reid Re-identification features are extracted, and N seq ×d act The motion features are extracted.

[0060] <Relationship modeling unit 14 of learning device 100> The relational modeling unit 14 receives the motion features and re-identified features output from the feature selection unit 12, and outputs motion features that have been modified in consideration of the relevance between the features.

[0061] Now, let the total number of features in an input sequence be N seq The process in the relational modeling unit 14 is defined, for example, as follows.

[0062] Let the set of motion features be X act ∈R N_seq×d_act , the re-identified feature set is X reid ∈R N_seq×d_reid The set of action features output by the relational modeling unit 14 is expressed as ^X tgt ∈R N_seq×d_act Let's say ^X tgt is obtained through the process defined by the following equations 1-4.

[0063]

number

[0064]

number

[0065]

number

[0066]

number

[0067] <Pooling unit 13 of learning device 100> The pooling unit 13 receives the action features output by the relational modeling unit 14 and outputs group behavior features for identifying group behaviors occurring in a sequence. seq ×d act The motion features of 1×d are pooled. act The collective behavior characteristics are extracted.

[0068] Any method can be used for the pooling process, for example, maximum pooling or average value pooling.

[0069] <Classification Unit 15 and Classification Unit 16 of Learning Device 100> The classification unit 16 receives the motion features output by the relational modeling unit 14 and classifies the target motion into one of the predetermined motion categories. The classification unit 15 receives the group behavior features output by the pooling unit 13 and classifies the group behavior into one of the predetermined group behavior categories.

[0070] Any method can be used for the classification process. Let the total number of action categories be N action Then, N seq ×d act The motion features defined by the matrix act ×d action Apply the transformation matrix from the right. In this case, each element can be interpreted as performing an operation corresponding to the index that has the maximum value in each row of the output matrix.

[0071] <Model parameter update unit 17 of learning device 100> The model parameter update unit 17 compares the position feature map output by the convolutional neural network 11, the size feature and re-identification feature output by the feature selection unit 12, and the action classification results and group behavior classification results output by the classification units 16 and 15 with the correct answer data, and updates the parameters of a part of the model or the entire model so that the total error is minimized. In the following, the error function related to the position feature map is defined as L hm , the error function related to the size feature is L size , the error function for the re-identified features is L reid , the error function related to the motion classification result is L action , the error function related to the group behavior classification result is L activity Let us assume that.

[0072] A known method can be used to calculate each of the error functions. For example, the error function L hm The focal loss and the error function L related to the size feature are disclosed in Non-Patent Document 8. size The L1 loss and the error function L related to the re-identification feature are disclosed in Non-Patent Document 8. reid The triplet loss, the error function L action related to the action classification result, and the error function L action related to the group behavior classification result are disclosed in Non-Patent Document 9. activity We can use cross entropy loss for this.

[0073] The overall error function is L hm , L size , L reid , L action , L activity Here, the weights corresponding to each term can be determined manually, or they can be optimized during the learning process as learning parameters. When optimizing during the learning process, the objective function is expressed as the following Equation 5. w hm , w size , w reid , w action , w activity are parameters that are optimized during training.

[0074]

number

[0075] <Details of each part of the inference device 200> Each unit of inference device 200 shown in FIG. 6 will now be described in detail.

[0076] Convolutional neural network 11, feature selection unit 12, relational modeling unit 14, and pooling unit 13 are the same in inference device 200 and learning device 100. Furthermore, collective behavior classification unit 15 in inference device 200 is the same as classification unit 15 in learning device 100. Below, functional units that are not present in learning device 100 or functional units that are different from those in learning device 100 will be described.

[0077] <Peak detection unit 18 of inference device 200> The peak detection unit 18 outputs the point position of the target object from the position feature map corresponding to each image frame among the outputs of the convolutional neural network 11. The point position can be output as a position in the position feature map where a value equal to or greater than a preset threshold is output. Considering that outputs of adjacent positions in the position feature map are likely to capture the same object, NMS (Non-Maximum Suppression) processing or the like may be performed in advance to suppress redundant outputs.

[0078] <Post-detection processing unit 19 of the inference device 200> The post-detection processing unit 19 outputs a target object detection result for each image frame from the point position data output by the peak detection unit 18 and the size feature output by the feature selection unit 12. If a certain point position output by the peak detection unit 19 is (x, y) and the size feature corresponding to that point is (w, h), the object detection result, i.e., a rectangle (x1, y1, x2, y2) is calculated as (xw / 2, yh / 2, x+w / 2, y+h / 2).

[0079] <Tracking section 20 of inference device 200> The tracking unit 20 receives as input the re-identification features output by the feature selection unit 12 and the detection results output by the post-detection processing unit 19, matches the detection results capturing the same individual across different image frames, and outputs the results as tracking results.

[0080] A known method can be used for the tracking process, for example, the method disclosed in Non-Patent Document 8 can be used.

[0081] <Action Classification Unit 16 of Inference Device 200> The action classification unit 16 classifies the target action into one of the predetermined action categories using the action features output by the relational modeling unit 14 and the tracking result output by the tracking unit 20 as input. Any method can be used for action classification, and for example, the same method as that described for the classification units 15 and 16 of the learning device 100 can be used.

[0082] Alternatively, if it is guaranteed that the same action is taken for each tracking result, the consistency of the action within each tracking result may be taken into consideration. A method for guaranteeing the consistency of the action can be realized, for example, by outputting an action label for each detection result constituting one tracking result in a manner similar to that described for the classification units 15 and 16 of the learning device 100, taking a majority vote within the tracking result, and replacing all detection action labels within the tracking result with the action label that appears most frequently.

[0083] Below, embodiments 2 and 3 will be described. However, embodiments 2 and 3 are based on embodiment 1, and the following mainly describes the parts that are different from embodiment 1.

[0084] Example 2 A learning device 100 in the second embodiment is shown in FIG. 7, and an inference device 200 in the second embodiment is shown in FIG.

[0085] In the second embodiment, the re-identified features output by the feature selection unit 12 are transformed in the relational modeling unit 14 while taking into account the relevance between features, with the motion features as auxiliary information. This is different from the first embodiment. The processing of the re-identified features by the relational modeling unit 14 using the motion features as auxiliary information is performed by the motion feature X act Re-identify feature X reid It can be defined as a reversal of roles between

[0086] Example 3 A learning device 100 in the third embodiment is shown in FIG. 9, and an inference device 200 in the third embodiment is shown in FIG.

[0087] In the third embodiment, the motion features and the re-identified features output by the feature selection unit 12 are transformed in the relational modeling units 14-1 and 14-2 in consideration of the relationship between the features, while the other is used as auxiliary information. This is different from the first and second embodiments. The processes of the relational modeling units 14-1 and 14-2 can be performed by the same methods as those shown in the first and second embodiments.

[0088] (Hardware configuration example) In this embodiment, the learning device 100 and the inference device 200 (collectively referred to as devices) can each be realized by, for example, causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0089] That is, the device can be realized by executing a program corresponding to the processing performed by the device, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0090] Fig. 11 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 11 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus BS.

[0091] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0092] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0093] (Summary of the embodiment) This specification discloses at least the following learning device, inference device, learning method, inference method, and program. (Section 1) A learning device that performs learning for behavior recognition, a convolutional neural network that inputs each image frame constituting a video sequence and outputs a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit that receives the motion feature map, the re-identification feature map, and the size feature map, and outputs motion features, re-identification features, and size features; a relational modeling unit that receives the operation features and the re-identified features and outputs features in which interactions between the features are taken into consideration; a first classification unit that outputs a group behavior classification result based on the features output from the relational modeling unit; a second classification unit that outputs an action classification result based on the features output from the relational modeling unit; a model parameter update unit that updates model parameters of the convolutional neural network, the feature selection unit, the relational modeling unit, the first classification unit, and the second classification unit so that errors between the position feature map, the size feature, the re-identification feature, the group behavior classification result, and the action classification result and ground truth data are minimized; A learning device comprising: (Section 2) The relational modeling unit transforms the behavioral features by using the re-identified features as auxiliary information, or transforms the re-identified features by using the behavioral features as auxiliary information. 2. A learning device as described in claim 1. (Section 3) The relational modeling unit includes a first relational modeling unit and a second relational modeling unit, the first relational modeling unit converts the behavioral features by using the re-identified features as auxiliary information, and the second relational modeling unit converts the re-identified features by using the behavioral features as auxiliary information. 2. A learning device as described in claim 1. (Section 4) An inference device that performs inference for behavior recognition, a convolutional neural network that inputs each image frame constituting a video sequence and outputs a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit that receives the point position data obtained from the position feature map, the motion feature map, the re-identification feature map, and the size feature map, and outputs motion features, re-identification features, and size features; a tracking unit which receives the point position data, the detection result obtained from the size feature, and the re-identification feature, and outputs a tracking result; a relational modeling unit that receives the operation features and the re-identified features and outputs features in which interactions between the features are taken into consideration; a group behavior classification unit that outputs a group behavior classification result based on the features output from the relational modeling unit; an action classification unit that outputs an action classification result based on the features output from the relational modeling unit and the tracking result; An inference device comprising: (Section 5) The relational modeling unit transforms the behavioral features by using the re-identified features as auxiliary information, or transforms the re-identified features by using the behavioral features as auxiliary information. 5. The inference apparatus of claim 4. (Section 6) The relational modeling unit includes a first relational modeling unit and a second relational modeling unit, the first relational modeling unit converts the behavioral features by using the re-identified features as auxiliary information, and the second relational modeling unit converts the re-identified features by using the behavioral features as auxiliary information. 5. The inference apparatus of claim 4. (Section 7) A learning method executed by a learning device that performs learning for behavior recognition, comprising: a convolutional neural network inputting each image frame of a video sequence and outputting a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit inputting the motion feature map, the re-identification feature map, and the size feature map, and outputting motion features, re-identification features, and size features; a step of a relational modeling unit receiving the operation features and the re-identified features and outputting features in which interactions between features are taken into consideration; a step of a first classification unit outputting a group behavior classification result based on the features output from the relational modeling unit; A second classification unit outputs an action classification result based on the features output from the relational modeling unit; updating model parameters of the convolutional neural network, the feature selection unit, the relational modeling unit, the first classifier, and the second classifier so that errors between the position feature map, the size feature, the re-identification feature, the group behavior classification result, and the action classification result and ground truth data are minimized; A learning method that provides: (Section 8) An inference method executed by an inference device that performs inference for activity recognition, comprising: a convolutional neural network inputting each image frame of a video sequence and outputting a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit inputting the point position data obtained from the position feature map, the motion feature map, the re-identification feature map, and the size feature map, and outputting a motion feature, a re-identification feature, and a size feature; A tracking unit receives the point position data, the detection result obtained from the size feature, and the re-identification feature, and outputs a tracking result; a step of a relational modeling unit receiving the operation features and the re-identified features and outputting features in which interactions between features are taken into consideration; a group behavior classification unit outputting a group behavior classification result based on the features output from the relational modeling unit; an action classification unit outputting an action classification result based on the features output from the relational modeling unit and the tracking result; An inference method comprising: (Section 9) A program for causing a computer to function as the learning device according to any one of claims 1 to 3. (Section 10) A program for causing a computer to function as the inference device according to any one of claims 4 to 6.

[0094] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0095] 11 Convolutional Neural Networks 12 Feature Selection Section 13 Pooling Department 14 Relational Modeling Section 15 Classification section 16 Classification Department 17 Model parameter update section 18 Peak detector 19 Post-detection processing section 20 Tracking Department 100 Learning Device 200 Reasoning device 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device

Claims

1. A learning device that performs learning for behavior recognition, a convolutional neural network that inputs each image frame constituting a video sequence and outputs a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit that receives the motion feature map, the re-identification feature map, and the size feature map, and outputs motion features, re-identification features, and size features; a relational modeling unit that receives the operation features and the re-identified features and outputs features in which interactions between the features are taken into consideration; a first classification unit that outputs a group behavior classification result based on the features output from the relational modeling unit; a second classification unit that outputs an action classification result based on the features output from the relational modeling unit; a model parameter update unit that updates model parameters of the convolutional neural network, the feature selection unit, the relational modeling unit, the first classification unit, and the second classification unit so that errors between the position feature map, the size feature, the re-identification feature, the group behavior classification result, and the action classification result and ground truth data are minimized; A learning device comprising:

2. The relational modeling unit transforms the behavioral features by using the re-identified features as auxiliary information, or transforms the re-identified features by using the behavioral features as auxiliary information. The learning device according to claim 1 .

3. The relational modeling unit includes a first relational modeling unit and a second relational modeling unit, the first relational modeling unit converts the behavioral features by using the re-identified features as auxiliary information, and the second relational modeling unit converts the re-identified features by using the behavioral features as auxiliary information. The learning device according to claim 1 .

4. An inference device that performs inference for behavior recognition, a convolutional neural network that inputs each image frame constituting a video sequence and outputs a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit that receives the point position data obtained from the position feature map, the motion feature map, the re-identification feature map, and the size feature map, and outputs motion features, re-identification features, and size features; a tracking unit which receives the point position data, the detection result obtained from the size feature, and the re-identification feature, and outputs a tracking result; a relational modeling unit that receives the operation features and the re-identified features and outputs features in which interactions between the features are taken into consideration; a group behavior classification unit that outputs a group behavior classification result based on the features output from the relational modeling unit; an action classification unit that outputs an action classification result based on the features output from the relational modeling unit and the tracking result; An inference device comprising:

5. The relational modeling unit transforms the behavioral features by using the re-identified features as auxiliary information, or transforms the re-identified features by using the behavioral features as auxiliary information.

5. The inference device according to claim 4.

6. The relational modeling unit includes a first relational modeling unit and a second relational modeling unit, the first relational modeling unit converts the behavioral features by using the re-identified features as auxiliary information, and the second relational modeling unit converts the re-identified features by using the behavioral features as auxiliary information.

5. The inference device according to claim 4.

7. A learning method executed by a learning device that performs learning for behavior recognition, comprising: a convolutional neural network inputting each image frame of a video sequence and outputting a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit inputting the motion feature map, the re-identification feature map, and the size feature map, and outputting motion features, re-identification features, and size features; a step of a relational modeling unit receiving the operation features and the re-identified features and outputting features in which interactions between features are taken into consideration; a step of a first classification unit outputting a group behavior classification result based on the features output from the relational modeling unit; A second classification unit outputs an action classification result based on the features output from the relational modeling unit; updating model parameters of the convolutional neural network, the feature selection unit, the relational modeling unit, the first classifier, and the second classifier so that errors between the position feature map, the size feature, the re-identification feature, the group behavior classification result, and the action classification result and ground truth data are minimized; A learning method that provides:

8. An inference method executed by an inference device that performs inference for activity recognition, comprising: a convolutional neural network inputting each image frame of a video sequence and outputting a motion feature map, a re-identification feature map, a size feature map, and a position feature map; a feature selection unit inputting the point position data obtained from the position feature map, the motion feature map, the re-identification feature map, and the size feature map, and outputting a motion feature, a re-identification feature, and a size feature; A tracking unit receives the point position data, the detection result obtained from the size feature, and the re-identification feature, and outputs a tracking result; a step of a relational modeling unit receiving the operation features and the re-identified features and outputting features in which interactions between features are taken into consideration; a group behavior classification unit outputting a group behavior classification result based on the features output from the relational modeling unit; an action classification unit outputting an action classification result based on the features output from the relational modeling unit and the tracking result; An inference method comprising:

9. A program for causing a computer to function as the learning device according to any one of claims 1 to 3.

10. A program for causing a computer to function as the inference device according to any one of claims 4 to 6.

Images