Behavioral location identification device, control method, and program

Abstract

The behavior location identification device (2000) acquires a target clip (10) and detects a person (40) from the target clip (10). The behavior location identification device (2000) generates a person clip (60) from the target clip (10) for each person (40) detected from the target clip (10), and extracts a feature map from each of the person clips (60). The behavior location identification device (2000) calculates a behavior score for each predefined behavior class based on the feature map extracted from the person clip (60), calculates a behavior score indicating the reliability of the behavior of the behavior class included in the target clip (10), and performs class activation mapping for each of the person clips (60) to locate each behavior whose behavior class has a behavior score equal to or greater than a threshold.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to localizing video-captured activities. [Background technology]

[0002] Activity localization is the process of locating an activity captured in an image, i.e., determining which activity is performed at which location in the image. US Pat. No. 6,399,433 discloses a system for detecting activities in videos using neural networks for loss prevention in the retail industry. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Publication No. 2021 / 0248377 [Non-patent literature]

[0004] [Non-Patent Document 1] Junnan Li, Jianquan Liu, Yongkang Wong, Shoji Nishimura, Mohan Kankanhalli, "Weakly-Supervised Multi-Person Action Recognition in 360° Videos", [online], February 9, 2020, [Retrieved November 10, 2021],<arXiv,https: / / arxiv.org / pdf / 2002.03266.pdf> Search from Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present disclosure to provide novel techniques for localizing activities captured on video. [Means for solving the problem]

[0006] The present disclosure provides an apparatus for identifying behaviors, the apparatus having at least one memory configured to store instructions and at least one processor, the processor being configured to execute the instructions to obtain a target clip, the target clip being a sequence of target images, the target clip being fisheye images of one or more people captured from a generally upward perspective, detect one or more people from the target clip, generate a person clip from the target clip for each person detected from the target clip, the person clip being a sequence of person images, each of the person clips being a partial region of the target image, and including the detected person corresponding to the person clip, extract a feature map from each of the person clips, and calculate, based on the feature map extracted from the person clips, for each predetermined behavior class, a behavior score indicating a confidence of the behavior of the behavior class included in the target clip, the behavior class being a type of behavior, and locating each behavior of the behavior class having the behavior score equal to or greater than a threshold by performing class activation mapping on each of the person clips.

[0007] The present disclosure further provides a computer-implemented control method, comprising: acquiring a target clip, which is a sequence of target images, each of which is a fisheye image captured from a generally upward viewpoint, of one or more persons; detecting one or more persons from the target clip; and generating a person clip from the target clip for each of the persons detected from the target clip, each of the person clips being a sequence of person images that is a partial area of ​​the target image and including the detected person corresponding to the person clip; extracting a feature map from each of the person clips; and calculating, based on the feature map extracted from the person clips, for each predetermined behavior class, a behavior score indicating a reliability of a behavior of the behavior class included in the target clip, the behavior class being a type of behavior, and locating each behavior of the behavior class having the behavior score equal to or greater than a threshold by performing class activation mapping on each of the person clips.

[0008] The present disclosure further provides a non-transitory computer-readable storage medium having a program stored thereon, the program causing a computer to execute the steps of: acquiring a target clip, which is a sequence of target images, each of which is a fisheye image of one or more people captured from a generally upward viewpoint; detecting one or more people from the target clip; and generating a person clip from the target clip for each of the people detected from the target clip, the person clip being a sequence of person images, each of which is a partial region of the target image, and including the detected person corresponding to the person clip; extracting a feature map from each of the person clips; and calculating, based on the feature map extracted from the person clips, for each predetermined behavior class, an behavior score indicating a reliability of the behavior of the behavior class included in the target clip, the behavior class being a type of behavior, and performing class activation mapping for each of the person clips to locate each behavior of the behavior class having the behavior score equal to or greater than a threshold. Effect of the Invention

[0009] In accordance with the present disclosure, novel techniques are provided for localizing video-captured activities. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an overview of a behavior location identifying device according to a first embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of a person clip. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of a behavior position specifying device. FIG. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of the behavior position specifying device. [Diagram 5] 10 is a flowchart illustrating an exemplary flow of a process performed by a behavior location identifying device. [Figure 6]FIG. 13 is a diagram showing a case where a feature map represents spatiotemporal features of a target person and its surrounding area. [Figure 7] FIG. 13 illustrates an example of a method for calculating an action score based on a feature map. [Figure 8] FIG. 13 illustrates an example of a class activation map. [Figure 9] 10 is a flowchart illustrating an exemplary flow of a process performed by a position specifying unit. [Figure 10] FIG. 11 shows a first example in which a target clip is modified to indicate the results of behavioral location. [Figure 11] FIG. 11 shows a second example in which a target clip is modified to indicate the results of behavior location. [Figure 12] FIG. 11 is a diagram showing an overview of a behavior location identifying device according to a second embodiment. [Figure 13] FIG. 11 is a block diagram showing an example of a functional configuration of a behavior position identifying device according to a second embodiment. [Figure 14] 11 is a flowchart showing a flow of processing performed by a behavior position identifying device of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An embodiment according to the present disclosure will be described below with reference to the drawings. The same elements are assigned the same reference numerals throughout the drawings, and redundant descriptions are omitted as necessary. In addition, unless otherwise specified, predetermined information (e.g., a predetermined value or a predetermined threshold value) is stored in advance in a storage device accessible by a computer that uses the information.

[0012] EMBODIMENT 1 <Summary> Fig. 1 shows an overview of the behavior location identifying device 2000 of the embodiment 1. Note that the overview shown in Fig. 1 shows an example of the operation of the behavior location identifying device 2000 in order to make the behavior location identifying device 2000 easier to understand, and does not limit or narrow the range of operations that the behavior location identifying device 2000 can take.

[0013] The activity localization device 2000 operates on a target clip 10 that is part of a video 20 and is formed of a sequence of video frames 22. Each video frame 22 in the target clip 10 is called a "target image 12." The video 20 is a sequence of video frames 22 generated by a fisheye camera 30.

[0014] The fisheye camera 30 includes a fisheye lens and is installed as a top-view camera that captures the target location from a perspective above. Thus, each video frame 22 (and each target image 12) is a fisheye top-view image of the target location captured from a perspective above.

[0015] The target location may be any location. For example, when the fisheye camera 30 is used as a surveillance camera, the target location is a location to be monitored, such as a facility or its surroundings. The facility to be monitored may be a station, a stadium, or the like.

[0016] It is sometimes preferable that the field of view of the fisheye camera 30 is substantially close to 360 degrees in the horizontal direction. However, the field of view of the fisheye camera 30 does not need to be exactly 360 degrees in the horizontal direction. The fisheye camera 30 may be installed so that the optical axis of the fisheye lens is approximately parallel to the vertical direction. However, the optical axis of the fisheye lens of the fisheye camera does not need to be exactly parallel to the vertical axis.

[0017] The activity location identifying device 2000 detects one or more people 40 from the target clip 10, detects one or more actions taken by the detected people 40, and locates the detected actions (i.e., identifies which type of action occurred in which area of ​​the target clip 10). There may be various activity classes, such as "walking", "drinking", "putting on a jacket", and "operating a phone". Specifically, the activity location identifying device 2000 may operate as follows.

[0018] The behavior location identification device 2000 acquires a target clip 10 and detects one or more people 40 from the target clip 10. The behavior location identification device 2000 then generates a person clip 60 for each person 40 detected from the target clip 10. Hereinafter, a person corresponding to a person clip 60 is referred to as a "target person". The person clips 60 of a target person are a sequence of person images 62, each of which includes the target person and is cropped from a corresponding target image 12. The cropping positions (the positions in the target image 12 where the person images 62 are cropped) and the dimensions of the person images 62 in a single person clip 60 are the same as each other.

[0019] 2 shows an example of a person clip 60. In this example, the target clip 10 includes three people 40-1 to 40-3. Therefore, person clips 60-1 to 60-3 are generated for the people 40-1 to 40-3, respectively.

[0020] The behavior location identifying device 2000 extracts a feature map from each person clip 60. The feature map represents the spatiotemporal features of the person clip 60. The behavior location identifying device 2000 then calculates a behavior score for each predefined behavior class based on the feature map extracted from the person clip 60 to generate a behavior score vector 50. The behavior score of a behavior class represents the reliability of one or more behaviors of the behavior class included in the target clip 10 (in other words, the reliability that one or more persons 40 in the target clip 10 are performing the behavior of the behavior class). The behavior score vector 50 is a vector having the behavior scores of each predefined behavior class as its elements.

[0021] Assume that there are three predefined behavioral classes A1, A2, and A3. In this case, the behavioral score vector is a three-dimensional vector v=(c1,c2,c3), where c1 represents the behavioral score of the behavioral class A1 (i.e., the confidence of one or more behaviors of the behavioral class A1 included in the target clip 10), c2 represents the behavioral score of the behavioral class A2 (i.e., the confidence of one or more behaviors of the behavioral class A2 included in the target clip 10), and c3 represents the behavioral score of the behavioral class A3 (i.e., the confidence of one or more behaviors of the behavioral class A3 included in the target clip 10).

[0022] The behavior score vector 50 does not indicate what type of behavior is performed in what area of ​​the target clip 10. Based on the feature map obtained from the target clip 10 and the behavior score vector 50, the behavior localization device 2000 localizes the behavior in the target clip 10 (i.e., identifies what type of behavior occurred in what area of ​​the target clip 10). This type of localization is called "behavior localization."

[0023] The behavior location identification device 2000 uses class activation mapping to perform behavior location for the target clip 10. Specifically, for each person clip 60 and for each behavior class detected from the target clip 10, the behavior location identification device 2000 performs class activation mapping to identify which area in the person image 62 of that person clip 60 includes which type of behavior. In this way, each of the detected behaviors of the person 40 is located.

[0024] <Examples of effects> As described above, the behavior location identification device 2000 provides a novel technique for identifying the location of behaviors captured in a video. Specifically, the behavior location identification device 2000 generates a person clip 60 for each person 40 detected from a target clip 10, extracts a feature map for each person clip 60, and calculates a behavior score vector indicating each behavior score of a predefined behavior class. Then, the behavior location identification device 2000 identifies the behavior class of the behavior included in each person clip 60 using class activation mapping, thereby identifying each behavior in the target clip 10.

[0025] The behavior location identification device 2000 will be described in detail below.

[0026] <Example of functional configuration> 3 is a block diagram showing an example of a functional configuration of the behavior position identifying device 2000. The behavior position identifying device 2000 includes an acquiring unit 2020, a person clip generating unit 2040, a feature extracting unit 2060, a score calculating unit 2080, and a position identifying unit 2100.

[0027] The acquisition unit 2020 acquires a target clip 10. The person clip generation unit 2040 generates a person clip 60 for each person 40 detected in the target clip 10. The feature extraction unit 2060 extracts a feature map from each of the person clips 60. The score calculation unit 2080 calculates a behavior score for each predefined behavior class and generates a behavior score vector 50 using the feature map. The location identification unit 2100 performs class activation mapping based on the behavior score vector 50 and the feature map to locate the detected behavior in the target clip 10.

[0028] <Example of hardware configuration> The behavior location identification device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer manufactured for realizing the behavior location identification device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0029] The behavior location identification device 2000 may be realized by installing an application on a computer. The application is realized by a program for causing the computer to function as the behavior location identification device 2000. In other words, the program implements the functional units of the behavior location identification device 2000.

[0030] Fig. 4 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes the behavior position identification device 2000. In Fig. 4, the computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0031] The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage element such as a random access memory (RAM) or a read only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN). In some implementations, the computer 500 is connected to the fisheye camera 30 via this network. The above-mentioned programs may be stored in the storage device 1080. The CPU 1040 executes the programs to realize each of the functional units of the behavior position identification device 2000.

[0032] The hardware configuration of the computer 1000 is not limited to that shown in Fig. 4. For example, as described above, the behavior location identification device 2000 may be realized by a plurality of computers. In this case, the computers may be connected to each other via a network.

[0033] One or more of the functional configurations of the behavior location identification device 2000 may be implemented in the fisheye camera 30. In this case, the fisheye camera 30 functions as the whole or a part of the computer 500. For example, when all of the functional configurations of the behavior location identification device 2000 are implemented in the fisheye camera 30, the fisheye camera 30 may analyze the target clip 10 generated by itself, detect behavior from the target clip 10, locate the detected behavior in the target clip 10, and output information indicating the result of the behavior location identification for the target clip 10. The fisheye camera 30 functioning as described above may be a network camera, an IP (internet protocol camera), or an intelligent camera.

[0034] <Processing flow> 5 is a flowchart showing an exemplary flow of processing performed by the behavior position identifying device 2000. The acquiring unit 2020 acquires a target clip 10 (S102). The person clip generating unit 2040 detects one or more people 40 from the target clip 10 (S104). The person clip generating unit 2040 generates a person clip 60 for each detected person 40 (S106). The feature extracting unit 2060 extracts a feature map from each person clip 60 (S108). The score calculating unit 2080 generates a behavior score vector 50 based on the feature map (S110). The position identifying unit 2100 identifies the position of the behavior detected from the target clip 10 (S112).

[0035] <Getting target clip 10: S102> The acquiring unit 2020 acquires a target clip 10 (S102). As described above, the target clip 10 is a part of the video 20 generated by the fisheye camera 30. The number of target images 12 in the target clip 10 may be predetermined. Since multiple sequences of a predetermined number of video frames 22 may be extracted from different parts of the video 20, the behavior location identifying device 2000 may treat each of the sequences as a target clip 10. In this way, the behavior location identifying device 2000 may perform behavior location identification for each of the different parts of the video 20.

[0036] There are various ways to acquire the subject clip 10. For example, the acquirer 2020 may acquire a video 20 and divide it into multiple sequences of a predetermined number of video frames 22, thereby acquiring multiple subject clips 10. The video 20 may be acquired by accessing a memory unit in which the video 20 is stored, or by receiving the video 20 transmitted from another computer, such as a fisheye camera 30.

[0037] In another example, the acquisition unit 2020 may periodically acquire one or more video frames 22. The acquisition unit 2020 then generates a target clip 10 with a predetermined number of acquired video frames 22. The acquisition unit 2020 can generate multiple target clips 10 by repeating the above-mentioned flow. The video frames 22 may be acquired in a manner similar to the method of acquiring the video 20 described above.

[0038] <Detection of Person 40: S104> The person clip generator 2040 detects one or more people 40 from the target clip 10 (S104). Note that there are various well-known methods for detecting people from a sequence of fisheye images, and the person clip generator 2040 may be configured to detect the people 40 from the target clip 10 using one of these methods. For example, a machine learning based model (hereinafter referred to as a "person detection model") is used to detect the people 40 from the target clip 10. The person detection model is configured to take a fisheye clip as an input, and is pre-trained to output a person in the input fisheye clip and a region (e.g., a bounding box) containing the person for each fisheye video frame in response to the input of the fisheye clip.

[0039] The person clip generating unit 2040 may detect the person 40 from the entire area of ​​the target clip 10, or may detect the person 40 from a partial area of ​​the target clip 10. In the latter case, for example, the person 40 is detected from a circular area in each of the target images 12 of the target clip 10. An example of this latter case will be described later as a second embodiment of the behavior position identifying device 2000.

[0040] <Creating Person Clip 60: S106> The person clip generator 2040 generates a person clip 60 for each person 40 detected from the target clip 10 (S106). The person clips 60 of a target person are a sequence of images called "person images 62," each of which is a partial area of ​​the corresponding target image 12 and includes the target person. Hereinafter, an area cut out from the target image 12 to generate the person image 62 is called a "cut-out area." Note that the dimensions (width and height) of the cut-out area may be defined in advance.

[0041] The person clip generator 2040 may operate as follows for each target person (person 40 detected from the target clip 10): The person clip generator 2040 rotates the target images 12 by the same angle relative to each other so that the target person appears approximately upright in each of the target images 12. For example, if the person clip generator 2040 detects a bounding box for each person 40, the rotation angle of the target image 12 for the target person may be determined based on the orientation of the bounding box of the target person in a representative image (e.g., the first image) of the target images 12.

[0042] Next, the person clip generator 2040 identifies crop locations (i.e., locations of crop regions) in each of the rotated target images 12 where the target person is included in the crop region. The person clip generator 2040 then generates person images 12 by cropping images from the identified crop regions of the rotated target images 12, thereby generating person clips 60 for the target person.

[0043] <Feature extraction: S108> The feature extraction unit 2060 extracts a feature map from each of the person clips 60 (S108). The feature map represents spatiotemporal features of the person clips 60. The feature map may be extracted using a neural network, such as a 3D convolutional neural network (CNN), configured to acquire clips and pre-trained to output spatiotemporal features of the input clips as a feature map in response to the input clips.

[0044] The feature map may represent the spatiotemporal characteristics of the entire region of the person clip 60, or may represent the characteristics of a portion of the region of the person clip 60. In the latter case, the feature map may represent the spatiotemporal characteristics of the target person and the region around it.

[0045] FIG. 6 illustrates a case where the feature map represents spatiotemporal features of a target person and its surrounding area. In FIG. 6, there is a network 70 that takes a person clip 60 as input and outputs a feature map 80. The network 70 can be any kind of 3D CNN, such as a 3D ResNet. The feature extraction unit 2060 also performs pooling, such as max pooling, on the bounding box of the target person across all person images 62 to obtain a binary mask. The feature extraction unit 2060 resizes the binary mask to the same width and height as the feature map 80, and multiplies the resized mask by the feature map 80 to obtain the feature map 90. Hereinafter, the feature map output from the feature extraction unit 2060 will be described as "feature map 90" regardless of whether it is masked by a binary mask or not.

[0046] <Calculation of behavior score: S110> The score calculation unit 2080 calculates behavior scores for the predefined behavior classes, thereby calculating the behavior score vector 50 (S110). The feature map 90 obtained from the person clip 60 is used to calculate the behavior score. In some implementations, the score calculation unit 2080 may operate as shown in FIG. 7. FIG. 7 shows an example of a method for calculating the behavior score based on the feature map 90. In this example, person clips 60-1 and 60-N are obtained from the target clip 10, and feature maps 90-1 to 90-N are obtained from each of the person clips 60-1 to 60-N.

[0047] First, the score calculation unit 2080 performs pooling, for example average pooling, on each of the feature maps 90. Then, each of the pooling results is input to the fully connected layer 200. The fully connected layer 200 is configured to obtain the results of pooling on the feature maps 90, and is pre-trained to output, for each pre-defined behavior class, a vector called an "intermediate vector 210" that represents the confidence of the behavior of that behavior class contained in the person clip 60 corresponding to the feature map 90.

[0048] Using the fully connected layer 200, intermediate vectors 210 are obtained for all of the feature maps 90. The score calculation unit 2080 then aggregates the intermediate vectors 210 into a behavior score vector 50. The intermediate vectors 210 may be aggregated using a log-sum-exponential function as disclosed by Non-Patent Document 1. Note that the behavior score vector 50 may be scaled so that the maximum range of each element is [0,1].

[0049] <Class Activation Mapping: S112> The position identifying unit 2100 performs class activation mapping such as CAM (class activation mapping), Grad-CAM, and SmoothGrad to identify the position of the behavior detected from the target clip 10. Class activation mapping is a method for finding one or more regions in an input image associated with a prediction result (behavior score of an behavior class in the case of the behavior position identifying device 2000). The position identifying unit 2100 performs class activation mapping for each behavior class detected from the target clip 10 and for each person clip 60 to identify where in the target clip 10 the behavior of the detected behavior class was performed.

[0050] The behavior classes detected from the target clip 10 are those that each have a behavior score equal to or greater than a predefined threshold Tc. Assume that there are three predefined behavior classes A1, A2, and A3, and their behavior scores are 0.8, 0.3, and 0.7, respectively, and the threshold Tc is 0.6. In this case, the behavior classes detected are behavior classes A1 and A3, because their behavior scores are greater than the threshold Tc.

[0051] FIG. 8 shows an example of a class activation map. This figure shows a class activation map generated for person clip P1. Person clip P1 includes the action "operate phone". Assume that the detected actions are "drink", "operate phone" and "walk". In this case, the locator 2100 generates a class activation map for each of these three detected actions. The darker an area in the class activation map is, the more relevant that area is to predicting the action score of the action class that corresponds to that map. Also, the darker a location in the class activation map is, the greater the value of the cell in the map that corresponds to that location.

[0052] In Figure 8, the class activation map for "operate the phone" has a dark and wide area, while the other two class activation maps do not have such areas. Therefore, it is possible to predict that the action included in the person clip P1 is "operate the phone". In addition, by superimposing the action class map for "operate the phone" on the person clip P1, it is possible to predict the area of ​​the person clip P1 where the action of "operate the phone" is performed.

[0053] FIG. 9 is a flowchart showing an exemplary flow of the process performed by the position identifying unit 2100. Steps S202 to S212 form a loop process L1 that is performed for each person clip 60. In step S202, the position identifying unit 2100 determines whether or not the loop process L1 has been performed for all of the person clips 60. If the loop process L1 has been performed for all of the person clips 60, the process shown in FIG. 9 ends. On the other hand, if the loop process L1 has not been performed for all of the person clips 60, the position identifying unit 2100 selects a person clip 60 for which the loop process L1 has not yet been performed. The person clip 60 selected here is referred to as a "person clip Pi."

[0054] Steps S204 to S208 form a loop process L2 that is performed for each detected behavior class. In step S204, the position identification unit 2100 determines whether or not the loop process L2 has been performed for all the detected behavior classes. If the loop process L2 has been performed for all the detected behavior classes, then S210 is performed. On the other hand, if the loop process L2 has not been performed for all the detected behavior classes, the position identification unit 2100 selects, from the detected behavior classes, a behavior class for which the loop process L2 has not yet been performed. The behavior class selected here is referred to as "behavior class Aj".

[0055] The position identification unit 2100 performs class activation mapping on the person clip Pi (S206) using the behavior scores of Aj indicated by the behavior score vector 80 obtained from the person clip Pi and the feature map 90. As described above, there are various types of class activation mapping, and any of them can be adopted.

[0056] For example, when Grad-CAM is employed, the class activation map can be generated in a manner similar to that disclosed in Non-Patent Document 1. Specifically, for each channel of the feature map 90 obtained from the person clip Pi, the position identification unit 2100 calculates the importance of the channel for predicting the behavior score of the behavior class Aj based on the gradient of the behavior score with respect to the channel. This can be formulated as follows:

number

[0057] The importance of each channel calculated above is then used as the weight for each channel to generate a class activation map as a weighted combination of the channels of feature map 90 for person clip P1, which can be formulated as follows:

number

[0058] Step S208 is the end of the loop process L2, and therefore step S204 is carried out next.

[0059] After the loop process L2 for the person clip Pi is completed, the locating unit 2100 has class activation maps obtained from the person clip Pi for all detected behavior classes. In step S210, the locating unit 2100 identifies the behavior class of the behavior performed by the target person in the person clip Pi and locates the behavior based on the obtained class activation maps. To do so, the locating unit 2100 identifies one of the class activation maps corresponding to the behavior class of the behavior performed by the target person in the person clip Pi.

[0060] The class activation map calculated for behavior class Aj can be said to contain an area that is highly associated with the behavior score of behavior class Aj only if the target person in person clip Pi performed the behavior of behavior class Aj. Therefore, the class activation map that is most associated with the behavior score (prediction result of behavior score) of the corresponding behavior class is the one that corresponds to the behavior class of the behavior performed in person clip P1.

[0061] Specifically, for example, the locator 2100 calculates the sum of the cells for each class activation map, and identifies which class activation map has the largest sum. The locator 2100 then identifies the behavior class corresponding to the class activation map with the largest sum as the behavior class of the behavior taken in the person clip 60. This can be formulated as follows:

number

[0062] <Output from behavior location identification device 2000> The behavior localization device 2000 may output the spatial and temporal behavior localization results, i.e., information called "output information" indicating what type of behavior is performed in what area of ​​the target clip 10 during what period. The information indicated by the output information may be of various types. In some implementations, the output information may include, for each behavior of a detected behavior class, a set of the behavior class of the behavior, the location where the behavior was performed (e.g., the location of the bounding box of the person 40 who performed the behavior), and the period where the behavior was performed (e.g., the frame number of the target clip 10). For example, the output information may include, for each person 40 detected from the target clip 10, the target clip 10 modified to indicate the bounding box of the person 40 including an annotation indicating the behavior class of the behavior performed by the person 40.

[0063] 10 shows a first example in which a target clip 10 is modified to show the results of the activity localization. In this example, the activity classes "drink", "operate phone" and "walk" are detected and localized. To show the result, a combination of bounding box 220 and annotation 230 is superimposed on the target clip 10. The bounding box 220 indicates the location of the detected person 40, and the corresponding annotation 230 indicates the activity class of the activity performed by this person 40.

[0064] In other implementations, the output information may include the modified object clip 10 with the class activation maps of the detected behavior classes superimposed thereon. Suppose a target person in person clip Pi has been identified as performing an action in behavior class Aj. In this case, the object image 12 corresponding to the person image 62 in person clip Pi is modified to have the class activation map generated for the combination of person clip Pi and behavior class Aj superimposed on it. The locations in the object image 12 where the class activation map is superimposed are the locations where the corresponding person image 62 is cropped.

[0065] Figure 11 shows a second example in which the target clip 10 is modified to show the results of activity localization. This figure assumes the same situation as Figure 10. However, in Figure 11, instead of the bounding box 220, a map 240 is superimposed on the target clip 10. The map 240 superimposed on a person is a class activation map generated for that person's person clip 60 and corresponding to the activity class of the activity performed by that person.

[0066] <On Optimizing Trainable Parameters> The behavior location identification device 2000 has trainable parameters such as weights in the network 70 and the fully connected layer 200. These trainable parameters are pre-optimized by repeatedly updating them using multiple training data (in other words, the behavior location identification device 2000 is pre-trained). The training data may include a combination of test clips of behavior score vectors 50 and ground truth data. The test clips are any clips that are generated by a top-view fisheye camera (preferably by the fisheye camera 30) and contain one or more people. The ground truth data are behavior score vectors that indicate the maximum confidence (e.g., 1 when the behavior score vector 50 is scaled to [0,1]) of each of the behavior classes contained in the test clip.

[0067] The trainable parameters are updated based on a loss representing the difference between the ground truth data and the behavior score vector 50 calculated by the behavior locator 2000 in response to the input of the test clip. The loss may be calculated using any kind of loss function, such as a cross-entropy loss function. The loss function may further include a regularization term. For example, since a target person may perform a single behavior in a person clip 60, it is preferable to add a penalty when the intermediate vector (the behavior score vector calculated from a single feature map 80) shows high confidence for multiple behavior classes. This kind of regularization term is disclosed in Non-Patent Document 1.

[0068] EMBODIMENT 2 Fig. 12 shows an overview of the behavior location identification device 2000 of the embodiment 2. Note that the overview shown in Fig. 12 shows an example of the operation of the behavior location identification device 2000 of the embodiment 2 in order to make it easier to understand the behavior location identification device 2000 of the embodiment 2, and does not limit or narrow the range of operations that the behavior location identification device 2000 of the embodiment 2 can take.

[0069] In the subject clip 10, people appear at different angles. Non-Patent Document 1 addresses this problem by converting video frames captured from a fisheye camera into a panoramic image and analyzing the panoramic image for activity detection and localization.

[0070] However, such conversion to a panoramic image may distort a person located near the center (the optical axis of the fisheye lens when viewed from above). Regarding this problem, in order to locate an action that occurred near the optical axis of the fisheye lens, the method performed by the action location identification device 2000 is considered to be more effective than the method described in Non-Patent Document 1, because the person located near the center is not distorted.

[0071] Therefore, the behavior location identifying device 2000 of the second embodiment generates two different types of clips from the target clip 10 and calculates behavior scores for these two clips by performing two different methods. By doing so, it is possible to more accurately identify the location of the behavior in the target clip 10. Hereinafter, these two types of clips are referred to as the "center clip 100" and the "panorama clip 110," respectively. In addition, the method performed on the center clip 100 is referred to as "fisheye processing," and the method performed on the panorama clip 110 is referred to as "panorama processing."

[0072] The central clip 100 is a sequence of central regions of the target images 12. To generate the central clip 100, the behavior location identification device 2000 extracts a central region 14 from each of the target images 12. The central clip 100 is generated as a sequence of central regions 14. The central region 14 is a circular region with a predefined radius whose center is located at a location corresponding to the optical axis of the fisheye camera 30. The position corresponding to the optical axis of the fisheye camera 30 may be detected by the method disclosed in Non-Patent Document 1. Hereinafter, each image included in the central clip 100 (i.e., the central region 14) is referred to as a "central image 112."

[0073] The panoramic clip 110 is a sequence of target images that have been converted into a panoramic image. To generate the panoramic clip 110, the behavior locating device 2000 converts each target image 12 into a panoramic image. The panoramic clip 110 is generated as a sequence of those panoramic images. The target images 12 may be converted into panoramic images using the method disclosed in Non-Patent Document 1. Hereinafter, each image included in the panoramic clip 110 is referred to as a "panoramic image 112."

[0074] The fisheye processing is a method of calculating a behavior score from the central clip 100, similar to the method in which the behavior location identification device 2000 of the eleventh embodiment calculates a behavior score for the target image 10. Specifically, the fisheye processing includes the steps of detecting one or more persons 40 from the central clip 100, generating a person clip 60 for each person 40 detected from the central clip 100, extracting a feature map 90 from each of the person clips 60, and calculating a behavior score based on the feature map 90. Hereinafter, a vector indicating the behavior score calculated for the central clip 100 is referred to as a "behavior score vector 130."

[0075] Panorama processing is a method of calculating an action score in a manner similar to that disclosed in Non-Patent Document 1. Specifically, panorama processing may be performed as follows. The behavior location identification device 2000 calculates a feature map (spatiotemporal features) of the panorama clip 110 by inputting the panorama clip 110 to a neural network such as a 3D CNN that can extract spatiotemporal features as a feature map from a sequence of images. Next, the behavior location identification device 2000 calculates a binary mask by human detection of the panorama clip 110, resizes the binary mask to the same width and height as the feature map, and multiplies the binary mask by the feature map to obtain a masked feature map. The masked feature map is divided into multiple blocks. The behavior location identification device 2000 inputs the masked feature map to a fully connected layer after performing pooling for each block, thereby obtaining a behavior score for each block. The behavior scores obtained for each block are aggregated into a single vector that indicates the behavior score of the entire panorama clip 110. Hereinafter, this vector will be referred to as the "behavior score vector 140."

[0076] As described above, the behavior location identifying device 2000 obtains the behavior score vector 130 as a result of the fisheye processing and the behavior score vector 140 as a result of the panoramic processing. The behavior location identifying device 2000 uses the behavior score vector 130 and the behavior score vector 140 to locate each behavior in the target clip 100.

[0077] In some implementations, the behavior location identifying device 2000 uses the behavior score vector 130 and the behavior score vector 140 separately as shown in FIG. 12. In this case, the behavior location identifying device 2000 performs behavior location on the central clip 100 using the behavior score vector 130 in a manner similar to the manner in which the behavior location identifying device 2000 of the eleventh embodiment performs behavior location on the target clip 10 using the behavior score vector 50. This allows the behavior detected from the central clip 110 to be located. In addition, the behavior location identifying device 2000 performs behavior location on the panoramic clip 110 using the behavior score vector 140 in a manner similar to the method disclosed in Non-Patent Document 1. This allows the behavior detected from the panoramic clip 110 to be located. Next, the behavior location identifying device 2000 aggregates the result of the behavior location identified on the central clip 100 and the result of the behavior location identified on the panoramic clip 110, thereby allowing the behavior of the behavior class detected on the entire target clip 10 to be located.

[0078] In another implementation, the behavior location identification device 2000 aggregates the behavior score vector 130 and the behavior score vector 140 into a single vector called an "aggregated behavior score vector." In this case, the behavior class whose behavior score is equal to or greater than the threshold Tc is treated as a detection target. Instead of using the behavior score vectors 130 and 140 separately, the behavior location identification device 2000 uses the aggregated behavior score vector to perform behavior location identification separately for the central clip 100 and the panoramic clip 110 for each detected behavior class, and aggregates the behavior location results.

[0079] Except that the aggregate behavior score vector is used for class activation mapping, the behavior localization performed for the center clip 100 in this case is the same as if the behavior score vectors 130 and 140 were used separately. Similarly, the behavior localization performed for the panoramic clip 110 in this case is the same as if the behavior score vectors 130 and 140 were used separately, except that the aggregate behavior score vector is used for class activation mapping.

[0080] <Example of functional configuration> FIG. 13 is a block diagram showing an example of the functional configuration of the behavior position identification device 2000 of the second embodiment. The behavior position identification device 2000 includes an acquisition unit 2020, a center clip generation unit 2120, a panoramic clip generation unit 2140, a fisheye processing unit 2160, a panoramic processing unit 2180, and a position identification unit 2100. The acquisition unit 2020 acquires the target clip 10 as described in the eleventh embodiment. The center clip generation unit 2120 generates the center clip 100 from the target clip 10. The panoramic clip generation unit 2140 generates the panoramic clip 110 from the target clip 10. The fisheye processing unit 2160 performs fisheye processing on the center clip 100 to calculate the behavior score vector 130. Although not shown in FIG. 13, the person clip generation unit 2040, the feature extraction unit 2060, and the score calculation unit 2080 are included in the fisheye processing unit 2160. The panorama processing unit 2180 performs panorama processing on the panorama clip 110 to calculate the behavior score vector 140. The position identifying unit 2100 uses the behavior score vector 130 and the behavior score vector 140 to identify the behavior position for the target clip 10.

[0081] <Example of hardware configuration> Like the behavior location identification device 2000 of the eleventh embodiment, the behavior location identification device 2000 of the second embodiment may have the hardware configuration shown in Fig. 3. However, the storage device 1080 of the second embodiment may further include a program that realizes the functional configuration of the behavior location identification device 2000 of the second embodiment.

[0082] <Processing flow> FIG. 14 is a flowchart showing a flow of processing performed by the behavior location identifying device 2000 of the second embodiment. The acquiring unit 2020 acquires a target clip 10 (S302). Between steps S302 and S316, there are two sequences of processing shown to be performed in parallel. The first sequence of processing includes steps S304-S308 and is performed to locate behavior in the central clip 100. Meanwhile, the second sequence of processing includes steps S310-S314 which are performed to locate behavior in the panoramic clip 110. Note that in other implementations, these sequences may be performed sequentially rather than in parallel.

[0083] The first sequence of processing is performed as follows: The center clip generator 2120 generates the center clip 100 (S304). The fisheye processor 2160 performs fisheye processing on the center clip 100 to calculate the behavior score vector 130 (S306). The locator 2100 uses the behavior score vector 130 to locate the behavior of the behavior class detected for the center clip 100 (S308).

[0084] The second flow of processing is performed as follows: The panoramic clip generator 2140 generates the center clip 100 (S310). The panoramic processor 2180 performs panoramic processing on the panoramic clip 110 to calculate the behavior score vector 140 (S312). The locator 2100 uses the behavior score vector 140 to locate the behavior of the behavior class detected for the panoramic clip 100 (S308).

[0085] The locator 2100 aggregates the results of the action location determination for the central clip 100 and the action location determination for the panoramic clip 110 .

[0086] 14, it is assumed that the behavior score vectors 130 and 140 are used separately to localize the behavior. However, as described above, the behavior localization may be performed using an aggregate behavior score vector instead of using the behavior score vectors 130 and 140 separately. In this case, after the fisheye and panoramic processing is completed, the aggregate behavior score vector is calculated, and then the behavior localization is performed for the center clip 100 and the panoramic clip 110 using the aggregate behavior score vector.

[0087] <Output from behavior location identification device 2000> The behavior location identifying device 2000 of the second embodiment may output output information similar to that output by the behavior location identifying device 2000 of the eleventh embodiment. However, the output information of the second embodiment may indicate an aggregated result of behavior location identification for the center clip 100 and the panoramic clip 110. For example, as shown in FIG. 10, when the output information includes the target clip 10 on which the bounding box 220 and the annotation 230 are superimposed, the behavior location identifying device 2000 generates the bounding box 220 and the annotation 230 for both the person whose behavior is identified based on the result of the fisheye processing and the person whose behavior is identified based on the result of the panoramic processing. The same applies to the case where the map 240 and the annotation 230 are superimposed on the target clip 10 as shown in FIG. 11.

[0088] In some cases, there may be a person included in both the central clip 100 and the panoramic clip 110. In this case, the localization unit 2100 may obtain class activation maps for the person from both the central clip 100 and the panoramic clip 110. The localization unit 2100 may combine these class activation maps and localize the person's activity based on the combined class activation map. The class activation maps may be combined by taking their intersection points or their midpoints.

[0089] <Trainable parameter optimization> The behavior locating device 2000 of the second embodiment further includes trainable parameters used for panoramic processing in addition to the trainable parameters mentioned in the eleventh embodiment. Similar to those adopted in the eleventh embodiment, the trainable parameters of the behavior locating device 2000 of the second embodiment may be pre-optimized using multiple training data. However, in this embodiment, the loss may be calculated using an aggregate behavior score vector instead of the behavior score vector 50. Specifically, the loss may be calculated to represent the difference between the ground truth data of the aggregate behavior score indicated by the training data and the aggregate behavior score calculated by the behavior locating device 2000 of the second embodiment according to the input of the test clip.

[0090] The program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs). The program may also be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0091] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present invention.

[0092] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) A behavior location identification device, at least one memory configured to store instructions; Executing the instructions, acquiring a target clip, the target clip being a sequence of target images, the target images being fisheye images of one or more people captured from a generally upward perspective; Detecting one or more people from the clip of interest; generating a person clip from the object clip for each person detected in the object clip, the person clip being a sequence of person images, each of the person clips being a region of the object image, the person clip including the detected person corresponding to the person clip; Extracting a feature map from each of said person clips; calculating an action score for each predetermined action class based on the feature map extracted from the person clip, the action score indicating a reliability of the action of the action class included in the target clip, the action class being a type of action; performing class activation mapping for each of the person clips to locate each activity in an activity class having the activity score equal to or greater than a threshold; and at least one processor configured to: (Appendix 2) The activity localization includes, for each person clip, for each of the behavior classes whose behavior score is greater than or equal to the threshold, generating a class activation map using the behavior score for that behavior class and the feature map extracted from the person clip; identifying the class activation map that exhibits the highest association with the behavioral score of the corresponding behavioral class; and identifying the behavior class corresponding to the identified class activation map as the behavior class of the behavior included in the person clip. (Appendix 3) The calculation of the behavioral score includes: calculating, for each person clip, an intermediate vector indicating a confidence level of the behavior of the behavior class contained in the person clip for each of the predetermined behavior classes; aggregating the intermediate vectors into a behavior score vector indicative of the behavior score of the predetermined behavior class. (Appendix 4) The extraction of the feature map and the calculation of the behavioral score are performed by a pre-trained neural network; The behavior localization device of Appendix 3, wherein the pre-trained neural network is trained using a training dataset including test clips and a vector indicating a maximum confidence for each of the behavior classes contained in the test clips, and the test clips include fisheye images of one or more people captured from a generally upward perspective. (Appendix 5) 5. The behavior location determination apparatus of claim 3 or 4, wherein the intermediate vectors are aggregated into the behavior score vector using a log-sum-exponential function. (Appendix 6) The at least one processor further executes the instructions to: generating a sequence of center clips, each of which is generated by cropping a center region from a corresponding said target image; generating a panoramic clip, each of the panoramic images being a sequence of panoramic images generated by converting a corresponding one of the target images into a panoramic image; locating the activity in the target clip by locating the activity in the central clip, locating the activity in the panoramic clip, and aggregating results of locating the activity in the central clip and locating the activity in the panoramic clip; The location of the action included in the central clip may include: detecting one or more people from the center clip; generating said people clips from said center clip for each of said people detected in said center clip; extracting the feature map from each of the people clips; calculating the behavior score for each of the predetermined behavior classes based on the feature map extracted from the person clips; and locating each behavior of a behavior class having the behavior score above a threshold by performing class activation mapping on each of the person clips. (Appendix 7) A control method performed by a computer, comprising the steps of: acquiring a target clip, the target clip being a sequence of target images, the target images being fisheye images of one or more people captured from a generally upward perspective; detecting one or more people from the clip of interest; and for each person detected from the object clip, generating a person clip from the object clip, the person clip being a sequence of person images, each of the person clips being a region of the object image, and including the detected person corresponding to the person clip; extracting a feature map from each said person clip; calculating, for each predetermined behavior class, an behavior score indicating a reliability of the behavior of the behavior class included in the target clip based on the feature map extracted from the person clip, wherein the behavior class is a type of behavior; The method includes the step of locating each behavior of a behavior class having the behavior score above a threshold by performing class activation mapping on each of the person clips. (Appendix 8) The activity localization includes, for each person clip, for each of the behavior classes whose behavior score is greater than or equal to the threshold, generating a class activation map using the behavior score for that behavior class and the feature map extracted from the person clip; identifying the class activation map that exhibits the highest association with the behavioral score of the corresponding behavioral class; and identifying the behavior class corresponding to the identified class activation map as the behavior class of the behavior included in the person clip. (Appendix 9) The calculation of the behavioral score includes: calculating, for each person clip, an intermediate vector indicating a confidence level of the behavior of the behavior class contained in the person clip for each of the predetermined behavior classes; aggregating the intermediate vectors into a behavior score vector indicative of the behavior score for the predetermined behavior class. (Appendix 10) The extraction of the feature map and the calculation of the behavioral score are performed by a pre-trained neural network; The control method of claim 9, wherein the pre-trained neural network is trained using a training dataset including test clips and a vector indicating a maximum confidence for each of the behavioral classes contained in the test clips, the test clips including fisheye images of one or more people captured from a generally upward perspective. (Appendix 11) 11. The control method of claim 9 or 10, wherein the intermediate vector is aggregated into the behavioral score vector using a log-sum-exponential function. (Appendix 12) generating a sequence of center clips, each of which is generated by cropping a center region from a corresponding said target image; generating a panoramic clip, which is a sequence of panoramic images, each of which is generated by converting a corresponding one of the target images into a panoramic image; locating the activity in the target clip by locating the activity in the central clip, locating the activity in the panoramic clip, and aggregating results of locating the activity in the central clip and locating the activity in the panoramic clip; The location of the action included in the central clip may include: detecting one or more people from the center clip; generating said people clips from said center clip for each of said people detected in said center clip; extracting the feature map from each of the people clips; calculating the behavior score for each of the predetermined behavior classes based on the feature map extracted from the person clips; A control method as described in any one of Appendix 7 to 11, comprising: locating each behavior of a behavior class having the behavior score above a threshold by performing class activation mapping for each of the person clips. (Appendix 13) A non-transitory computer-readable storage medium having a program stored therein, the program being configured to: acquiring a target clip, the target clip being a sequence of target images, the target images being fisheye images of one or more people captured from a generally upward perspective; detecting one or more people from the clip of interest; generating a person clip from the object clip for each person detected in the object clip, the person clip being a sequence of person images, each of the person clips being a region of the object image, the person clip including the detected person corresponding to the person clip; extracting a feature map from each said person clip; calculating an action score for each predetermined action class based on the feature map extracted from the person clip, the action score indicating a reliability of the action of the action class included in the target clip, the action class being a type of action; A non-transitory computer-readable storage medium for causing a user to perform a step of locating each behavior of a behavior class having the behavior score equal to or greater than a threshold by performing class activation mapping on each of the person clips. (Appendix 14) The activity localization includes, for each person clip, for each of the behavior classes whose behavior score is greater than or equal to the threshold, generating a class activation map using the behavior score for that behavior class and the feature map extracted from the person clip; identifying the class activation map that exhibits the highest association with the behavioral score of the corresponding behavioral class; and identifying the behavior class corresponding to the identified class activation map as the behavior class of the behavior included in the person clip. (Appendix 15) The calculation of the behavioral score includes: calculating, for each person clip, an intermediate vector indicating a confidence level of the behavior of the behavior class contained in the person clip for each of the predetermined behavior classes; aggregating the intermediate vectors into a behavior score vector indicative of the behavior score for the predetermined behavior class. (Appendix 16) The extraction of the feature map and the calculation of the behavioral score are performed by a pre-trained neural network; The storage medium of claim 15, wherein the pre-trained neural network is trained using a training dataset including test clips and a vector indicating a maximum confidence for each of the behavioral classes contained in the test clips, the test clips including fisheye images of one or more people captured from a generally upward perspective. (Appendix 17) 17. The storage medium of claim 15 or 16, wherein the intermediate vector is aggregated into the behavior score vector using a log-sum-exponential function. (Appendix 18) The program causes the computer to generating a sequence of center clips, each of which is generated by cropping a center region from a corresponding said target image; generating a panoramic clip, which is a sequence of panoramic images, each of which is generated by converting a corresponding one of the target images into a panoramic image; locating the activity in the target clip by locating the activity in the central clip, locating the activity in the panoramic clip, and aggregating results of locating the activity in the central clip and locating the activity in the panoramic clip; The location of the action included in the central clip may include: detecting one or more people from the center clip; generating said people clips from said center clip for each of said people detected in said center clip; extracting the feature map from each of the people clips; calculating the behavior score for each of the predetermined behavior classes based on the feature map extracted from the person clips; 18. A storage medium as claimed in any one of appendices 13 to 17, further comprising: locating each behaviour of a behaviour class having the behaviour score above a threshold by performing class activation mapping for each of the person clips. [Explanation of symbols]

[0093] 10 Target Clips 12 Target image 14 Central area 20 Videos 22 Video Frames 30 Fisheye Camera 40 people 50, 130, 140 Behavioral score vector 50 60 People Clips 62 Portraits 70 Network 80, 90 feature maps 100 Center Clip 102 Center Image 110 Panorama Clip 112 Panorama Images 200 fully connected layers 210 Intermediate Vector 220 Bounding Box 230 Comments 240 Maps 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Behavioral location identification device 2020 Acquisition Department 2040 Person Clip Generation Unit 2060 Feature Extraction Unit 2080 Score Calculation Department 2100 Location identification part 2120 Center Clip Generator 2140 Panorama Clip Generator 2160 Fisheye Processing Unit 2180 Panorama Processing Unit

Claims

1. A behavior location identification device, at least one memory configured to store instructions; Executing the instructions, acquiring a target clip, the target clip being a sequence of target images, the target images being fisheye images of one or more people captured from a generally upward perspective; Detecting one or more people from the clip of interest; generating a person clip from the object clip for each person detected in the object clip, the person clip being a sequence of person images, each of the person clips being a region of a portion of the object image, the person clip including the detected person corresponding to the person clip; Extracting a feature map from each of said person clips; calculating an action score for each predetermined action class based on the feature map extracted from the person clip, the action score indicating a reliability of the action of the action class included in the target clip, the action class being a type of action; performing class activation mapping for each of the person clips to locate each activity in an activity class having the activity score equal to or greater than a threshold; and at least one processor configured to:

2. The activity localization includes, for each person clip, for each of the behavior classes whose behavior score is greater than or equal to the threshold, generating a class activation map using the behavior score for that behavior class and the feature map extracted from the person clip; identifying the class activation map that exhibits the highest association with the behavioral score of the corresponding behavioral class; and identifying the behavior class corresponding to the identified class activation map as the behavior class of the behavior included in the person clip.

3. The calculation of the behavioral score includes: calculating, for each of the person clips, an intermediate vector indicating a confidence level of the behavior of the behavior class contained in the person clip for each of the predetermined behavior classes; The behavior location identifying apparatus according to claim 1 or 2, further comprising: aggregating the intermediate vectors into a behavior score vector indicative of the behavior score of the predetermined behavior class.

4. The extraction of the feature map and the calculation of the behavioral score are performed by a pre-trained neural network; 4. The behavior location identification device of claim 3, wherein the pre-trained neural network is trained using a training dataset including test clips and vectors that indicate maximum confidence for each of the behavior classes contained in the test clips, and the test clips include fisheye images of one or more people captured from a generally upward perspective.

5. The behavior location determining apparatus according to claim 3 or 4, wherein the intermediate vectors are aggregated into the behavior score vector using a log-sum-exponential function.

6. The at least one processor further executes the instructions to: generating a sequence of center clips, each of which is generated by cropping a center region from a corresponding said target image; generating a panoramic clip, each of the panoramic images being a sequence of panoramic images generated by converting a corresponding one of the target images into a panoramic image; locating the activity in the target clip by locating the activity in the central clip, locating the activity in the panoramic clip, and aggregating results of locating the activity in the central clip and locating the activity in the panoramic clip; The location of the action included in the central clip may include: detecting one or more people from the central clip; generating said people clips from said center clip for each of said people detected in said center clip; extracting the feature map from each of the people clips; calculating the behavior score for each of the predetermined behavior classes based on the feature map extracted from the person clips; The apparatus for identifying behavior localization according to claim 1 , further comprising: performing class activation mapping for each of the person clips to locate each behavior of a behavior class having the behavior score equal to or greater than a threshold.

7. A control method performed by a computer, comprising the steps of: acquiring a target clip, the target clip being a sequence of target images, the target images being fisheye images of one or more people captured from a generally upward perspective; detecting one or more people from the clip of interest; and for each person detected from the object clip, generating a person clip from the object clip, the person clip being a sequence of person images, each of the person clips being a region of the object image, and including the detected person corresponding to the person clip; extracting a feature map from each said person clip; calculating, for each predetermined behavior class, an behavior score indicating a reliability of the behavior of the behavior class included in the target clip based on the feature map extracted from the person clip, wherein the behavior class is a type of behavior; A method of control comprising the step of locating each activity of an activity class having said activity score above a threshold by performing class activation mapping for each said person clip.

8. The activity localization includes, for each person clip, for each of the behavior classes whose behavior score is greater than or equal to the threshold, generating a class activation map using the behavior score for that behavior class and the feature map extracted from the person clip; identifying the class activation map that exhibits the highest association with the behavioral score of the corresponding behavioral class; and identifying the behavior class corresponding to the identified class activation map as the behavior class of the behavior included in the person clip.

9. The calculation of the behavioral score includes: calculating, for each of the person clips, an intermediate vector indicating a confidence level of the behavior of the behavior class contained in the person clip for each of the predetermined behavior classes; A method according to claim 7 or 8, further comprising aggregating the intermediate vectors into a behaviour score vector indicative of the behaviour scores for the predetermined behaviour class.

10. The extraction of the feature map and the calculation of the behavioral score are performed by a pre-trained neural network; 10. The control method of claim 9, wherein the pre-trained neural network is trained using a training dataset including test clips and a vector indicating a maximum confidence for each of the behavior classes contained in the test clips, the test clips including fisheye images of one or more people captured from a generally upward perspective.

11. The method of claim 9 or 10, wherein the intermediate vectors are aggregated into the behavior score vector using a log-sum-exponential function.

12. generating a sequence of center clips, each of which is generated by cropping a center region from a corresponding said target image; generating a panoramic clip, which is a sequence of panoramic images, each of which is generated by converting a corresponding one of the target images into a panoramic image; locating the activity in the target clip by locating the activity in the central clip, locating the activity in the panoramic clip, and aggregating results of locating the activity in the central clip and locating the activity in the panoramic clip; The location of the action included in the central clip may include: detecting one or more people from the central clip; generating said people clips from said center clip for each of said people detected in said center clip; extracting the feature map from each of the people clips; calculating the behavior score for each of the predetermined behavior classes based on the feature map extracted from the person clips; 12. The method of claim 7, further comprising: locating each behavior of a behavior class having the behavior score above a threshold by performing class activation mapping for each of the person clips.

13. A non-transitory computer-readable storage medium having a program stored therein, the program being configured to: acquiring a target clip, the target clip being a sequence of target images, the target images being fisheye images of one or more people captured from a generally upward perspective; detecting one or more people from the clip of interest; generating a person clip from the object clip for each person detected in the object clip, the person clip being a sequence of person images, each of the person clips being a region of the object image, the person clip including the detected person corresponding to the person clip; extracting a feature map from each said person clip; calculating an action score for each predetermined action class based on the feature map extracted from the person clip, the action score indicating a reliability of the action of the action class included in the target clip, the action class being a type of action; A non-transitory computer-readable storage medium for causing a user to perform a step of locating each behavior of a behavior class having the behavior score equal to or greater than a threshold by performing class activation mapping on each of the person clips.

14. The activity localization includes, for each person clip, for each of the behavior classes whose behavior score is greater than or equal to the threshold, generating a class activation map using the behavior score for that behavior class and the feature map extracted from the person clip; identifying the class activation map that exhibits the highest association with the behavioral score of the corresponding behavioral class; and identifying the behavior class corresponding to the identified class activation map as the behavior class of the behavior included in the people clip.

15. The calculation of the behavioral score includes: calculating, for each of the person clips, an intermediate vector indicating a confidence level of the behavior of the behavior class contained in the person clip for each of the predetermined behavior classes; aggregating the intermediate vectors into a behavior score vector indicative of the behavior score for the predetermined behavior class.

16. The extraction of the feature map and the calculation of the behavioral score are performed by a pre-trained neural network; 16. The storage medium of claim 15, wherein the pre-trained neural network is trained using a training dataset including test clips and vectors that indicate maximum confidence for each of the behavioral classes contained in the test clips, the test clips including fisheye images of one or more people captured from a generally upward perspective.

17. The storage medium of claim 15 or 16, wherein the intermediate vectors are aggregated into the behavior score vector using a log-sum-exponential function.

18. The program causes the computer to generating a sequence of center clips, each of which is generated by cropping a center region from a corresponding said target image; generating a panoramic clip, which is a sequence of panoramic images, each of which is generated by converting a corresponding one of the target images into a panoramic image; locating the activity in the target clip by locating the activity in the central clip, locating the activity in the panoramic clip, and aggregating results of locating the activity in the central clip and locating the activity in the panoramic clip; The location of the action included in the central clip may include: detecting one or more people from the central clip; generating said people clips from said center clip for each of said people detected in said center clip; extracting the feature map from each of the people clips; calculating the behavior score for each of the predetermined behavior classes based on the feature map extracted from the person clips; and locating each behavior of a behavior class having the behavior score above a threshold by performing class activation mapping on each of the person clips.

Images