Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program capable of detecting action of an object with high accuracy regardless of difference in input to a spatiotemporal action detection model in a pre-learning stage and an inference stage.SOLUTION: A data generation device for training includes: an action calculation unit for calculating an action feature quantity regarding action of each object reflected in a video detected in each frame of the video; an input unit for receiving an input of a character string that shows arbitrary action; a character string calculation unit for calculating a character string feature quantity about the character string; and a score calculation unit for calculating, for each object, a detection score indicating a degree to which the object takes action indicated by the character string based on the action feature quantity and the character string feature quantity.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program for detecting an object performing a specific action. [Background technology]

[0002] There is a demand for technology that can detect the behavior of people, animals, and other objects in video. Such technology is used, for example, to detect when people in video captured by a surveillance camera are engaging in specific, suspicious behavior in order to prevent crime, or to detect what plays players are making in sports footage.

[0003] As a technology that can meet such a demand, for example, there is a technology proposed in Non-Patent Document 1. Non-Patent Document 1 discloses a technology for learning a spatio-temporal action detection model using a video in which one action label is assigned to the entire video. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Arnab, A., Sun, C., Nagrani, A. and Schmid, C.: Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos, ECCV (2020) Summary of the Invention [Problem to be solved by the invention]

[0005] In order to train a detection model that can accurately detect what actions an object in each frame of a video is taking, it is necessary to assign correct labels indicating actions to each frame of the video used as learning data. However, it is very costly (labor and time) to assign correct labels to objects that appear in each frame of a large amount of video used as learning data.

[0006] The present disclosure aims to provide an information processing device, an information processing method, and a program capable of detecting the behavior of an object with high accuracy regardless of differences in input to a spatiotemporal behavior detection model in the pre-learning stage and the inference stage. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present disclosure includes a behavior calculation unit that calculates behavioral features related to the behavior of each object appearing in a moving image detected for each frame of the moving image, an input unit that accepts input of a string indicating an arbitrary behavior, a string calculation unit that calculates a string feature related to the string, and a score calculation unit that calculates, for each of the objects, a detection score indicating the degree to which the object is performing the behavior indicated by the string based on the behavior features and the string features.

[0008] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which calculates behavioral features related to the behavior of each object appearing in a video detected for each frame of the video, accepts input of a string indicating an arbitrary behavior, calculates string features related to the string, and calculates, for each object, a detection score indicating the degree to which the object is engaging in the behavior indicated by the string based on the behavior features and the string features.

[0009] A program according to one aspect of the present disclosure causes a computer to execute the following steps: calculating behavioral features relating to the behavior of each object appearing in a video detected for each frame of the video; accepting input of a string indicating an arbitrary behavior; calculating string features relating to the string; and calculating a detection score indicating the degree to which the object is engaging in the behavior indicated by the string based on the behavior features and the string features. Effect of the Invention

[0010] The present disclosure can detect the behavior of an object with high accuracy regardless of differences in input to a spatiotemporal behavior detection model in the pre-learning stage and the inference stage. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining a system configuration of a monitoring system including an information processing device according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a diagram for explaining a configuration of an information processing device; [Diagram 3] A diagram for explaining the configuration and processing flow of a specific behavior detection program in the pre-learning stage. [Figure 4] A schematic diagram showing the XY coordinates of joint points among the feature point information extracted by the feature point extraction unit for a person walking indoors. [Diagram 5] A diagram for explaining the configuration and processing flow of a specific behavior detection program in the inference stage. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0013] [System Configuration] First, a system configuration of a monitoring system 1 including an information processing device 100 according to an embodiment of the present disclosure will be described.

[0014] 1, the surveillance system 1 includes an information processing device 100 and a surveillance camera 110 connected by a cable 120. A plurality of surveillance cameras 110 may be connected to the information processing device 100. The information processing device 100 and the surveillance camera 110 do not need to be connected by the cable 120, and may be connected wirelessly, for example.

[0015] The surveillance camera 110 is installed facing a predetermined shooting direction and continuously shoots the monitored location. The video captured by the surveillance camera 110 has a plurality of frame images. The video captured by the surveillance camera 110 is transmitted to the information processing device 100 as digital video data. The video captured by the surveillance camera 110 may be read out from the surveillance camera 110 by the information processing device 100 at any time.

[0016] The information processing device 100 executes a detection process using video images acquired from the surveillance camera 110 to detect an object exhibiting a specific behavior.

[0017] An object is, for example, a person appearing in a video, but may also be a moving object other than a person, such as a vehicle in a video of road traffic, a production machine in a video of a factory, or livestock in a video of a farm.

[0018] The specific behavior is, for example, a behavior designated by a user of the monitoring system 1. When executing the detection process, the user of the monitoring system 1 can designate the behavior that the user wants the monitoring system 1 to detect. The user may designate the behavior via, for example, an input unit 103 (described in detail later) included in the information processing device 100.

[0019] Examples of specific actions include the following. For example, if the user intends to use the surveillance system 1 installed in a store to detect crimes, the user sets theft, violence, etc. as specific actions. If the user intends to use the surveillance system 1 installed in a store to check the purchasing behavior of customers, the user sets actions such as picking up an item or putting it in a basket as specific actions. If the user intends to use the surveillance system 1 installed on a road to monitor traffic, the user sets congestion caused by a large number of vehicles, or vehicle accidents, etc. as specific actions.

[0020] [Configuration of information processing device 100] Next, a description will be given of the configuration of information processing device 100 according to the present embodiment. Fig. 2 is a diagram for explaining the configuration of information processing device 100 according to the present embodiment.

[0021] The information processing device 100 is a so-called computer, and includes a main body unit 101, a display unit 102, and an input unit 103.

[0022] The display unit 102 is a device that allows the information processing device 100 to present information to the user. As the display unit 102, for example, a liquid crystal display (LCD: Liquid Display Panel) can be used.

[0023] The display unit 102 may include multiple display screens. The display unit 102 can be used to display video images captured by the surveillance camera 110 and to display the results of detection of specific behaviors using the video images.

[0024] The input unit 103 is a device that allows the information processing device 100 to receive user input instructions. The input unit 103 is a keyboard and a pointing device. As the pointing device, a mouse may be used, or a device other than a mouse, such as a trackball, may be used.

[0025] A touch pad may also be used as the input unit 103. In this case, the touch pad is attached so as to cover the display screen of the display unit 102, whereby the display unit 102 and the input unit 103 form a touch panel.

[0026] 2, the main body 101 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a NIC (Network Interface Card) 204, and a storage unit 205. These components are connected to each other via an internal bus 211 so as to be able to communicate with each other.

[0027] When the main body 101 is powered on, the CPU 201 reads and starts a boot program from the ROM 202, and executes software such as an OS (Operating System) and a specific behavior detection program read from the storage unit 205, using the RAM 203 as a working storage area.

[0028] The ROM 202 and the RAM 203 are semiconductor memories. In particular, the ROM 202 is a non-volatile semiconductor memory.

[0029] The NIC 204 executes a process for communicating with other devices via a communication network such as a LAN (Local Area Network) or the Internet. This communication may be wired communication or wireless communication. In addition, the communication may be performed by directly connecting another device to the information processing device 100, such as a USB (Universal Serial Bus) device.

[0030] The storage unit 205 is a large-capacity storage device, and may be not only a storage device built into the information processing device 100, such as a hard disk drive (HDD), but also an external storage device, such as a cloud storage.

[0031] In addition to storing the above-mentioned software, the memory unit 205 also stores video images captured by the surveillance camera 110, as well as parameters and learning data of a neural network, which is a machine learning model used in the specific behavior detection program, as described below, and detection results by the specific behavior detection program.

[0032] CPU 201 accesses display unit 102, input unit 103, and surveillance camera 110 via internal bus 211. In addition, video images captured by surveillance camera 110 are stored in storage unit 205 via cable 120 and internal bus 211.

[0033] [Specific behavior detection program] Next, a description will be given of a specific behavior detection program executed in the information processing device 100. The specific behavior detection program performs a specific behavior detection process that detects an object exhibiting a specific behavior by using a machine learning model, in particular, a neural network.

[0034] The specific behavior detection program has a configuration for an inference stage for executing the specific behavior detection process, and a configuration for a pre-learning stage for realizing the inference stage. In the following description, the configuration of the specific behavior detection program will be described separately for the pre-learning stage and the inference stage.

[0035] (Pre-learning stage configuration) 3 is a diagram for explaining the configuration and processing flow of the specific behavior detection program in the pre-learning stage. The specific behavior detection program has a feature point extraction unit 301, a behavior calculation unit 302, a behavior aggregation unit 303, a behavior classification learning unit 304, a character string calculation unit 305, and a distance learning unit 306 as components in the pre-learning stage.

[0036] In the pre-learning stage, a learning dataset is input to the specific behavior detection program. The learning dataset is a set of a large amount of video data and behavior label data that indicates what behaviors the objects in the video are performing.

[0037] The video data included in the training dataset does not have to be video captured by the surveillance camera 110. In other words, the training dataset may include video captured at a location other than where the surveillance camera 110 is installed.

[0038] As described above, one action label included in the learning dataset is paired with one video. The action label is a label that indicates, in short, what action the object is performing in the video as a whole. The action label is composed of a character string that indicates an action, such as "running on treadmill," "wrestling," or "belly dancing." In other words, if the video included in the learning dataset is a video of objects wrestling as a whole, "wrestling" is assigned as the action label.

[0039] Since the learning dataset does not have to be images captured by the surveillance camera 110, an already existing dataset can be used as the learning dataset used in pre-learning of the specific behavior detection program of the information processing device 100 according to the present disclosure. This makes it possible to significantly reduce the effort required for generating a learning dataset, such as acquiring video data using the surveillance camera 110 for pre-learning and assigning behavior labels to the video data acquired from the surveillance camera 110.

[0040] When the feature point extraction unit 301 receives an input of a video, it extracts feature points of an object (e.g., a person) for each frame constituting the video by neuro-computation using a machine learning model, and generates feature point information related to the feature points. The feature points are, for example, joint points constituting the skeleton of the object.

[0041] The feature point information includes, for example, at least any one of the time when the feature point extraction unit 301 detected the feature point (joint point), the coordinates of the joint point, a likelihood score indicating that the joint point has been detected with high probability, a label indicating the type of object, an attribute indicating the type of joint point, and an attribute indicating the appearance of the object.

[0042] The coordinates of a joint point are the X and Y coordinate values ​​of the joint point in a frame and the frame number of the frame.

[0043] The likelihood score of a joint point is the probability that the joint point is likely detected. If it is certain that the joint point is located at the detected coordinate, the probability is close to 1, and if it is not certain, the probability is small.

[0044] The label representing the type of object is information representing the class to which the object including the joint point belongs. For example, if the label value is "0", the object represents a "person", if it is "1", it represents a "shovel", if it is "2", it represents a "racket", etc.

[0045] The attribute of a joint point is the class to which the joint point belongs. For example, if the attribute of a joint point is "0", the joint point is an "elbow", if it is "1", it is a "wrist", if it is "2", it is a "shoulder", etc.

[0046] The attribute representing the appearance of an object is a vector that abstractly quantifies information that identifies the appearance of the object that includes the joint point.

[0047] The type of object, the type of joint, and the appearance of the object may be represented by a vector quantity having multiple values ​​as components.

[0048] For example, OpenPose can be used as the feature point extraction unit 301. OpenPose is software developed by the Center for Technology Transfer and Enterprise Creation (CTTEC) of Carnegie Mellon University in the United States, and is capable of detecting the joint points of multiple people and the connection relationships between the joint points in real time (see Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, Internet https: / / arxiv.org / abs / 1812.08008).

[0049] In OpenPose, coordinates corresponding to the positions of a person's nose, heart, left and right shoulders, elbows, wrists, waists, knees, ankles, eyes, and ears are extracted as joint points. Note that the feature point extraction unit 301 according to the present disclosure is not limited to OpenPose, and information related to joint points may be extracted using a machine learning model other than OpenPose.

[0050] It is desirable for the feature point extraction unit 301 to include at least one of a Convolutional Neural Network (CNN) and a self-attention mechanism.

[0051] The feature point extraction unit 301 outputs feature point information extracted from the video image to the behavior calculation unit 302 .

[0052] Fig. 4 is a diagram showing a model of the XY coordinates of joint points of a person walking indoors, which are included in the feature point information extracted by feature point extraction unit 301. As shown in Fig. 4, the joint coordinates of the person walking indoors are extracted for each of frames #1 to #4, constituting the feature point information.

[0053] In addition to joint points, the feature points extracted by the feature point extraction unit 301 may be points included in the contour of an object, for example, vertices of a circumscribing rectangle that surrounds an object. In this case, the coordinates of the vertex are the XY coordinate values ​​of the vertex in the frame and the frame number. The likelihood score of the vertex is a probability that the vertex is likely to be detected. The label of the object is information indicating the class to which the object surrounded by the circumscribing rectangle that has the vertex belongs. The attribute of the vertex indicates the position of the vertex in the circumscribing rectangle that has the vertex. For example, if the attribute of the vertex is "0", the vertex is the "upper left" vertex of the circumscribing rectangle, and if it is "1", the vertex is the "upper right" vertex.

[0054] When extracting information about vertices, for example, YOLO (You Only Look Once) can be used. YOLO is a deep neural network (DNN) that extracts a bounding box surrounding an object included in an image and a predicted class probability to which the object belongs.

[0055] DNN refers to a multi-layer neural network, particularly one with four or more layers. DNNs are expected to be able to extract behavioral features with high accuracy, and are therefore used in a variety of fields beyond image recognition processing (Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection”, Internet<https: / / arxiv.org / abs / 1506.02640> (See ). Machine learning models other than YOLO may be used to extract information about the vertices of the bounding rectangle.

[0056] The behavior calculation unit 302 extracts behavior features based on the feature point information acquired from the feature point extraction unit 301. The behavior features are, for example, vector quantities. The vectors are vectors that abstractly digitize information that identifies the behavior of an object, such as a person, related to the behavior features. The behavior features are vectors of fixed length in the sense that the number of components is fixed.

[0057] A deep neural network (DNN) can be used for the action calculation unit 302. In this embodiment, as the DNN, for example, Structured Keypoint Pooling can be used to extract action features from feature point information (Ryo Hachiuma, Fumiaki Sato, Taiki Sekii, “Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling”, Internet<https: / / arxiv.org / abs / 2303.15270> However, in the present disclosure, the DNN is not limited to the above example, and any appropriate DNN may be used.

[0058] The PointNet used in the behavior calculation unit 302 is, for example, a Permutation Equivariant neural network.

[0059] In the pre-learning stage, the behavior calculation unit 302 performs machine learning using feature point information extracted by the feature point extraction unit 301 based on video images included in a learning dataset prepared in advance. Parameters set by the machine learning are stored in, for example, the storage unit 205.

[0060] The behavior calculation unit 302 outputs the behavior feature amount to the behavior aggregation unit 303 for each frame and for each object appearing in each frame.

[0061] The behavior summarizing unit 303 summarises the behavior feature amounts calculated for each frame by the behavior calculation unit 302 over the entire video, and calculates one aggregated feature amount for the entire video.

[0062] The behavior aggregating unit 303 calculates the aggregated feature amount by using, for example, Max Pooling. Max Pooling is a process of aggregating by extracting the maximum value in a certain region in the feature amount space.

[0063] The behavior aggregation unit 303 aggregates individual behavior features from the behavior features acquired from the behavior calculation unit 302, for example, for each frame or for each object, and further aggregates the behavior features for the entire video, thereby calculating an aggregated feature.

[0064] The behavior aggregator 303 is, for example, a permutation invariant neural network.

[0065] The behavior classification learning unit 304 performs behavior classification to classify aggregated feature amounts in units of video images into specific classes.

[0066] The character string calculation unit 305 calculates character string features based on the action labels included in the learning dataset. As described above, the action labels are character strings that indicate what action the object is taking in the entire video. The character string calculation unit 305 calculates character string features from the character strings of the action labels using, for example, Transformer.

[0067] Transformer performs various processes, such as morphological analysis, on an input string of characters and calculates features, including the relationships between words contained in the string.

[0068] The Transformer may be configured as a machine learning model that has been trained in advance using a plurality of character strings at a stage prior to the pre-training stage. Furthermore, the character strings used for training the Transformer are not limited to character strings that constitute action labels, and may be a large number of general character strings.

[0069] The distance learning unit 306 performs contrastive learning on the distance between features based on the aggregated features output by the behavior aggregation unit 303 and the character string features output by the character string calculation unit 305. Contrastive learning is one of the self-supervised learning methods, and can perform self-learning based on a huge amount of data using a mechanism for comparing data without labeling. In contrastive learning, features are learned so that the aggregated features are closer to character string features of correct behavior labels and farther from character string features of different (incorrect) behavior labels. This makes it possible to represent the aggregated features obtained by aggregating the behavior features of each object in the entire image and the character string features corresponding to the character strings of the behavior labels in the same feature space. The parameters set by machine learning are stored in, for example, the storage unit 205.

[0070] As described above, the specific behavior detection program performs pre-learning using a learning data set of video images and only one behavior label assigned to the entire video image. The pre-learning of the specific behavior detection program is a multitask learning including behavior classification by the behavior classification learning unit 304 and contrast learning by the distance learning unit 306.

[0071] This allows the specific behavior detection program to efficiently learn when a video and a behavior label are input so that it can publicly determine whether the behavior label is appropriate as a string indicating the behavior of the entire video.

[0072] (Inference stage configuration) After the above-mentioned learning in the pre-learning stage, in the inference stage, the specific behavior detection program detects an object that is performing an arbitrary behavior specified by the user from among the objects appearing in the video. The arbitrary behavior specified by the user is a behavior indicated by an arbitrary character string input by the user via the input unit 103. In other words, in the inference stage, the behavior that the specific behavior detection program should detect is unrelated to the behavior indicated by the behavior label for each video included in the learning dataset in the pre-learning stage. In other words, in the inference stage, even if the user specifies a behavior that is completely different from the behavior indicated by the behavior label learned in the pre-learning stage, the specific behavior detection program detects an object performing the specified behavior.

[0073] Here, the input in the inference stage is a video image captured by the surveillance camera 110 and a character string input by the user via the input unit 103. As described above, the character string input by the user is a character string indicating an arbitrary behavior that the user wishes to detect. The character string input by the user may be the same as the character string indicating multiple types of behavior labels included in the learning dataset used for learning the specific behavior detection program in the pre-learning stage, or may be a completely different character string.

[0074] As a specific example, if the behavioral label strings contained in the training dataset are "running on treadmill," "wrestling," and "belly dancing" as in the example described above, the string entered by the user may be a string that is completely unrelated to the behavioral label strings, such as "violence."

[0075] 5 is a diagram for explaining the configuration and processing flow of the specific behavior detection program in the inference stage. The specific behavior detection program in the inference stage has a feature point extraction unit 401, a behavior calculation unit 402, a character string calculation unit 403, a score calculation unit 404, and a detection result output unit 405.

[0076] The feature point extraction unit 401 performs the same operation as the feature point extraction unit 301 in the pre-learning stage. That is, when the feature point extraction unit 401 receives an input of a moving image, it extracts feature points of an object (e.g., a person) for each frame constituting the moving image by neuro-computation using a machine learning model, and generates feature point information related to the feature points. The feature points are, for example, joint points constituting the skeleton of an object.

[0077] The behavior calculation unit 402 extracts behavior features using a trained model to which parameters trained in the pre-learning stage are applied, based on the feature point information acquired from the feature point extraction unit 401. Here, the trained model used by the behavior calculation unit 402 in the inference stage is fixed by applying the parameters of the behavior calculation unit 302 to which machine learning was performed in the pre-learning stage. In other words, the behavior calculation unit 402 does not perform new learning in the inference stage, but calculates and outputs behavior features based on the input feature point information.

[0078] In this way, since the behavior calculation unit 402 does not perform new learning in the inference stage, the information processing device 100 requires fewer calculation resources in the inference stage. This makes it possible to provide the information processing device 100 at low cost.

[0079] The behavior calculation unit 402 outputs the behavior feature amount for each object shown in each frame to the score calculation unit 404.

[0080] The character string calculation unit 403 calculates character string features based on a character string input by a user, using a trained model to which parameters trained in the pre-learning stage are applied. The trained model used by the character string calculation unit 403 in the inference stage is fixed by applying parameters of the character string calculation unit 305 to which machine learning was performed in the pre-learning stage. In other words, like the behavior calculation unit 402, the character string calculation unit 403 does not perform new learning in the inference stage, but calculates and outputs character string features based on an input character string.

[0081] In this way, since the character string calculation unit 403 does not perform new learning in the inference stage, the information processing device 100 requires fewer calculation resources in the inference stage. This makes it possible to provide the information processing device 100 at low cost.

[0082] The score calculation unit 404 calculates a detection score for each object based on the behavior feature for each object acquired from the behavior calculation unit 402 and the character string feature of an arbitrary character string designated by the user acquired from the character string calculation unit 403.

[0083] The score calculation unit 404 represents, for example, the behavior feature and the character string feature in the same feature space and calculates the cosine similarity between the behavior feature and the character string feature. That is, the detection score calculated by the score calculation unit 404 is the cosine similarity between the behavior feature and the character string feature.

[0084] The detection result output unit 405 outputs the detection result based on the detection score acquired from the score calculation unit 404. The method of outputting the detection result by the detection result output unit 405 is not particularly limited in the present disclosure, but the detection result output unit 405 may, for example, extract an object whose detection score is equal to or greater than a predetermined threshold, and output the extracted object and the detection score as a set.

[0085] The detection result output by the detection result output unit 405 is displayed, for example, on the display unit 102. In the present disclosure, the display method on the display unit 102 is not limited, but for example, it is sufficient if a list of detected objects (objects determined to be performing a behavior specified by the user) and their detection scores is displayed.

[0086] <Action and effect> As described above, the information processing device 100 according to an embodiment of the present disclosure includes a behavior calculation unit 402 that calculates behavioral features related to the behavior of each object appearing in a moving image detected for each frame of the moving image, an input unit 103 that accepts input of a string indicating an arbitrary behavior, a string calculation unit 403 that calculates string features related to the string, and a score calculation unit 404 that calculates a detection score indicating the degree to which the object is performing the behavior indicated by the string based on the behavior features and the string features.

[0087] According to the information processing device 100 according to the embodiment of the present disclosure, the trained model used by the behavior calculation unit 402 to calculate the behavior feature amount is trained in advance using training data unrelated to the behavior indicated by the character string accepted by the input unit 103. For this reason, as the training data for the trained model, a video image including a behavior unrelated to the behavior indicated by the character string accepted by the input unit 103 can be used. As a result, even if a behavior completely different from the behavior indicated by the behavior label learned in the pre-training stage is input to the input unit 103 in the inference stage, an object performing the behavior indicated by the input character string can be detected without any problem. Therefore, as the training data for training the trained model used by the behavior calculation unit 402 to calculate the behavior feature amount, there is no need to perform training specialized for the behavior indicated by the character string that the input unit 103 will accept in the inference stage in the pre-training stage. For this reason, for example, an existing large-scale data set can be used as the training data for the training model in the pre-training stage, so that the training cost of the trained model can be reduced.

[0088] Furthermore, the information processing device 100 according to the embodiment of the present disclosure provides the following effects. It is assumed that the video images input to the time-space behavior detection model are different between the pre-learning stage and the inference stage. For example, if the appearance of an object captured in the video images used for learning in the pre-learning stage differs from the appearance of an object captured in the video images input in the inference stage (e.g., different clothing), the difference in appearance may affect the behavior detection accuracy of the time-space behavior detection model.

[0089] In the information processing device 100 according to the embodiment of the present disclosure, the input to the behavior calculation unit 402 is, for example, feature point information indicating the skeleton of an object, and is not a moving image. A situation in which the appearance of the input in the pre-learning stage of the information processing device 100 differs significantly from that of the input in the inference stage is unlikely to occur. This makes it possible to prevent the occurrence of so-called domain shift (a significant decrease in inference performance due to differences in the properties of data used in learning and data to be inferred in the inference stage) in the information processing device 100.

[0090] In the information processing device 100 according to the embodiment of the present disclosure, the behavior calculation unit 402 has a trained model that is trained in advance to input feature point information and output behavior feature amounts. This trained model uses parameters trained by the behavior calculation unit 302 in the pre-training stage, and the parameters are fixed in the inference stage.

[0091] Furthermore, in the information processing device 100 according to the embodiment of the present disclosure, in the pre-learning stage, the behavior calculation unit 302 performs learning to calculate behavior features based on feature point information using various video images included in the learning dataset. Also, in the pre-learning stage, the distance learning unit 306 performs contrast learning based on aggregate features in which behavior features are aggregated across the entire video image and character string features. Furthermore, in a stage prior to the pre-learning stage, the character string calculation unit 305 performs learning to calculate character string features using multiple character strings. Due to these pre-learnings, the behavior calculation unit 402, the character string calculation unit 403, and the score calculation unit 404 do not perform further learning in the inference stage, so that unnecessary learning costs in the inference stage can be omitted.

[0092] In the information processing device 100 according to the embodiment of the present disclosure, the character strings input in the pre-learning stage are character strings of action labels indicating actions of video images included in a learning dataset, whereas the character strings input to the information processing device 100 in the inference stage are arbitrary character strings designated by a user through the input unit 103 or the like.

[0093] As described above, there may be no relationship between the character string of the behavior label input in the pre-learning stage and the character string input to the information processing device 100 in the inference stage. That is, an arbitrary character string designated by the user in the inference stage may not be the character string of the behavior label learned in the pre-learning stage. In other words, the character string input to the information processing device 100 in the inference stage may be unknown to the trained model of the information processing device 100.

[0094] With this configuration, the learning dataset used in the pre-learning stage of the information processing device 100 may be a learning dataset that includes video images with behavioral labels that are not related to character strings specified by the user in the inference stage. Therefore, as described above, the cost of preparing the learning dataset is low, and since it is not necessary to learn in advance about character strings specified by the user in the inference stage, the learning cost can be reduced.

[0095] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure described in the claims.

[0096] In the above-described embodiment, the information processing device 100 and the surveillance camera 110 are separate from each other, but the present disclosure is not limited to this. For example, the information processing device may include a surveillance camera.

[0097] In the above-described embodiment, each component of the information processing device 100 may be arranged on a cloud, that is, the information processing device 100 may be configured as a so-called cloud server.

[0098] In the above-described embodiment, the information processing device 100 has the feature point extraction units 301, 401, but the information processing device of the present disclosure may not have a configuration for extracting feature points from an input video. In this case, the information processing device according to the present disclosure may receive, as an input, feature point information extracted by an external device based on a video, and perform various processes in the pre-learning stage and the inference stage based on the feature point information. [Explanation of symbols]

[0099] 1. Surveillance System 100 Information processing device 101 Main body 102 Display section 103 Input section 110 Surveillance Camera 120 Cable 201 CPU 202 Display section 203 Input section 204 NIC 205 Storage section 211 Internal Bus 301 Feature Point Extraction Unit 302 Behavior calculation department 303 Action Aggregation Department 304 Behavior Classification Learning Department 305 String calculation section 306 Distance Learning Department 401 Feature point extraction unit 402 Behavior Calculation Department 403 String calculation part 404 Score Calculation Department 405 Detection result output section

Claims

1. a behavior calculation unit that calculates a behavior feature amount related to a behavior of each object captured in the video detected for each frame of the video; an input unit that accepts input of a character string indicating an arbitrary action; a character string calculation unit for calculating a character string feature amount relating to the character string; a score calculation unit that calculates, for each of the objects, a detection score indicating a degree to which the object is performing the behavior indicated by the character string based on the behavior feature amount and the character string feature amount; An information processing device comprising:

2. The behavior calculation unit calculates the behavior feature amount based on feature point information related to feature points of the object. The information processing device according to claim 1 .

3. The behavior calculation unit has a trained model that is trained in advance to input the feature point information and output the behavior feature amount. The information processing device according to claim 2 .

4. The behavior calculation unit calculates the behavior feature amount using feature extraction neuro-operation. The information processing device according to claim 3 .

5. The behavior calculation unit is composed of a deep neural network (DNN) having permutation-equivariant characteristics. The information processing device according to claim 4.

6. the behavior calculation unit is trained in advance by a contrastive learning that compares behavior features of objects captured in a plurality of video images with character string features that are character string features included in behavior labels that are given in advance to each of the plurality of video images; The information processing device according to claim 3 .

7. the behavior calculation unit has been trained in advance by the contrastive learning in which an aggregate feature obtained by aggregating behavior features of each object in each frame of the video over the entire video is compared with the character string feature, The information processing device according to claim 6.

8. The aggregated features are calculated using feature aggregated neuro-computing. The information processing device according to claim 7.

9. The aggregate feature is calculated using a deep neural network (DNN) having permutation-invariant properties. The information processing device according to claim 8.

10. the score calculation unit calculates a cosine similarity between the behavior feature and the character string feature to obtain the detection score. The information processing device according to claim 1 .

11. The feature points are joint points for each of the objects. The information processing device according to claim 2 .

12. The feature point is at least one of the points included in the contour of each of the objects. The information processing device according to claim 2 .

13. The feature points are vertices of a bounding rectangle of the object. The information processing device according to claim 12.

14. The feature point is at least one of points included in the contour in a polar coordinate system having a center position of the object as an origin. The information processing device according to claim 12.

15. The feature point information includes a time when the object was detected. The information processing device according to claim 2 .

16. The feature point information is a likelihood score indicating that the feature point is likely to be detected; and a feature vector indicating at least one of a type of the object, a type of the feature points, and an appearance of the object; Including at least one of The information processing device according to claim 2 .

17. 1. A computer-implemented information processing method, comprising: calculating a behavior feature amount relating to a behavior of each object appearing in the video, the behavior feature amount being detected for each frame of the video; Accepts input of a string indicating any action, Calculating character string features for the character string; calculating, for each of the objects, a detection score indicating a degree to which the object is performing the behavior indicated by the character string based on the behavior feature amount and the character string feature amount; Information processing methods.

18. A step of calculating a behavior feature amount relating to a behavior of each object captured in the video, the behavior feature amount being detected for each frame of the video; A step of accepting an input of a character string indicating an arbitrary action; calculating a character string feature amount for the character string; calculating, for each of the objects, a detection score indicating a degree to which the object is performing the behavior indicated by the character string based on the behavior feature and the character string feature; A program that causes a computer to execute the following.

Images