Information processing apparatus, information processing method and program

Abstract

To provide an information processing apparatus, an information processing method and a program, allowing for reducing the burden on a user when realizing the recognition of a time-series behavior.SOLUTION: A method includes: acquiring a plurality of video images to recognize a plurality of behaviors of a subject from the acquired videos images and performing, as recognition processing of a single behavior, each processing of human-body detection, human-body tracking, position presumption and behavior recognition from a video image determined as an object of processing, using a model obtained by deep learning. An information processing apparatus furthermore generates a list of single-behavior information, stores a list of single-behavior information generated and determines whether or not there remains a video image to be determined as an object of processing.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In video human behavior recognition technology, a time-series image of multiple frames with a human region cut out is generally used as input, features of the human region are extracted, and behavior is determined from the time-series features. A model trained by machine learning is used to extract the features, and training data in which time-series images are labeled with behavioral labels such as "raising a hand" or "squatting" is used to train the model.

[0003] In Non-Patent Document 1, action recognition is realized by acquiring a model through deep learning using a data set in which general actions are defined as action labels. Patent Document 1 also proposes a learning method in which detailed actions such as drinking water are defined, and higher-level actions such as bringing a hand close to the face are defined by clustering action labels based on the similarity of time-series features, and then used for learning.

[0004] However, when applying behavior recognition to various use cases, the definition of the time-series behavior to be recognized differs depending on the application. For example, when recognizing a customer's purchasing behavior, it is necessary to recognize the behavior of picking up a product from a shelf and looking at the label, and when analyzing factory work, it is necessary to recognize the behavior of picking up a part from a tray and attaching it. Thus, it is a heavy burden on the user to prepare all the actions for all use cases as behavior labels in advance. Therefore, a method of recognizing time-series behavior that reduces the burden on the user is desired. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2022-72444 [Non-patent literature]

[0006] [Non-Patent Document 1] Sijie Yan et al, "Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition", AAAI 2018, Article No.: 912, page 7444-7452, February 2018 [Non-Patent Document 2] Alexey Bochkovskiy et al, "YOLOv4: Optimal Speed ​​and Accuracy of Object Detection", CVPR2020, April 2020 [Non-Patent Document 3] Yifu Zhang et al, "FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking", CVPR2020, April 2020 [Non-Patent Document 4] Ke Sun et al, "Deep High-Resolution Representation Learning for Human Pose Estimation", CVPR2019, February 2019 [Non-Patent Document 5] Ce Zheng et al, "3D Human Pose Estimation with Spatial and Temporal Transformers" ICCV2021, March 2021 Summary of the Invention [Problem to be solved by the invention]

[0007] When recognizing actions using a machine learning model that has been trained by assigning action labels of the time-series actions to be recognized to videos, it is necessary to assign all action labels, which places a heavy burden on the user.

[0008] An object of the present invention is to reduce the burden on a user when realizing time-series behavior recognition. [Means for solving the problem]

[0009] In order to achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: an acquisition means for acquiring a plurality of videos, a recognition means for recognizing a plurality of behaviors of a subject from the videos, a first evaluation means for evaluating a time-series correlation between the recognized behaviors, a grouping means for grouping the set of behaviors based on the evaluation of the time-series correlation, and an output means for outputting the grouped set of behaviors. Effect of the Invention

[0010] To reduce the burden on users when realizing time-series behavior recognition. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram showing an example of a hardware configuration of the information processing device according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the first embodiment. [Diagram 3] FIG. 4 is a diagram showing an example of a score used in recognition of a single action according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a single action information list according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a time-series behavior graph according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a time-series label according to the first embodiment. [Figure 7] 11 is a flowchart showing an example of a single behavior information list generation process according to the first embodiment. [Figure 8] 5 is a flowchart showing an example of a graph generation process according to the first embodiment. [Figure 9] FIG. 4 is a diagram showing an example of a selection GUI by a user according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing an example of an editing screen for a time-series action according to the first embodiment. [Figure 11] 4 is a flowchart showing an example of a generation process of a time-series action according to the first embodiment. [Figure 12] 4 is a flowchart showing an example of a time-series behavior recognition process according to the first embodiment. [Figure 13] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 14] FIG. 11 is a diagram showing an example of a search GUI according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of a search result display GUI according to the second embodiment. [Figure 16] 11 is a flowchart showing an example of a search process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] [Embodiment 1] The information processing device 100 according to this embodiment recognizes a plurality of actions (single actions) of a subject from a video and evaluates the time-series correlation of the recognized single actions. Next, based on the evaluation of the time-series correlation, the information processing device 100 generates and outputs a set of single actions to which a label (time-series label) used in learning a machine learning model is assigned. Hereinafter, such an information processing device 100 will be described.

[0014] The hardware configuration of the information processing device 100 according to the present embodiment is shown in FIG. 1. FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to the present embodiment. The information processing device 100 includes an input unit 101, a display unit 102, a network I / F unit 103, a CPU 104, a RAM 105, a ROM 106, a HDD 107, and a data bus 108. The CPU 104 reads out a control computer program stored in the ROM 106, loads it into the RAM 105, and executes various control processes. The RAM 105 is used as a temporary storage area for storing a program executed by the CPU 104, a work memory, and the like. The network I / F unit 103 connects to an external device (not shown) so that various information can be transmitted and received via the Internet. The HDD 107 stores various information such as image data, setting parameters, or various programs. Image data and the like received from an external device via the network I / F unit 203 are transmitted and received to the CPU 104, the RAM 105, and the ROM 106 via the data bus 108.

[0015] Image processing of image data is realized by CPU 104 executing an image processing program stored in ROM 106 or HDD 107. In addition, HDD 107 is capable of inputting data from an external device via network I / F unit 103.

[0016] The input unit 101 is, for example, a keyboard and mouse, or a touch panel, and receives input from a user. The display unit 102 is, for example, a liquid crystal display, and displays information such as the results of various processes. The input unit 101 and the display unit 102 are also connected to other functional units via a data bus 108 so as to be able to communicate with each other.

[0017] The functions of the information processing device 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the information processing device 100 according to this embodiment. The information processing device 100 has a video acquisition unit 201, a behavior recognition unit 202, a graph generation unit 203, a label generation unit 204, and a time-series behavior recognition unit 205.

[0018] The image acquisition unit 201 acquires an image to be processed. The image acquisition unit 201 according to this embodiment may acquire an image stored in the HDD 107, or may acquire an image captured by an external camera or an image stored externally via the network I / F unit 103.

[0019] The behavior recognition unit 202 processes the video acquired by the video acquisition unit 201, recognizes single behaviors of the subject in the video, and generates time-series single behavior information that compiles the single behavior information together with time-series information. Here, the single behavior information is the recognition result of a general-purpose single behavior that is divided into one type of motion (e.g., several seconds in time), such as "reaching out a hand forward" or "looking at one's hand." An example of a detection score assigned by the behavior recognition unit 202 when recognizing a single behavior is shown in FIG. 3.

[0020] In this embodiment, the detection score is a scalar value ranging from 0 to 1, and the closer to 1 the detected behavior is, the higher the likelihood that the behavior is the detected behavior. The single behavior information list refers to information that lists information indicating a single behavior in a video in chronological order, as shown in Fig. 4. Furthermore, the single behavior label is associated with an image of a human region or a joint position used when learning the model, along with chronological information.

[0021] The graph generating unit 203 evaluates the time-series correlation of the single actions recognized by the action recognition unit 202. The graph generating unit 203 also groups the set of single actions based on the evaluation of the time-series correlation. In this embodiment, the graph generating unit 203 evaluates the time-series correlation between the single actions in the single action information list that is listed in chronological order and stored in the HDD, and groups the single actions based on the evaluation. The set of single actions according to this embodiment is an action defined by combining a plurality of single actions, and is, for example, an action that is divided into actions lasting several seconds to several minutes. Such a set of single actions may be referred to as a "time-series action" below. Note that, in this description, the result of grouping based on the evaluation of the correlation is shown in a graph (time-series action graph).

[0022] The evaluation process of the time-series correlation of single actions by the graph generating unit 203 will be described below. The time-series correlation of single actions according to this embodiment is evaluated as, for example, the probability that single actions occur consecutively in time series. That is, for example, the graph generating unit 203 can evaluate the probability that one single action occurs next to another single action based on the single action information list stored in the HDD 107. In this case, the graph generating unit 203 may group the single actions as a set when the probability is equal to or greater than a predetermined threshold. The evaluation process of the time-series correlation performed by the graph generating unit 203 and the generation process of the graph based on the evaluation will be described later with reference to FIG. 5 and FIG. 8.

[0023] The evaluation of the time-series correlation according to this embodiment is based on the probability that single actions occur consecutively in a time series. The evaluation of the correlation according to this embodiment is expressed by a graph using nodes and edges as shown in Fig. 5. In Fig. 5, the single actions are represented by nodes for each type, and single actions with high correlation are connected by edges.

[0024] The label generating unit 204 outputs a set of single actions (time-series actions) grouped by the graph generating unit 203. The label generating unit 204 can group, for example, a predetermined number of actions as the time-series actions. For example, the label generating unit 204 may select one single action (for example, based on a user input) and generate a time-series action by grouping a predetermined number of actions including the selected single action based on the time-series correlation between the single actions. Details of the generation process of the time-series actions will be described later with reference to FIG. 11.

[0025] Also, for example, the label generating unit 204 may generate a label (time-series label) that associates a label name with a time-series action by presenting the time-series action to the user via a GUI as shown in Fig. 10 described later and receiving an input of a label name for the presented set. Here, the time-series label is information that lists a set of single actions grouped in chronological order and their label names as shown in Fig. 6.

[0026] Furthermore, the label generating unit 204 can generate labels to be used for learning based on a user input via the input unit 101.

[0027] The time-series behavior recognition unit 205 recognizes the time-series behavior of the subject from the video based on the time-series label. The time-series behavior recognition unit 205 according to the present embodiment can determine whether the video contains the time-series behavior of the subject based on the matching rate between the behavior detected from the video to be processed and the set of single behaviors indicated by the time-series label. Such a determination process based on the matching rate will be described later with reference to FIG. 12.

[0028] Furthermore, the time-series behavior recognition unit 205 may recognize time-series behavior from a video using a learning model that has been trained to recognize the time-series behavior generated by the label generation unit 204. Such a learning model can be obtained, for example, by updating parameters of the learning model using a general machine learning technique to recognize and output the time-series behavior in a video, using a video including time-series behavior to which a time-series label has been added as correct answer data.

[0029] Next, the generation process of the single behavior information list in this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the generation process of the single behavior information list by the information processing device 100. The process shown in Fig. 7 starts, for example, when a start instruction is input by the user (for example, when a start operation is performed via the input unit 101).

[0030] In S701, the video acquisition unit 201 acquires a video. In this embodiment, it is assumed that one video to be processed in S702 to S704 is acquired in S701, and multiple videos are processed by the loop processing of S701 to S705. However, it is also possible to adopt an embodiment in which multiple videos are acquired in S701, and the videos to be processed are selected in sequence to perform a series of processes.

[0031] In S702, the behavior recognition unit 202 recognizes multiple single behaviors of the subject from the acquired video. The behavior recognition unit 202 according to this embodiment performs each process of human body detection, human body tracking, posture estimation, and behavior recognition from the video to be processed as a single behavior recognition process, using a model acquired by deep learning. Here, the single behavior recognition process by the behavior recognition unit 202 can be performed by a known technology that recognizes the behavior of the subject in the image. An example of each process performed by the behavior recognition unit 202 will be described below.

[0032] In the human body detection process, the behavior recognition unit 202 can obtain a circumscribing rectangle of a person by using the method of Non-Patent Document 2 with an image as an input. In the human body tracking process, the behavior recognition unit 202 can obtain an identifier for each person by using an image and a circumscribing rectangle of a person as an input by using the method of Non-Patent Document 3. In the posture estimation process, the behavior recognition unit 202 can obtain two-dimensional and three-dimensional joint positions of a person by using an image and a circumscribing rectangle of a person as an input by using the methods of Non-Patent Documents 4 and 5. In the behavior recognition process, the behavior recognition unit 202 can recognize a single behavior of a person by using three-dimensional joint positions of the person in multiple frames as an input by using the method of Non-Patent Document 1. Here, in order to input the three-dimensional joint positions of a person in multiple frames together, the human body detection results and joint position information for the same person are combined between frames based on the identifier of the person obtained by the human body tracking process. At this time, it is possible to recognize a single behavior by inputting frame information of a section (for example, several seconds) including an action defined as a single behavior together into a model.

[0033] In this embodiment, the single behavior detection process is described as performing each of the processes of human body detection, human body tracking, posture estimation, and behavior recognition. However, this is not limited to this as long as it is possible to detect a single behavior of a subject in an image. For example, in order to recognize a single behavior, a posture estimation method that directly obtains the joint positions of a human body using only an image as input, or a behavior recognition method that recognizes behavior using an image sequence in which a human region is cut out as input, may be used.

[0034] In S703, the behavior recognition unit 202 generates a single behavior information list. Here, the behavior recognition unit 202 generates a single behavior information list (time-series single behavior information) by listing the single behavior information generated by the behavior recognition unit 202 in chronological order based on the identifier for each person. In S704, the behavior recognition unit 202 stores the generated single behavior information list in the HDD 107.

[0035] In S705, the information processing device 100 determines whether or not there is any video to be processed. If there is any video to be processed, the process returns to S701, and if there is no video to be processed, the process ends. Here, for example, it may be determined that there is any video to be processed when an end instruction is given by the user.

[0036] As described above, the graph generating unit 203 according to this embodiment groups single behaviors based on an evaluation of the time-series correlation of the single behaviors (generates a time-series behavior graph). Hereinafter, the generation process of the time-series behavior graph by the information processing device 100 according to this embodiment will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the generation process of the time-series behavior graph by the graph generating unit 203 according to this embodiment. The process shown in FIG. 8 may be automatically started when a start condition, for example, the process in FIG. 7 is completed, is satisfied, or may be started in response to a start instruction from a user.

[0037] In S801, the graph generating unit 203 acquires one single action information list stored in the HDD 107 as a processing target for S802 to S804.

[0038] In S802, the graph generating unit 203 adds a new node based on the acquired single behavior information list. The graph generating unit 203 extracts single behavior labels whose detection scores are equal to or greater than a preset threshold (e.g., 0.9) from the single behavior information included in the single behavior information list, and adds the extracted single behavior labels (types of single behavior) as nodes in the graph. If there are multiple single behavior labels whose detection scores are equal to or greater than a preset threshold from the single behavior information, a node may be added for each single behavior label, or a node may be added for a combination of single behavior labels that simultaneously exceed the threshold as one label of simultaneous behavior. Conversely, if there is no single behavior label whose detection score is equal to or greater than a preset threshold from the single behavior information, a node may be added with the single behavior label name of the unknown behavior as the section in which the behavior was unknown.

[0039] 8, correlation values ​​are added to edges between nodes in S803, and single actions indicated by those nodes are grouped based on the correlation values ​​in S805. Such processing will be described below.

[0040] In S803, the graph generating unit 203 adds a correlation value to the edge between the nodes. Here, the graph generating unit 203 processes pairs of single behavior information that are in a chronological order among the single behavior information included in the single behavior information list. Then, for the pair to be processed, the node from which the label was extracted in S802 is identified, and the correlation value is added by 1 to the edge between the nodes. The correlation value addition process is performed for each pair of single behavior information. Also, if an edge does not exist between the nodes, an edge is added.

[0041] Here, the edge includes directional information (edge ​​direction). That is, even if there is an edge between a first single action and a second single action, different edges are generated and correlation values ​​are added when the second single action occurs from the first single action and when the second single action occurs from the second single action. However, if the single actions occur consecutively, it may be determined that the correlation is strong regardless of the order, and the edge may be generated without including directional information.

[0042] In S804, the graph generating unit 203 determines whether or not there remains a single action information list to be processed. If there remains a single action information list to be processed, the process returns to S801, and if there remains no single action information list, the process proceeds to S805. Note that, although all single action information lists stored in the HDD 107 are assumed to be processed here, it is also possible to use only single action information lists that satisfy a predetermined condition, such as processing only single action information lists acquired within a predetermined period.

[0043] In S805, the graph generation unit 203 groups the single actions based on the correlation values, reflects the grouping results in the time-series action graph, and ends the processing in Fig. 8. Here, for example, the graph generation unit 203 normalizes the correlation values ​​for each edge added in S803 so that the maximum value is 1 and the minimum value is 0, and can group pairs of single actions whose normalized correlation values ​​are equal to or greater than a predetermined threshold value (e.g., 0.6). In this case, in the time-series action graph, edges between grouped nodes are left and other edges are deleted.

[0044] An example of a time-series behavior graph generated in this way is shown in Figure 5. In Figure 5, for multiple nodes showing a single behavior, edges are drawn between nodes with strong time-series correlations, and nodes that do not have other nodes with such strong correlations are drawn alone. In the following, a node connected to a certain node by an edge is referred to as an adjacent node.

[0045] The label generation unit 204 according to this embodiment outputs a set of single actions grouped by the graph generation unit 203, and generates a time-series label by giving a label name to the set based on a user input. An example of a UI for generating a time-series label by the label generation unit 204 according to this embodiment will be described below with reference to Figs. 9 and 10.

[0046] FIG. 9 is an example of a UI in which a user selects a frame in a video and selects a label assignment target of a time-series behavior. The label assignment target selection UI 900 is composed of a subject selection screen 901, a sequence bar 902, and a frame designation pointer 903. The subject selection screen 901 displays a video and the output of each process by the behavior recognition unit 202. For example, the label generation unit 204 displays a person circumscribing rectangle 906 and a joint position 907 included in the single behavior information superimposed on an image including a subject 904 and an object 905 on the subject selection screen 901. Here, the user can select a display frame to be displayed as the subject selection screen 901 by moving the frame designation pointer 903 via the input unit 101. In addition, the user can instruct extraction of single behavior information of the subject within the person circumscribing rectangle 906 by selecting inside the person circumscribing rectangle 906 on the subject selection screen 901.

[0047] FIG. 10 is a diagram showing an example of a GUI that presents a time-series action to a user and accepts input of a label name for the presented time-series action. The time-series label registration UI 1000 is composed of a time-series action confirmation screen 1001, a time-series label name input form 1005, and a time-series label registration button 1006. In the single action confirmation screen 1002, an image showing each single action included in the time-series action is displayed. In addition, in the single action label confirmation screen 1003, a label name corresponding to the image of the single action displayed in the single action confirmation screen 1002 is displayed. In the example of FIG. 10, an image showing joint position information included in the single action information is displayed in the single action confirmation screen 1002, and a person's movement corresponding to the label of the single action can be presented to the user. In addition, in FIG. 10, each single action included in the time-series action is displayed in order along the time series.

[0048] A user can input an operation to the time-series behavior confirmation screen 1001 via the input unit 101 to perform editing of the single behavior included in the time-series behavior. An example of an editing operation by accepting such a user input will be described below. For example, the time-series order of the single behavior included in the time-series behavior can be edited by the user's drag operation on the single behavior label confirmation screen 1003. Also, for example, the single behavior information can be deleted from the time-series behavior by operating the single behavior information delete button 1004. Furthermore, for example, the time-series label name can be input in the time-series label name input form 1005, and the time-series label registration button 1006 can be pressed to edit and save the time-series label for the displayed time-series behavior.

[0049] In FIG. 10, an example in which only one set of extracted time-series behaviors is displayed is shown, but multiple time-series behaviors may be output and displayed on the time-series label registration UI 1000. In that case, the same operation may be possible for each of the multiple displayed time-series behaviors, or the operation may be possible for one time-series behavior selected by the user. Also, instead of displaying each single behavior side by side as in FIG. 10, the time-series behavior may be displayed in the form of a graph as in FIG. 5. In that case, only the nodes and edges corresponding to the time-series behavior may be displayed as a graph, or the nodes and edges corresponding to the time-series behavior may be highlighted (for example, displayed in color, bold, or blinking) in the entire graph as in FIG. 5.

[0050] In the example of Fig. 10, the information indicating the joint position information is displayed in chronological order on the single action confirmation screen 1002, but this is not particularly limited as long as the information can present a single action included in the time-series action to the user. For example, the information displayed in chronological order may be information synthesized by adjusting the viewpoint using CG of a human model based on three-dimensional joint position coordinates. Also, instead of displaying images separately in chronological order, the single actions included in the time-series information may be displayed as images in the same frame.

[0051] As described above, the label generation unit 204 according to this embodiment outputs a set of single actions grouped by the graph generation unit 203, and generates a time-series label by assigning a label name to the set based on a user input. Here, a process of selecting a set of single actions (time-series actions) to be grouped from the generated time-series action graph will be described. The label generation unit 204 according to this embodiment sets a single action for which a time-series correlation with other single actions is evaluated based on a user input. In the example shown in FIG. 11 described later, a frame in a video is specified based on a user input, and a single action recognized from a subject in the frame is set as a single action for which a time-series correlation with other single actions is evaluated.

[0052] Fig. 11 is a flowchart showing an example of a time-series label generation process by the information processing device 100 according to the present embodiment. The process shown in Fig. 11 may be started, for example, when a user performs an operation to start editing the time-series labels, or when a time-series action is generated by the graph generating unit 203.

[0053] In S1101, the label generation unit 204 sets a single action to be processed (to be grouped with other single actions). Here, the label generation unit 204 can set a single action to be processed (a frame including an action to which a time-series label is to be assigned) based on a user operation to specify a frame via the frame specification pointer 903 of the label assignment target selection UI 900.

[0054] In S1102, the label generation unit 204 sets a subject to be processed (for extracting time-series behavior). Here, the label generation unit 204 can set the subject to be processed based on a user operation to select a subject via the subject selection screen 901 of the label assignment target selection UI 900. Also, for example, the label generation unit 204 may automatically set a subject that satisfies a predetermined condition as the processing target, such as setting a subject having the largest person circumscribing rectangle included in the single behavior information as the processing target.

[0055] In S1103, the label generation unit 204 searches for and extracts single action information that satisfies the conditions from the HDD 107 based on the settings in S1101 and S1102.

[0056] In S1104, the label generation unit 204 extracts time-series actions based on the extracted single action information and the time-series action graph. Here, the label generation unit 204 searches the time-series action graph for a node that corresponds to the single action label included in the extracted single action information. Next, the label generation unit 204 adds to the set of single actions the single action indicated by the adjacent node whose correlation value with the edge of the searched node is equal to or greater than a preset threshold value. Furthermore, the label generation unit 204 repeats the process of searching for similarly highly correlated nodes from the adjacent node and adding them to the set of single actions (for example, until the number of single actions included in the set reaches a predetermined number).

[0057] In this case, the edge has the above-mentioned edge direction, and when an adjacent node is added to the set of single actions, time-series information according to the direction is held. In this case, the process of adding (grouping) a node (single action) that follows the searched node in the time series to the set and the process of adding a node that precedes the searched node in the time series to the set are performed separately. Here, the label generation unit 204 can perform a process of generating a set of single actions so that the total number of the searched node, the node that precedes the searched node in the time series, and the node that follows the searched node in the time series is a predetermined number. If an adjacent node whose correlation value satisfies the threshold value is not found before the predetermined number is reached, the set of single actions at that time is determined as the time-series action. In addition, if there are multiple adjacent nodes whose correlation value is equal to or greater than the threshold value for one node, a separate time-series action is extracted for each of them.

[0058] In addition, although it has been described here that both the process of adding a chronologically preceding node to a set and the process of adding a chronologically succeeding node to a set are performed, it is also possible to perform only one of them. In this case, the label generating unit 204 can obtain, as a user input, which of these processes (or both) to perform, for example, in S1101 to S1102.

[0059] Note that, although the description will be made assuming that one single action to be processed is set in S1101, for example, multiple single actions may be set here, and a set may be generated to include all the set single actions. Also, the cost of an edge may be set so that the higher the correlation value for the edge, the smaller the cost, and a group of nodes included in a path with the smallest cost may be selected as the single action to be included in the time-series action using a graph optimization technique or the like.

[0060] In S1105, the label generation unit 204 sets a label name for the time-series action of the generated set, and ends the processing in Fig. 11. Here, the label generation unit 204 accepts an editing operation of the time-series action and the time-series label name by the user via, for example, the time-series label registration UI 1000 shown in Fig. 10, and registers the time-series label in the HDD 107 based on the content of the operation.

[0061] According to this processing, the subject's actions can be recognized from multiple videos, and a set of actions can be grouped and output based on an evaluation of the time-series correlation between the recognized actions. Therefore, when realizing the recognition of time-series actions, it is possible to reduce the burden on the user by grouping and presenting actions that are highly correlated in time series.

[0062] As described above, in this embodiment, the frames in the video to be processed are specified based on the user's input, but the present invention is not limited to such processing as long as a set of single actions evaluated to have a high chronological correlation can be extracted. For example, the label generation unit 204 may set a period for extracting time-series actions in the video, and extract a set of single actions evaluated to have a strong chronological correlation during the set period (using a graph optimization method, etc.) as time-series actions. Alternatively, for example, the graph generation unit 203 may use a graph optimization method to extract a set of all single actions in chronological order for which edges are generated as shown in FIG. 6 as time-series actions. In this case, the condition for a high correlation may be set more strictly, for example, the threshold value of the normalized correlation value used in grouping may be set to 0.9 or more.

[0063] Fig. 12 is a flowchart showing an example of a process of recognizing a time-series action from a video by the time-series action recognition unit 205 according to this embodiment. The process shown in Fig. 12 is performed in the same manner as in Fig. 7 except that S1201 to S1203 are performed instead of S704 to S705, and therefore a duplicated description will be omitted.

[0064] In S1201, the time-series behavior recognition unit 205 acquires a pre-registered time-series label from the HDD 107. In S1202, the time-series behavior recognition unit 205 compares the single behavior information list generated in S703 with the time-series label acquired in S1201 to determine whether or not a time-series behavior has occurred in the video to be processed.

[0065] The time-series behavior recognition unit 205 can determine that a time-series behavior is occurring in the video to be processed, for example, when the single behavior information list generated from the video contains a set of single behaviors indicated by the time-series labels (i.e., there are ones with matching content and order).

[0066] The time-series behavior recognition unit 205 may also evaluate the matching rate between the behavior detected from the video to be processed and the set of single behaviors indicated by the time-series label, and may determine whether or not the time-series behavior is occurring in the video based on the matching rate. Here, the above-mentioned determination may be performed based on a time-series behavior detection score calculated so that the score is set to the maximum (1) when the above-mentioned contents and order are completely matched, and the score varies depending on the matching rate. For example, when the single behavior information list of the video "pick up a product and look at the label" includes the single behaviors "reach forward, hold something, bring hand close to face, bend hand and hold, look at hand" and the time-series behavior is a set of single behaviors "reach forward, bring hand close to face, bend hand and hold, look at hand", the time-series behavior recognition unit 205 may calculate the time-series behavior detection score to be 0.8, assuming that the content rate of the time-series behavior in the single behavior information list is 80%. In this case, the time-series behavior recognition unit 205 may determine that the time-series behavior is occurring in the video to be processed when the calculated time-series behavior detection score is equal to or greater than a predetermined threshold value.

[0067] In S1203, the time-series behavior recognition unit 205 determines whether or not there is any video to be processed. If there is any video to be processed, the process returns to S701, and if there is no video to be processed, the process ends. Here, for example, if the user issues an end instruction, it may be determined that there is any video to be processed.

[0068] Although the subject according to the present embodiment has been described as a human body, the subject is not limited to this so long as the subject's behavior can be recognized in the video. For example, the subject may be a vehicle, an animal, or a part of a human body.

[0069] [Embodiment 2] The information processing device 1300 according to this embodiment can group a set of single actions based on an evaluation of a time-series correlation by the same process as the information processing device 100. In addition, the information processing device 1300 according to this embodiment obtains a selection of a single action by a user, and presents grouped single actions regarding the selected single action.

[0070] 13 is a block diagram showing an example of a functional configuration of an information processing device 1300 according to this embodiment. The information processing device 1300 according to this embodiment has the same configuration as the information processing device 100 of the first embodiment except that it has a search unit 1301 instead of the label generation unit 204 and the time-series behavior recognition unit 205, and therefore a duplicated description will be omitted.

[0071] The search unit 1301 obtains a selection of a single action by a user. Then, the search unit 1301 presents other single actions that are grouped with the single action selected by the user to form a time-series action as related actions of the selected single action based on an evaluation of a time-series correlation.

[0072] Hereinafter, the related behavior search process by the information processing device 1300 according to this embodiment will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a diagram showing an example of a GUI for performing a search for related behavior by the search unit 1301 according to this embodiment. Fig. 15 is a diagram showing an example of a GUI for displaying such search results.

[0073] 14 is a diagram showing an example of a GUI for selecting a single behavior as a search criterion and executing a search. The related behavior search UI 1400 is composed of a single behavior confirmation screen 1401, a single behavior check box 1402, and a search instruction button 1403. The search unit 1301 displays the single behavior information as a criterion on the related behavior search UI 1400. When there are multiple candidates for the single behavior information as a criterion, the search unit 1301 may display all of them. The process of setting the single behavior (candidates) as a criterion can be performed in the same manner as the setting of the processing target in S1101 in the first embodiment, for example.

[0074] The user checks the candidates for single action information on a single action confirmation screen 1401, and selects the single action to be searched for with a single action check box 1402. Furthermore, the user operates a search instruction button 1403 to instruct the information processing device 1300 to perform a search process.

[0075] FIG. 15 is an example of a UI that displays the related behavior search results. The search result confirmation UI 1500 displays a related behavior confirmation screen 1501. The related behavior confirmation screen 1501 displays the single behavior output as a search result of the related behavior, and allows the user to confirm it. The search result confirmation UI 1500 may also display the chronological relationship of the related behavior, or the chronological relationship of the related behavior with respect to the reference single behavior. In the related behavior confirmation screen 1501, when there are multiple single behaviors output as the search result, it is possible to display the single behaviors in order of the degree of relevance (here, correlation value). At this time, the correlation value with respect to the reference single behavior may be displayed simultaneously on the related behavior confirmation screen 1501. Note that, here, all other single behaviors that are grouped with the single behavior selected by the user to become chronological behaviors are displayed as related behaviors, but those with a correlation value with respect to the selected single behavior that is less than a threshold value may be hidden.

[0076] 16 is a flowchart showing an example of a related behavior search process by the information processing device 1300 according to this embodiment. The process shown in FIG. 16 may be started, for example, when a user performs a search start operation, or when a time-series behavior is generated by the graph generating unit 203.

[0077] In S1601, the label generation unit 204 sets a single action to be processed (to be grouped with other single actions). S1601 can be executed in the same manner as S1101 in the first embodiment.

[0078] In S1602, the label generation unit 204 sets a subject to be processed (to extract a time-series behavior). S1602 can be executed in the same manner as S1102 in the first embodiment.

[0079] In S1603, the search unit 1301 searches for and extracts single action information satisfying the conditions from the HDD 107 based on the settings in S1601 and S1602, and displays the extracted single action information on the related action search UI 1400.

[0080] In S1604, the search unit 1301 selects a reference single action from the single action information extracted in S1603. Here, the reference single action is selected based on the user's input via the related action search UI 1400 as described above.

[0081] In S1605, the search unit 1301 extracts related actions for the single action selected as a criterion. Here, the search unit 1301 extracts time-series actions based on the single action information of the selected single action and the time-series action graph by the same process as S1104 in the first embodiment, and can extract the single action included in the time-series action as a related action. Here, the search unit 1301 may save the correlation value of each related action for the selected single action as a relevance value and display it on the search result confirmation UI 1500. In S1606, the search unit 1301 displays a list of the related actions extracted in S1605 on the search result confirmation UI 1500, and ends the process of FIG. 16.

[0082] The disclosure of this specification includes the following information processing device, information processing method, and program.

[0083] (Item 1) An acquisition means for acquiring a plurality of images; A recognition means for recognizing a plurality of behaviors of a subject from the video; A first evaluation means for evaluating a time-series correlation between the recognized actions; grouping means for grouping the set of actions based on the assessment of the time series correlation; an output means for outputting the set of grouped actions; An information processing device comprising: (Item 2) the first evaluation means evaluates a probability that a first behavior of the subject recognized from the plurality of images occurs consecutively from the first behavior as a time-series correlation between the first behavior of the subject recognized from the plurality of images and the second behavior of the subject recognized from the plurality of images; 2. The information processing device according to item 1, wherein the grouping means groups the first action and the second action when the probability is equal to or greater than a predetermined threshold. (Item 3) 2. The information processing device according to item 1, wherein the grouping means groups a predetermined number of actions as the set of actions. (Item 4) A selection means for selecting a third behavior from the behaviors recognized by the recognition means, 4. The information processing device according to item 3, wherein the grouping means groups the predetermined number of actions including the third action as the set of actions. (Item 5) The information processing device described in item 4, characterized in that, in order to group the set of actions, the grouping means performs a first grouping process of adding an action that chronologically follows the third action to the set of actions, and a second grouping process of adding an action that chronologically precedes the third action to the set of actions. (Item 6) 5. The information processing device according to item 4, wherein the selection means selects the third behavior from the behaviors recognized by the recognition means based on a user input. (Item 7) The system further includes a presentation unit that presents at least one of the actions recognized by the recognition unit to a user. 7. The information processing device according to item 6, wherein the selection means selects the third action from the at least one action presented by the presentation means based on a user input. (Item 8) 2. The information processing device according to item 1, further comprising a control means for displaying the set of grouped actions on a display device. (Item 9) 9. The information processing device according to item 8, wherein the control means causes the display device to display the set of actions in a chronological order. (Item 10) a generation unit for generating a graph in which each type of recognized behavior of the subject is treated as a node and the evaluation of the time-series correlation is treated as an edge between the nodes; 9. The information processing device according to item 8, wherein the control means causes the display device to display the set of actions as the graph. (Item 11) A labeling means for labeling the set of actions; a determining means for determining whether the set of labeled actions occurs in an input video based on actions included in the set of actions; 2. The information processing device according to item 1, further comprising: (Item 12) The method further includes a second evaluation means for evaluating a matching rate between a behavior of the subject recognized from the input video and a behavior included in the set of behaviors, Item 12. The information processing device according to item 11, wherein the determination means determines whether or not the set of labeled actions occurs in the input video based on the matching rate. (Item 13) acquiring a plurality of images; Recognizing a plurality of behaviors of a subject from the video; evaluating temporal correlations between the recognized behaviors; grouping the set of actions based on the assessment of the temporal correlation; outputting the set of grouped actions; An information processing method comprising: (Item 14) A program for causing a computer to function as the information processing device according to any one of items 1 to 12.

[0084] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0085] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0086] 201: video acquisition unit, 202: behavior recognition unit, 203: graph generation unit, 204: label generation unit, 205: time-series behavior recognition unit

Claims

1. An acquisition means for acquiring a plurality of images; A recognition means for recognizing a plurality of behaviors of a subject from the video; A first evaluation means for evaluating a time-series correlation between the recognized actions; grouping means for grouping the set of actions based on the assessment of the time series correlation; an output means for outputting the set of grouped actions; An information processing device comprising:

2. the first evaluation means evaluates a probability that a first behavior of the subject recognized from the plurality of images will occur consecutively from the first behavior in a time series manner as a time series correlation between the first behavior of the subject recognized from the plurality of images and the second behavior of the subject recognized from the plurality of images; The information processing apparatus according to claim 1 , wherein said grouping means groups said first action and said second action when said probability is equal to or greater than a predetermined threshold value.

3. The information processing apparatus according to claim 1 , wherein the grouping means groups a predetermined number of actions as the set of actions.

4. a selection means for selecting a third behavior from the behaviors recognized by the recognition means, The information processing apparatus according to claim 3 , wherein the grouping means groups the predetermined number of actions including the third action as the set of actions.

5. 5. The information processing device according to claim 4, characterized in that, in order to group the set of actions, the grouping means performs a first grouping process of adding an action that chronologically follows the third action to the set of actions, and a second grouping process of adding an action that chronologically precedes the third action to the set of actions.

6. The information processing apparatus according to claim 4 , wherein the selection means selects the third behavior from the behaviors recognized by the recognition means on the basis of a user's input.

7. The system further includes a presentation unit that presents at least one of the actions recognized by the recognition unit to a user. The information processing apparatus according to claim 6 , wherein the selection means selects the third action from the at least one action presented by the presentation means on the basis of a user's input.

8. The information processing apparatus according to claim 1 , further comprising a control means for displaying the set of grouped actions on a display device.

9. 9. The information processing apparatus according to claim 8, wherein said control means causes said display device to display said set of actions in a chronological order.

10. a generation unit for generating a graph in which each type of recognized behavior of the subject is treated as a node and the evaluation of the time-series correlation is treated as an edge between the nodes; 9. The information processing apparatus according to claim 8, wherein said control means causes said display device to display said set of actions as said graph.

11. A labeling means for labeling the set of actions; a determination means for determining whether the set of labeled actions occurs in an input video based on actions included in the set of actions; The information processing device according to claim 1 , further comprising:

12. The method further includes a second evaluation means for evaluating a matching rate between a behavior of the subject recognized from the input video and a behavior included in the set of behaviors, The information processing apparatus according to claim 11 , wherein the determining means determines whether or not the set of labeled actions occurs in the input video based on the coincidence rate.

13. acquiring a plurality of images; Recognizing a plurality of behaviors of a subject from the video; evaluating temporal correlations between the recognized behaviors; grouping the set of actions based on the assessment of the temporal correlation; outputting the set of grouped actions; An information processing method comprising:

14. A program for causing a computer to function as the information processing device according to any one of claims 1 to 12.

Images