Behavior recognition device, method, and program

The behavior recognition device addresses the challenge of understanding work scenarios by analyzing the relationship between people and objects, enabling accurate scene information extraction and improving work status analysis through a comprehensive analysis of interactions and locations.

WO2024236779A1PCT designated stage expired Publication Date: 2024-11-21NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/018468
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current behavior recognition technologies struggle to accurately understand work scenarios by failing to consider the relationship between people and objects, and are not suitable for environments with multiple interactions or varying distances, making it difficult to grasp scene information effectively.

Method used

A behavior recognition device and method that includes an object detection unit, an action recognition unit, and a relationship analysis unit, which analyzes the relationship between detected objects and actions to output scene information, allowing for the consideration of location and interaction scenarios, including multiple people and objects.

Benefits of technology

Enables accurate grasping of scene information related to actions, improving the understanding of work status by analyzing the relationship between people and objects, regardless of location or interaction, and handling scenarios with multiple people and objects effectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023018468_21112024_PF_FP_ABST
    Figure JP2023018468_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A behavior recognition device according to one embodiment includes: a detection unit that detects an object including a person captured in a moving image; a behavior recognition unit that recognizes the behavior of the person captured in the moving image and detected by the detection unit; and a relationship analysis unit that analyzes the relationship between the object detected by the detection unit and the behavior recognized by the behavior recognition unit, thereby outputting scene information related to the object detected by the detection unit and the behavior recognized by the behavior recognition unit.
Need to check novelty before this filing date? Find Prior Art

Description

Behavior recognition device, method, and program

[0001] FIELD Embodiments of the present invention relate to an apparatus, method, and program for activity recognition.

[0002] Due to the labor shortage, various industries are seeking to digitize their operations. In order to digitize operations, it is necessary to understand how these operations are being performed, i.e., to understand the scene information related to their behavior. Understanding operations enables analysis and improvement, such as visualizing the behavior of store employees to analyze and improve operations, visualizing the behavior of medical professionals to understand medical procedures and other actions, or visualizing unsafe behavior of factory workers to help with safety measures.

[0003] However, it is difficult to accurately grasp the status of work. For example, observing workers on the floor of a store, hospital, or factory is time-consuming. Even if observation is performed using video from a surveillance camera, the volume of video footage is enormous, which is also time-consuming. Furthermore, the main functions for detecting work status from surveillance camera video are generally motion detection or person detection.

[0004] In order to automatically acquire the state of a person's behavior using a camera, it is conceivable that the actual behavior of a person can be visualized using the behavior recognition technology described in Non-Patent Document 1, for example, and used for business analysis and improvement. Also, Non-Patent Document 2, for example, describes a method that takes into account such interactions between a person and an object.

[0005] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He, “SlowFast Networks for Video Recognition”, ICCV2019. Georgia Gkioxari, Ross Girshick, Piotr Dollar, Kaiming He, “Detecting and Recognizing Human-Object Interactions”, CVPR2018.

[0006] As described above, the behavior recognition technology described in Non-Patent Document 1 can be used to understand the behavior of individual people, but this technology does not take into account the relationship between people and objects, and is therefore not suitable for understanding work status. To understand this work status, it is necessary to recognize objects related to work, such as measuring blood pressure when a person picks up a blood pressure monitor, and actions taken with respect to these objects, and to understand what work the worker is about to perform.

[0007] Furthermore, the method described in Non-Patent Document 2 that takes into account interactions between people and objects mainly assumes that the object is nearby or that there is one-to-one interaction between the person and the object, and therefore it is not possible to set the distance to the target object, and location information cannot be utilized when performing a specific task in a certain location.

[0008] Furthermore, this method is based on the assumption that one person interacts with an object, so it is not suitable for detecting scenes in which multiple people interact or do not interact at all, and it is costly to build a dataset.

[0009] The present invention has been made in light of the above circumstances, and its purpose is to provide a behavior recognition device, method, and program that are capable of appropriately grasping scene information related to behavior.

[0010] A behavior recognition device according to one aspect of the present invention includes a detection unit that detects objects, including people, that appear in a moving image; a behavior recognition unit that recognizes the behavior of the people that appear in the moving image and are detected by the detection unit; and a relationship analysis unit that analyzes the relationship between the objects detected by the detection unit and the behavior recognized by the behavior recognition unit, and outputs scene information related to the objects detected by the detection unit and the behavior recognized by the behavior recognition unit.

[0011] A behavior recognition method according to one aspect of the present invention is a method performed by a behavior recognition device, and includes: detecting objects, including people, that appear in a moving image using a detection unit of the behavior recognition device; recognizing behavior of the people that appear in the moving image and are detected by the detection unit using a behavior recognition unit of the behavior recognition device; and analyzing the relationship between the objects detected by the detection unit and the behavior recognized by the behavior recognition unit using a relationship analysis unit of the behavior recognition device, thereby outputting scene information related to the objects detected by the detection unit and the behavior recognized by the behavior recognition unit.

[0012] According to the present invention, scene information relating to behavior can be appropriately grasped.

[0013] FIG. 1 is a diagram showing an application example of a behavior recognition device according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of the procedure of the processing operation of the behavior recognition device. FIG. 3 is a flowchart showing an example of the procedure of the processing operation of a relationship analysis unit of the behavior recognition device. FIG. 4 is a diagram showing an example of extraction condition information in a table format. FIG. 5 is a flowchart showing an example of the procedure of the processing operation by the relationship analysis unit of the behavior recognition device when specific extraction condition information is given. FIG. 6 is a flowchart showing an example of the processing operation related to learning of a relational analysis model by the relationship analysis unit of the behavior recognition device. FIG. 7 is a block diagram showing an example of the hardware configuration of the behavior recognition device according to an embodiment of the present invention.

[0014] An embodiment of the present invention will be described below with reference to the drawings. In this embodiment, in addition to the action information output by the action recognition model such as that described in Non-Patent Document 1, related objects are detected by object detection, and the relationship between the action and the related object is analyzed based on specific extraction conditions, and the task state, i.e., scene information, is grasped regardless of the presence or absence of location information or interaction.

[0015] In this embodiment, the current work state can be estimated from the relationship between object information and behavior information obtained from media information such as video images, thereby making it possible to grasp the work state based on the relationship between people and objects.

[0016] Furthermore, when analyzing the above relationships, information about the work location can be taken into account by making it possible to set conditions such as specifying the range in the video or the distance between people and objects. To take into account both cases with and without interaction, the following relationship features and relationship analysis model are used to handle scenes in which multiple people interact with multiple related objects.

[0017] 1 is a diagram showing an application example of an action recognition device according to an embodiment of the present invention. Next, each unit of this action recognition device will be described. As shown in FIG. 1, the action recognition device 100 includes an object detection unit 10, an action recognition unit 20, a relationship analysis unit 30, and a relationship analysis model DB (database) 40.

[0018] The object detection unit 10 receives video segments as input, detects object information indicating objects appearing in the video segments for each frame, and outputs the object information to the relationship analysis unit 30. Of the object information, the object detection unit 10 detects person information indicating people appearing in the video segments for each frame, and outputs the person information to the behavior recognition unit 20. This person information is information related to people, with an object label such as "person," and is rectangular information of people detected for each frame of the video.

[0019] A video segment is a unit of a group of frames in a video. For example, a group of frames for five seconds may be defined as one segment. When considering a long-term video of several hundred or several thousand frames, a group of frames sampled at regular intervals may be defined as a segment.

[0020] The behavior recognition unit 20 receives video segments and person information as input, and outputs behavior information to the relationship analysis unit 30. Details of the behavior information will be described later. The relationship analysis unit 30 receives object information, behavior information, and external extraction condition information as input, and outputs scene information to the outside via communication with the relationship analysis model DB 40.

[0021] 2 is a flowchart showing an example of the processing procedure of the behavior recognition device. The object detection unit 10 receives video segments as input (step S11). The object detection unit 10 detects object information using the video segments, outputs the object information to the relation analysis unit 30, and outputs person information from the object information to the behavior recognition unit 20 (step S12).

[0022] Here, an object detection model such as that disclosed in the above-mentioned Non-Patent Document 2 may be used to detect object information. Object detection is performed for each frame, and object information is information obtained during object detection, such as the object label, rectangle information, and score of the object for each frame. The rectangle information is information representing the area in which the detected object is surrounded by a rectangle, and it is sufficient that the information included is sufficient to represent the rectangle.

[0023] The behavior recognition unit 20 receives video segments and person information as input (step S13). The behavior recognition unit 20 obtains behavior information from the video segments and person information and outputs this behavior information to the relationship analysis unit 30 (step S14). The behavior information here is behavior labels (sometimes referred to as behavior recognition labels) such as "walk," "touch," and "hold" obtained from the video segments and person information, and can be obtained using, for example, the method described in Non-Patent Document 1 above. Alternatively, a posture may be detected from rectangular information of a person, and the behavior label may be obtained from this posture. The behavior label may also be obtained in segments of 30 frames or the like.

[0024] Next, the relationship analysis unit 30 receives the object information, the behavior information, and the extraction condition information (step S15). Here, the extraction condition information is information indicating conditions for constructing the relationship features, which are features used for analyzing the relationship between people and objects. Details will be described later.

[0025] The relationship analysis unit 30 outputs scene information based on the object information, behavior information, and extraction condition information (step S16). Here, the scene information may be, for example, "product replacement in progress" for a retail store or "unsafe scene" for a factory. This scene information may be expressed as a one-hot vector or the like according to the number of scenes to be classified.

[0026] Next, the details of the processing operation of the relationship analysis unit 30 will be described. Fig. 3 is a flowchart showing an example of the procedure of the processing operation of the relationship analysis unit of the behavior recognition device. Fig. 4 is a diagram showing an example of extraction condition information in a table format. The relationship analysis unit 30 receives object information from the object detection unit 10, behavior information from the behavior recognition unit 20, and extraction condition information from the outside (step S21).

[0027] The relationship analysis unit 30 constructs relationship features from the object information and the behavior information based on the extraction condition information (step S22). This extraction condition information specifies conditions for detecting people and objects, as shown in FIG. 4, and is information used by the relationship analysis unit 30 to construct relationship features under these conditions. For example, the designated area shown in FIG. 4 is used to specify a rectangle within a frame, and to extract frames in which a designated object is detected within the range of this rectangle by the relationship analysis unit 30. Here, the designated object can be specified by an object label, as shown in FIG. 4.

[0028] Furthermore, as shown in FIG. 4, by setting a reference object, a related object, and a distance threshold in the extraction condition information, the relationship analysis unit 30 can calculate the distance between the center coordinates of a rectangle when the reference object is detected and the center coordinates of a rectangle when the related object is detected, and extract frames where the distance is equal to or less than the distance threshold.

[0029] The relationship analysis unit 30 can then construct relationship features by utilizing the extraction condition information, such as by using the calculated distances. The relationship analysis unit 30 can construct relationship features by using the number of appearances of objects in the target video segment, the position information of objects related to the object information, the number of appearances of behavior labels, etc.

[0030] Information used to construct the relational features may include the size of the object, the movement of the object, or the variance of the movement. In this way, by using features that take into account multiple people and related objects as relational features, it is possible to handle interactions between multiple people and multiple objects.

[0031] Finally, the relation analysis unit 30 inputs the constructed relation features into a relation analysis model stored in the relation analysis model DB 40, acquires the scene information output from the relation analysis model, and outputs it to the outside (step S23). This relation analysis model is a model related to a classification method such as a Gradient Boosting Decision Tree (GBDT), and is a model that can visualize the relationships between each feature. The relation analysis model may also be a neural network model that can visualize features using self-attention.

[0032] 5 is a flowchart showing an example of a procedure for processing operations by the relationship analysis unit of the behavior recognition device when specific extraction condition information is provided. First, the relationship analysis unit 30 receives object information from the object detection unit 10, behavior information from the behavior recognition unit 20, and specific extraction condition information from an external source (step S31). Based on the received extraction condition information, the relationship analysis unit 30 extracts frames in which a specified object is detected in a specified region (step S32).

[0033] The relationship analysis unit 30 calculates the distance between the reference object and the related object from the object information and the behavior information (step S33). The relationship analysis unit 30 extracts a behavior label from the behavior information (step S34). The relationship analysis unit 30 constructs a relationship feature using the position information of the related object in the object information, the distance between the reference object and the related object, and the behavior label (step S35). The relationship analysis unit 30 inputs the constructed relationship feature into a relationship analysis model and outputs the scene information output from this relationship analysis model to the outside (step S36).

[0034] 6 is a flowchart showing an example of a processing operation procedure for training a relational analysis model by the relational analysis unit of the behavior recognition device. First, the relational analysis unit 30 receives object information used to train parameters of the relational analysis model from the object detection unit 10, behavior information used to train the parameters from the behavior recognition unit 20, extraction condition information used to train the parameters from an external source, and correct answer information for scene information from an external source (step S41). Based on the extraction condition information, the relational analysis unit 30 constructs relational features used to train the relational analysis model from the object information and behavior information (step S42).

[0035] The relation analysis unit 30 uses the constructed relation features as input and the received scene information as output, thereby learning the parameters of the relation analysis model so that the difference between the output scene information and the correct answer information approaches 0 (step S43). The relation analysis unit 30 outputs and stores this learned relation analysis model in the relation analysis model DB 40 (step S44).

[0036] As described above, by using the behavior recognition device according to this embodiment, it is possible to detect scene information from video images that takes into account the relationship between people and objects. This makes it possible to estimate procedures that include abstract actions based on basic actions that represent physical actions.

[0037] Fig. 7 is a block diagram showing an example of the hardware configuration of an action recognition device according to an embodiment of the present invention. In the example shown in Fig. 7, the action recognition device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0038] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0039] An input device 300 and an output device 400, which are attached to the behavior recognition device 100 and used by a user or the like, are connected to the input / output interface 113. The input / output interface 113 takes in operation data input by a user or the like through the input device 300, such as a keyboard, a touch panel, a touchpad, or a mouse, and outputs output data to the output device 400, which includes a display device using a liquid crystal or an organic electroluminescence (EL) display, for display. The input device 300 and the output device 400 may be devices built into the behavior recognition device 100, or may be input devices and output devices of other information terminals that can communicate with the behavior recognition device 100 via a network NW.

[0040] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and stores programs necessary to execute various control processes, etc., according to one embodiment.

[0041] The data memory 112 is a tangible storage medium that is a combination of, for example, the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and created during various processing steps.

[0042] The behavior recognition device 100 according to an embodiment of the present invention may be configured as a data processing device having a software-based processing function unit. The storage area used as a work memory or the like by the behavior recognition device 100 may be configured by using the data memory 112 shown in Fig. 7. However, these configured storage areas are not essential components within the behavior recognition device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage system such as a database server located in the cloud.

[0043] The processing function unit can be realized by having the hardware processor 111A read and execute a program stored in the program memory 111B, but the processing function unit may also be realized in various other forms, including an integrated circuit such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0044] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0045] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0046] 100... Behavior recognition device 10... Object detection unit 20... Behavior recognition unit 30... Relational analysis unit 40... Relational analysis model DB (database)

Claims

1. A behavior recognition device comprising: a detection unit that detects objects, including people, that appear in a moving image; a behavior recognition unit that recognizes the behavior of people that appear in the moving image and are detected by the detection unit; and a relationship analysis unit that analyzes the relationship between the objects detected by the detection unit and the behavior recognized by the behavior recognition unit, and outputs scene information related to the objects detected by the detection unit and the behavior recognized by the behavior recognition unit.

2. The behavior recognition device of claim 1, wherein the relationship analysis unit constructs features of the relationship between the object detected by the detection unit and the behavior recognized by the behavior recognition unit based on information that defines conditions for constructing the features, and outputs the scene information by analyzing the relationship between the object detected by the detection unit and the behavior recognized by the behavior recognition unit based on the constructed features.

3. The behavior recognition device described in claim 2, wherein the relationship analysis unit calculates a distance between a first object appearing in the video and a second object related to the first object based on the object detected by the detection unit, the behavior recognized by the behavior recognition unit, and information defining conditions for constructing the features, and constructs features of the relationship between the object detected by the detection unit and the behavior recognized by the behavior recognition unit based on the calculated distance.

4. The behavior recognition device of claim 2, wherein the relationship analysis unit outputs the scene information using a model that inputs features of the relationship between an object and an action and outputs the scene information, constructs features of the relationship between the object detected by the detection unit and the action recognized by the behavior recognition unit, which are features used for learning parameters of the model, and learns the parameters based on the difference between the scene information output from the model based on the constructed relationship and correct information for the scene information.

5. The behavior recognition device according to claim 2, wherein the conditions defining the conditions for constructing the features include the range in which the objects and people appear in the video and the conditions for the objects to be detected by the detection unit.

6. A method performed by a behavior recognition device, comprising: detecting an object, including a person, that appears in a moving image using a detection unit of the behavior recognition device; recognizing the behavior of the person that appears in the moving image and is detected by the detection unit using the behavior recognition unit of the behavior recognition device; and analyzing the relationship between the object detected by the detection unit and the behavior recognized by the behavior recognition unit using a relationship analysis unit of the behavior recognition device, thereby outputting scene information related to the object detected by the detection unit and the behavior recognized by the behavior recognition unit.

7. A behavior recognition processing program that causes a processor to function as each part of the behavior recognition device according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Feeling generator and feeling generation method

    JP1999265239A

  • Scene recognizing device

    JP2000293685A

  • Image display device

    JP2018055619A

  • Method, identification device and program for identifying potential associates of at least one target person

    JP2020530970A

  • Video analyzing system and video analyzing method

    JP2022048475A