Information processing device, information processing method, and program

Abstract

To provide a causal interaction detection device capable of extracting posture information about each person detected from video data.SOLUTION: A causal interaction detection device 2000 includes a posture extraction unit 2020, a model generation unit 2040, and a correlation detection unit 2060. The posture extraction unit 2020 extracts posture information about each person from video data. Person posture information indicates the posture of a person in time series. The model generation unit 2040 generates a change model about each person. The correlation detection unit 2060 detects one or more collections of a plurality of persons correlated with each other at a posture change time on the basis of the change model.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates generally to a technique for detecting causal interactions between multiple people from video.

Background Art

[0002] Causal interaction between people is an interaction in which two or more people are involved, and represents the case where there is a causal relationship in their interaction. Various interactions between people, such as one person's action or state affecting another person's action or state, become causal interactions.

[0003] Non-Patent Document 1 and Non-Patent Document 2 disclose techniques for detecting causal interactions between people from video data. Non-Patent Document 1 discloses a system for detecting causal interactions between people based on the concept of Granger causality. According to the definition of Granger causality, time series data {x(t)} is considered to have Granger causality with respect to other time series data {y(t)} when y(t) can be better inferred by knowing the past values of x(t). The system of Non-Patent Document 1 uses, as time series data, the trajectories of key points of people's heads in scenes within video data. According to Non-Patent Document 1, the trajectory of the key point of a person's head in a scene is represented as a linear combination with the key points of the heads of other people in the scene, and the problem of finding causal interactions is considered as the problem of identifying a sparse graph.

[0004] Non-Patent Document 2 discloses a system that uses feature points of the skeletons of people to recognize the interaction between two people. In Non-Patent Document 2, a support vector machine (SVM) is pre-trained to input a video containing the interaction between two people and classify their interaction into one of a predetermined interaction class.

Prior Art Documents

Non-Patent Documents

[0005] [Non-Patent Document 1] Mustafa Ayazoglu, Burak Yilmaz, Mario Sznaier, and Octavia Camps, "Finding Causal Interactions in Video Sequences," 2013 IEEE International Conference on Computer Vision, December 1, 2013. [Non-Patent Document 2] Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L. Berg, and Dimitris Samaras, "Two-person interaction detection using body-pose features and multiple instance learning," 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, June 16, 2012. [Overview of the project] [Problems that the invention aims to solve]

[0006] The system described above cannot detect various interactions between people. Specifically, the system disclosed in Non-Patent Literature 1 cannot detect interactions between people when the skeletal posture of a person changes significantly without a large change in the trajectory of the head's keypoint. The reason for this problem is that Non-Patent Literature 1 uses trajectory information of only one keypoint of each person in the video data scene. Here, the trajectories of multiple keypoints of one person cannot be represented by a linear combination of multiple keypoints of other people (the relationship is not linear), so the method disclosed in Non-Patent Literature 1 cannot be directly extended to multiple keypoints.

[0007] Regarding Non-Patent Document 2, the interpersonal interactions that the system disclosed therein can detect are limited to one predetermined type. This is because the system's SVM needs to be pre-trained using training data representing one of the known types of interactions. Therefore, this system has difficulty detecting unknown types of interpersonal interactions.

[0008] One of the purposes of this disclosure is to provide a technology for detecting various causal interactions between people. [Means for solving the problem]

[0009] The causal interaction detection device provided in this disclosure comprises at least one processor and a memory element for storing instructions. The at least one processor is configured to extract posture information for each person detected from video data by executing the instructions, the posture information showing the posture of the person in time series, generate a change model representing the change in posture over time for each person based on the posture information, determine for each of one or more sets of a plurality of people whether the times of posture change of the people in that set are correlated with each other, and detect the people whose times of posture change are determined to be correlated with each other as people who have a causal interaction with each other.

[0010] The control method further provided in this disclosure is performed by a computer. The control method includes extracting posture information for each person detected from video data, the posture information representing the person's posture over time, generating a change model for each person based on the posture information that represents the change in posture over time, determining for each of one or more sets of a plurality of persons whether the times of posture change of the persons in that set are correlated with each other, and detecting persons whose times of posture change are determined to be correlated with each other as persons who have a causal interaction with each other.

[0011] The non-temporary computer-readable media further provided by this disclosure contain a program that causes a computer to execute the control methods of this disclosure. [Effects of the Invention]

[0012] This disclosure provides a technology for detecting various causal interactions between people. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 shows an overview of the causal interaction detection device according to Embodiment 1. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the causal interaction detection device of Embodiment 1. [Figure 3] Figure 3 is a block diagram showing an example of the hardware configuration of a computer that implements a causal interaction detection device. [Figure 4] Figure 4 is a flowchart illustrating an example of the processing flow performed by the causal interaction detection device of Embodiment 1. [Figure 5] Figure 5 shows an example of a method for calculating the dissimilarity between a set of actual postures of a person and a reference movement. [Figure 6] Figure 6 shows a case where two people are interacting with each other. [Figure 7A] Figure 7A shows a model of a person's transformation. [Figure 7B] Figure 7B shows a model of changes in a person. [Modes for carrying out the invention]

[0014] Hereafter, embodiments relating to this disclosure will be described with reference to the drawings. The same elements are assigned the same reference numerals across multiple drawings, and redundant explanations are omitted where appropriate.

[0015] Embodiment 1 <Overview> FIG. 1 shows an overview of the causal interaction detection device according to Embodiment 1. Here, FIG. 1 only shows an example of operations that the causal interaction detection device can perform, and does not limit the operations of the causal interaction detection device.

[0016] The causal interaction detection device is used to detect causal interactions between a plurality of persons 20 imaged in video data 30. To detect the causal interactions of the persons 20, the causal interaction detection device analyzes the video data 30 and generates a model of the temporal change of the posture (hereinafter, change model) for each person 20. Further, the causal interaction detection device identifies the correlation of the times when large posture changes occur for the plurality of persons 20 in the video data 30 by comparing the change models of the plurality of persons 20 with each other. Then, the causal interaction detection device generates a set of one or more persons 20 (hereinafter, detection set). Each detection set indicates a plurality of persons 20 having causal interactions. Specifically, a plurality of persons 20 whose times of posture change are correlated with each other are included in the same detection set. For example, the causal interaction detection device detects that there is a causal interaction between them when the times of posture change overlap each other among the plurality of persons 20.

[0017] For example, in FIG. 1, four persons 20-1 to 20-4 are detected from the video data 30. The change models 40-1 to 40-4 represent the temporal changes of the postures for the persons 20-1 to 20-4, respectively. By comparing the change model 40-1 with the other change models 40-2 to 40-4, it can be seen that there is no other person 20 whose time of large posture change is correlated with the time of large posture change of the person 20-1. The same applies to the person 20-2.

[0018] On the other hand, by comparing the change model 40-3 and the change model 40-4, it can be seen that the person 20-3 is making a large posture change during a time period overlapping with the time when the person 20-4 makes a large posture change. Therefore, the causal interaction detection device identifies that there is a causal interaction between the persons 20-3 and 20-4.

[0019] <Examples of effects and benefits> According to the causal interaction detection device of Embodiment 1, a change model 40 representing the temporal change in posture is generated for each person 20 detected from the video data 30. Based on the generated change model 40, the causal interaction detection device detects multiple people 20 whose posture changes are temporally correlated with each other, and it is considered that these people 20 have a causal interaction.

[0020] This method does not require a pre-trained model trained to detect a predetermined type of interaction between people. Therefore, the causal interactions that the causal interaction detection device can detect are not limited to a predetermined type of interaction. Furthermore, as will be described later, the causal interaction detection device is inherently capable of handling poses represented by multiple keypoints. Therefore, the causal interactions that the causal interaction detection device can detect are not limited to those in which pose is represented by a single keypoint. Thus, the causal interaction detection device can detect various types of causal interactions.

[0021] The following provides a more detailed explanation of the causal interaction detection device.

[0022] <Example of functional configuration> Figure 2 is a block diagram showing an example of the functional configuration of the causal interaction detection device 2000 of Embodiment 1. The causal interaction detection device 2000 includes a posture extraction unit 2020, a model generation unit 2040, and a correlation detection unit 2060. The posture extraction unit 2020 extracts posture information for each person 20 from the video data 30. The posture information for person 20 shows the posture of person 20 in chronological order. The model generation unit 2040 generates a change model 40 for each person 20. The correlation detection unit 2060 detects one or more sets of multiple people 20 that are correlated with each other in terms of the time of posture change, based on the change model 40.

[0023] <Example of hardware configuration for the causal interaction detection device 2000> The causal interaction detection device 2000 can be implemented using one or more computers. Each of these computers may be a dedicated computer for implementing the causal interaction detection device 2000, or it may be a general-purpose computer such as a personal computer (PC), a server machine, or a portable device. The causal interaction detection device 2000 can be implemented by installing an application on a computer. The application is implemented as a program that makes the computer function as the causal interaction detection device 2000. In other words, the program implements the functional components of the causal interaction detection device 2000.

[0024] Figure 3 is a block diagram showing an example of the hardware configuration of a computer 1000 that implements a causal interaction detection device 2000. In Figure 3, the computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120.

[0025] Bus 1020 is a data communication path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data from each other. The processor 1040 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). Memory 1060 is a primary storage element such as RAM (Random Access Memory) or ROM (Read Only Memory). Storage device 1080 is a secondary storage element such as a hard disk, SSD (Solid State Drive), or memory card. The input / output interface 1100 is an interface between the policy generation device 100 and peripheral devices (such as a keyboard, mouse, or display device). The network interface 1120 is an interface between the policy generation device 100 and the network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0026] The storage device 1080 can store the aforementioned computer program. The processor 1040 executes its computer program to realize each functional component of the causal interaction detection device 2000.

[0027] The hardware configuration of computer 1000 is not limited to the configuration shown in Figure 3. For example, as mentioned above, the causal interaction detection device 2000 can be implemented using multiple computers. In this case, these computers can be connected to each other via a network.

[0028] <Processing flow> Figure 4 is a flowchart showing an example of the processing flow performed by the causal interaction detection device 2000 of Embodiment 1. The posture extraction unit 2020 acquires video data 30 (S102). The posture extraction unit 2020 extracts posture information for each person 20 from the video data 30 (S104). The model generation unit 2040 generates a change model 40 for each person 20 based on the extracted posture information of the person 20 (S106). The correlation detection unit 2060 generates one or more sets of people 20 that are correlated with each other in terms of the time of posture change (S108).

[0029] <Acquisition of video data 30: S102> The attitude extraction unit 2020 acquires video data 30 (S102). There are various ways to acquire the video data 30. For example, the attitude extraction unit 2020 acquires the video data 30 from the camera that generates the video data 30. Alternatively, for example, the attitude extraction unit 2020 acquires the video data 30 from the storage device where the camera stores the video data 30.

[0030] The camera generating the video data 30 is any camera capable of capturing images of multiple people 20. For example, it could be a surveillance camera set up at a location to be monitored. Alternatively, it could be a portable camera attached to a person or object (such as a drone) patrolling a designated location.

[0031] The video data generated by the camera may be divided into multiple video data 30. For example, each video data of a predetermined length (e.g., 1 minute) generated by the camera may be treated as video data 30. Here, the video data may be divided into multiple video data 30 such that parts of two adjacent video data 30 overlap with each other.

[0032] <Extraction of posture information: S104> The posture extraction unit 2020 extracts posture information for each person 20 from the video data 30 (S104). The posture information for person 20 represents the time series of person 20's posture (in other words, a time axis sequence of person 20's postures). For example, the posture information represents the posture of person 20 for each time frame of the video data 30. In this case, the posture extraction unit 2020 calculates the posture of person 20 for each time frame of the video data 30 and generates posture information that shows the sequence of the calculated postures of person 20. Note that the posture information is not required to show the posture of person 20 for each frame. For example, the posture information may show the posture of person 20 every few frames.

[0033] To calculate the posture of each person 20, the posture extraction unit 2020 detects the person 20 from the frames of the video data 30. For example, the detected person 20 is represented by the coordinates of its bounding rectangle in the frame. Furthermore, the posture extraction unit 2020 tracks these bounding rectangles of people across multiple frames of the video data 30. In this way, multiple bounding rectangles representing the same person 20 are identified across multiple frames. The posture extraction unit 2020 extracts the coordinates of the skeletal keypoints of each detected and tracked person 20. As a result, the posture extraction unit 2020 generates posture information for each person 20 detected from the video data 30, which includes a time series of the coordinates of the skeletal keypoints of that person 20.

[0034] Here, the order of operations may differ. For example, instead of tracking the bounding rectangle of the person and then extracting the coordinates of the skeletal keypoints, the coordinates of the skeletal keypoints may be extracted first and then tracked across multiple frames of the video data 30.

[0035] <Generating a change model: S106> The model generation unit 2040 generates a change model 40 for each person 20 in the video data 30 based on the posture information extracted for each person 20 (S106). Specifically, for each person 20 detected from the video data 30, the model generation unit 2040 models the change in the posture of that person 20 as a function of time.

[0036] The change in posture at time t can be modeled by comparing posture Pt at time t with a reference posture Pref and calculating a dissimilarity score that represents how much posture Pt differs from the reference posture Pref. This allows tracking how much a person's posture changes over time.

[0037] <<Details about standard posture>> The reference pose Pref represents the posture of person 20 that is considered normal in the scene (in part or all of the video data 30). There are various ways to determine the reference pose based on the scenario. For example, the reference pose of person 20 can be defined by the posture of that person in one of several initial frames in which that person appears in the video data 30 (for example, the posture in one of the first five frames in which that person appears). In this way, the reference pose is defined independently for each of the multiple people 20. Here, the first few frames may not contain all of the skeletal keypoints of person 20. For example, if person 20 is at the edge of the frame, only some of their keypoints may be visible. Therefore, it is preferable to select from a fixed set of initial frames (for example, the 5th or 10th).

[0038] Another method for determining a reference posture is to use knowledge about the scene captured in the video data 30 and select the most common posture among several possible actions in that scene. For example, suppose the video data 30 is generated by a surveillance camera installed in a place where many people are walking on a sidewalk. In this case, the most common posture in that scene is "standing upright." Therefore, the posture representing standing upright can be used as the reference posture. Another example is a scene in a cycling lane where many people are cycling. In this case, the posture representing "cycling," where the person's back is bent and their hands are gripping the handlebars of a bicycle, would be an appropriate reference posture. In these cases, the reference posture is pre-stored in a memory device accessible from the causal interaction detection device 2000.

[0039] The reference posture does not need to be determined based on a single frame; it may be determined by a sequence of frames. In other words, the reference posture may be defined by a sequence of postures (e.g., actions). In this case, the reference posture may also be called the reference action. When the examples described above are extended to include multiple frames, they may correspond to actions such as walking or cycling. Specifically, for video data 30 where walking is a common action, the reference posture may be defined by a sequence of postures representing the walking action (e.g., the movement of the hands and feet while walking). On the other hand, for video data 30 where cycling is a common action, the reference posture may be defined by a sequence of postures representing the cycling action (e.g., the movement of the feet while cycling).

[0040] When a reference motion is used, the dissimilarity between the posture of person 20 and the reference motion is calculated by considering the set of postures of person 20 in the form of a sliding window. Figure 5 shows an example of how to calculate the dissimilarity between the set of actual postures of person 20 and the reference motion. In this example, the reference motion is defined by a column of three reference postures. Therefore, the size of the sliding window is 3. The stride of the sliding window is 4.

[0041] Firstly, the model generation unit 2040 calculates the dissimilarity between the first set of real poses of person 20, including the first to third real poses of person 20, by comparing them with a reference motion. Specifically, the distance d11 between the first real pose and the first reference pose, the distance d12 between the second real pose and the second reference pose, and the distance d13 between the third real pose and the third reference pose are calculated. Based on these calculated distances, the dissimilarity D1 between the first set of real poses and the reference poses is calculated. In this example, the dissimilarity is calculated as the sum of the distances between the real poses and the reference poses. Dissimilarity D1 = d11 + d12 + 13.

[0042] Next, the model generation unit 2040 compares a second set of actual postures of the person 20 with a reference movement in a similar manner. In this example, since the stride is 4, the second set of actual postures includes the 5th through 7th actual postures.

[0043] The reference posture may be fixed for all frames, or it may be updated as a function of time. In the latter case, if the posture of person 20 changes to a new posture and this new posture continues for a long period of time, the reference posture is updated to this new posture. For example, suppose a person walks and then sits down, and remains sitting for a long period of time before performing any other actions. In this case, the initial reference posture would correspond to the "standing" posture, and subsequently, the reference posture may be updated to "sitting" because this is the person's new normal state. The reference posture can be updated by determining how long person 20 has been in the current state. Specifically, for example, if person 20 changes to a new posture different from the current posture, and person 20 continues that new posture for a predetermined length of time or longer, the reference posture is updated to the new posture.

[0044] <<Details on dissimilarity>> The dissimilarity of the poses of person 20 can be calculated, for example, as the distance between that pose and a reference pose. There are various ways to represent the distance between two poses, such as cosine distance or weighted distance. When using weighted distance, each keypoint of person 20 is assigned a separate weight.

[0045] A learning-based method can be used to calculate the dissimilarity between the target pose (the pose for which dissimilarity is calculated) and the reference pose. Specifically, the model generation unit 2040 has a pre-trained regression model that takes a pair of target poses and a reference pose as input and outputs the dissimilarity between them. Each of these regression models is pre-trained using multiple training data sets that associate each pair of target poses and a reference pose with the dissimilarity of that pair (in other words, the dissimilarity that should be output by the regression model into which that pair was input).

[0046] <Detection of causal interactions: S108> The correlation detection unit 2060 detects one or more sets of multiple individuals 20 whose posture change times are correlated with each other (S108). This means that the correlation detection unit 2060 finds the relationship between the times when each person's posture changes significantly. If there is a correlation between the times when multiple people's postures change significantly, those people are likely interacting with each other. Here, "significant posture change" can be defined as a posture change whose dissimilarity is above a predetermined threshold.

[0047] For example, the correlation detection unit 2060 sequentially selects any set of multiple people 20 and determines whether the time of posture changes of the multiple people 20 included in the selected set has a predetermined time correlation by comparing their change models 40. If it is determined that they have the predetermined time correlation, the correlation detection unit 2060 treats the selected set as a detected set. This means that a causal interaction is identified between the multiple people 20 included in the selected set. If it is determined that they do not have the predetermined time correlation, the correlation detection unit 2060 does not treat the selected set as a detected set. This means that a causal interaction is identified between the multiple people 20 included in the selected set.

[0048] There can be various temporal correlations in the posture changes of multiple individuals 20 who are thought to have a causal interaction. One example of such a correlation is the overlap of large posture changes on the time axis. For example, suppose the posture of a certain individual 20 changes significantly in a certain time window. Furthermore, suppose this time window overlaps with a time window in which the posture of another individual 20 changes significantly. In this case, the timing of the posture changes of those individuals 20 is thought to be correlated, and there is a high probability that these two individuals 20 have a causal interaction.

[0049] Therefore, the correlation detection unit 2060 determines whether the time windows in which the postures of multiple people 20 in the selected set change significantly overlap for a predetermined length of time or longer by comparing their change models 40. If it is determined that these time windows overlap for a predetermined length of time or longer, the correlation detection unit 2060 treats the selected set as a detected set.

[0050] Examples of interactions where time windows overlap include handshakes and hugs. In these actions, the time windows of the posture changes of the individuals involved overlap significantly because these actions occur almost simultaneously. Therefore, their postures can change almost simultaneously. For actions such as pushing, punching, and kicking, overlap is possible, but the degree of overlap is smaller because one person's action occurs after another person's action (i.e., the resulting posture change always begins after the causative posture change has already started).

[0051] As mentioned above, the length of the time window overlap may depend on the actions of person 20. Therefore, the threshold for detecting the overlap between time windows can be predetermined based on the type of causal interaction that the causal interaction detection device 2000 should detect.

[0052] Multiple individuals 20 can have causal interactions even if their time windows for large posture changes do not overlap. For example, a causal action may end simultaneously with or slightly before the start of a resulting action. In other words, there may be an interval of a certain magnitude between a causal action and a resulting action. Therefore, the correlation that "for individuals 20 in the selected set, the interval between time windows for large posture changes is below a predetermined threshold" can be used as another predetermined time correlation. In this case, for individuals 20 in the selected set, the correlation detection unit 2060 detects time windows with large posture changes and calculates the interval between time windows. If the calculated interval is below a predetermined threshold, the correlation detection unit 2060 treats the selected set as the detected set.

[0053] <<Other elements for detecting correlation>> The correlation detection unit 2060 can improve the accuracy of detecting causal interactions between the individuals 20 by utilizing factors other than the temporal correlation of posture changes. One such factor could be the distance between the individuals 20. Even if there is a temporal correlation between large posture changes of the individuals 20, there will be no causal interaction if those individuals are far apart from each other. Therefore, the correlation detection unit 2060 can take the distance between the individuals 20 into consideration.

[0054] For example, the correlation detection unit 2060 determines whether or not there is a causal interaction between the individuals 20 in the selected set only if the distance between those individuals 20 is below a predetermined threshold. In other words, if the distance between the individuals 20 is greater than the predetermined threshold, the correlation detection unit 2060 determines that there is no causal interaction between the individuals 20, regardless of the time correlation in the changes in posture.

[0055] The determination of the distance between the people 20 in the selected set may be performed after the determination of the time correlation of the changes in the posture of the people 20. In this case, the correlation detection unit 2060 determines for each detection set whether the distance between the people 20 in the detection set is below a predetermined threshold. The correlation detection unit 2060 then determines that the people 20 in the detection set have a causal interaction if the distance between the people 20 in the detection set is below the predetermined threshold.

[0056] Determining the distance between 20 people within the detection set can be done using a trained model. For example, the trained model is trained to use pairs of person images to identify whether they are interacting based on the distance between them.

[0057] The direction that person 20 is facing can also be used as a factor to improve the accuracy of detecting whether or not there is a causal interaction between people 20. Specifically, if there is a causal interaction between people 20, person 20 is likely to be facing the other person 20. On the other hand, if person 20 is facing the direction of the determination, even if the change in person 20's posture has a temporal correlation, there is likely no causal interaction between people 20. Therefore, by considering the direction that person 20 is facing, the accuracy of detecting causal interactions between people 20 can be improved.

[0058] For example, the correlation detection unit 2060 determines whether there is a time correlation between the posture changes of the people 20 in the selected set, but only when the target parts of the people 20 are facing each other. The target parts are, for example, the head, torso, or eyes of the person 20. Specifically, when the target parts of the people 20 are facing each other, the difference in the direction in which the target parts are facing is equal to or close to 180 degrees. Therefore, for example, the correlation detection unit 2060 calculates the difference D in the direction in which the target parts of the people 20 are facing and determines whether D satisfies "180-m<=D<=180+m". m is a predetermined margin. Here, the parameter m is a real number greater than 0 and less than 180 (for example, m=45).

[0059] If the difference D satisfies the above conditions, the correlation detection unit 2060 determines that the people 20 are facing each other, and then compares their change models 40 to determine whether there is a predetermined time correlation between their posture changes. On the other hand, if the difference D does not satisfy the above conditions, the correlation detection unit 2060 determines that the people 20 are not facing each other. Therefore, their change models 40 are not compared. Note that known techniques can be applied to calculate the direction in which a person's body part (head, torso, or eyes, etc.) in the video data is facing.

[0060] Even if the individuals 20 are not facing each other, there may still be cases where they are interacting with each other. For example, interacting individuals 20 may be facing a common object. Therefore, the correlation detection unit 2060 can use the condition "the individuals 20 are facing a common object" instead of the condition "the individuals 20 are facing each other."

[0061] Similar to the determination of the distance between the individuals 20 in the selected set, the determination of the direction the individuals 20 in the selected set are facing may be performed after the determination of whether or not there is a temporal correlation between the changes in the posture of the individuals 20. Alternatively, the determination of the direction the individuals 20 in the selected set are facing may be performed using a trained model. For example, the trained model may be trained to use pairs of person images to determine whether or not they are interacting based on the direction they are facing.

[0062] As mentioned earlier, using additional factors to determine whether or not there is a causal interaction between people is particularly useful in cases where there are multiple groups of people, and within each group, people interact causally. Considering characteristics such as distance and orientation helps to distinguish these people into multiple groups, rather than falsely identifying them as all interacting with each other and belonging to a single group.

[0063] <Output based on detection results> The causal interaction detection device 2000 can generate and output output information based on the detected set of persons 20. There can be various types of information to be output. For example, the output information may indicate one or more sets of persons 20 that have been determined to have a causal interaction. In the output information, a person 20 may be represented by the bounding rectangle of that person in the corresponding frame of the video data 30. Alternatively, for example, a person 20 may be represented by a partial image of the corresponding frame (for example, the image region of the bounding rectangle of that person in the frame). Alternatively, for example, each person 20 may be represented by including each frame of the video data 30 edited to show the bounding rectangle of the person 20 to be represented by the output information.

[0064] The output information may further include the type of social group to which the person 20 in the detection set belongs, using additional features such as age, gender, clothing, or possessions of the person 20. To this end, a trained model can be used that takes image and scene information of the person 20 in the detection set as input and extracts features from it that are useful for classifying those people 20 into one of several social groups.

[0065] The output information described above can be output in various ways. For example, the causal interaction detection device 2000 displays the output information on a display device by outputting it to the display device. The display device is observed, for example, by a security guard in the security room. Alternatively, for example, the causal interaction detection device 2000 may transmit the output information to another computer, such as a mobile terminal, used by a security guard at the site or in the security room, or by the operator of the causal interaction detection device 2000. Alternatively, for example, the causal interaction detection device 2000 may store the output information in a storage device for later use.

[0066] <Example> The following describes an example of the operation of the causal interaction detection device 2000. Herein, the operation of the causal interaction detection device 2000 described below is just one example of the various possible operations of the causal interaction detection device 2000, and the operation of the causal interaction detection device 2000 is not limited to the following example.

[0067] Figure 6 shows a case where two people 20 are interacting with each other. The possible interaction here is "pushing," with person 20-5 pushing person 20-6. Note that the following explanation is also applicable to any interaction.

[0068] Figure 6 shows several frames of video data 30 input to the causal interaction detection device 2000, where an interaction exists. Here, the video data 30 in Figure 6 is cropped and centered around people 20-5 and 20-6 for ease of illustration. Initially, people 20-5 and 20-6 are standing still. Then, person 20-5 approaches person 20-6 and pushes person 20-6. This pushing motion moves person 20-6 backward.

[0069] In Figure 6, figures 20-5 and 20-6 are represented by their skeletal posture coordinates. Fifteen skeletal keypoints are considered: head, nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, left hip, right knee, left knee, right ankle, and left ankle. Each keypoint is associated with coordinates (x,y) representing its pixel position in the image frame. Alternatively, three-dimensional skeletal keypoints of the figures can be used instead of the two-dimensional skeletal keypoints shown in Figure 6.

[0070] A person can be located anywhere in the image, and their scale may differ between frames based on the distance from the camera to that person. Therefore, in order to correctly compare different postures, it is preferable to normalize the skeletal posture by converting and scaling it to a fixed size.

[0071] Next, in order to understand how each person's posture changes over time, a reference posture Pref is first selected for each person 20. In this example, the posture of person 20 (standing) in the first frame is selected as the reference posture Pref for that person 20. Furthermore, in this example, the dissimilarity (i.e., the degree of posture change) between the posture of person 20 in a given frame and the reference posture is calculated for each frame using the cosine distance between the posture vector of person 20 in that frame and the posture vector of the reference posture. Here, the posture vector of a person in a given posture is a matrix containing the coordinates of the person's skeletal keypoints in that frame. The aforementioned cosine distance can be calculated using the following formula.

number

[0072] Equation (1) provides a set of dissimilarity scores for each person 20 as a function of the frame number, namely D(p_1, p_ref), D(p_2, p_ref), ..., and D(p_N, p_ref), where N is the total number of frames in the video data 30. This set of dissimilarity scores represents how much the posture in each frame differs from the reference posture. Therefore, this set of dissimilarity scores can be used as a change model 40.

[0073] Figures 7A and 7B show the change models 40 for figures 20-5 and 20-6, respectively. From Figure 7A, it can be seen that the posture of figure 20-5 does not change much compared to the reference posture in the first and last frames, but changes significantly between frame number 50 and frame number 80 due to figure 20-5 pushing 20-6. Similarly, from Figure 7B, it can be seen that the posture of figure 20-6 changes rapidly and significantly between frame number 70 and 90 because figure 20-6 is pushed by figure 20-5.

[0074] Next, by considering the times when there are significant posture changes for both individuals 20, the time correlation between the change models 40 in Figures 7A and 7B is calculated. By setting the dissimilarity threshold to 0.5, all frames with a dissimilarity greater than 0.5 are classified as frames with significant posture changes. Therefore, for individual 20-5, a significant posture change occurs between frame numbers 50 and 80, while for individual 20-6, a significant posture change occurs between frame numbers 70 and 90. Thus, significant posture changes occur for both individuals 20-5 and 20-6 between frame numbers 70 and 80 (i.e., the time window for significant posture changes in change model 40-5 overlaps with the time window for significant posture changes in change model 40-6). Therefore, a causal interaction between individuals 20-5 and 20-6 is detected between frames 70 and 80.

[0075] Although this disclosure has been described above with reference to embodiments, this disclosure is not limited to the embodiments described above. Those skilled in the art will understand that various modifications can be made to the configurations and details of this disclosure within the scope of the invention.

[0076] Programs can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Programs may also be provided to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0077] Some or all of the above embodiments may also be described as follows, but are not limited to the following: <Note> (Note 1) For each person detected from the video data, posture information showing their posture over time is extracted. For each of the one or more sets of the aforementioned individuals, based on the posture information, it is determined whether the time intervals for changes in the posture of the individuals within that set are correlated with each other. A causal interaction detection device that detects individuals whose posture change times are determined to be correlated with each other as individuals who have a causal interaction with each other. (Note 2) Determine whether the first time window overlaps with the second time window. The first time window is the period during which the degree of change in the posture of the first person is greater than or equal to a threshold. The second time window is the period during which the degree of change in the posture of the second person is greater than or equal to the threshold. The causal interaction detection device described in Appendix 1, which determines that the time of the posture change of the first person is correlated with the time of the posture change of the second person when the first time window overlaps with the second time window. (Note 3) Determine whether the interval between the first time window and the second time window is less than or equal to the first threshold. The first time window is the period during which the degree of change in the posture of the first person is greater than or equal to the second threshold. The second time window is the period during which the degree of change in the posture of the second person is greater than or equal to the second threshold. The causal interaction detection device according to Appendix 1, which determines that the time of the posture change of the first person is correlated with the time of the posture change of the second person when the interval is less than or equal to the first threshold. (Note 4) Determine whether the distance between the first person and the second person is below a threshold. The causal interaction detection device according to Appendix 1, which determines that the first person does not have a causal interaction with the second person when the distance is determined to be greater than the threshold. (Note 5) Determine whether the first person is interested in the second person. The causal interaction detection device described in Appendix 1, which determines that the first person does not have a causal interaction with the second person when the first person is not facing the second person. (Note 6) The causal interaction detection device according to any one of Appendix 1 to 5, wherein the posture information represents the posture of the person in a certain frame of the video data using the coordinates of a plurality of keypoints of the person detected from that frame. (Note 7) Based on the aforementioned posture information, a change model representing the temporal change in posture is generated for each of the aforementioned individuals. The causal interaction detection device according to any one of the appendices 1 to 6, wherein the change model of the person represents the change in the person's posture in a certain frame of the video data by the degree of dissimilarity between the person's posture in that frame and a reference posture. (Note 8) The causal interaction detection device according to Appendix 7, wherein the dissimilarity of the person in a certain frame of the video data is calculated by the distance between the person's posture in that frame and the reference posture. (Note 9) For each person detected from the video data, posture information showing their posture over time is extracted. For each of the one or more sets of the aforementioned individuals, based on the posture information, it is determined whether the time intervals for changes in the posture of the individuals within that set are correlated with each other. A control method performed by a computer, which includes detecting individuals whose posture change times are determined to be correlated with each other as individuals who have a causal interaction with each other. (Note 10) For each person detected from the video data, posture information showing their posture over time is extracted. For each of the one or more sets of the aforementioned individuals, based on the posture information, it is determined whether the time intervals for changes in the posture of the individuals within that set are correlated with each other. A program that causes a computer to detect individuals whose posture change times are determined to be correlated with each other as individuals who have a causal interaction with each other. [Explanation of Symbols]

[0078] 20 people 30 video data 40 Variation Models 1000 computers 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface 2000 Causal Interaction Detection Device 2020 Posture extraction part 2040 Model Generation Unit 2060 Correlation detection unit

Claims

1. A means for acquiring posture information indicating the posture of a person detected from video data, A detection means for detecting multiple individuals whose posture change times are correlated with each other, based on graph data relating to time generated from the acquired posture information, An output means that outputs information indicating the multiple persons that have been detected, An information processing apparatus having a determination means that determines whether the times of posture changes of each person detected from the video data are correlated with each other, based on the posture information acquired by the acquisition means.

2. The information processing apparatus according to claim 1, wherein the output means outputs information indicating the positions of the plurality of persons in the frames in which the plurality of persons are detected among the video data.

3. The information processing apparatus according to claim 2, wherein the output means outputs a bounding rectangle of the plurality of persons as information indicating the position of the plurality of persons.

4. The determination means is Determine whether the first time window overlaps with the second time window. The first time window is the period during which the degree of change in the posture of the first person is greater than or equal to a threshold. The second time window is the period during which the degree of change in the posture of the second person is greater than or equal to the threshold. The information processing device according to claim 1, wherein, when the first time window overlaps with the second time window, it is determined that the time of the posture change of the first person is correlated with the time of the posture change of the second person.

5. The determination means is Determine whether the interval between the first time window and the second time window is less than or equal to the first threshold. The first time window is the period during which the degree of change in the posture of the first person is greater than or equal to the second threshold. The second time window is the period during which the degree of change in the posture of the second person is greater than or equal to the second threshold. The information processing device according to claim 1, wherein, when the interval is less than or equal to the first threshold, it is determined that the time of the posture change of the first person is correlated with the time of the posture change of the second person.

6. The determination means is Determine whether the distance between the first person and the second person is below a threshold. The information processing apparatus according to claim 1, wherein if the distance is determined to be greater than the threshold, it is determined that the first person has no causal interaction with the second person.

7. The determination means is Determine whether the first person is suitable for the second person. The information processing device according to claim 1, which determines that the first person does not have a causal interaction with the second person when the first person is not facing the second person.

8. A step of obtaining posture information indicating the posture of a person detected from video data, The steps include detecting multiple individuals whose posture change times are correlated with each other, based on graph data relating to time generated from the acquired posture information, The steps include outputting information indicating the multiple persons that were detected, An information processing method performed by a computer, comprising: a determination step of determining whether the times of posture changes for each person detected from the video data are correlated with each other, based on the posture information acquired in the acquisition step.

9. A process to obtain posture information indicating the posture of a person detected from video data, A process to detect multiple individuals whose posture change times are correlated with each other, based on graph data relating to time generated from the acquired posture information, A process that outputs information indicating the multiple persons that were detected, A program that causes a computer to perform a process to determine whether the times of posture changes for each person detected from the video data are correlated with each other, based on the posture information acquired in the aforementioned acquisition process.

Images