Information processing program, information processing method, and information processing device

Abstract

To detect a situation on which measures need to be done from picture data.SOLUTION: An information processor acquires picture data having objects including a person and an object, and specifies the relations between the objects in the acquired picture data by inputting the acquired picture data into a first mechanical learning model. The information processor specifies an action of a person in the picture data by using the feature amount of the person in the acquired picture data. The information processor predicts an action or a state that the person is going to make, by inputting the specified action of the person and the specified relation into a probability model.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Behavior recognition technology is known that recognizes human behavior from video data. For example, a technology is known that recognizes the movements and actions of a person from video data captured by a camera or the like, using skeletal information of the person in the video data. In recent years, the introduction of human behavior recognition has progressed along with the spread of self-checkouts in supermarkets and convenience stores, and the spread of surveillance cameras in schools, trains, public facilities, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 049216 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since the behavior of a person recognized by the behavior recognition technology indicates the behavior that the person has currently or previously performed, it may be too late to take measures after recognizing that the person has performed a certain behavior.

[0005] In one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that can detect a situation requiring countermeasures in advance from video data. [Means for solving the problem]

[0006] In the first proposal, the information processing program causes a computer to execute the following process: acquire video data containing objects including people and objects; input the acquired video data into a first machine learning model to identify the relationships between each of the objects in the acquired video data; identify the behavior of the people in the video data using features of the people included in the acquired video data; and input the identified behavior of the people and the identified relationships into a probabilistic model to predict the future behavior or state of the people. [Effects of the Invention]

[0007] According to one embodiment, a situation requiring countermeasures can be detected in advance from video data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a behavior prediction system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an information processing device that realizes behavior prediction according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating a specific example of behavior prediction. [Figure 4] FIG. 4 is a functional block diagram of the information processing apparatus according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a facial expression recognition rule. [Figure 6] FIG. 6 is a diagram illustrating an example of higher-level behavior specifying rules. [Figure 7] FIG. 7 is a diagram illustrating an example of a probabilistic model. [Figure 8] FIG. 8 is a diagram illustrating the training data. [Figure 9] FIG. 9 is a diagram illustrating machine learning of a relationship model. [Figure 10] FIG. 10 is a diagram for explaining the generation of a skeleton recognition model. [Figure 11] FIG. 11 is a diagram illustrating an example of generating a facial expression recognition model. [Figure 12] FIG. 12 is a diagram showing an example of camera placement. [Figure 13] FIG. 13 is a diagram illustrating the movement of the marker. [Figure 14] FIG. 14 is a diagram illustrating an example of generating higher-level behavior specification rules. [Figure 15] FIG. 15 is a diagram illustrating an example of generating a probabilistic model. [Figure 16] FIG. 16 is a diagram for explaining the identification of the relationship. [Figure 17] FIG. 17 is a diagram for explaining the identification of a relationship by HOID. [Figure 18] FIG. 18 is a diagram illustrating a specific example of identifying a person's current behavior. [Figure 19] FIG. 19 is a diagram illustrating another example of identifying a person's current behavior. [Figure 20] FIG. 20 is a diagram illustrating the prediction of a person's behavior. [Figure 21] FIG. 21 is a diagram illustrating prediction of a person's behavior using a probabilistic model. [Figure 22] FIG. 22 is a flowchart showing the flow of the behavior prediction process. [Figure 23] FIG. 23 is a diagram illustrating an example of a solution that applies behavior prediction related to people and objects. [Figure 24] FIG. 24 is a diagram illustrating an example of a solution that applies person-to-person behavior prediction. [Figure 25] FIG. 25 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes in detail embodiments of the information processing program, information processing method, and information processing device disclosed herein with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the embodiments can be combined as appropriate within a consistent range. [Example]

[0010] <Overall structure> Fig. 1 is a diagram illustrating an example of the overall configuration of a behavior prediction system according to Example 1. As illustrated in Fig. 1, the behavior prediction system includes a store 1, which is an example of a space, a plurality of cameras 2 installed in different locations within the store 1, and an information processing device 10 that analyzes video data.

[0011] Each of the multiple cameras 2 is an example of a surveillance camera that captures an image of a specific area within the store 1, and transmits the captured image data to the information processing device 10. In the following description, the image data may be referred to as "image data." The image data includes multiple frames in a time series. Each frame is assigned a frame number in ascending chronological order. One frame is image data of a still image captured by the camera 2 at a certain timing.

[0012] The information processing device 10 is an example of a computer that analyzes each image data captured by each of the multiple cameras 2. Each of the multiple cameras 2 and the information processing device 10 are connected using various networks, such as the Internet or a dedicated line, whether wired or wireless.

[0013] In recent years, surveillance cameras have been installed not only in stores1 but also in towns and on train platforms, and various services aimed at creating a safe and secure society are being offered using the video data acquired by these cameras. For example, services are being offered that detect shoplifting, accidents, suicides by jumping, and other incidents to help with post-incident handling. However, all of the services currently offered are for post-incident detection, and it is difficult to say that they are effectively utilizing video data from the perspective of preventing incidents before they occur, as they can detect signs of shoplifting, possible suspicious individuals, signs of illness such as seizures, and signs of dementia or Alzheimer's disease that are difficult to identify at first glance.

[0014] Therefore, in Example 1, we will explain an information processing device 10 that combines "behavior analysis" that analyzes a person's current facial expressions and behavior with "context sensing" that detects the surrounding environment, objects, and the relationships between them to realize "behavior prediction" that predicts a person's future behavior and inner thoughts.

[0015] 2 is a diagram illustrating an information processing device 10 that realizes behavior prediction according to Example 1. As illustrated in Fig. 2, the information processing device 10 performs relationship identification and person behavior recognition for video data, and performs person behavior prediction using the identified relationship and the recognized behavior.

[0016] Specifically, the information processing device 10 acquires video data containing objects including people and objects. The information processing device 10 then identifies the relationships between each of the objects in the video data using a relationship model that identifies the relationships between the objects in the video data. Meanwhile, the information processing device 10 identifies the current behavior of the person in the video data using feature quantities of the person included in the video data. The information processing device 10 then inputs the identified current behavior of the person and the identified relationships into a probabilistic model to predict the person's future behavior, such as signs of shoplifting, or the person's condition, such as Alzheimer's disease.

[0017] For example, as shown in FIG. 2, the information processing device 10 inputs video data into a relationship model and identifies relationships between people in the video data or relationships between people and things (objects) in the video data.

[0018] The information processing device 10 also uses a behavior analyzer and a facial expression analyzer to recognize a person's current behavior. Specifically, the behavior analyzer inputs video data into a trained skeleton recognition model to acquire the person's skeleton information, which is an example of a feature. The facial expression recognizer inputs video data into a trained facial expression recognition model to acquire the person's facial expression information, which is an example of a feature. The information processing device 10 then refers to predetermined behavior identification rules to recognize the person's current behavior that corresponds to the combination of the identified person's skeleton information and facial expression information.

[0019] Then, the information processing device 10 inputs the relationship between people or the relationship between a person and an object, and the person's current behavior into a probabilistic model, which is an example of a model using a hidden Markov model, and obtains the result of predicting the person's future behavior.

[0020] Here, the behavior predicted by the information processing device 10 can be short-term prediction or long-term prediction. Fig. 3 is a diagram illustrating a specific example of behavior prediction. As shown in Fig. 3, the behavior prediction made by the information processing device is not limited to "behavior" that can be determined from a person's appearance, such as purchasing behavior or shoplifting, but also includes "emotions" and "conditions" that are difficult to determine from a person's appearance, such as illness, and are influenced by a person's inner self.

[0021] Specifically, the information processing device 10 makes ultra-short-term predictions from a few seconds to a few minutes into the future, predicting the occurrence and necessity of "human assistance by robots" and "online communication assistance." As short-term predictions from a few hours into the future, the information processing device 10 predicts the occurrence of sudden events such as "purchasing behavior in a store," "crimes such as shoplifting and stalking," and "suicidal behavior," as well as events involving little movement from the current location. As medium-term predictions from a few days into the future, the information processing device 10 predicts the occurrence of planned crimes such as "police station attacks" and "domestic violence." As hyper-instrumental predictions from a few months into the future, the information processing device 10 predicts the occurrence of latent events (conditions) that cannot be seen with the naked eye, such as "improvements in academic performance, sales, etc." and "prediction of diseases such as Alzheimer's."

[0022] In this way, the information processing device 10 can detect situations that require countermeasures in advance from video data, and can provide services that aim to create a safe and secure society.

[0023] <Functional configuration> 4 is a functional block diagram illustrating a functional configuration of the information processing device 10 according to the first embodiment. As shown in FIG.

[0024] The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface, etc. For example, the communication unit 11 receives video data, etc. from each camera 2, and outputs the processing results of the information processing device 10, etc. to a pre-designated device, etc.

[0025] The storage unit 20 is a processing unit that stores various data and programs executed by the control unit 30, and is realized by, for example, a memory, a hard disk, etc. The storage unit 20 stores a video data DB 21, a training data DB 22, a relationship model 23, a skeleton recognition model 24, a facial expression recognition model 25, facial expression recognition rules 26, higher-level behavior identification rules 27, and a probability model 28.

[0026] The video data DB21 is a database that stores video data captured by each of the multiple cameras 2 installed in the store 1. For example, the video data DB16 stores video data for each camera 2 or for each time period during which the video was captured.

[0027] The training data DB 22 is a database that stores various training data used to generate graph data and various machine learning models such as a skeleton recognition model 24, a facial expression recognition model 25, and a probabilistic model 28. The training data stored here includes supervised training data to which correct answer information is added, and unsupervised training data to which correct answer information is not added.

[0028] The relationship model 23 is an example of a machine learning model that identifies the relationship between objects included in video data. Specifically, the relationship model 23 is a model for HOID (Human Object Interaction Detection) generated by machine learning that identifies the relationship between people or the relationship between a person and an object.

[0029] For example, when identifying the relationship between persons, a model for HOID is used as the relationship model 23, which identifies and outputs a first class indicating a first person and first area information indicating the area in which the first person appears, a second class indicating a second person and second area information indicating the area in which the second person appears, and the relationship between the first class and the second class, depending on the input of a frame in the video data.

[0030] In addition, when identifying the relationship between a person and an object, a model for HOID is used as the relationship model 23, which identifies and outputs a first class indicating a person and first area information indicating the area where the person appears, a second class indicating an object and second area information indicating the area where the object appears, and the relationship between the first class and the second class.

[0031] The relationships shown here are merely examples, and are not limited to simple relationships such as "holding," but also include complex relationships such as "holding product A in the right hand," "stalking the person walking in front," and "looking behind." As the relationship model 23, the above two HOID models may be used separately, or a single HOID model generated to identify both person-to-person relationships and person-to-object relationships may be used. The relationship model 23 is generated by the control unit 30, which will be described later, but a pre-generated model may also be used.

[0032] The skeleton recognition model 24 is an example of a machine learning model that generates skeleton information, which is an example of a feature of a person. Specifically, the skeleton recognition model 24 outputs two-dimensional skeleton information in response to input image data. For example, the skeleton recognition model 24 is an example of a deep learning device that estimates two-dimensional joint positions (skeletal coordinates) of the head, wrists, waist, ankles, etc. from two-dimensional image data of a person, and recognizes basic movements and rules defined by a user.

[0033] By using this skeleton recognition model 24, it is possible to recognize a person's basic movements and obtain the position of the ankles, the direction of the face, and the direction of the body. Basic movements include, for example, walking, running, and stopping. User-defined rules include transitions of skeleton information corresponding to each action up to picking up a product. The skeleton recognition model 24 is generated by the control unit 30, which will be described later, but it is also possible to use data generated in advance.

[0034] The facial expression recognition model 25 is an example of a machine learning model that generates facial expression information related to facial expressions, which is an example of a person's feature quantity. Specifically, the facial expression recognition model 25 is a machine learning model that estimates AUs (Action Units), which is a method of breaking down and quantifying facial expressions based on facial parts and facial muscles. In response to input image data, the facial expression recognition model 25 outputs a facial expression recognition result such as "AU1:2, AU2:5, AU4:1, ..." expressed in terms of the occurrence intensity (e.g., a five-point scale) of each AU from AU1 to AU28 that are set to identify facial expressions. Note that the facial expression recognition model 25 is generated by the control unit 30, which will be described later, but may also use data generated in advance.

[0035] The facial expression recognition rules 26 are rules for recognizing facial expressions using the output results of the facial expression recognition model 25. FIG. 5 is a diagram showing an example of the facial expression recognition rules 26. As shown in FIG. 5, the facial expression recognition rules 26 store "facial expressions" and "estimation results" in association with each other. The "facial expressions" are the facial expressions of the recognition target, and the "estimation results" are the intensities of each AU from AU1 to AU28 corresponding to each facial expression. The example in FIG. 5 shows that when "AU1 has intensity 2, AU2 has intensity 5, and AU3 has intensity 0...", the expression is recognized as "smiling." The facial expression recognition rules 26 are data registered in advance by an administrator or the like.

[0036] The higher-level behavior identification rule 27 is a rule for identifying a person's current behavior. Fig. 6 is a diagram showing an example of the higher-level behavior identification rule 27. As shown in Fig. 6, the higher-level behavior identification rule 27 is a rule that associates a current behavior with a transition of elemental behaviors that are executed to identify the current behavior.

[0037] In the example of Figure 6, it is defined that the current behavior is identified as XX when element behavior B, element behavior A, element behavior P, and element behavior J are performed in this order. For example, the current behavior is "behavior showing interest in product A," element behavior B is "stopping," element behavior A is "looking at product A," element behavior P is "picking up product A," and element behavior J is "putting product A into the basket."

[0038] Furthermore, each elemental behavior is associated with a basic movement and a facial expression. For example, for elemental behavior B, the basic movement is defined as "as a time series pattern between time t1 and time t3, the basic movement of the whole body transitions to basic movements 02, 03, 03, the basic movement of the right arm transitions to basic movements 27, 25, 25, and the basic movement of the face transitions to basic movements 48, 48, 48," and the facial expression is defined as "as a time series pattern between time t1 and time t3, facial expression H continues."

[0039] Note that notations such as basic action 02 are written using identifiers that identify each basic action, for the sake of explanation, such as stopping, raising arms, and crouching. Similarly, notations such as facial expression H are written using identifiers that identify each facial expression, for the sake of explanation, such as smiling and angry faces. Note that the higher-level behavior identification rules 27 are generated by the control unit 30, which will be described later, but may also use data generated in advance.

[0040] The probabilistic model 28 is an example of a probabilistic model that predicts a person's future behavior and state from basic movements and facial expression information. For example, the probabilistic model 28 can be a hidden Markov model (HMM), in which values ​​of observable variables depending on the state are determined in a situation where the state cannot be directly observed. In this embodiment, the state of the hidden Markov model corresponds to future behavior, and the variable values ​​correspond to current behavior (or a combination of skeletal information and facial expressions) and observation information including relationships. Furthermore, transition probabilities are defined between each state, and a probability distribution representing the relationship between the state and the observation information is defined between the state and the observation information.

[0041] Fig. 7 is a diagram showing an example of the probabilistic model 28. As shown in Fig. 7, the probabilistic model 28 is a hidden Markov model that includes, as states, actions A, B, and C that are prediction targets, and that learns the transition probabilities between the states and the probability distribution between the states and observable information (observation information).

[0042] For example, the hidden Markov model in Fig. 7 indicates that for behavior A, the probability of transitioning from behavior A to behavior A is 30%, the probability of transitioning from behavior A to behavior B is 60%, and the probability of transitioning from behavior A to behavior C is 10%. Furthermore, the hidden Markov model in Fig. 7 indicates that observation information A-1 and observation information A-2 are associated with behavior A, and that when in the state of behavior A, the probability that observation information A-1 is observed is 60%, and the probability that observation information A-2 is observed is 40%.

[0043] That is, by using such a hidden Markov model, the information processing device 10 can estimate behavior using observation information from the past to the present, and predict future behavior (future behavior) based only on the transition probability of the hidden Markov model for the probability of the estimated behavior.

[0044] The observation information used in the hidden Markov model can be changed as desired. For example, a hidden Markov model can be adopted that uses "current behavior" and "relationships" as observation information and "behavior to be predicted" as each state. That is, when the observation information "current behavior and relationships" obtained from video data is input, the hidden Markov model uses this observation information to estimate "behavior" in the near future or at present. Then, the information processing device 10 predicts and outputs the "behavior" with the highest transition probability from this "behavior" as a future behavior according to the hidden Markov model.

[0045] In this way, the information processing device 10 re-predicts the current behavior identified from skeletal information, etc., using a hidden Markov model, and can perform future behavior prediction after increasing the reliability of the current behavior, so improved accuracy can be expected.

[0046] As another example, a hidden Markov model can be employed that uses "skeletal information and facial expressions up to the present" and "relationships" as observation information, and "behavior to be predicted" as each state. That is, when the hidden Markov model receives observation information "current skeletal information, facial expressions, and relationships" obtained from video data, it estimates current "behavior" from this observation information. Then, according to the hidden Markov model, the information processing device 10 predicts and outputs the "behavior" with the highest transition probability from this current "behavior" as a future behavior.

[0047] In this way, the information processing device 10 can predict future behavior after directly predicting current behavior using a hidden Markov model from skeletal information and the like, and therefore, an improvement in prediction speed can be expected.

[0048] 4, the control unit 30 is a processing unit that controls the entire information processing device 10, and is realized by, for example, a processor. The control unit 30 has a pre-processing unit 40 and an operational processing unit 50. The pre-processing unit 40 and the operational processing unit 50 are realized by electronic circuits included in the processor, processes executed by the processor, etc.

[0049] <Pre-processing unit 40> The pre-processing unit 40 is a processing unit that, prior to the operation of behavior prediction, generates each model, rule, etc., using the training data stored in the storage unit 20. The pre-processing unit 40 has a relationship model generation unit 41, a skeleton recognition model generation unit 42, a facial expression recognition model generation unit 43, a rule generation unit 44, and a probabilistic model generation unit 45.

[0050] (Generating a relationship model) The relationship model generation unit 41 is a processing unit that generates the relationship model 23 using training data stored in the training data DB 22. Here, as an example, an example will be described in which a model for HOID using a neural network or the like is generated as the relationship model 23. Note that, as an example only, the generation of a model for HOID that specifies the relationship between a person and an object will be described, but a model for HOID that specifies the relationship between people can also be generated in a similar manner.

[0051] First, we will explain the training data used for machine learning of the HOID model. Fig. 8 is a diagram explaining the training data. As shown in Fig. 8, each training data has image data as input data and correct answer information set for the image data.

[0052] The correct answer information includes a class of the person to be detected (first class), a class of the object to be purchased or operated by the person (second class), a relationship class indicating the interaction between the person and the object, and a Bbox (Bounding Box: object area information) indicating the area of ​​each class. That is, information about the object held by the person is set as the correct answer information. Note that the interaction between a person and an object is an example of the relationship between a person and an object. When used to identify the relationship between people, a class indicating the other person is used as the second class, area information of the other person is used as the area information of the second class, and the relationship between people is used as the relationship class.

[0053] Next, machine learning of the HOID model using training data will be described. FIG. 9 is a diagram illustrating machine learning of the relationship model 23. As shown in FIG. 9, the relationship model generation unit 41 inputs training data into the HOID model and obtains the output result of the HOID model. This output result includes the person class, object class, and relationship (interaction) between the person and object detected by the HOID model. The relationship model generation unit 41 then calculates error information between the correct answer information of the training data and the output result of the HOID model, and performs machine learning of the HOID model by error backpropagation so as to reduce the error. Note that the generation of the HOID model is merely an example, and other methods can be used, or the model can be generated manually by an administrator, etc.

[0054] (Generation of skeleton recognition model 24) The skeleton recognition model generation unit 42 is a processing unit that uses training data to generate the skeleton recognition model 24. Specifically, the skeleton recognition model generation unit 42 generates the skeleton recognition model 24 by supervised learning using training data with correct answer information (labels).

[0055] FIG. 10 is a diagram illustrating generation of the skeleton recognition model 24. As shown in FIG. 10, the skeleton recognition model generation unit 42 inputs image data of basic movements to which labels of basic movements have been assigned to the skeleton recognition model 24, and executes machine learning of the skeleton recognition model 24 so as to reduce the error between the output result of the skeleton recognition model 24 and the label. For example, the skeleton recognition model 24 is a neural network. The skeleton recognition model generation unit 42 executes machine learning of the skeleton recognition model 24 to change the parameters of the neural network. The skeleton recognition model 24 inputs explanatory variables, which are image data (for example, image data of a person performing the basic movement), to the neural network. Then, the skeleton recognition model 24 generates a machine learning model in which the parameters of the neural network have been changed so as to reduce the error between the output result output by the neural network and correct data, which is the label of the basic movement.

[0056] As training data, image data with labels such as "walking," "running," "stopping," "standing," "standing in front of a shelf," "picking up a product," "turning head to the right," "turning head to the left," "looking up," and "tilting head down" can be used. Note that the generation of the skeleton recognition model 24 is merely an example, and other methods can be used. Furthermore, the behavior recognition disclosed in Japanese Patent Application Laid-Open No. 2020-71665 and Japanese Patent Application Laid-Open No. 2020-77343 can also be used as the skeleton recognition model 24.

[0057] (Generation of facial expression recognition model 25) The facial expression recognition model generation unit 43 is a processing unit that uses training data to generate the facial expression recognition model 25. Specifically, the facial expression recognition model generation unit 43 generates the facial expression recognition model 25 by supervised learning using training data with correct answer information (labels).

[0058] Here, the generation of the facial expression recognition model 25 will be described with reference to Fig. 11 to Fig. 13. Fig. 11 is a diagram illustrating an example of the generation of the facial expression recognition model 25. As shown in Fig. 11, the facial expression recognition model generation unit 43 generates training data and performs machine learning on image data captured by the RGB (Red, Green, Blue) camera 25a and the IR (Infrared) camera 25b.

[0059] As shown in FIG. 11, first, the RGB camera 25a and the IR camera 25b are directed toward the face of a person with markers. For example, the RGB camera 25a is a general digital camera that receives visible light and generates an image. For example, the IR camera 25b senses infrared light. The markers are, for example, IR reflective (retroreflective) markers. The IR camera 25b can perform motion capture by utilizing IR reflection from the markers. In the following description, the person to be imaged will be referred to as the subject.

[0060] In the training data generation process, the facial expression recognition model generation unit 43 acquires image data captured by the RGB camera 25a and the results of motion capture by the IR camera 25b. Then, the facial expression recognition model generation unit 43 generates image data 122 in which markers are deleted from the AU generation intensities 121 and the captured image data by image processing. For example, the generation intensities 121 may be data in which the generation intensity of each AU is expressed using a five-level rating from A to E, and annotated such as "AU1:2, AU2:5, AU4:1, ...".

[0061] In the machine learning process, the facial expression recognition model generation unit 43 performs machine learning using the image data 122 and the AU occurrence intensities 121 output from the training data generation process, and generates the facial expression recognition model 25 for estimating the AU occurrence intensities from the image data. The facial expression recognition model generation unit 43 can use the AU occurrence intensities as labels.

[0062] Here, the arrangement of the cameras will be described with reference to FIG. 12. FIG. 12 is a diagram showing an example of the arrangement of the cameras. As shown in FIG. 12, a plurality of IR cameras 25b may constitute a marker tracking system. In this case, the marker tracking system can detect the positions of the IR reflective markers by stereo photography. Furthermore, it is assumed that the relative positional relationships between the plurality of IR cameras 25b are corrected in advance by camera calibration.

[0063] Furthermore, multiple markers are attached to the subject's face to be imaged, covering AU1 to AU28. The positions of the markers change according to changes in the subject's facial expression. For example, marker 401 is placed near the base of the eyebrows. Furthermore, markers 402 and 403 are placed near the facial line. The markers may be placed on the skin corresponding to one or more AUs and the movement of facial muscles. Furthermore, the markers may be placed to avoid areas of the skin where texture changes are significant due to wrinkles, etc.

[0064] Furthermore, the subject wears device 25c with reference point markers attached outside the facial contour. Even if the subject's facial expression changes, the position of the reference point markers attached to device 25c remains unchanged. Therefore, facial expression recognition model generation unit 43 can detect changes in the positions of the markers attached to the face based on changes in their relative positions from the reference point markers. Furthermore, by setting the number of reference markers to three or more, facial expression recognition model generation unit 43 can identify the positions of the markers in three-dimensional space.

[0065] The device 25c is, for example, a headband. Alternatively, the device 25c may be a VR headset, a mask made of a hard material, or the like. In this case, the facial expression recognition model generation unit 43 can use the rigid surface of the device 25c as a reference point marker.

[0066] When the IR camera 25b and the RGB camera 25a are used to capture images, the subject's facial expression changes. This allows the subject's facial expression to change over time as an image. The RGB camera 25a may also capture video. The video can be considered as a number of still images arranged in time series. The subject may change their facial expression freely, or may change their facial expression according to a predetermined scenario.

[0067] The generation intensity of an AU can be determined based on the amount of movement of the marker. Specifically, the facial expression recognition model generation unit 43 can determine the generation intensity based on the amount of movement of the marker calculated based on the distance between a position preset as a determination criterion and the position of the marker.

[0068] Here, the movement of the marker will be described using FIG. 13. FIG. 13 is a diagram illustrating the movement of the marker. (a), (b), and (c) in FIG. 13 are images captured by the RGB camera 25a. The images are assumed to be captured in the order of (a), (b), and (c). For example, (a) is an image when the subject has no expression. The facial expression recognition model generation unit 43 can consider the position of the marker in image (a) as a reference position with a movement amount of 0. As shown in FIG. 13, the subject is frowning. At this time, the position of the marker 401 moves downward in accordance with the change in facial expression. At this time, the distance between the position of the marker 401 and the reference marker attached to the device 25c increases.

[0069] In this way, the facial expression recognition model generation unit 43 identifies image data showing a certain facial expression of the subject and the intensity of each marker when that expression is made, and generates training data with the explanatory variable being "image data" and the objective variable being "intensity of each marker." The facial expression recognition model generation unit 43 then generates the facial expression recognition model 25 through supervised learning using the generated training data. For example, the facial expression recognition model 25 is a neural network. The facial expression recognition model generation unit 43 changes the parameters of the neural network by executing machine learning of the facial expression recognition model 25. The facial expression recognition model 25 inputs the explanatory variables into the neural network. The facial expression recognition model 25 then generates a machine learning model in which the parameters of the neural network are changed so as to reduce the error between the output result output from the neural network and the correct data, which is the objective variable.

[0070] Note that the generation of the facial expression recognition model 25 is merely an example, and other methods can be used. Furthermore, as the facial expression recognition model 25, the behavior recognition disclosed in Japanese Patent Laid-Open No. 2021-111114 can also be used.

[0071] (Generation of higher-level behavior specification rule 27) 4, the rule generation unit 44 is a processing unit that uses past history, etc. to generate higher-level behavior identification rules 27. Specifically, the rule generation unit 44 generates the higher-level behavior identification rules 27 by identifying, from various past video data, transitions in movements and facial expressions that occur before a person performs a certain behavior.

[0072] 14 is a diagram illustrating an example of generating a higher-level behavior identification rule. As shown in FIG. 14, the rule generation unit 44 extracts a plurality of past image data acquired by going back a predetermined time from image data in which a certain behavior XX was performed to the image data in question. Then, the rule generation unit 44 detects basic actions and facial expressions for each of the past image data acquired by going back using a trained model, image analysis, etc.

[0073] Thereafter, the rule generation unit 44 identifies the transitions of elemental actions (transitions of basic movements and transitions of facial expressions) detected up to the time of action XX. For example, the rule generation unit 44 identifies "transitions of basic movements of the whole body, transitions of basic movements of the right arm, and transitions of basic movements of the face between times t1 and t3" and "continuation of facial expression H between times t1 and t3" as elemental action B. Furthermore, the rule generation unit 44 identifies "transitions of basic movements of the right arm and a change from facial expression H to facial expression I between times t4 and t7" as elemental action A.

[0074] In this way, the rule generation unit 44 identifies the order of elemental action B, elemental action A, elemental action P, and elemental action J as the transition of elemental actions leading to action XX. Then, the rule generation unit 44 generates a higher-level action identification rule 27 that associates "action XX" with "the transition of elemental action B, elemental action A, elemental action P, and elemental action J," and stores it in the storage unit 20.

[0075] It should be noted that the generation of the higher level behavior specification rules 27 is merely an example, and other methods may be used, or the higher level behavior specification rules 27 may be generated manually by an administrator or the like.

[0076] (Generation of probabilistic model 28) The probabilistic model generation unit 45 is a processing unit that generates a probabilistic model 28 using training data generated by aggregating past events, past experiences, etc. FIG. 15 is a diagram illustrating an example of generating the probabilistic model 28. As shown in FIG. 15, the probabilistic model generation unit 45 uses training data, which is time-series data of a combination of a certain "action" and "observation information" at the time the action occurred (see (1) in FIG. 15), to learn various probabilities of a hidden Markov model (HMM) (see (2) in FIG. 15).

[0077] For example, the probabilistic model generation unit 45 uses training data to learn the HMM by updating the probability distribution obtained from observation information that indicates the relationship between the behavior transition probability, which probabilistically expresses the state transition in which a certain behavior is likely to occur after another behavior, and the feature values ​​(pairs of skeletal information and facial expressions, or current behavior) of people in the video data.

[0078] The learning method can be any of various known methods. The probabilistic model 28 is not limited to the HMM, and various models that can predict a latent state from observable information can also be used.

[0079] <Operation Processing Unit 50> Returning to Figure 4, the operational processing unit 50 has an acquisition unit 51, a relationship identification unit 52, a behavior identification unit 53, and a behavior prediction unit 54, and is a processing unit that executes behavior prediction processing to predict the future behavior of people appearing in the video data using each model and each rule prepared in advance by the pre-processing unit 40.

[0080] The acquisition unit 51 is a processing unit that acquires video data from each camera 2 and stores the data in the video data DB 21. For example, the acquisition unit 51 may acquire the data from each camera 2 at any time, or may acquire the data periodically.

[0081] (Identifying Relationships) The relationship identification unit 52 is a processing unit that executes a relationship identification process that identifies relationships between people or between people and objects that appear in the video data, using the relationship model 23. Specifically, the relationship identification unit 52 inputs each frame included in the video data into the relationship model 23, and identifies the relationship according to the output result of the relationship model 23. Then, the relationship identification unit 52 outputs the identified relationship to the behavior prediction unit 54.

[0082] FIG. 16 is a diagram illustrating the identification of relationships. As shown in FIG. 16, the relationship identification unit 52 inputs frame 1 into the machine-learned relationship model 23 to identify a first person class, a second person class, and the relationship between the people. As another example, the relationship identification unit 52 inputs a frame into the machine-learned relationship model 23 to identify a person class, an object class, and the relationship between the person and the object. In this way, the relationship identification unit 52 uses the relationship model 23 to identify the relationship between people or the relationship between a person and an object for each frame.

[0083] Fig. 17 is a diagram illustrating the identification of relationships using HOID. As shown in Fig. 17, the relationship identification unit 52 inputs each frame (image data) included in the video data into HOID (relationship model 23) and obtains the output result of HOID. Specifically, the relationship identification unit 52 obtains the person Bbox, the person class name, the object Bbox, the object class name, the probability value of the interaction between the person and the object, and the class name of the interaction between the person and the object.

[0084] As a result, for example, the relationship identification unit 52 identifies "person (customer)" and "person (store clerk)" as person classes, and identifies the relationship between the "person (customer)" and the "person (store clerk)" as "the store clerk talks to the customer." The relationship identification unit 52 also performs the above relationship identification process on each of the subsequent frames such as frame 2 and frame 3, thereby identifying the relationship "talking" or the relationship "handing over" for each frame.

[0085] As another example, the relationship identification unit 52 inputs the frame into the machine-learned relationship model 23 to identify a person class, an object class, and a relationship between a person and an object. For example, the relationship identification unit 52 identifies a "customer" as a person class and a "product" as an object class, and identifies the relationship between the "customer" and the "product" as "the customer owns the product."

[0086] (Identifying current behavior) The behavior identification unit 53 is a processing unit that identifies a person's current behavior from the video data. Specifically, for each frame in the video data, the behavior identification unit 53 acquires skeletal information of each part of the person using the skeleton recognition model 24, and identifies the person's facial expression using the facial expression recognition model 25. Then, the behavior identification unit 53 identifies the person's behavior using the skeletal information of each part of the person and the person's facial expression identified for each frame, and outputs the identified behavior to the behavior prediction unit 54.

[0087] FIG. 18 is a diagram illustrating a specific example of identifying a person's current behavior. As shown in FIG. 18, the behavior identification unit 53 inputs frame 1, which is image data, to the skeleton recognition model 24 and the facial expression recognition model 25. The skeleton recognition model 24 generates skeleton information for each body part in response to the input of frame 1 and outputs the movement of each body part in accordance with the skeleton information for each body part. For example, by using the skeleton recognition model 24, the behavior identification unit 53 can obtain movement information for each body part, such as "face: facing forward, arms: raising, legs: walking, ...". Furthermore, in response to the input of frame 1, the facial expression recognition model 25 outputs the occurrence intensities of each AU from AU1 to AU28, "AU1:2, AU2:5, AU4:1, ...", as the facial expression recognition result. The behavior identification unit 53 then compares the facial expression recognition result with the facial expression recognition rules 26 to identify an expression, such as "smile."

[0088] The behavior identification unit 53 also executes the above identification process for each frame subsequent to frame 2 and frame 3, and identifies, for each frame, the movement information and facial expression of each body part of the person appearing in the frame.

[0089] The behavior identification unit 53 then performs the above-described identification process for each frame to identify transitions in the movements of each body part of the person and transitions in the facial expression. Thereafter, the behavior identification unit 53 compares the transitions in the movements of each body part of the person and transitions in the facial expression with each element behavior of the higher-level behavior identification rule 27 to identify element behavior B.

[0090] Furthermore, the behavior identification unit 53 identifies transitions of elemental behaviors by repeatedly identifying elemental behaviors from the video data. Then, the behavior identification unit 53 compares the transitions of elemental behaviors with the higher-level behavior identification rules 27, thereby being able to identify the current behavior XX of the person appearing in the video data.

[0091] In the example of FIG. 18, both the movement of each body part and the facial expression are identified for each frame, but the present invention is not limited to this. For example, a person's facial expression is affected by changes in the person's inner self, so the facial expression when performing a certain action does not necessarily represent the inner self at the time of the action. In other words, when a person's facial expression changes after performing a certain action, it is often the case that the facial expression changes before and after performing the action. Therefore, the action identification unit 53 can also identify the facial expression using a frame different from the frame used to identify the movement of each body part.

[0092] FIG. 19 is a diagram illustrating another example of identifying a person's current behavior. FIG. 19 illustrates an example in which frame 1, frame 2, and frame 3 are treated as one processing unit, and behavior identification is performed for each frame, and facial expression recognition is performed for the latest frame (frame 3 in this example). As shown in FIG. 19, the behavior identification unit 53 performs skeleton recognition using the skeleton recognition model 24 for frames 1, 2, and 3, as in FIG. 18, and identifies the behavior of each body part for each frame. Meanwhile, the behavior identification unit 53 inputs frame 3 into the facial expression recognition model 25 to identify the person's facial expression.

[0093] Thereafter, the behavior identifying unit 53 identifies the component behavior and the current behavior, similarly to Fig. 18. Note that the example described here is merely illustrative, and the behavior identifying unit 53 may identify the movement of each body part in each frame and recognize facial expression in the first frame. Furthermore, while the behavior identifying unit 53 identifies the movement in each frame, it may also identify facial expressions and facial expression transitions that occur between frames using multiple frames (frames 1 to 3 in Fig. 19) for facial expression recognition.

[0094] (Predicting future behavior) The behavior prediction unit 54 is a processing unit that predicts the future behavior of a person using the person's current behavior and relationships. Specifically, the behavior prediction unit 54 inputs the relationships identified by the relationship identification unit 52 and the person's current behavior identified by the behavior identification unit 53 into the probabilistic model 28, and predicts the person's future behavior. The behavior prediction unit 54 then transmits the prediction result to the administrator's terminal or displays it on a display or the like.

[0095] FIG. 20 is a diagram illustrating prediction of a person's behavior. As shown in FIG. 20, the behavior prediction unit 54 acquires the relationship "hold" identified at that time point at frame 1, acquires the relationship "holding a product in the right hand" identified at that time point at frame 2, and acquires the relationship "holding" identified at that time point and current behavior XX at frame 3. The behavior prediction unit 54 then predicts the person's behavior by inputting the latest relationship and current behavior XX into a probabilistic model 28 (HMM). Note that the relationships shown here are merely examples, and when a model for HOID is used, relationships that can identify "who is doing what and how" are identified, such as "person A is holding product B."

[0096] Fig. 21 is a diagram illustrating prediction of a person's behavior using the probabilistic model 28. As shown in Fig. 21, the behavior prediction unit 54 inputs observation information including behavior transitions and relationship transitions up to the present into an HMM to estimate an "behavior." Then, the behavior prediction unit 54 identifies the "behavior" with the highest transition probability from the estimated "behavior" according to the transition probability of each state of the probabilistic model 28, and predicts the identified "behavior" as the person's future behavior.

[0097] To give a specific example, when the behavior prediction unit 54 obtains observation information that "a person is holding a screwdriver" at the current time, it inputs the observation information from the past to the present into the HMM to estimate the behavior of "a person picking up a screwdriver" as the behavior at the current time. Then, the behavior prediction unit 54 identifies the behavior "a person tightening a screw" that has the highest transition probability from the current behavior "a person picking up a screwdriver" to other behaviors, based only on the transition probabilities of the HMM. As a result, the behavior prediction unit 54 predicts "a person tightening a screw" as a future behavior a certain time later.

[0098] The observation information shown in Fig. 21 is not limited to the transition of observation information, but the latest observation information (current behavior and relationship) can also be used. Also, the observation information can use a "combination of skeletal information and facial expression" instead of the current behavior. Which information to use is determined by the configuration (learning content) of the HMM.

[0099] 20 illustrates an example in which the behavior prediction unit 54 performs behavior prediction using the current behavior and the latest facial expression, but the present invention is not limited to this. As described above, a person's facial expression is significantly affected by changes in the person's inner self, so the latest behavior does not necessarily represent the current facial expression. Therefore, as shown in FIG. 20, the behavior prediction unit 54 can also perform behavior prediction using the current behavior identified by the latest frame 3 and at least one relationship recognized up to frame 3 or a change in the relationship from frame 1 to frame 3.

[0100] At this time, when a current behavior is identified in a first frame, which is an example of image data at a certain time, and a relationship is identified in a second frame, the behavior prediction unit 54 determines whether a second frame is detected within a preset number of frames or a preset time range from the time the first frame was detected. Then, when the behavior prediction unit 54 determines that the second frame was detected within the preset range, it predicts the person's future behavior or state based on the person's behavior included in the first frame and the relationship included in the second frame.

[0101] In other words, the behavior prediction unit 54 predicts a person's future behavior or state using the current behavior and relationships detected at relatively close timing. Note that the preset range can be set arbitrarily, and either the current behavior or the relationships may be identified first.

[0102] <Processing flow> 22 is a flowchart showing the flow of the behavior prediction process. It is assumed here that the pre-processing has already been completed. As shown in FIG. 22, when the operation processing unit 50 acquires one frame (S101: Yes), it inputs the frame to the relationship model 23, and based on the output result of the relationship model 23, it identifies the objects shown in the frame (S102), and identifies the relationships between the objects (S103).

[0103] Then, the operation processing unit 50 inputs the frame into the skeleton recognition model 24 to acquire, for example, skeleton information of the person indicating the movement of each body part (S104). Note that if no person is captured in the frame in S103, the operation processing unit 50 omits S104.

[0104] Furthermore, the operational processing unit 50 inputs the frame into the facial expression recognition model 25 and identifies the facial expression of the person from the output result and the facial expression recognition rule 26 (S105). Note that if no person is captured in the frame in S103, the operational processing unit 50 omits S105.

[0105] Thereafter, the operation processing unit 50 uses the skeletal information and facial expression of the person to identify the relevant element behavior from the higher-level behavior identification rules 27 (S106). Here, if the person's current behavior has not been identified (S107: No), the operation processing unit 50 repeats S101 and subsequent steps for the next frame.

[0106] On the other hand, if the current behavior of the person is identified (S107: Yes), the operation processing unit 50 inputs the current behavior and the identified relationship into the probabilistic model 28 to predict the person's future behavior (S108). Thereafter, the operation processing unit 50 outputs the result of the behavior prediction (S109).

[0107] <Example> Next, a description will be given of specific examples of solutions that contribute to realizing a safe and secure society using behavior prediction by the above-described information processing device 10. Here, a solution using the relationship between people and objects and a solution using the relationship between people will be described.

[0108] (Solutions that utilize the relationship between people and things) Fig. 23 is a diagram illustrating an example of a solution that applies behavior prediction related to people and objects. Fig. 23 illustrates an example of behavior prediction using video data captured by a surveillance camera in a supermarket or the like. Note that the processing described below is executed across one frame or multiple frames within one piece of video data.

[0109] 23, the information processing device 10 uses the relationship model 23 to identify information about people and objects, such as "Person A and Product A, Person B and cart, Person C and wallet, Person D," and information about relationships, such as "Person A's relationship to Product A is 'holding,'" "Person B's relationship to the cart is 'pushing,'" and "Person C's relationship to the wallet is 'touching,'" from frames in the video data. Here, since no object is detected for Person D, no relationship is identified.

[0110] In addition, the information processing device 10 performs skeleton recognition using the skeleton recognition model 24 and facial expression recognition using the facial expression recognition model 25, and uses the recognition results to identify person A's current behavior as "holding product A," person B's current behavior as "pushing a cart," person C's current behavior as "walking," and person D's current behavior as "stopping."

[0111] Then, the information processing device 10 predicts the future behavior of person A, "high possibility of purchasing product A," the future behavior of person B, "high possibility of shoplifting," and the future behavior of person C, "high possibility of leaving the store without making a purchase," by performing behavior prediction using the current behavior and the relationship. Here, since the relationship with person D has not been specified, person D is not subject to behavior prediction.

[0112] In other words, the information processing device 10 identifies customers moving around a specific area of ​​the video data, which is an area of ​​the store shelves, and the products that the customers are going to purchase, identifies the type of behavior the customers have toward the products (e.g., looking, holding, etc.) as a relationship, and predicts the customer's purchasing behavior toward the products (e.g., purchase, shoplifting, etc.).

[0113] In this way, the information processing device 10 can use the above-mentioned behavior prediction to help analyze purchasing behavior such as actions and routes leading up to a purchase, purchase marketing, etc. Furthermore, the information processing device 10 can detect a person who is likely to commit a crime such as shoplifting, like person B, and can be used to prevent crimes by strengthening surveillance of that person, etc.

[0114] (Solutions that utilize relationships between people) Fig. 24 is a diagram illustrating an example of a solution that applies person-to-person behavior prediction. Fig. 24 illustrates an example of behavior prediction using video data captured at night by a surveillance camera on a street or the like. Note that the processing described below is executed across one frame or multiple frames within one piece of video data.

[0115] As shown in FIG. 24, the information processing device 10 uses the relationship model 23 to identify information about people, "Person A (female: in her 20s) and Person B (male: in his 40s)," and information about the relationship, "Person A's relationship with Person B is 'close,'" and "Person B's relationship with Person A is 'stalking,'" from frames in the video data.

[0116] In addition, the information processing device 10 performs skeleton recognition using the skeleton recognition model 24 and facial expression recognition using the facial expression recognition model 25, and uses the recognition results to identify person A's current behavior of "walking in front of person B" and person B's current behavior of "hiding."

[0117] Then, the information processing device 10 predicts the future behavior of person A, "high possibility of being attacked by person B," and the future behavior of person B, "high possibility of attacking person A," by using the current behavior and the relationship to predict the behavior.

[0118] In other words, assuming that person A is the victim and person B is the perpetrator, the information processing device 10 can predict whether person B will commit a crime against person A based on the perpetrator's relationship with the victim, "stalking." As a result, the information processing device 10 can detect locations where a crime is likely to occur using the above-mentioned behavior prediction, and take preventative measures such as dispatching police officers. This can also be useful in considering countermeasures such as increasing the number of streetlights in such locations.

[0119] <Effects> As described above, the information processing device 10 can predict signs of accidents or crimes, rather than the occurrence of such accidents or crimes, and can therefore detect situations requiring countermeasures in advance from video data. Furthermore, the information processing device 10 can perform behavior prediction from video data captured by general cameras such as surveillance cameras, and therefore can be introduced into existing systems without requiring a complex system configuration or new devices. Furthermore, because the information processing device 10 is introduced into an existing system, costs can be reduced compared to building a new system. Furthermore, the information processing device 10 can predict not only simple behaviors that are continuous from past and present behaviors, but also complex behaviors of individuals that cannot be simply identified from past and present behaviors. This allows the information processing device 10 to improve the accuracy of predicting individuals' future behaviors.

[0120] Furthermore, since the information processing device 10 can realize behavior prediction using two-dimensional image data without using three-dimensional image data, the processing speed can be increased compared to processing using laser sensors that have been used recently. Furthermore, the high-speed processing of the information processing device 10 allows it to quickly detect situations that require countermeasures in advance. [Example]

[0121] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0122] <Numbers, etc.> The numerical examples, number of cameras, label names, rule examples, behavior examples, and state examples used in the above embodiments are merely examples and can be changed as desired. The process flow described in each flowchart can also be changed as appropriate within a consistent range. While the above embodiments use a store as an example, the present invention is not limited to this and can also be applied to, for example, warehouses, factories, classrooms, train cars, and airplane cabins. The relationship model 23 is an example of a first machine learning model, the skeleton recognition model 24 is an example of a second machine learning model, and the facial expression recognition model 25 is an example of a third machine learning model.

[0123] <System> The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0124] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown. In other words, all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0125] Furthermore, all or any part of the processing functions performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0126] <Hardware> Fig. 25 is a diagram illustrating an example of a hardware configuration. As shown in Fig. 25, an information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 25 are connected to each other via a bus or the like.

[0127] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0128] The processor 10d reads out a program that executes the same processing as each processing unit shown in FIG. 4 from the HDD 10b or the like and loads it into the memory 10c, thereby operating a process that executes each function described in FIG. 4 or the like. For example, this process executes the same function as each processing unit possessed by the information processing device 10. Specifically, the processor 10d reads out a program having the same function as the pre-processing unit 40, the operational processing unit 50, or the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processing as the pre-processing unit 40, the operational processing unit 50, or the like.

[0129] In this way, the information processing device 10 operates as an information processing device that executes a behavior prediction method by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 10. For example, the above-described embodiment may also be applied in the same way to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0130] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and may be read out from the recording medium and executed by a computer. [Explanation of symbols]

[0131] 10. Information processing equipment 11 Communications Department 20 Memory section 21 Video Data DB 22 Training Data DB 23 Relationship Model 24 Skeleton Recognition Model 25 Facial Expression Recognition Model 26 Facial Expression Recognition Rules 27. Higher Order Behavior Identification Rules 28 Probabilistic Models 30 Control Unit 40 Pre-processing section 41 Relationship model generation unit 42 Skeleton recognition model generation unit 43 Facial Expression Recognition Model Generation Unit 44 Rule Generation Unit 45 Probabilistic model generation unit 50 Operation Processing Unit 51 Acquisition Department 52 Relationship Identification Unit 53 Behavior Specification Department 54 Behavior Prediction Department

Claims

1. Acquire video data having targets including people and objects; inputting the acquired video data into a first machine learning model to identify relationships between objects in the acquired video data; Identifying a current action of a person in the video data using features including facial expressions of the person included in the acquired video data and skeletal information indicating joint positions of the person; inputting the identified person's current behavior and the identified relationships into a probabilistic model to predict future behaviors to be performed by the person whose current behavior is identified, including emotions and states influenced by the person's inner world; An information processing program that causes a computer to execute a process.

2. The behavior of the identified person is Among the plurality of frames constituting the video data, a first frame is included, The identified relationship is: Among the plurality of frames constituting the video data, a second frame is included, The prediction process includes: determining whether the second frame is detected within a predetermined number of frames or a predetermined time range from the time when the first frame is detected; The information processing program according to claim 1, characterized in that, when it is determined that the second frame is detected within a predetermined number of frames or time range, future actions of the person are predicted based on the actions of the person contained in the first frame and the relationships contained in the second frame.

3. The process of identifying the behavior includes: A second machine learning model is obtained by changing the parameters of the neural network so that an error between an output result output by the neural network when explanatory variables that are image data are input to the neural network and correct data that are labels of actions is reduced; inputting the video data into the second machine learning model to identify the movements of each body part of the person; obtaining a third machine learning model by changing parameters of the neural network so as to reduce an error between an output result output from the neural network when an explanatory variable, which is image data including the facial expression of the person, is input to the neural network and correct answer data indicating a target variable, which is the intensity of each marker in the facial expression of the person; inputting the video data into the third machine learning model to generate marker intensities for the person; and identifying facial expressions of the person using the generated markers; The information processing program according to claim 1, characterized in that the current behavior of the person in the video data is identified by comparing the movements of each part of the identified person's body, the facial expressions of the identified person, and predetermined rules.

4. The first machine learning model is a model for HOID (Human Object Interaction Detection) generated by machine learning to identify a first class indicating a person and first area information indicating an area in which the person appears, a second class indicating an object and second area information indicating an area in which the object appears, and a relationship between the first class and the second class; The process of identifying the relationship includes: inputting the video data into a model for the HOID; As an output of the HOID model, the first class and the first area information, the second class and the second area information, and the relationship between the first class and the second class are obtained for people and objects appearing in the video data; 2. The information processing program according to claim 1, further comprising: determining a relationship between the person and the object based on the acquired result.

5. the person is a customer moving through a predetermined area of ​​the video data; the object is a product that the customer is interested in purchasing; the relationship is a type of behavior of the person with respect to the product, 5. The information processing program according to claim 4, wherein the prediction process predicts the customer's purchase behavior regarding a product as a future behavior of the person.

6. The first machine learning model is a model for HOID (Human Object Interaction Detection) generated by machine learning to identify a first class indicating a first person and first area information indicating an area in which the first person appears, a second class indicating a second person and second area information indicating an area in which the second person appears, and a relationship between the first class and the second class; The process of identifying the relationship includes: inputting the video data into a model for the HOID; As an output of the HOID model, for each person appearing in the video data, the first class and the first area information, the second class and the second area information, and a relationship between the first class and the second class are obtained; 2. The information processing program according to claim 1, further comprising: determining a relationship between the people based on the acquired result.

7. The first person is the perpetrator, The second person is the victim, the relationship is a type of behavior of the first person toward the second person; 7. The information processing program according to claim 6, wherein the prediction process predicts a criminal act by the first person against the second person as a future action to be taken by the person.

8. The prediction process includes: inputting the identified current behavior of the person and the identified relationship into the probabilistic model to estimate a behavior or state; An information processing program according to any one of claims 1 to 7, characterized in that the behavior or state with the highest probability of state transition from the estimated behavior or state is predicted as the person's future behavior according to the probabilistic model.

9. Acquire video data having targets including people and objects; inputting the acquired video data into a first machine learning model to identify relationships between objects in the acquired video data; Identifying a current action of a person in the video data using features including facial expressions of the person included in the acquired video data and skeletal information indicating joint positions of the person; inputting the identified person's current behavior and the identified relationships into a probabilistic model to predict future behaviors to be performed by the person whose current behavior is identified, including emotions and states influenced by the person's inner world; An information processing method in which processing is performed by a computer.

10. Acquire video data having targets including people and objects; inputting the acquired video data into a first machine learning model to identify relationships between objects in the acquired video data; Identifying a current action of a person in the video data using features including facial expressions of the person included in the acquired video data and skeletal information indicating joint positions of the person; inputting the identified person's current behavior and the identified relationships into a probabilistic model to predict future behaviors to be performed by the person whose current behavior is identified, including emotions and states influenced by the person's inner world; An information processing device having a control unit.

Images