Information processing program, information processing method, and information processing device

Abstract

To facilitate learning a model that recognizes a specific motion with high accuracy.SOLUTION: An information processing device 100 specifies pieces of positional information on respective body regions of paired two body regions among body regions of a first person in video in which the first person is captured. The information processing device 100 generates a first element 110 indicating the sum of the pieces of positional information on the respective body regions of the specified right and left paired two body regions. The information processing device 100 generates a third element 130 indicating an absolute value of a second element 120 indicating a difference between the pieces of positional information on the respective body regions of the specified right and left paired two body regions. The information processing device 100 learns a model based on the generated first element 110 and the generated third element 130.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, there is a technique for detecting skeletal information of a person in each frame of a video by analyzing a video in which a person is reflected. There is also a technique for learning and using a machine learning model that recognizes a person's behavior according to the input skeletal information of the person.

[0003] As a prior art, for example, among the coordinates corresponding to each of a plurality of body parts, with respect to a reference axis formed by connecting the coordinates corresponding to a plurality of predetermined reference parts, the coordinates corresponding to each of the plurality of body parts are converted into coordinates that are line-symmetric. Also, for example, there is a technique for determining the presence or absence of a symmetric line segment figure based on pixels in a region corresponding to the periphery of a pair of vertical lines and oblique lines in an input image and pixels in a region outside the periphery.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art, it may be difficult to accurately recognize a specific behavior of a person using a machine learning model. For example, it is difficult to learn a machine learning model that accurately recognizes an action of "grasping an object with the right hand" and an action of "grasping an object with the left hand" as the same action of "grasping an object with one hand".

[0006] In one aspect, the present invention aims to make it easier to train models that accurately recognize specific behaviors. [Means for solving the problem]

[0007] According to one embodiment, an information processing program, information processing method, and information processing device are proposed that acquire video footage of a first person, analyze the acquired video footage to identify the positional information of two pairs of body parts of the first person in the video footage, generate a first component representing the sum of the positional information of the identified body parts, generate a third component representing the absolute value of the second component representing the difference between the positional information of the identified body parts, and learn a model that outputs posture information of a second person from video footage of a second person based on the generated first component and the generated third component. [Effects of the Invention]

[0008] According to one embodiment, it becomes possible to make it easier to train a model that accurately recognizes specific behaviors. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is an explanatory diagram showing one embodiment of the information processing method according to the embodiment. [Figure 2] Figure 2 is an explanatory diagram showing an example of the information processing system 200. [Figure 3] Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. [Figure 4] Figure 4 is a block diagram showing an example of the hardware configuration of the video capture device 201. [Figure 5] Figure 5 is a block diagram showing an example of the functional configuration of the information processing device 100. [Figure 6] Figure 6 is an explanatory diagram showing the operation flow of the information processing device 100. [Figure 7]FIG. 7 is an explanatory diagram showing an example of the stored content of the pair management table 700. [Figure 8] FIG. 8 is an explanatory diagram showing an example of obtaining the coordinate information of each skeleton. [Figure 9] FIG. 9 is an explanatory diagram (part 1) showing an example of converting coordinate information. [Figure 10] FIG. 10 is an explanatory diagram (part 2) showing an example of converting coordinate information. [Figure 11] FIG. 11 is an explanatory diagram (part 1) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 12] FIG. 12 is an explanatory diagram (part 2) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 13] FIG. 13 is an explanatory diagram (part 3) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 14] FIG. 14 is an explanatory diagram (part 4) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 15] FIG. 15 is an explanatory diagram (part 5) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 16] FIG. 16 is an explanatory diagram (part 6) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 17] FIG. 17 is an explanatory diagram (part 7) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 18] FIG. 18 is an explanatory diagram (part 8) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 19] FIG. 19 is an explanatory diagram (part 9) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 20] FIG. 20 is an explanatory diagram (part 10) showing a numerical example representing the effect of the information processing apparatus 100. [Figure 21] FIG. 21 is a flowchart showing an example of the learning processing procedure. [Figure 22] FIG. 22 is a flowchart showing an example of the recognition processing procedure.

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, embodiments of an information processing program, an information processing method, and an information processing apparatus according to the present invention will be described in detail with reference to the drawings.

[0011] (An example of the information processing method according to the embodiment) FIG. 1 is an explanatory diagram showing an example of the information processing method according to the embodiment. The information processing apparatus 100 is a computer for learning a model. The information processing apparatus 100 is, for example, a server or a PC (Personal Computer).

[0012] The model is, for example, an opportunity learning model that recognizes a person's behavior according to the input skeletal information of the person. The skeletal information includes, for example, coordinate values representing the positions of each of the plurality of skeletons possessed by the person. The position of the skeleton is, for example, a position such as the neck, head, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, or left foot.

[0013] The model can be used, for example, to recognize the behavior of a person who is a customer and conduct marketing, to recognize the behavior of a person who is a care recipient and detect a fall of the person, or to recognize the behavior of a person shown in a surveillance camera and discover a suspicious person. Therefore, it is desired to learn a model that can accurately recognize a person's behavior according to the input skeletal information of the person.

[0014] However, conventionally, it may be difficult to learn a model that can accurately recognize a person's behavior according to the input skeletal information of the person. Therefore, even when using the learned model, it may be difficult to accurately recognize a specific behavior of a person.

[0015] For example, when recognizing a person's actions, it is sometimes preferable to recognize similar actions involving two symmetrical body parts as the same action. Specifically, it may be preferable to recognize the action of "grabbing an object with the right hand" and the action of "grabbing an object with the left hand" as the same action of "grabbing an object with one hand."

[0016] Traditionally, it has been difficult to train a model that can accurately recognize the same type of action for each of two symmetrical body parts as the same action. Specifically, it is difficult to train a model that can accurately recognize the action of "grabbing an object with the right hand" and the action of "grabbing an object with the left hand" as the same action of "grabbing an object with one hand."

[0017] In contrast, one possible approach is to train a model based on training data in which two sets of skeletal information corresponding to two different body parts (a pair of left and right-handed body parts) performing the same type of action are associated with labels indicating the same action. Specifically, one could prepare first training data in which the first set of skeletal information for the first action, "grabbing an object with the right hand," is associated with the label indicating the action of "grabbing an object with one hand." Specifically, one could prepare second training data in which the second set of skeletal information for the second action, "grabbing an object with the left hand," is associated with the label indicating the action of "grabbing an object with one hand." Specifically, one could train a model based on the first and second sets of training data prepared.

[0018] In this method, two sets of training data are treated as training data containing labels indicating the same action, even though the skeletal information contained in each set of training data has different features. As a result, fluctuations in the training data tend to be large, and there is a problem in that it is difficult to train a model that can accurately recognize the same type of action for each of two symmetrical body parts as the same action.

[0019] Another possible approach is to learn two models that correspond to different actions performed by two different body parts in a pair. Specifically, one could learn a first model capable of recognizing the first action, "grabbing an object with the right hand," and a second model capable of recognizing the second action, "grabbing an object with the left hand." More precisely, whether the first or second action is recognized, it could be treated as the same action, "grabbing an object with one hand."

[0020] This method requires training two models to accurately recognize the action of "grabbing an object with one hand," which leads to an increase in the amount of training data needed, resulting in increased processing load and processing time during training. Furthermore, it increases the processing load and processing time required for accuracy verification or maintenance of the two trained models. Therefore, it leads to increased development and operational costs when attempting to recognize the action of "grabbing an object with one hand."

[0021] Another possible approach is to process the skeletal information of two corresponding body parts when the same type of action is performed on different body parts of a pair, treat them as identical skeletal information, and then train the model.

[0022] Specifically, one approach is to set the coordinate value representing the hand position as the coordinate value with the larger absolute value between the coordinate value representing the right hand position and the coordinate value representing the left hand position. More specifically, one approach is to process the skeletal information of each of the two sets of skeletal information so that the combination of the coordinate values ​​representing the right hand position and the coordinate values ​​representing the left hand position is replaced with the coordinate value representing the hand position, and then train the model.

[0023] In this method, the coordinate value with the smaller absolute value between the coordinate value representing the position of the right hand and the coordinate value representing the position of the left hand is not reflected in the model. Therefore, there is a problem in that it is difficult to train a model that can accurately recognize the same type of action for each of the two body parts that are a pair as the same action.

[0024] Specifically, one approach is to set the coordinate value of the right hand that is further from the center of the body as the coordinate value of the first hand, and the coordinate value of the left hand that is closer to the center of the body as the coordinate value of the second hand. Specifically, one approach is to process the skeletal information so that the coordinate values ​​of the right hand and the left hand are replaced with the coordinate value of the first hand that is further from the center of the body and the coordinate value of the second hand that is closer to the center of the body. Specifically, one approach is to train a model based on the processed skeletal information of each hand.

[0025] In this method, when processing multiple skeletal data points in a time series and attempting to train a model based on this processed data, the model may not be able to learn properly. Specifically, the relative distances between the coordinate values ​​indicating the position of the right hand from the center of the body and the coordinate values ​​indicating the position of the left hand may be reversed in the middle of the time series. As a result, the hand treated as the first hand and the hand treated as the second hand may be reversed in the middle of the time series. Furthermore, the processed multiple skeletal data points may represent physically unnatural behaviors of a person, such as one hand appearing instantaneously in a different location instead of moving continuously. Consequently, there is a problem in that it is difficult to train a model that can accurately recognize the same type of action for each of two symmetrical body parts as the same action.

[0026] Therefore, this embodiment describes an information processing method that can facilitate the training of a machine learning model that accurately recognizes specific actions.

[0027] In Figure 1, the information processing device 100 identifies the positional information of each of two pairs of body parts that the first person possesses in the video footage in which the first person is shown. The video footage includes, for example, one or more frames. The body parts include, for example, the neck, head, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, or left foot.

[0028] A pair of body parts is, for example, a combination of a right hand and a left hand. Positional information represents, for example, a combination of multiple component values ​​in different axes that represent the position of the body part in three-dimensional space. The axes are, for example, called the X-axis, Y-axis, and Z-axis. The position of the body part in three-dimensional space is, for example, the position of the skeleton representing the body part in three-dimensional space. Positional information may also be, for example, a vector.

[0029] Positional information may represent, for example, a combination of multiple component values ​​in different axes that represent the position of a body part in two-dimensional space. The axes are, for example, called the X-axis and the Y-axis. The two-dimensional space may correspond to, for example, a frame or a defined area of ​​size in which a person is captured within the frame. The position of a body part in two-dimensional space is, for example, the position of the skeleton representing the body part in two-dimensional space. Positional information may be, for example, a vector. Positional information may be, for example, polar coordinates.

[0030] The information processing device 100 identifies the positional information of each of two paired body parts of the first person in each frame of a video showing the first person. Specifically, the information processing device 100 acquires a video showing the first person. Specifically, the information processing device 100 identifies the positional information by analyzing the acquired video and generating positional information of each of two left-right paired body parts of the first person in each frame of the video.

[0031] (1-1) The information processing device 100 generates a first component 110 that represents the sum of the positional information of each of the two identified left and right body parts. For example, the information processing device 100 generates a first component 110 that represents the sum of a vector 101 representing the positional information of the right hand body part and a vector 102 representing the positional information of the left hand body part. The first component 110 is, for example, a vector.

[0032] Specifically, the information processing device 100 generates a first component 110 which represents the sum of a vector 101 representing the position information of the right hand and a vector 102 representing the position information of the left hand in each frame of the video showing the first person. This allows the information processing device 100 to obtain a first component 110 that represents the commonalities in the position information of the two paired body parts, and can serve as a guideline for learning the model.

[0033] (1-2) The information processing device 100 generates a third component 130 which represents the absolute value of a second component 120 that represents the difference in positional information of each of the two identified left and right body parts. For example, the information processing device 100 generates a third component 130 which represents the absolute value of a second component 120 that represents the difference between a vector 101 that represents the positional information of the right hand body part and a vector 102 that represents the positional information of the left hand body part. The second component 120 is, for example, a vector.

[0034] The third component 130 is, for example, a vector with a predetermined direction whose magnitude is the absolute value of the second component 120. The predetermined direction is determined, for example, according to a predetermined rule, based on the direction of the second component 120. The predetermined rule is a rule that determines the direction of the third component to be either the same direction as the second component or the opposite direction of the second component. The predetermined rule is a rule that determines the direction of the third component to be either the same direction as the second component or the opposite direction of the second component, with one of the axial direction components being positive.

[0035] Specifically, the information processing device 100 generates a third component 130 that represents the absolute value of the second component 120, which represents the difference between the vector 101 representing the position information of the right hand body part and the vector 102 representing the position information of the left hand body part in each frame of the video showing the first person. This allows the information processing device 100 to obtain a third component 130 that represents the characteristics of the differences in the position information of each body part and can serve as a guideline when training the model. Furthermore, the information processing device 100 can generate a third component 130 with adjusted orientation so that the generated third component 130 matches even if the movements of the two body parts in a left-right pair are reversed.

[0036] (1-3) The information processing device 100 learns a model based on the generated first component 110 and the generated third component 130. The second person may be, for example, the same person as the first person. The second person may be, for example, a different person from the first person.

[0037] The model has the function of outputting posture information of a second person from video footage in which the second person is shown. For example, the model has the function of outputting posture information of a second person in response to input of explanatory variables that include the positional information of two paired body parts of the second person in video footage in which the second person is shown.

[0038] The model may be, for example, a neural network. The model may also be, for example, a mathematical formula. The model may also be, for example, a tree structure. The posture information may be, for example, information indicating whether or not a second person is in a specific posture. The posture information may also be, for example, information indicating whether or not a second person is in a posture corresponding to a specific action. The posture information may also be, for example, information indicating whether or not a second person has performed a specific action.

[0039] The information processing device 100 learns a model based on the generated first component 110 and the generated third component 130 in each frame of the video showing the first person. Specifically, the information processing device 100 obtains labels indicating the correct actions of the first person in each frame of the video showing the first person. Specifically, for each frame of the video showing the first person, the information processing device 100 generates training data that associates input samples containing the first component 110 and the third component 130 in that frame with labels indicating the correct actions of the first person.

[0040] Specifically, the information processing device 100 trains a model based on the generated training data using logistic regression. Specifically, the information processing device 100 may train a model based on the generated training data using a method other than logistic regression. Specifically, the information processing device 100 may train a model based on the generated training data using backpropagation.

[0041] As a result, the information processing device 100 can learn a model that has the function of outputting posture information of a second person from an image showing the second person. For example, the information processing device 100 can convert the positional information of each of two body parts that are a left-right pair into a combination of a first component 110 and a third component 130 having predetermined properties.

[0042] The predetermined properties include, for example, the property that even if the movements of two body parts in a left-right pair are reversed, the combination of the first component 110 and the third component 130 remains the same. The predetermined properties include, for example, the property that the characteristics of the movements of two body parts in a left-right pair are reflected in the combination of the first component 110 and the third component 130. The predetermined properties include, for example, the property that the first component 110 changes continuously and the third component 130 changes continuously along a time series.

[0043] Therefore, the information processing device 100 can learn a model that can accurately estimate the posture information of a second person from an image showing the second person by utilizing the combination of the first component 110 and the third component 130. Using the learned model, the information processing device 100 can recognize similar actions related to each of two body parts that are paired left and right as the same action.

[0044] Since the information processing device 100 only needs to train a single model, it can suppress an increase in the amount of training data to be prepared, thereby suppressing an increase in processing load and processing time during training. The information processing device 100 can suppress an increase in processing load and processing time for accuracy verification or maintenance of the trained model. The information processing device 100 can suppress an increase in development costs and operational costs when making specific actions recognizable.

[0045] (1-4) The information processing device 100 may acquire video footage of the second person and use a trained model to acquire posture information of the second person. For example, by analyzing the acquired video footage of the second person, the information processing device 100 identifies the positional information of each of two pairs of body parts that the second person possesses in each frame of the video footage.

[0046] The information processing device 100 generates a fourth component, for example, which represents the sum of the positional information of two identified left-right paired body parts in each frame of the video showing the second person. The information processing device 100 also generates a sixth component, for example, which represents the absolute value of the fifth component, which represents the difference in the positional information of two identified left-right paired body parts in each frame of the video showing the second person.

[0047] The information processing device 100, for example, uses a learned model based on the generated fourth component and the generated sixth component to acquire posture information indicating whether the second person is in a posture corresponding to a specific action. This allows the information processing device 100 to generate posture information with high accuracy. The information processing device 100 can then make the posture information available for use.

[0048] This explanation describes the case where the information processing device 100 operates independently, but it is not limited to this. For example, the information processing device 100 may collaborate with other computers. For example, multiple computers may collaborate to realize the functions of the information processing device 100. Specifically, the functions of the information processing device 100 may be realized on the cloud.

[0049] Here, we have described a case in which the information processing device 100 analyzes an image of a first person, but it is not limited to this case. For example, the information processing device 100 may be able to communicate with another computer that is analyzing an image of a first person. In this case, the information processing device 100 identifies the location information by receiving from the other computer the location information of each of two pairs of body parts that the first person possesses in the image of the first person.

[0050] Here, we have described a case where the information processing device 100 learns a model based on the first component and the third component it generates, but it is not limited to this. For example, the information processing device 100 may transmit the first component and the third component it generates to another computer. In this case, the other computer learns a model based on the first component and the third component it receives.

[0051] This explanation describes a case where the information processing device 100 uses a learned model to acquire posture information of a second person, but it is not limited to this case. For example, the information processing device 100 may transmit the learned model to another computer. In this case, the other computer acquires video footage of the second person and uses the received model to acquire posture information of the second person.

[0052] (An example of information processing system 200) Next, using Figure 2, we will describe an example of an information processing system 200 to which the information processing device 100 shown in Figure 1 is applied.

[0053] Figure 2 is an explanatory diagram showing an example of an information processing system 200. In Figure 2, the information processing system 200 includes an information processing device 100, one or more video capturing devices 201, and one or more client devices 202.

[0054] In the information processing system 200, the information processing device 100 and the video capture device 201 are connected via a wired or wireless network 210. The network 210 can be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. Also in the information processing system 200, the information processing device 100 and the client device 202 are connected via a wired or wireless network 210.

[0055] The information processing device 100 is a computer for learning a model. The information processing device 100 stores, for example, a first machine learning model. The first machine learning model has a function to output positional information of body parts of a person shown in a video, in response to the input video. Body parts include, for example, the neck, head, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, or left foot. The positional information indicates the position of the body part in three-dimensional space. The positional information indicates, for example, a combination of multiple component values ​​in different axial directions that represent the position of the body part in three-dimensional space. The position is, for example, the position of the skeleton representing the body part in three-dimensional space.

[0056] Position information may, for example, indicate the position of a body part in two-dimensional space. Position information may, for example, indicate a combination of multiple component values ​​in different axial directions that represent the position of a body part in two-dimensional space. Position is, for example, the position of the skeleton representing the body part in two-dimensional space. Position information may, for example, be a vector. Position information may, for example, be polar coordinates.

[0057] Specifically, the first machine learning model has the function of outputting positional information of body parts of a person in a given frame, in response to the input of each frame of a video. The first machine learning model is, for example, an AI (Artificial Intelligence) model. The first machine learning model can be implemented, for example, by a neural network, mathematical formulas, or a tree structure.

[0058] The information processing device 100 acquires, for example, an image of a first person. There may be multiple people who could be the first person. The information processing device 100 may also acquire, for example, an image of each of the multiple people who could be the first person. Specifically, the information processing device 100 acquires the image of the first person by receiving it from the video recording device 201.

[0059] The information processing device 100, for example, analyzes the acquired video footage of the first person to identify the positional information of each of two pairs of body parts that belong to the first person in each frame of the video footage. Specifically, the information processing device 100 uses a first machine learning model to identify the positional information of each of two pairs of body parts that belong to the first person in each frame of the acquired video footage of the first person.

[0060] The information processing device 100 generates a first component that represents the sum of the positional information of two identified left-right pairs of body parts in each frame of the video showing the first person. The information processing device 100 generates a third component that represents the absolute value of the second component that represents the difference in the positional information of two identified left-right pairs of body parts in each frame of the video showing the first person. The information processing device 100 learns a model based on the generated first component and the generated third component in each frame of the video showing the first person.

[0061] The information processing device 100 acquires, for example, video footage of a second person. The second person is, for example, a person whose actions are to be determined. Specifically, the information processing device 100 acquires video footage of the second person by receiving it from the video recording device 201.

[0062] The information processing device 100, for example, analyzes the acquired video footage of the second person to identify the positional information of each of two pairs of body parts that the second person possesses in each frame of the video footage. Specifically, the information processing device 100 uses a first machine learning model to identify the positional information of each of two pairs of body parts that the second person possesses in each frame of the acquired video footage of the second person, based on the video footage.

[0063] The information processing device 100 generates a fourth component, for example, in each frame of the video showing the second person, which represents the sum of the positional information of two identified left-right paired body parts. The information processing device 100 generates a sixth component, for example, in each frame of the video showing the second person, which represents the absolute value of the fifth component, which represents the difference in the positional information of two identified left-right paired body parts. The information processing device 100 uses a learned model to obtain posture information indicating whether the second person is in a posture corresponding to a specific action, for example, in each frame of the video showing the second person, based on the generated fourth component and the generated sixth component.

[0064] The information processing device 100 outputs, for example, the acquired posture information. The output format may be, for example, display on a screen, print to a printer, transmit to another computer, or store in a memory area. Specifically, the information processing device 100 transmits the acquired posture information to the client device 202. The information processing device 100 is managed, for example, by an administrator who manages the information processing system 200. The information processing device 100 may be, for example, a server or a PC (Personal Computer).

[0065] The video recording device 201 is a computer for capturing images of a specific area and generating images of people. The video recording device 201 includes, for example, a camera having multiple image sensors, and the camera captures a specific area where people may be present. The video recording device 201 generates, for example, an image of a specific person and transmits it to the information processing device 100.

[0066] Specifically, the video recording device 201 generates video footage of a first person and transmits it to the information processing device 100. Specifically, the video recording device 201 generates video footage of a second person and transmits it to the information processing device 100. The video recording device 201 is, for example, a smartphone. The video recording device 201 may also be, for example, a fixed-point camera. The video recording device 201 may also be, for example, a drone.

[0067] The client device 202 is a computer used by an operator who wishes to access the posture information of a second person. The client device 202 receives, for example, the posture information of the second person from the information processing device 100. The client device 202 outputs the received posture information of the second person in a format accessible to the operator. Output formats include, for example, display on a screen, printing to a printer, transmission to another computer, or storage in a memory area. The client device 202 may be, for example, a PC, tablet, or smartphone.

[0068] This section describes a case where the information processing device 100 is a different device from the video recording device 201, but it is not limited to this case. For example, the information processing device 100 may have the functionality of a video recording device 201 and may operate as a video recording device 201. This section describes a case where the information processing device 100 is a different device from the client device 202, but it is not limited to this case. For example, the information processing device 100 may have the functionality of a client device 202 and may operate as a client device 202.

[0069] (Examples of applications of Information Processing System 200) Next, an example of the application of the information processing system 200 will be described. The information processing system 200 can be applied, for example, to determining whether a person captured in video footage taken by a surveillance camera has performed a specific action equivalent to a suspicious act, a prohibited act, or a criminal act. Specific actions could include, for example, violent acts such as hitting someone with one hand, or prohibited acts such as touching an exhibit with one hand. Exhibits could be, for example, plants, animals, or works of art. In this case, the video capture device 201 is, for example, a surveillance camera. The worker could be, for example, a security guard or a police officer. In this case, the information processing system 200 can accurately determine whether the person has performed a specific action equivalent to a suspicious act, a prohibited act, or a criminal act. Therefore, the information processing system 200 can make it easier to deter specific actions equivalent to suspicious acts, prohibited acts, or criminal acts.

[0070] Furthermore, the information processing system 200 can be applied, for example, to determine whether a person captured in video footage taken by a fixed-point camera installed in a nursing home has performed a specific action, thereby supporting the work of caregivers working in the nursing home. Specific actions could include, for example, limping or falling and placing one hand on the floor. In this case, the person in question is, for example, a person requiring care who is staying in a nursing home. The video capture device 201 is, for example, a fixed-point camera. The operator is, for example, a caregiver. In this case, the information processing system 200 can accurately determine whether the person in question has performed a specific action, making it easier for the operator to ensure the safety of the person in question.

[0071] Furthermore, the information processing system 200 could be applied, for example, to a marketing campaign where a fixed-point camera installed in a store selling groceries or other goods captures footage of a target person who has performed a specific action. The specific action could be, for example, picking up a product with one hand. In this case, the target person is, for example, a customer visiting the store. The video capture device 201 is, for example, a fixed-point camera. The operator is, for example, a marketing professional. In this case, the information processing system 200 can accurately determine whether the target person has performed the specific action, enabling the operator to conduct marketing with accuracy.

[0072] Furthermore, the information processing system 200 can be applied, for example, to a case where content is provided to a person who has visited an entertainment facility, based on whether that person has performed a specific action as captured in video footage taken by a fixed-point camera installed in the entertainment facility. The specific action could be, for example, moving one hand in a specific pattern. The content could be, for example, entertainment videos, machinery and equipment within the facility, or music. In this case, the person in question is, for example, a child. The video capture device 201 is, for example, a fixed-point camera. In this case, the information processing system 200 can accurately determine whether or not the person in question has performed a specific action, and can facilitate the appropriate provision of content.

[0073] (Example of hardware configuration of information processing device 100) Next, an example of the hardware configuration of the information processing device 100 will be described using Figure 3.

[0074] Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. In Figure 3, the information processing device 100 includes a CPU (Central Processing Unit) 301, a memory 302, and a network interface 303. The information processing device 100 also includes a recording medium interface 304, a recording medium 305, a display 306, and an input device 307. Each component is connected by a bus 300.

[0075] Here, the CPU 301 is responsible for the overall control of the information processing device 100. The memory 302 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and flash ROM. Specifically, for example, flash ROM and ROM store various programs, and RAM is used as the work area for the CPU 301. Programs stored in memory 302 are loaded into the CPU 301, causing the CPU 301 to execute the coded processes.

[0076] The network interface 303 is connected to network 210 via a communication line, and then connects to other computers via network 210. The network interface 303 manages the internal interface with network 210 and controls the input and output of data from other computers. The network interface 303 is, for example, a modem or a LAN adapter.

[0077] The recording medium interface (I / F) 304 controls the reading and writing of data to the recording medium 305 according to the control of the CPU 301. The recording medium interface (I / F) 304 is, for example, a disk drive, an SSD (Solid State Drive), or a USB (Universal Serial Bus) port. The recording medium 305 is a non-volatile memory that stores the data written under the control of the recording medium interface (I / F) 304. The recording medium 305 is, for example, a disk, semiconductor memory, or USB memory. The recording medium 305 may be detachable from the information processing device 100.

[0078] Display 306 displays data such as cursors, icons, toolboxes, documents, images, or functional information. Display 306 is, for example, a CRT (Cathode Ray Tube), a liquid crystal display, or an organic EL (Electroluminescence) display. Input device 307 has keys for inputting characters, numbers, or various instructions, and performs data input. Input device 307 is, for example, a keyboard or a mouse. Input device 307 may also be, for example, a touch panel input pad or a numeric keypad.

[0079] The information processing device 100 may have, in addition to the components described above, a camera, for example. Furthermore, the information processing device 100 may have, in addition to the components described above, a printer, scanner, microphone, or speaker, for example. Also, the information processing device 100 may have multiple recording medium interfaces 304 and recording mediums 305, for example. Furthermore, the information processing device 100 does not necessarily have, for example, a display 306 or an input device 307. Also, the information processing device 100 does not necessarily have, for example, recording medium interfaces 304 and recording mediums 305.

[0080] (Example hardware configuration of video recording device 201) Next, we will describe an example of the hardware configuration of the video capture device 201 using Figure 4.

[0081] Figure 4 is a block diagram showing an example of the hardware configuration of the video recording device 201. In Figure 5, the video recording device 201 includes a CPU 401, memory 402, network interface 403, recording medium interface 404, recording medium 405, and camera 406. Each component is connected by a bus 400.

[0082] Here, the CPU 401 is responsible for the overall control of the video capture device 201. The memory 402 includes, for example, ROM, RAM, and flash ROM. Specifically, for example, flash ROM and ROM store various programs, and RAM is used as the work area for the CPU 401. Programs stored in memory 402 are loaded into the CPU 401, causing the CPU 401 to execute the coded processes.

[0083] The network interface 403 is connected to network 210 via a communication line, and then connects to other computers via network 210. The network interface 403 manages the internal interface with network 210 and controls the input and output of data from other computers. The network interface 403 is, for example, a modem or a LAN adapter.

[0084] The recording medium interface (I / F) 404 controls the reading and writing of data to the recording medium (SSD) 405 according to the control of the CPU 401. The recording medium interface (I / F) 404 is, for example, a disk drive, SSD, or USB port. The recording medium (SSD) 405 is a non-volatile memory that stores the data written under the control of the recording medium interface (I / F) 404. The recording medium (SSD) 405 is, for example, a disk, semiconductor memory, or USB memory. The recording medium (SSD) 405 may be detachable from the video capture device (SSD) 201.

[0085] Camera 406 has multiple image sensors and generates images of a specific area captured by the multiple image sensors. For example, if a person is present in a specific area, camera 406 will generate an image of that person. Camera 406 is, for example, a digital camera. Camera 406 is, for example, a fixed-point camera. Camera 406 may be, for example, movable. Camera 406 is, for example, a surveillance camera.

[0086] In addition to the components described above, the video recording device 201 may also have, for example, a keyboard, mouse, display, printer, scanner, microphone, speaker, etc. Furthermore, the video recording device 201 may have multiple recording media I / F 404 and recording media 405. Alternatively, the video recording device 201 may not have recording media I / F 404 or recording media 405.

[0087] (Example hardware configuration for client device 202) The hardware configuration example for client device 202 is specifically the same as the hardware configuration example for information processing device 100 shown in Figure 3, so a detailed explanation is omitted.

[0088] (Example of the functional configuration of the information processing device 100) Next, an example of the functional configuration of the information processing device 100 will be described using Figure 5.

[0089] Figure 5 is a block diagram showing an example of the functional configuration of the information processing device 100. The information processing device 100 includes a storage unit 500, an acquisition unit 501, an identification unit 502, a generation unit 503, a learning unit 504, a recognition unit 505, and an output unit 506.

[0090] The storage unit 500 is implemented by a storage area such as the memory 302 or recording medium 305 shown in Figure 3. The following description will focus on the case where the storage unit 500 is included in the information processing device 100, but is not limited to this case. For example, the storage unit 500 may be included in a device different from the information processing device 100, and the contents of the storage unit 500 may be accessible from the information processing device 100.

[0091] The acquisition unit 501 to the output unit 506 function as an example of a control unit. Specifically, the acquisition unit 501 to the output unit 506 realize their functions, for example, by having the CPU 301 execute a program stored in a storage area such as the memory 302 or recording medium 305 shown in Figure 3, or by using the network I / F 303. The processing results of each functional unit are stored in a storage area such as the memory 302 or recording medium 305 shown in Figure 3.

[0092] The memory unit 500 stores various information that is referenced or updated in the processing of each functional unit. The memory unit 500 stores, for example, an image of a person. Specifically, the memory unit 500 stores a first image of a first person. The first person is, for example, a person whose posture information is known. The posture information indicates, for example, whether the posture of the first person is a specific posture. The posture information may also indicate, for example, whether the posture of the first person corresponds to a posture for a specific action. The posture information may also indicate, for example, whether the first person has performed a specific action. Specifically, the first person is a person whose performance of a specific action is known. The first image of the first person includes, for example, one or more frames. The first image of the first person is acquired, for example, by the acquisition unit 501.

[0093] Specifically, the memory unit 500 stores a correct label indicating whether or not the first person performed a specific action, associated with the first video footage in which the first person is shown. Specifically, the memory unit 500 stores a correct label indicating whether or not the first person performed a specific action in each frame of the first video footage in which the first person is shown. The correct labels are acquired, for example, by the acquisition unit 501. Specifically, the memory unit 500 may also store a correct label indicating whether or not the first person performed a specific action, associated with the entire first video footage in which the first person is shown.

[0094] Specifically, the memory unit 500 stores a second video image in which a second person is shown. The second person is, for example, a person whose posture information is to be estimated. The posture information indicates, for example, whether the posture of the second person is a specific posture or not. The posture information may also indicate, for example, whether the posture of the second person is a posture corresponding to a specific action or not. The posture information may also indicate, for example, whether the second person has performed a specific action or not. Specifically, the second person is a person whose performance of a specific action is to be determined. The second video image in which the second person is shown includes, for example, one or more frames. The second video image in which the second person is shown is acquired, for example, by the acquisition unit 501.

[0095] The memory unit 500 stores, for example, the positional information of two pairs of body parts of the first person in the first video image in which the first person is shown. Examples of body parts include the neck, head, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, or left foot. The two body parts are, for example, a combination of different body parts that are paired left and right. Left and right here refers to the left and right sides of the body, when the front of the body is considered forward and the top of the head is considered upward, rather than the left and right sides in the video. Specifically, the two body parts are a combination of the right hand and the left hand.

[0096] Positional information represents, for example, a combination of multiple component values ​​in different axes that indicate the position of a body part in three-dimensional space. These axes are, for example, called the X-axis, Y-axis, and Z-axis. The position of a body part in three-dimensional space is, for example, the position of the skeleton representing the body part in three-dimensional space. Positional information may also be, for example, a vector.

[0097] Positional information may represent, for example, a combination of multiple component values ​​in different axes that represent the position of a body part in two-dimensional space. The axes are, for example, called the X-axis and the Y-axis. The two-dimensional space may correspond to, for example, a frame or a defined area of ​​size in which a person is captured within the frame. The position of a body part in two-dimensional space is, for example, the position of the skeleton representing the body part in two-dimensional space. Positional information may be, for example, a vector. Positional information may be, for example, polar coordinates.

[0098] Specifically, the memory unit 500 stores the positional information of two pairs of body parts of the first person in each frame of the first video in which the first person is shown. The positional information is identified, for example, by the identification unit 502. The positional information may also be acquired, for example, by the acquisition unit 501.

[0099] The memory unit 500 stores, for example, the positional information of two pairs of body parts of the second person in the second video image in which the second person is shown. Examples of body parts include the neck, head, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, or left foot. The two body parts are, for example, a combination of different body parts that are paired left and right. Specifically, the two body parts are a combination of the right hand and the left hand.

[0100] Positional information represents, for example, a combination of multiple component values ​​in different axes that indicate the position of a body part in three-dimensional space. These axes are, for example, called the X-axis, Y-axis, and Z-axis. The position of a body part in three-dimensional space is, for example, the position of the skeleton representing the body part in three-dimensional space. Positional information may also be, for example, a vector.

[0101] Positional information may represent, for example, a combination of multiple component values ​​in different axes that represent the position of a body part in two-dimensional space. The axes are, for example, called the X-axis and the Y-axis. The two-dimensional space may correspond to, for example, a frame or a defined area of ​​size in which a person is captured within the frame. The position of a body part in two-dimensional space is, for example, the position of the skeleton representing the body part in two-dimensional space. Positional information may be, for example, a vector. Positional information may be, for example, polar coordinates.

[0102] Specifically, the memory unit 500 stores the positional information of two pairs of body parts of the second person in each frame of the second video in which the second person is shown. The positional information is identified, for example, by the identification unit 502. The positional information may also be acquired, for example, by the acquisition unit 501.

[0103] The memory unit 500 stores, for example, a component representing the sum of the positional information of each of two paired body parts among the body parts that a person possesses. The memory unit 500 stores, for example, a component representing the difference in the positional information of each of two paired body parts among the body parts that a person possesses. The memory unit 500 stores, for example, a component representing the absolute value of the component representing the difference in the positional information of each of two paired body parts among the body parts that a person possesses.

[0104] Specifically, the memory unit 500 stores a first component that represents the sum of the positional information of two paired body parts among the body parts possessed by the first person. The first component is generated, for example, by the generation unit 503. Specifically, the memory unit 500 stores a second component that represents the difference between the positional information of two paired body parts among the body parts possessed by the first person. The second component is generated, for example, by the generation unit 503. The memory unit 500 stores a third component that represents the absolute value of the second component, which represents the difference between the positional information of two paired body parts among the body parts possessed by the first person. The third component is generated, for example, by the generation unit 503. The first component, the second component, and the third component are components of information that represent the characteristics of the combination of positional information of two paired body parts.

[0105] Specifically, the memory unit 500 stores a fourth component that represents the sum of the positional information of each of two paired body parts among the body parts possessed by the second person. The fourth component is generated, for example, by the generation unit 503. Specifically, the memory unit 500 stores a fifth component that represents the difference between the positional information of each of two paired body parts among the body parts possessed by the second person. The fifth component is generated, for example, by the generation unit 503. The memory unit 500 stores a sixth component that represents the absolute value of the fifth component that represents the difference between the positional information of each of two paired body parts among the body parts possessed by the second person. The sixth component is generated, for example, by the generation unit 503. The fourth component, the fifth component, and the sixth component are components of information that represent the characteristics of the combination of positional information of each of the two paired body parts.

[0106] The memory unit 500 stores, for example, a model. The model has a function to output posture information of a person from an image of the person. The posture information indicates, for example, whether the person's posture is a specific posture. The posture information may also indicate, for example, whether the person's posture corresponds to a specific action. The posture information may also indicate, for example, whether the person has performed a specific action.

[0107] The model specifically has the function of outputting posture information of a second person in response to inputs of the fourth component and the sixth component. The model may also specifically have the function of outputting posture information of a second person in response to inputs of the fourth component, the fifth component and the sixth component. The model is, for example, an AI model. The model can be implemented, for example, by a neural network, mathematical formulas, or a tree structure. The model is trained, for example, by the learning unit 504.

[0108] The acquisition unit 501 acquires various types of information used in the processing of each functional unit. The acquisition unit 501 stores the acquired information in the storage unit 500 or outputs it to each functional unit. The acquisition unit 501 may also output the information stored in the storage unit 500 to each functional unit. The acquisition unit 501 acquires various types of information, for example, based on user input. The acquisition unit 501 may also receive various types of information from a device other than the information processing device 100, for example.

[0109] The acquisition unit 501 acquires, for example, an image of a person. Specifically, the acquisition unit 501 acquires a first image of a first person. More specifically, the acquisition unit 501 acquires the first image of the first person by receiving it from another computer. The other computer is, for example, the video recording device 201. More specifically, the acquisition unit 501 may acquire the first image of the first person by receiving input of the first image of the first person based on user input.

[0110] Specifically, the acquisition unit 501 acquires a correct label indicating whether or not the first person performed a specific action, associated with the first video footage in which the first person is shown. Alternatively, the acquisition unit 501 may acquire a correct label indicating whether or not the first person performed a specific action in each frame of the first video footage in which the first person is shown.

[0111] Specifically, the acquisition unit 501 acquires a second video showing a second person. More specifically, the acquisition unit 501 acquires the second video showing a second person by receiving it from another computer. The other computer is, for example, the video recording device 201. More specifically, the acquisition unit 501 may acquire the second video showing a second person by receiving input of the second video showing a second person based on user operation input.

[0112] The acquisition unit 501 may, for example, acquire the position information of each of two paired body parts among the body parts of the first person in the first video in which the first person is shown. Specifically, the acquisition unit 501 acquires the position information of each of two paired body parts among the body parts of the first person in each frame of the first video in which the first person is shown. More specifically, the acquisition unit 501 acquires the position information of each of two paired body parts among the body parts of the first person in each frame of the first video in which the first person is shown, when the identification unit 502 does not identify the position information of each body part.

[0113] The acquisition unit 501 acquires, for example, the position information of each of two paired body parts of the second person in the second video in which the second person is shown. Specifically, the acquisition unit 501 stores the position information of each of two paired body parts of the second person in each frame of the second video in which the second person is shown. More specifically, the acquisition unit 501 stores the position information of each of two paired body parts of the second person in each frame of the second video in which the second person is shown, when the identification unit 502 does not identify the position information of each body part.

[0114] The acquisition unit 501 may receive a start trigger to initiate processing in any of the functional units. A start trigger may be, for example, a predetermined operation input by a user. A start trigger may also be, for example, the receipt of predetermined information from another computer. A start trigger may also be, for example, the output of predetermined information by any of the functional units.

[0115] The acquisition unit 501 may, for example, receive the acquisition of a first video showing a first person as a start trigger to initiate processing by the identification unit 502, the generation unit 503, and the learning unit 504. The acquisition unit 501 may, for example, receive the acquisition of a second video showing a second person as a start trigger to initiate processing by the identification unit 502, the generation unit 503, and the recognition unit 505.

[0116] The acquisition unit 501 may, for example, receive as a start trigger to begin processing between the generation unit 503 and the learning unit 504 the acquisition of positional information for each of two paired body parts of the first person in the first video showing the first person. The acquisition unit 501 may, for example, receive as a start trigger to begin processing between the generation unit 503 and the recognition unit 505 the acquisition of positional information for each of two paired body parts of the second person in the second video showing the second person.

[0117] The identification unit 502 identifies the positional information of each of two pairs of body parts of a person in the video by analyzing the video acquired by the acquisition unit 501. For example, the identification unit 502 identifies the positional information of each of two pairs of body parts of a person in the video by analyzing the first video of a first person acquired by the acquisition unit 501.

[0118] Specifically, the identification unit 502 analyzes the first video footage of the first person acquired by the acquisition unit 501 to identify positional information in the first video footage, which represents the position of each body part in multidimensional space, with each component value in a different axial direction. More specifically, the identification unit 502 analyzes the first video footage of the first person acquired by the acquisition unit 501 to identify positional information in each frame of the first video footage, which represents the position of each body part in multidimensional space, with each component value in a different axial direction. As a result, the identification unit 502 can obtain information that represents the characteristics of the first person's posture and can serve as a guide for learning a model to estimate the posture information of the first person.

[0119] Specifically, the identification unit 502 may analyze the first video footage of the first person acquired by the acquisition unit 501 to identify the position of the first person's skeleton in the first video footage, and then identify the positional information of each body part based on the identified position of the first person's skeleton. In this way, the identification unit 502 can utilize the method for identifying the position of the skeleton to obtain information that can serve as a guide for learning a model to represent the characteristics of the first person's posture and estimate the posture information of the first person.

[0120] The identification unit 502, for example, analyzes the second video footage of the second person acquired by the acquisition unit 501 to identify the positional information of two pairs of body parts of the second person in the second video footage.

[0121] Specifically, the identification unit 502 analyzes the second video footage of the second person acquired by the acquisition unit 501 to identify positional information in the second video footage, which represents the position of each body part in multidimensional space, with each component value in a different axial direction. More specifically, the identification unit 502 analyzes the second video footage of the second person acquired by the acquisition unit 501 to identify positional information in each frame of the second video footage, which represents the position of each body part in multidimensional space, with each component value in a different axial direction. As a result, the identification unit 502 can obtain information that represents the characteristics of the second person's posture and can be used to estimate the posture information of the second person using a model.

[0122] Specifically, the identification unit 502 may analyze the second video footage of the second person acquired by the acquisition unit 501 to identify the position of the second person's skeleton in the second video footage, and then identify the positional information of each body part based on the identified position of the second person's skeleton. In this way, the identification unit 502 can utilize the method for identifying the position of the skeleton to represent the characteristics related to the second person's posture and obtain information to be used when estimating the second person's posture information using a model.

[0123] The generation unit 503 generates a first component representing the sum of the positional information of each body part, based on the positional information of each of two paired body parts of the body parts of the first person identified by the identification unit 502. The generation unit 503 calculates an index value for each axis direction using the sum of the component values ​​in that axis direction indicated by the positional information of each body part, and generates a first component by combining the calculated index values. The index value is, for example, the value obtained by dividing the sum of the component values ​​in the axis direction by a specified value. The specified value is, for example, √2. Specifically, for each frame of the first video showing the first person, the generation unit 503 calculates an index value for each axis direction using the sum of the component values ​​in that axis direction indicated by the positional information of each body part, and generates a first component by combining the calculated index values.

[0124] The generation unit 503 may, for example, generate the sum of the vectors representing the position information of each body part as the first component, if the position information of each body part is a vector. In this case, the first component is a vector. Specifically, the generation unit 503 generates the sum of the vectors representing the position information of each body part as the first component for each frame of the first video showing the first person. In this way, the generation unit 503 can obtain a first component that represents the features concerning the commonality of the position information of each body part of a pair of left and right body parts, and can serve as a guideline when training the model.

[0125] The generation unit 503 generates a second component that represents the difference in the positional information of each body part, based on the positional information of each body part of a pair of body parts identified by the identification unit 502. The generation unit 503 calculates an index value for each axis, for example, using the difference in the component values ​​in the axis direction indicated by the positional information of each body part, and generates a second component by combining the calculated index values. The index value is, for example, the value obtained by dividing the difference in the component values ​​in the axis direction by a specified value. The specified value is, for example, √2. Specifically, for each frame of the first video showing the first person, the generation unit 503 calculates an index value for each axis, for each axis, using the difference in the component values ​​in the axis direction indicated by the positional information of each body part, and generates a second component by combining the calculated index values.

[0126] For example, if the position information of two paired body parts of the first person is a vector, the generation unit 503 may generate the difference between the vectors indicated by the position information of each body part as the second component. Specifically, for each frame of the first video showing the first person, the generation unit 503 generates the difference between the vectors indicated by the position information of each body part as the second component. In this case, the second component is a vector. As a result, the generation unit 503 can obtain a second component that represents the characteristics of the difference in the position information of two paired body parts and can serve as a guideline when learning the model.

[0127] The generation unit 503 generates a third component that represents the absolute value of the second component representing the difference in the positional information of each body part, based on the positional information of each of two pairs of body parts of the first person identified by the identification unit 502. The generation unit 503 calculates an index value for each axis direction using the absolute value of the difference in the component values ​​in that axis direction indicated by the positional information of each body part, and generates a third component by combining the calculated index values. The index value is, for example, the value obtained by dividing the absolute value of the difference in the component values ​​in that axis direction by a specified value. The specified value is, for example, √2. Specifically, for each frame of the first video showing the first person, the generation unit 503 calculates an index value for each axis direction using the absolute value of the difference in the component values ​​in that axis direction indicated by the positional information of each body part, and generates a third component by combining the calculated index values.

[0128] The generation unit 503 generates a third component, for example, if the positional information of two paired body parts of a first person is a vector, the third component is a vector with a predetermined direction whose magnitude is the absolute value of the second component vector. In this case, the third component is a vector. The absolute value of the second component vector indicates the length of the second component vector. The predetermined direction is determined, for example, according to a predetermined rule, based on the direction of the difference between the second component vectors.

[0129] The prescribed rule determines the direction of the third component vector to be either the same direction as the second component vector, or the opposite direction to the second component vector. The prescribed rule determines the direction of the third component vector to be unified to one of the following directions, either the same direction as the second component vector, or the opposite direction to the second component vector, such that the directional component in one of the axes is positive.

[0130] Specifically, the generation unit 503 generates a third component for each frame of the first video showing the first person, which has a magnitude equal to the absolute value of the second component vector and a predetermined direction. This allows the generation unit 503 to obtain a third component that represents the differences in the positional information of two paired left and right body parts, while being independent of the reversal of the movement of the two paired left and right body parts. The generation unit 503 can obtain a third component that can serve as a guideline when training the model.

[0131] The generation unit 503 generates a fourth component representing the sum of the positional information of each body part, based on the positional information of each of two paired body parts of the second person identified by the identification unit 502. The generation unit 503 calculates an index value for each axis direction using the sum of the component values ​​in that axis direction indicated by the positional information of each body part, and generates a fourth component by combining the calculated index values. The index value is, for example, the value obtained by dividing the sum of the component values ​​in the axis direction by a specified value. The specified value is, for example, √2. Specifically, for each frame of the second video showing the second person, the generation unit 503 calculates an index value for each axis direction using the sum of the component values ​​in that axis direction indicated by the positional information of each body part, and generates a fourth component by combining the calculated index values.

[0132] The generation unit 503 may, for example, generate the sum of the vectors representing the position information of each body part as the fourth component, if the position information of each body part is a vector. In this case, the fourth component is a vector. Specifically, the generation unit 503 generates the sum of the vectors representing the position information of each body part as the fourth component for each frame of the second video showing the second person. As a result, the generation unit 503 can obtain a fourth component that represents the characteristics of the commonality of the position information of two body parts that are a left-right pair, and can serve as a guideline when estimating the posture information of the second person using a model.

[0133] The generation unit 503 generates a fifth component that represents the difference in the positional information of each body part, based on the positional information of each body part of a pair of body parts identified by the identification unit 502. The generation unit 503 calculates an index value for each axis direction using the difference in the component values ​​in that axis direction indicated by the positional information of each body part, and generates a fifth component by combining the calculated index values. The index value is, for example, the value obtained by dividing the difference in the component values ​​in the axis direction by a specified value. The specified value is, for example, √2. Specifically, for each frame of the second video showing the second person, the generation unit 503 calculates an index value for each axis direction using the difference in the component values ​​in that axis direction indicated by the positional information of each body part, and generates a fifth component by combining the calculated index values.

[0134] For example, if the position information of two paired body parts of the second person is a vector, the generation unit 503 may generate the difference between the vectors indicated by the position information of each body part as the fifth component. In this case, the fifth component is a vector. Specifically, for each frame of the second video showing the second person, the generation unit 503 generates the difference between the vectors indicated by the position information of each body part as the fifth component. As a result, the generation unit 503 can obtain a fifth component that represents the characteristics of the difference in the position information of two left-right paired body parts, and can serve as a guideline when estimating the posture information of the second person using a model.

[0135] The generation unit 503 generates a sixth component that represents the absolute value of the fifth component representing the difference in the positional information of each body part, based on the positional information of each of two pairs of body parts of the second person identified by the identification unit 502. The generation unit 503 calculates an index value for each axis direction using the absolute value of the difference in the component values ​​in that axis direction indicated by the positional information of each body part, and generates a sixth component by combining the calculated index values. The index value is, for example, the value obtained by dividing the absolute value of the difference in the component values ​​in that axis direction by a specified value. The specified value is, for example, √2. Specifically, for each frame of the second video showing the second person, the generation unit 503 calculates an index value for each axis direction using the absolute value of the difference in the component values ​​in that axis direction indicated by the positional information of each body part, and generates a sixth component by combining the calculated index values.

[0136] The generation unit 503 generates a vector with a predetermined direction as the sixth component, for example, if the positional information of two paired body parts of a second person is a vector, the absolute value of the fifth component vector is its magnitude. In this case, the sixth component is a vector. The absolute value of the fifth component vector indicates the length of the fifth component vector. The predetermined direction is determined, for example, according to a predetermined rule, based on the direction of the difference between the fifth component vectors.

[0137] The prescribed rule determines the direction of the sixth component vector to be either the same direction as the fifth component vector, or the opposite direction to the fifth component vector. The prescribed rule determines the direction of the sixth component vector to be unified to one of the following directions, either the same direction as the fifth component vector, or the opposite direction to the fifth component vector, such that the axial component is positive.

[0138] Specifically, the generation unit 503 generates a sixth component for each frame of the second video showing the second person, which has a magnitude equal to the absolute value of the fifth component vector and a predetermined direction. This allows the generation unit 503 to obtain a sixth component that represents the differences in the positional information of two paired body parts, while being independent of the reversal of movement of the two paired body parts. The generation unit 503 can then obtain a sixth component that can serve as a guideline when estimating the posture information of the second person using a model.

[0139] The learning unit 504 learns a model that outputs posture information of the second person from a second video showing the second person, based on the first component and the third component it generates. For example, the learning unit 504 generates training data that associates combinations of the first component and the third component with the correct labels acquired by the acquisition unit 501.

[0140] Specifically, the learning unit 504 generates training data that associates the combination of the generated first component and the generated third component with the correct label acquired by the acquisition unit 501 for each frame of the first video showing the first person. Alternatively, the learning unit 504 may generate training data that associates the combination of the time series of the generated first component and the time series of the generated third component in the first video showing the first person with the correct label corresponding to the entire first video acquired by the acquisition unit 501.

[0141] The learning unit 504, for example, trains a model based on the generated training data. Specifically, the learning unit 504 trains a model based on the generated training data using logistic regression. The information processing device 100 may, specifically, train a model based on the generated training data using backpropagation.

[0142] This allows the learning unit 504 to learn a model. For example, the learning unit 504 can learn a model that can accurately estimate the posture information of a second person from a video showing the second person. Using the learned model, the learning unit 504 can recognize similar actions of two paired body parts as the same action.

[0143] The learning unit 504 may learn a model that outputs posture information of a second person from a second video showing the second person, based on the first component, second component, and third component that it has generated. For example, the learning unit 504 generates learning data that associates combinations of the first component, second component, and third component with the correct labels acquired by the acquisition unit 501.

[0144] Specifically, the learning unit 504 generates training data that associates the combination of the generated first component, the generated second component, and the generated third component with the correct label acquired by the acquisition unit 501 for each frame of the first video showing the first person. Specifically, the learning unit 504 may generate training data that associates the combination of the time series of the generated first component, the time series of the generated second component, and the time series of the generated third component in the first video showing the first person with the correct label corresponding to the entire first video.

[0145] The learning unit 504, for example, trains a model based on the generated training data. Specifically, the learning unit 504 trains a model based on the generated training data using logistic regression. The information processing device 100 may, specifically, train a model based on the generated training data using backpropagation.

[0146] This allows the learning unit 504 to learn a model. For example, the learning unit 504 can learn a model that can accurately estimate the posture information of a second person from a video showing the second person. Using the learned model, the learning unit 504 can recognize similar actions of two paired body parts as the same action.

[0147] The recognition unit 505 acquires posture information of the second person using a learned model based on the generated fourth component and the generated sixth component. The recognition unit 505 generates input data that includes, for example, the combination of the generated fourth component and the generated sixth component as explanatory variables. The input data may further include, for example, the position information of non-paired body parts of the second person as explanatory variables.

[0148] Specifically, the recognition unit 505 generates input data that includes, as explanatory variables, a combination of the generated fourth component and the generated sixth component for each frame of the second video showing the second person. Specifically, the recognition unit 505 may also generate input data that includes, as explanatory variables, a combination of the time series of the generated fourth component and the time series of the generated sixth component in the second video showing the second person.

[0149] The recognition unit 505, for example, inputs the generated input data into a trained model, uses the trained model to estimate the posture information of the second person, and obtains the posture information of the second person output from the trained model. As a result, the recognition unit 505 can accurately determine the posture information of the second person.

[0150] The recognition unit 505 may acquire posture information of the second person using a learned model based on the generated fourth component, the generated fifth component, and the generated sixth component. For example, the recognition unit 505 generates input data that includes the combination of the generated fourth component, the generated fifth component, and the generated sixth component as explanatory variables.

[0151] Specifically, the recognition unit 505 generates input data that includes, as explanatory variables, a combination of the generated fourth component, the generated fifth component, and the generated sixth component for each frame of the second video showing the second person. Specifically, the recognition unit 505 may also generate input data that includes, as explanatory variables, a combination of the time series of the generated fourth component, the time series of the generated fifth component, and the time series of the generated sixth component in the second video showing the second person.

[0152] The recognition unit 505, for example, inputs the generated input data into a trained model, uses the trained model to estimate the posture information of the second person, and obtains the posture information of the second person output from the trained model. As a result, the recognition unit 505 can accurately determine the posture information of the second person.

[0153] The output unit 506 outputs the processing result of at least one of the functional units. The output format can be, for example, display on a screen, print to a printer, transmit to an external device via the network interface 303, or store in a storage area such as the memory 302 or recording medium 305. This allows the output unit 506 to notify the user of the processing result of at least one of the functional units, thereby improving the usability of the information processing device 100.

[0154] The output unit 506 outputs, for example, the model trained by the learning unit 504. Specifically, the output unit 506 transmits the model trained by the learning unit 504 to another computer. The other computer is, for example, the client device 202. This allows the output unit 506 to make the useful model available to the other computer.

[0155] The output unit 506 outputs, for example, the posture information of the second person acquired by the recognition unit 505. Specifically, the output unit 506 outputs the posture information of the second person in a way that is accessible to the user. Specifically, the output unit 506 may also transmit the posture information of the second person to another computer. The other computer may be, for example, the client device 202. This makes the posture information of the second person available to the user.

[0156] Here, we have described a case in which the information processing device 100 includes an acquisition unit 501, a specific unit 502, a generation unit 503, a learning unit 504, a recognition unit 505, and an output unit 506, but it is not limited to this. For example, the information processing device 100 may not include any of the functional units, but may be able to communicate with another computer that includes such functional unit. Specifically, the information processing device 100 may not include the recognition unit 505.

[0157] (Operation flow of the information processing device 100) Next, we will explain the operation flow of the information processing device 100 using Figure 6.

[0158] Figure 6 is an explanatory diagram showing the operation flow of the information processing device 100. In Figure 6, (6-1) the information processing device 100 receives learning video data 601, which includes multiple frames showing a person, from the video recording device 201. Based on the user's operation input, the information processing device 100 acquires annotation data 600, which includes correct labels indicating the actions of the person corresponding to each frame of the learning video data 601. The correct labels indicate, for example, whether or not the person is performing a specific action.

[0159] (6-2) The information processing device 100 stores a DL (Deep Learning) model 610. The DL model 610 has the function of estimating the coordinate information of each of the multiple skeletons of a person shown in a frame from the frame of the video data. Skeletons correspond to body parts. Body parts include, for example, the neck, head, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, or left foot.

[0160] The coordinate information includes, for example, a combination of multiple component values ​​in different axes that represent the position of each skeleton in three-dimensional space. Specifically, the coordinate information includes a combination of component values ​​in the X-axis direction, the Y-axis direction, and the Z-axis direction. The DL model 610 has a function that, for example, outputs the coordinate information of each of the multiple skeletons of a person shown in a frame, in response to the input of a frame of video data.

[0161] (6-3) The information processing device 100 performs the person recognition process 611 to recognize the person shown in each frame of the training video data 601 based on the received training video data 601. The information processing device 100 performs the skeleton estimation process 612 to obtain coordinate information for each of the multiple skeletons of the recognized person in each frame of the training video data 601 using the DL model 610.

[0162] (6-4) The information processing device 100 identifies a pair of skeletal structures from the skeleton of a person, which are a left-right pair. The pair may be, for example, a combination of the right hand skeleton corresponding to the right hand and the left hand skeleton corresponding to the left hand. The pair may also be, for example, a combination of the right elbow skeleton corresponding to the right elbow and the left elbow skeleton corresponding to the left elbow. The information processing device 100 extracts the coordinate information of each of the identified pair of skeletal structures from the coordinate information of each of the multiple skeletal structures acquired in each frame of the learning video data 601.

[0163] (6-5) The information processing device 100 performs the coordinate transformation 613 to convert the combination of coordinate information of each of the two extracted skeletons in each frame of the learning video data 601 into a combination of coordinate information in a special Cartesian coordinate system. The special Cartesian coordinate system is a Cartesian coordinate system for uniformly handling the symmetrical movements of each of the two skeletons of a left-right pair.

[0164] Specifically, one could consider a Cartesian coordinate system in which the X-axis of one skeleton and the X-axis of the other skeleton are orthogonal to each other, and adopt another Cartesian coordinate system with a 45-degree line as one of its axes as a special type of Cartesian coordinate system. The 45-degree line corresponds to a straight line defined, for example, by the formula "component value of the X-axis of one skeleton = component value of the X-axis of the other skeleton".

[0165] Similarly, specifically, one could consider a Cartesian coordinate system in which the Y-axis of one skeleton and the Y-axis of the other skeleton are orthogonal to each other, and adopt another Cartesian coordinate system with a 45-degree line as one of its axes as a special Cartesian coordinate system. The 45-degree line corresponds to a straight line defined, for example, by the formula "component value of the Y-axis of one skeleton = component value of the Y-axis of the other skeleton".

[0166] Similarly, specifically, one could consider a Cartesian coordinate system in which the Z-axis of one skeleton and the Z-axis of the other skeleton are orthogonal, and adopt another Cartesian coordinate system with a 45-degree line as one of its axes as a special Cartesian coordinate system. The 45-degree line corresponds to a straight line defined, for example, by the formula "component value of the Z-axis of one skeleton = component value of the Z-axis of the other skeleton". A specific example of a special Cartesian coordinate system will be described later using Figures 9 and 10. A specific example of a transformation will be described later using Figures 9 and 10.

[0167] (6-6) The information processing device 100 generates model training data in which correct labels are associated with combinations of two coordinate information in a transformed special Cartesian coordinate system for each frame of the training video data 601. The information processing device 100 trains the action recognition model 620 based on the generated model training data by performing the machine learning process 614.

[0168] The action recognition model 620 has the function of outputting a label indicating whether or not a person is performing a specific action in a frame of video data in which the person is shown, in response to input of features related to the posture of the person in that frame. Features related to the posture of a person are, for example, a combination of two pieces of coordinate information in a special Cartesian coordinate system. As a result, the information processing device 100 can obtain an action recognition model 620 that can accurately estimate a label indicating whether or not a person is performing a specific action in a frame of video data in which the person is shown.

[0169] (6-7) The information processing device 100 receives evaluation video data 602, which includes multiple frames in which a person is shown, from the video recording device 201.

[0170] (6-8) The information processing device 100 performs the person recognition process 615 to recognize the person shown in each frame of the evaluation video data 602 based on the received evaluation video data 602. The information processing device 100 performs the skeleton estimation process 616 to obtain the coordinate information of each of the multiple skeletons of the recognized person in each frame of the evaluation video data 602 using the DL model 610.

[0171] (6-9) The information processing device 100 identifies a pair of skeletal structures from the skeleton of the person, which are a left-right pair. The pair may be, for example, a combination of the right hand skeleton corresponding to the right hand and the left hand skeleton corresponding to the left hand. The pair may also be, for example, a combination of the right elbow skeleton corresponding to the right elbow and the left elbow skeleton corresponding to the left elbow. The information processing device 100 extracts the coordinate information of each of the identified pair of skeletal structures from the coordinate information of each of the multiple skeletal structures acquired in each frame of the evaluation video data 602.

[0172] (6-10) The information processing device 100 performs the coordinate transformation 617 to convert the combination of coordinate information of each of the two extracted skeletons in each frame of the evaluation video data 602 into a combination of coordinate information in a special Cartesian coordinate system. The special Cartesian coordinate system is a Cartesian coordinate system for uniformly handling the symmetrical movements of each of the two skeletons of a left-right pair.

[0173] (6-11) The information processing device 100 generates model input data in which, for each frame of the evaluation video data 602, a combination of two coordinate information in the transformed special Cartesian coordinate system is used as an explanatory variable. By performing the action recognition process 618, the information processing device 100 uses the action recognition model 620 to obtain a label indicating whether or not a person is performing a specific action in each frame of the evaluation video data 602, based on the model input data in that frame. As a result, the information processing device 100 can accurately estimate a label indicating whether or not a person is performing a specific action in a frame of video in which a person is shown.

[0174] (An example of the operation of the information processing device 100) Next, an example of the operation of the information processing device 100 will be explained using Figures 7 to 20. First, using Figure 7, an example of the contents of the pair management table 700 stored by the information processing device 100 will be explained in order to identify pairs of left and right skeletons from among the multiple skeletons that a person possesses. The pair management table 700 is implemented, for example, by a storage area such as the memory 302 or recording medium 305 of the information processing device 100 shown in Figure 3.

[0175] Figure 7 is an explanatory diagram showing an example of the contents stored in the pair management table 700. As shown in Figure 7, the pair management table 700 has fields for skeleton 1 and skeleton 2. The pair management table 700 stores pair information as record 700-a by setting information in each field for each pair formed by combining the two skeletons. a is an arbitrary integer.

[0176] The Skeleton 1 field is set to one of the multiple skeletons that the person possesses. The type is indicated, for example, by the name of a body part. The Skeleton 2 field is set to another type of skeleton that the person possesses, which, when combined with the above-mentioned skeleton, forms a pair of two skeletons, but is different from the above-mentioned skeleton. This allows the information processing device 100 to refer to the pair management table 700 and identify pairs of left and right skeletons from the multiple skeletons that the person possesses.

[0177] The following describes an example in which the information processing device 100 acquires training video data 601 containing multiple frames in which a person is shown, generates multiple training data based on the training video data 601, and trains the action recognition model 620 based on the generated multiple training data. The information processing device 100 stores the DL model 610.

[0178] The information processing device 100 receives the learning video data 601 from the video recording device 201. Based on the user's input, the information processing device 100 acquires annotation data 600, which includes correct labels indicating the actions of the people corresponding to each frame of the learning video data 601. By performing the person recognition process 611, the information processing device 100 recognizes the people shown in each frame of the learning video data 601 based on the received learning video data 601.

[0179] Next, using Figure 8, we will explain an example in which the information processing device 100 performs the skeletal estimation 612 process and uses the DL model 610 to obtain coordinate information for each of the multiple skeletons of the recognized person in each frame of the training video data 601.

[0180] Figure 8 is an explanatory diagram illustrating an example of acquiring coordinate information for each skeleton. As indicated by the reference numeral 800 in Figure 8, the information processing device 100 acquires coordinate information for each skeleton in a three-dimensional Cartesian coordinate system in which the front direction of the person is the X-axis, the side direction of the person is the Y-axis, and the up-down direction of the person is the Z-axis.

[0181] In the following explanation, the X-axis relating to the skeleton of the left half of the body will be denoted as the Xl-axis, as shown by reference numeral 810 in Figure 8, and the X-axis relating to the skeleton of the right half of the body will be denoted as the Xr-axis, as shown by reference numeral 820 in Figure 8, to distinguish between the X-axes relating to each skeleton.

[0182] Similarly, as shown by the reference numeral 810 in Figure 8, the Y-axis relating to the skeleton of the left half of the body is denoted as the Yl-axis, and as shown by the reference numeral 820 in Figure 8, the Y-axis relating to the skeleton of the right half of the body is denoted as the Yr-axis, thus distinguishing the Y-axes relating to each skeleton.

[0183] Similarly, as shown by the reference numeral 810 in Figure 8, the Z-axis relating to the skeleton of the left half of the body is denoted as the Zl-axis, and as shown by the reference numeral 820 in Figure 8, the Z-axis relating to the skeleton of the right half of the body is denoted as the Zr-axis, thus distinguishing the Z-axis relating to each skeleton.

[0184] In the following explanation, the coordinate information of the skeleton belonging to the left half of the body is assumed to be a combination of, for example, the coordinate value xl of the Xl axis, the coordinate value yl of the Yl axis, and the coordinate value zl of the Zl axis. Similarly, in the following explanation, the coordinate information of the skeleton belonging to the right half of the body is assumed to be a combination of, for example, the coordinate value xr of the Xr axis, the coordinate value yr of the Yr axis, and the coordinate value zr of the Zr axis.

[0185] Specifically, the information processing device 100 acquires coordinate information (xl, yl, zl) of at least the left hand skeleton, which corresponds to the left hand, from among the multiple skeletons of the person in each frame of the learning video data 601. Specifically, the information processing device 100 acquires coordinate information (xr, yr, zr) of at least the right hand skeleton, which corresponds to the right hand, from among the multiple skeletons of the person in each frame of the learning video data 601.

[0186] The information processing device 100 refers to the pair management table 700 to identify pairs of skeletons in a person, specifically the right hand skeleton corresponding to the right hand and the left hand skeleton corresponding to the left hand. The information processing device 100 extracts the coordinate information of the left hand skeleton (xl, yl, zl) and the coordinate information of the right hand skeleton (xr, yr, zr) from the coordinate information of each of the acquired skeletons in each frame of the learning video data 601.

[0187] Next, using Figures 9 and 10, we will describe an example of how the information processing device 100 transforms the extracted coordinate information. Specifically, we will describe an example in which the information processing device 100 transforms a combination of coordinate information of the left hand skeleton (xl, yl, zl) and coordinate information of the right hand skeleton (xr, yr, zr) into a combination of two coordinate information in a special Cartesian coordinate system.

[0188] Figures 9 and 10 are explanatory diagrams illustrating an example of coordinate information transformation. In Figure 9, the information processing device 100 identifies the combinations of corresponding coordinate values ​​from the coordinate information of the left hand skeleton (xl, yl, zl) and the coordinate information of the right hand skeleton (xr, yr, zr) in each frame of the learning video data 601.

[0189] Specifically, the information processing device 100 identifies the combination (xl, xr) of the coordinate value xl of the left hand skeleton and the coordinate value xr of the right hand skeleton in each frame of the learning video data 601. Similarly, the information processing device 100 specifically identifies the combination (yl, yr) of the coordinate value yl of the left hand skeleton and the coordinate value yr of the right hand skeleton in each frame of the learning video data 601. Similarly, the information processing device 100 specifically identifies the combination (zl, zr) of the coordinate value zl of the left hand skeleton and the coordinate value zr of the right hand skeleton in each frame of the learning video data 601.

[0190] Here, in the orthogonal coordinate system 900 between the Xl and Xr axes, consider the case where point 902 of combination (xl',xr') exists symmetrically to point 901 of combination (xl,xr) with respect to the 45-degree line 910 represented by the equation xr=xl. The posture of the person corresponding to point 901 of combination (xl,xr) and the posture of the person corresponding to point 902 of combination (xl',xr') are each the left and right inversions of each other.

[0191] Here, in order to treat the same type of action using each of the two skeletal structures in a pair as the same action, it is desirable to treat the symmetrical posture or movement of a person as identical. Therefore, it is conceivable to set up a new Cartesian coordinate system so that the combination (xl, xr) and the combination (xl', xr') are converted to the same index value. Now, we will move on to the explanation of Figure 10 and describe the new Cartesian coordinate system 1000.

[0192] As shown in Figure 10, the new Cartesian coordinate system 1000 is a Cartesian coordinate system with an Xa axis and an Xb axis. The Xa axis corresponds to the 45-degree line 910 of the Cartesian coordinate system 900. The Xb axis corresponds to a straight line that passes through the origin of the Cartesian coordinate system 900 and is perpendicular to the 45-degree line 910 of the Cartesian coordinate system 900. The relationship between the combination (xl, xr) and the component value xa of the Xa axis in the Cartesian coordinate system 1000 is given by xa = (xl + xr) / √2. The relationship between the combination (xl, xr) and the component value xb of the Xb axis in the Cartesian coordinate system 1000 is given by xb = |xl - xr| / √2.

[0193] Here, (xl+xr) / √2 = (xl'+xr') / √2. Also, |xl-xr| / √2 = |xl'-xr'| / √2. Therefore, in the Cartesian coordinate system 1000, the combination (xl,xr) and the combination (xl',xr') can be converted to the same index value.

[0194] Furthermore, the coordinate value xa=(xl+xr) / √2 is considered capable of representing the commonalities of the combination (xl,xr). On the other hand, since |xl-xr| / √2=|xl'-xr'| / √2, xb=|xl'-xr'| / √2 is considered capable of representing the differences of the combination (xl,xr) while treating the symmetrical posture or movement of a person as identical.

[0195] The information processing device 100 converts the combination (xl, xr) in each frame of the training video data 601 into the combination (xa, xb). This allows the information processing device 100 to treat symmetrical postures or movements of a person as identical while obtaining feature quantities that accurately represent the characteristics of a person's posture or movement. Specifically, the information processing device 100 can convert the combination (xl, xr) into the combination (xa, xb) while retaining information other than the distinction between left and right.

[0196] Furthermore, the information processing device 100 can obtain a combination (xa,xb) such that the combination (xa,xb) changes continuously over time. The combination (xa,xb) corresponds to point 1001. In this way, the information processing device 100 can obtain a combination (xa,xb) that is suitable as a feature for training the action recognition model 620, which can recognize the same type of action for each of two left-right paired skeletons as the same action.

[0197] Similarly, the information processing device 100 converts the combination (yl,yr) in each frame of the training video data 601 to the combination (ya,yb). Similarly, the information processing device 100 converts the combination (zl,zr) in each frame of the training video data 601 to the combination (za,zb). As a result, the information processing device 100 can obtain features suitable for training the action recognition model 620 for each of the X, Y, and Z axes.

[0198] The information processing device 100 combines the transformed combinations in each frame of the training video data 601 to identify the total combination (xa, ya, za, xb, yb, zb), and associates the corresponding ground truth label with the frame to generate model training data. The information processing device 100 then performs machine learning processing 614 to train the action recognition model 620 based on the generated model training data. As a result, the information processing device 100 can train the action recognition model 620 to accurately recognize the same type of action for each of the two left-right skeletal pairs as the same action.

[0199] Next, we will explain numerical examples illustrating the effects of the information processing device 100 using Figures 11 to 20.

[0200] Figures 11 to 20 are explanatory diagrams showing numerical examples illustrating the effects of the information processing device 100. In Figure 11, Table 1100 shows the coordinate information of the left hand skeleton (xl, yl, zl) and the right hand skeleton (xr, yr, zr) for each frame per second of the left-facing video data showing a scene in which a person is crouching down and picking up an object with their "left hand".

[0201] Table 1100 has fields for seconds, left-hand xl, left-hand yl, left-hand zl, right-hand xr, right-hand yr, and right-hand zr. The seconds field is set to the number of seconds indicating which frame of the left-facing video data is at.

[0202] The left hand xl field is set to the coordinate value xl of the left hand skeleton with respect to the Xl axis in the frame of the specified number of seconds. The unit of the coordinate value xl is, for example, pixels. The left hand yl field is set to the coordinate value yl of the left hand skeleton with respect to the Yl axis in the frame of the specified number of seconds. The unit of the coordinate value yl is, for example, pixels. The left hand zl field is set to the coordinate value zl of the left hand skeleton with respect to the Zl axis in the frame of the specified number of seconds. The unit of the coordinate value zl is, for example, pixels.

[0203] The field for right hand xr contains the coordinate value xr of the right hand skeleton with respect to the Xr axis in the frame of the specified number of seconds. The unit of the coordinate value xr is, for example, pixels. The field for right hand yr contains the coordinate value yr of the right hand skeleton with respect to the Yr axis in the frame of the specified number of seconds. The unit of the coordinate value yr is, for example, pixels. The field for right hand zr contains the coordinate value zr of the right hand skeleton with respect to the Zr axis in the frame of the specified number of seconds. The unit of the coordinate value zr is, for example, pixels. Next, we will move on to the explanation of Figure 12.

[0204] In Figure 12, Graph 1200 shows the time series of coordinate information (xl, yl, zl) for the left hand skeleton shown in Figure 11, and the time series of coordinate information (xr, yr, zr) for the right hand skeleton shown in Figure 11. The vertical axis of Graph 1200 is pixels. The horizontal axis of Graph 1200 is seconds. Next, we will move on to the explanation of Figure 13.

[0205] In Figure 13, the left-facing video data showing a person crouching and picking up an object with their "left hand" is flipped horizontally, and this flipped video data is treated as the right-facing video data showing a scene where the person crouches and picks up an object with their "right hand". Table 1300 shows the coordinate information of the left hand skeleton (xl, yl, zl) and the right hand skeleton (xr, yr, zr) for each frame of the right-facing video data showing the person crouching and picking up an object with their "right hand".

[0206] Table 1300 has fields for seconds, left-hand xl, left-hand yl, left-hand zl, right-hand xr, right-hand yr, and right-hand zr. The contents of each field in Table 1300 are the same as those of each field in Table 1100, so we will omit the explanation. Next, we will move on to the explanation of Figure 14.

[0207] In Figure 14, Graph 1400 shows the time series of coordinate information (xl, yl, zl) for the left hand skeleton shown in Figure 13, and the time series of coordinate information (xr, yr, zr) for the right hand skeleton shown in Figure 13. The vertical axis of Graph 1400 is pixels. The horizontal axis of Graph 1400 is seconds. Next, we will move on to the explanation of Figure 15.

[0208] In Figure 15, Table 1500 shows the coordinate information (xa,ya,za,xb,yb,zb) obtained by converting the coordinate information (xl,yl,zl) of the left hand skeleton and the coordinate information (xr,yr,zr) of the right hand skeleton in each frame of the leftward-facing video data.

[0209] Table 1500 has fields for seconds, xa, ya, za, xb, yb, and zb. The seconds field is set to the number of seconds, indicating which frame of the left-facing video data is in.

[0210] The xa field is set to the coordinate value xa related to the Xa axis in the frame of the specified number of seconds. The unit of the coordinate value xa is, for example, pixels. The ya field is set to the coordinate value ya related to the Ya axis in the frame of the specified number of seconds. The unit of the coordinate value ya is, for example, pixels. The za field is set to the coordinate value za related to the Za axis in the frame of the specified number of seconds. The unit of the coordinate value za is, for example, pixels.

[0211] The xb field contains the coordinate value xb for the Xb axis in the frame of the specified number of seconds. The unit of the coordinate value xb is, for example, pixels. The yb field contains the coordinate value yb for the Yb axis in the frame of the specified number of seconds. The unit of the coordinate value yb is, for example, pixels. The zb field contains the coordinate value zb for the Zb axis in the frame of the specified number of seconds. The unit of the coordinate value zb is, for example, pixels.

[0212] Here, the coordinate information (xa,ya,za,xb,yb,zb) obtained by converting the coordinate information (xl,yl,zl) of the left hand skeleton and the coordinate information (xr,yr,zr) of the right hand skeleton in each frame of the right-facing video data per second will be the same as the values ​​shown in Table 1500. Next, we will move on to the explanation of Figure 16.

[0213] In Figure 16, Graph 1600 shows the time series of coordinate information (xa, ya, za, xb, yb, zb) for left-facing video data. The vertical axis of Graph 1600 represents pixels. The horizontal axis of Graph 1600 represents seconds. Similarly, Graph 1610 shows the time series of coordinate information (xa, ya, za, xb, yb, zb) for right-facing video data. The vertical axis of Graph 1610 represents pixels. The horizontal axis of Graph 1610 represents seconds.

[0214] As shown in Figures 15 and 16, the time series of coordinate information (xa, ya, za, xb, yb, zb) in the leftward-facing video data is identical to the time series of coordinate information (xa, ya, za, xb, yb, zb) in the rightward-facing video data.

[0215] As a result, the information processing device 100 can accurately represent the characteristics of a person's posture or movement while treating symmetrical postures or movements of a person as identical based on the coordinate information (xa, ya, za, xb, yb, zb). The information processing device 100 can retain information other than the distinction between left and right in the coordinate information (xa, ya, za, xb, yb, zb). Therefore, based on the coordinate information (xa, ya, za, xb, yb, zb), the information processing device 100 can learn an action recognition model 620 that can accurately recognize the same type of action relating to each skeleton of two left-right paired skeletons as the same action.

[0216] Next, we will move on to explaining Figures 17 and 18 and compare the method of the information processing device 100 with conventional methods. A conventional method, for example, is one in which the coordinate value with the larger absolute value between the coordinate value indicating the position of the right hand and the coordinate value indicating the position of the left hand is adopted as the coordinate value indicating the position of the hands.

[0217] In Figure 17, Table 1700 shows the coordinate information (xi, yi, zi) obtained by converting the coordinate information (xl, yl, zl) of the left hand skeleton and the coordinate information (xr, yr, zr) of the right hand skeleton in each frame of leftward-facing video data using a conventional method. xi is the larger of the absolute values ​​of xl and xr. yi is the larger of the absolute values ​​of yl and yr. zi is the larger of the absolute values ​​of zl and zr.

[0218] Table 1700 has fields for seconds, xi, yi, and zi. The seconds field is set to the number of seconds indicating which frame of the left-facing video data is in.

[0219] The xi field is set to the coordinate value xi in the frame of the specified number of seconds. The unit of the coordinate value xi is, for example, pixels. The yi field is set to the coordinate value yi in the frame of the specified number of seconds. The unit of the coordinate value yi is, for example, pixels. The zi field is set to the coordinate value zi in the frame of the specified number of seconds. The unit of the coordinate value zi is, for example, pixels.

[0220] Here, the coordinate information (xi,yi,zi) obtained by converting the coordinate information (xl,yl,zl) of the left hand skeleton and the coordinate information (xr,yr,zr) of the right hand skeleton in each frame of the rightward-facing video data using the conventional method is identical to the values ​​shown in Table 1700. Next, we will move on to the explanation of Figure 18.

[0221] In Figure 18, Graph 1800 shows the time series of coordinate information (xi, yi, zi) for left-facing video data. The vertical axis of Graph 1800 represents pixels. The horizontal axis of Graph 1800 represents seconds. Similarly, Graph 1810 shows the time series of coordinate information (xi, yi, zi) for right-facing video data. The vertical axis of Graph 1810 represents pixels. The horizontal axis of Graph 1810 represents seconds.

[0222] As shown in Figures 17 and 18, the time series of coordinate information (xi, yi, zi) in left-facing video data and the time series of coordinate information (xi, yi, zi) in right-facing video data are identical. However, in conventional methods, the coordinate information (xi, yi, zi) may lack information other than the distinction between left and right, regarding the coordinate information of the left hand skeleton (xl, yl, zl) and the coordinate information of the right hand skeleton (xr, yr, zr).

[0223] Specifically, as shown in Figures 12 and 14, around 7 seconds, the z-coordinate value of one hand decreases, but the z-value of the other hand does not decrease compared to the z-coordinate value of the other hand, as can be seen in Graphs 1200 and 1400. In contrast, as shown in Figure 18, in the time series of coordinate information (xi, yi, zi), around 7 seconds, the characteristic that the z-coordinate value of one hand decreases remains in the coordinate value zi, but the characteristic that the z-value of the other hand does not decrease is lost.

[0224] On the other hand, as shown in Figure 16, the information processing device 100 can represent the commonalities and differences between the coordinate values ​​zl and zr using the coordinate values ​​za and zb. Therefore, the information processing device 100 can retain in the coordinate information (za, zb) the characteristic that around 7 seconds, the z coordinate value of one hand becomes smaller, but the z coordinate value of the other hand does not become smaller compared to the z coordinate value of that one hand.

[0225] Therefore, compared to conventional methods, the information processing device 100 can more easily learn an action recognition model 620 that can accurately recognize the same type of action for each of the two pairs of skeletons as the same action. Next, we will move on to explaining Figures 19 and 20, and describe the case in which the information processing device 100 learns and tests the action recognition model 620 based on the ground truth labels corresponding to the left-facing video data and the ground truth labels corresponding to the right-facing video data.

[0226] In Figure 19, Table 1900 shows the correct labels corresponding to the leftward-facing video data. Table 1900 has fields for time and action. The time field is set to the number of seconds, indicating which frame of the leftward-facing video data it is. The action field is set to the correct label indicating whether or not the person performed a specific action in the above frame. The specific action is the person bending down and picking up an object with one hand. If the correct label has a value of 0, it indicates that the specific action was not performed. If the correct label has a value of 1, it indicates that the specific action was performed.

[0227] Table 1910 shows the correct labels corresponding to the rightward-facing video data. Table 1910 has fields for time and action. The time field is set to the number of seconds, indicating which frame of the rightward-facing video data it is. The action field is set to the correct label indicating whether or not the person performed a specific action in the above frame. The specific action is the person bending down and picking up an object with one hand. If the correct label has a value of 0, it indicates that the specific action was not performed. If the correct label has a value of 1, it indicates that the specific action was performed.

[0228] Here, frames where seconds mod 3 = 1 are not used for training, and frames where seconds mod 3 = 1 are used for testing. Specifically, frames where seconds mod 3 = 1 are seconds = 1, 4, 7, 10, 13, and 16. The information processing device 100 is assumed to have trained the action recognition model 620 based on model training data that combines coordinate information (xa, ya, za, xb, yb, zb) and the correct label for each frame used for training.

[0229] Furthermore, as a comparison point for the action recognition model 620, we assume that the conventional model was trained based on conventional training data, which combines coordinate information (xi, yi, zi) obtained using conventional methods with the correct labels for each frame of the training target. Now, let's move on to the explanation of Figure 20.

[0230] In Figure 20, the information processing device 100 uses the action recognition model 620 to estimate the presence or absence of action in each frame of the subject under test. Specifically, the information processing device 100 obtains the estimation result of the presence or absence of action by inputting coordinate information (xa, ya, za, xb, yb, zb) for each frame of the subject under test into the action recognition model 620. The estimation results of the presence or absence of action using the action recognition model 620 are shown in Table 2000.

[0231] Table 2000 has fields for Time, Ground Answer, and Estimated Result. The Time field is set to the number of seconds modulo 3 = 1. The Ground Answer field is set to the ground answer label for the frame of the above number of seconds. The Estimated Result field is set to the estimated result of whether or not an action occurred for the frame of the above number of seconds, using the action recognition model 620. An estimated result of 0 indicates that the specific action did not occur. An estimated result of 1 indicates that the specific action occurred.

[0232] Table 2010 shows the results of estimating the presence or absence of an action using a conventional model, for comparison with the results of estimating the presence or absence of an action using the action recognition model 620. Table 2010 has fields for time, ground truth, and estimation result. The time field is set to the number of seconds modulo 3 = 1. The ground truth field is set to the ground truth label for the frame of the above number of seconds. The estimation result field is set to the estimation result of the presence or absence of an action using the conventional model for the frame of the above number of seconds. An estimation result of 0 indicates that the specific action was not performed. An estimation result of 1 indicates that the specific action was performed.

[0233] Thus, the conventional method results in misestimation, leading to a Recall of 1.0 and a Precision of 0.5 (F-value of 0.667). Specifically, as mentioned above, the conventional method lacks the characteristic that around 7 seconds, the z-coordinate value of one hand decreases while the z-coordinate value of the other hand does not decrease, which is thought to be the cause of the misestimation.

[0234] In contrast, the information processing device 100 can use the action recognition model 620 to estimate the presence or absence of action and set the result to a value that matches the correct label. The information processing device 100 can achieve Recall=Precision=1.0 (F-score 1.0). Thus, compared to conventional methods, the information processing device 100 can learn an action recognition model 620 that can recognize human actions with higher accuracy, and can recognize human actions with higher accuracy.

[0235] (Learning process steps) Next, an example of a learning process procedure executed by the information processing device 100 will be described using Figure 21. The learning process is realized, for example, by the CPU 301 shown in Figure 3, storage areas such as memory 302 and recording medium 305, and network I / F 303.

[0236] Figure 21 is a flowchart showing an example of a learning process procedure. In Figure 21, the information processing device 100 acquires learning video data 601 (step S2101).

[0237] Next, the information processing device 100 uses a DL model to recognize a person shown in each frame of the learning video data 601 based on the learning video data 601, and calculates the coordinate information of each of the multiple skeletons of the person in each frame (step S2102).

[0238] Next, the information processing device 100 extracts the coordinate information of each of the two skeletons in each pair of skeletons from the coordinate information of each skeleton of the multiple skeletons of the person in each frame (step S2103).

[0239] Next, the information processing device 100 selects one of the two pairs of skeletons that form a left-right pair as the target for processing (step S2104). Next, the information processing device 100 converts the combination of x coordinates, y coordinates, and z coordinates of the coordinate information of the two skeletons of the selected pair into a combination of coordinates in a special Cartesian coordinate system that is independent of left-right orientation (step S2105).

[0240] Next, the information processing device 100 determines whether there are any pairs of left and right skeletons remaining that have not yet been selected (step S2106). If there are any pairs of left and right skeletons remaining (step S2106: Yes), the information processing device returns to the process in step S2104. On the other hand, if there are no pairs of left and right skeletons remaining (step S2106: No), the information processing device proceeds to the process in step S2107.

[0241] In step S2107, the information processing device 100 learns a machine learning model based on the combination of input samples that include the transformed coordinate combinations as explanatory variables and annotation data that serves as the ground truth (step S2107).

[0242] Next, the information processing device 100 saves the learned machine learning model (step S2108). Then, the information processing device 100 terminates the learning process.

[0243] (Recognition processing procedure) Next, an example of a recognition process procedure performed by the information processing device 100 will be described using Figure 22. The recognition process is realized, for example, by the CPU 301 shown in Figure 3, storage areas such as memory 302 and recording medium 305, and network I / F 303.

[0244] Figure 22 is a flowchart showing an example of the recognition processing procedure. In Figure 22, the information processing device 100 acquires evaluation video data 602 (step S2201).

[0245] Next, the information processing device 100 uses a DL model to recognize the person shown in each frame of the evaluation video data 602 based on the evaluation video data 602, and calculates the coordinate information of each of the multiple skeletons of the person in each frame (step S2202).

[0246] Next, the information processing device 100 extracts the coordinate information of each of the two skeletons in each pair of skeletons from the coordinate information of each skeleton of the multiple skeletons of the person in each frame (step S2203).

[0247] Next, the information processing device 100 selects one of the two pairs of skeletons that form a left-right pair as the target for processing (step S2204). Next, the information processing device 100 converts the combination of x coordinates, y coordinates, and z coordinates of the coordinate information of the two skeletons of the selected pair into a combination of coordinates in a special Cartesian coordinate system that is independent of left-right orientation (step S2205).

[0248] Next, the information processing device 100 determines whether there are any pairs of left and right skeletons remaining that have not yet been selected (step S2206). If there are any pairs of left and right skeletons remaining (step S2206: Yes), the information processing device returns to the process in step S2204. On the other hand, if there are no pairs of left and right skeletons remaining (step S2206: No), the information processing device proceeds to the process in step S2207.

[0249] In step S2207, the information processing device 100 uses the learned machine learning model to perform action recognition based on input data that includes the transformed coordinate combination as explanatory variables (step S2207).

[0250] Next, the information processing device 100 outputs the result of the action recognition (step S2208). Then, the information processing device 100 terminates the recognition process.

[0251] Here, the information processing device 100 may execute some steps of the flowcharts in Figures 21 and 22 in a different order. Alternatively, the information processing device 100 may omit some steps of the flowcharts in Figures 21 and 22.

[0252] As explained above, the information processing device 100 can acquire video footage of the first person. By analyzing the acquired video footage, the information processing device 100 can identify the positional information of two pairs of body parts belonging to the first person in the video. The information processing device 100 can generate a first component representing the sum of the positional information of the identified body parts. The information processing device 100 can generate a third component representing the absolute value of the second component, which represents the difference between the positional information of the identified body parts. Based on the generated first component and the generated third component, the information processing device 100 can learn a model that outputs the posture information of the second person from video footage of the second person. This makes it easier for the information processing device 100 to learn a model that accurately estimates the posture information of the second person from video footage of the second person.

[0253] According to the information processing device 100, a model can be trained based on the generated first component, second component, and generated third component. This makes it easier for the information processing device 100 to train a model that accurately estimates the posture information of a second person from an image showing the second person, while also considering the second component.

[0254] According to the information processing device 100, by analyzing the acquired video, it is possible to identify positional information that represents the position of each body part in the video, with multiple component values ​​in different axial directions. This allows the information processing device 100 to handle cases where the positional information represents a position in a multidimensional space with multiple component values ​​in different axial directions. The information processing device 100 can learn a model that utilizes positional information that represents a position in a multidimensional space with multiple component values ​​in different axial directions.

[0255] According to the information processing device 100, an index value can be calculated for each axis direction using the sum of the axial component values ​​indicated by the position information of each body part, and a first component can be generated by combining the calculated index values. According to the information processing device 100, an index value can be calculated for each axis direction using the difference between the axial component values ​​indicated by the position information of each body part, and a third component can be generated by combining the absolute values ​​of the calculated index values. As a result, the information processing device 100 can accurately generate the first component and the third component when the position information indicates multiple component values ​​in different axes that represent a position in a multidimensional space.

[0256] According to the information processing device 100, an index value can be calculated for each axis direction using the difference in axial component values ​​indicated by the position information of each body part, and a second component can be generated by combining the calculated index values. According to the information processing device 100, a model can be learned based on the generated first component, the generated second component, and the generated third component. As a result, the information processing device 100 can accurately generate the second component when the position information indicates multiple component values ​​in different axes, each representing a position in a multidimensional space.

[0257] According to the information processing device 100, two body parts can be assigned a combination of different body parts that are paired left and right. This allows the information processing device 100 to learn a model that can recognize similar actions of a second person using different body parts that are paired left and right as identical actions.

[0258] According to the information processing device 100, by analyzing an image of a second person, it is possible to identify the positional information of two pairs of body parts of the second person in the image. According to the information processing device 100, it is possible to generate a fourth component representing the sum of the positional information of the two pairs of body parts, and a sixth component representing the absolute value of the fifth component representing the difference between the positional information of the two pairs of body parts. According to the information processing device 100, based on the generated fourth component and the generated sixth component, the posture information of the second person can be obtained using a learned model. As a result, the information processing device 100 can accurately estimate the posture information of the second person.

[0259] According to the information processing device 100, by analyzing the acquired video, the position of the first person's skeleton in the video can be identified, and based on the identified position of the skeleton, the positional information of each body part can be identified. In this way, the information processing device 100 can identify the positional information of each body part related to the posture of the first person by utilizing the method for identifying the position of the skeleton.

[0260] According to the information processing device 100, the posture information can include information indicating whether or not the second person is in a posture corresponding to a specific action. This allows the information processing device 100 to determine whether or not the second person is in a posture corresponding to a specific action, and to determine whether or not the second person has performed a specific action.

[0261] The information processing method described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a PC or workstation. The information processing program described in this embodiment is recorded on a computer-readable recording medium and executed by being read from the recording medium by the computer. The recording medium can be a hard disk, flexible disk, CD (Compact Disc)-ROM, MO (Magneto Optical Disc), DVD (Digital Versatile Disc), etc. Furthermore, the information processing program described in this embodiment may be distributed via a network such as the Internet.

[0262] With regard to the embodiments described above, the following additional information is disclosed.

[0263] (Note 1) Obtain video footage showing the first person, By analyzing the acquired video footage, the positional information of each of two pairs of body parts belonging to the first person in the video footage is identified. A first component is generated that represents the sum of the positional information of each of the identified body parts. A third component is generated that represents the absolute value of the second component, which represents the difference in the positional information of each of the identified body parts. Based on the generated first component and the generated third component, a model is trained to output the posture information of a second person from a video showing the second person. An information processing program characterized by having a computer perform the processing.

[0264] (Note 2) The learning process described above is: The information processing program according to Appendix 1, characterized in that it learns the model based on the generated first component, the generated second component, and the generated third component.

[0265] (Appendix 3) The process specified above is: The information processing program according to Appendix 1 or 2, characterized by analyzing the acquired video to identify positional information that represents the position of each body part in multidimensional space in the video, with each component value in a different axial direction.

[0266] (Note 4) The process for generating the first component is: For each of the aforementioned axial directions, an index value is calculated using the sum of the component values ​​in the axial direction indicated by the position information of each body part, and the first component is generated by combining the calculated index values. The process for generating the aforementioned third component is: The information processing program according to Appendix 3, characterized in that it calculates an index value for each of the aforementioned axial directions using the difference in the axial component values ​​indicated by the position information of each body part, and generates the third component by combining the absolute values ​​of the calculated index values.

[0267] (Note 5) For each of the axial directions, an index value is calculated using the difference in the axial component values ​​indicated by the position information of each body part, and the second component is generated by combining the calculated index values. The computer is made to perform the process, The learning process described above is: The information processing program according to Appendix 4, characterized in that it learns the model based on the generated first component, the generated second component, and the generated third component.

[0268] (Appendix 6) The information processing program according to Appendix 1 or 2, characterized in that the two body parts are a combination of different body parts that are paired on the left and right sides.

[0269] (Note 7) The model has a function to output posture information of the second person in response to inputs of a fourth component representing the sum of the positional information of two paired body parts among the body parts of the second person, and a sixth component representing the absolute value of the fifth component representing the difference between the positional information of two paired body parts. By analyzing the video in which the second person appears, among the body parts of the second person in the video, a fourth component indicating the sum of the position information of each of two paired body parts, and a sixth component indicating the absolute value of the difference between the position information of each of the two paired body parts are generated. Based on the generated fourth component and the generated sixth component, using the learned model, posture information of the second person is obtained. An information processing program according to Appendix 1 or 2, characterized in that the computer is caused to execute the processing.

[0270] (Appendix 8) The specifying process is By analyzing the obtained video, the position of the skeleton of the first person in the video is specified, and based on the specified position of the skeleton, the position information of each of the body parts is specified. An information processing program according to Appendix 1 or 2, characterized by this.

[0271] (Appendix 9) The posture information indicates whether the second person is in a posture corresponding to a specific action. An information processing program according to Appendix 1 or 2, characterized by this.

[0272] (Appendix 10) Obtain a video in which the first person appears, By analyzing the obtained video, among the body parts of the first person in the video, the position information of each of two paired body parts is specified. Generate a first component indicating the sum of the specified position information of each of the body parts. Generate a third component indicating the absolute value of the difference between the specified position information of each of the body parts. Based on the generated first component and the generated third component, learn a model that outputs the posture information of the second person from the video in which the second person appears. An information processing method characterized in that a computer executes the processing.

[0273] (Appendix 11) Obtain a video in which the first person appears. By analyzing the acquired video footage, the positional information of each of two pairs of body parts belonging to the first person in the video footage is identified. A first component is generated that represents the sum of the positional information of each of the identified body parts. A third component is generated that represents the absolute value of the second component, which represents the difference in the positional information of each of the identified body parts. Based on the generated first component and the generated third component, a model is trained to output the posture information of a second person from a video showing the second person. An information processing device characterized by having a control unit. [Explanation of Symbols]

[0274] 100 Information Processing Devices 101,102 vectors 110 Component 1 120 Second component 130 Third component 200 Information Processing Systems 201 Video Recording Device 202 Client Devices 210 Network 300,400 buses 301,401 CPU 302,402 memory 303,403 Network I / F 304,404 Recording medium I / F 305,405 recording media 306 displays 307 Input device 406 Camera 500 storage section 501 Acquisition Department 502 Specific part 503 Generation part Room 504, Learning Department 505 Recognition part 506 Output section 600 Annotation Data 601 Learning video data 602 Evaluation video data 610 DL model 611,615 Person recognition 612,616 Skeleton estimation 613,617 Coordinate transformation 614 Machine learning 618 Action recognition 620 Action recognition model 700 Pair management table 800,810,820 Code 900,1000 Cartesian coordinate system 901,902,1001 Point 910 45-degree line 1100,1300,1500,1700,1900,1910,2000,2010 Table 1200,1400,1600,1610,1800,1810 Graph

Claims

1. The first process involves acquiring video footage of the first person, A second process involves analyzing the acquired video footage to identify the positional information of two paired body parts of the first person in the video footage. A third process generates a first component representing the sum of the positional information of each of the identified body parts, A fourth process that generates a third component representing the absolute value of the second component representing the difference in positional information of each of the identified body parts, A fifth process involves learning a model that outputs posture information of a second person from a video showing the second person, based on the first and third components generated. An information processing program characterized by causing a computer to execute it.

2. The fifth process is, The information processing program according to claim 1, characterized in that it learns the model based on the generated first component, the second component, and the generated third component.

3. The second process described above is: The information processing program according to claim 1 or 2, characterized in that, by analyzing the acquired video, it identifies positional information representing multiple component values ​​in different axial directions that represent the position of each body part in multidimensional space in the video.

4. The third process described above is: For each of the aforementioned axial directions, an index value is calculated using the sum of the component values ​​in the axial direction indicated by the position information of each body part, and the first component is generated by combining the calculated index values. The fourth process is, The information processing program according to claim 3, characterized in that it calculates an index value for each of the aforementioned axial directions using the difference in the axial component values ​​indicated by the position information of each body part, and generates the third component by combining the absolute values ​​of the calculated index values.

5. The information processing program according to claim 1 or 2, characterized in that the two body parts are a combination of different body parts that are paired on the left and right sides.

6. We obtained video footage showing the first person, By analyzing the acquired video footage, the positional information of each of two paired body parts of the first person in the video footage is identified. A first component is generated that represents the sum of the positional information of each of the identified body parts. A third component is generated that represents the absolute value of the second component, which represents the difference in the positional information of each of the identified body parts. Based on the generated first component and the generated third component, a model is trained to output posture information of the second person from a video showing the second person. An information processing method characterized in that the processing is performed by a computer.

7. We obtained video footage showing the first person, By analyzing the acquired video footage, the positional information of each of two paired body parts of the first person in the video footage is identified. A first component is generated that represents the sum of the positional information of each of the identified body parts. A third component is generated that represents the absolute value of the second component, which represents the difference in the positional information of each of the identified body parts. Based on the generated first component and the generated third component, a model is trained to output posture information of the second person from a video showing the second person. An information processing device characterized by having a control unit.

Images