Information Processing Program, Information Processing Method, and Information Processing Apparatus

Abstract

To achieve a reduction of process burdens when recognizing object actions.SOLUTION: An object action is formed by combining an element action 1 and an element action 2 which have a time interval within an effective time. Data 110 that indicates a relationship between element actions regarding the element action 1 and the element action 2 in an object period is acquired. An information processing device 100 acquires an effective time corresponding to the object action. The information processing device 100 sets a first divided section and a second divided section by dividing the object period. The information processing device 100 searches for a combination of the element action 1 and the element action 2 which form the object action and which have the time interval within the effective time among the element actions in each divided section of the first divided section and the second divided section based on the acquired data 110.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, there has been a technique of using a model obtained by machine learning to recognize a person, an object, etc. shown in a moving image, and further recognizing the behavior of the recognized person, the relationship between the recognized persons, and the relationship between the recognized person and the object. The model is, for example, a DNN (Deep Neural Network) or the like.

[0003] As a prior art, for example, there is one that identifies the type of basic movement based on the movement of feature points corresponding to a predetermined part or joint of the body of a subject included in consecutive frames. Further, for example, from a plurality of target pose data corresponding to a plurality of target image data, two or more target pose data are extracted as target trajectory data representing the transition of the subject's pose, and based on the target trajectory data, there is a technique of identifying the movement of the subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art, there is a problem that the processing burden for recognizing a specific behavior of a person tends to increase. For example, when recognizing a specific behavior formed by two or more behaviors, the model is learned using a moving image instead of a still image, and the processing burden tends to increase.

[0006] On one side, the present invention aims to reduce the processing burden involved in recognizing the target behavior.

Means for Solving the Problem

[0007] According to one embodiment, data indicating the relationship between element actions for a plurality of element actions during a target period is acquired, an effective time corresponding to the target action is acquired, and based on the acquired data, for each divided section set by dividing the target period according to the acquired effective time, an information processing program, an information processing method, and an information processing apparatus are proposed that search for combinations of two or more element actions that form the target action among the plurality of element actions.

Effect of the Invention

[0008] According to one aspect, it becomes possible to reduce the processing burden involved in recognizing the target behavior.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0010] Hereinafter, with reference to the drawings, embodiments of an information processing program, an information processing method, and an information processing apparatus according to the present invention will be described in detail.

[0011] (An Example of the Information Processing Method According to the Embodiment) FIG. 1 is an explanatory diagram showing an example of the information processing method according to the embodiment. The information processing apparatus 100 is a computer for facilitating the recognition of a target action. The target action is, for example, a relatively complex action. Specifically, it is considered that the more elemental actions that form the action, the more complex the action is. The target action is, for example, an action by a person. The information processing apparatus 100 is, for example, a server or a PC (Personal Computer).

[0012] Conventionally, in an attempt to recognize the behavior of an object shown in a moving image using a model such as a DNN obtained by machine learning, there is a problem that the processing burden for recognizing the behavior of the object tends to increase. For example, when training a model such as a DNN, it is preferable to prepare thousands or more training data, and the processing burden tends to increase. Further, for example, when training a model capable of recognizing a complex behavior formed by a combination of relatively many elemental behaviors, since the model is trained in consideration of the time series using a moving image instead of a still image, the processing burden tends to increase.

[0013] On the other hand, a method can be considered in which a DNN that detects the skeletal coordinates of a person is used to recognize the temporal change in the skeletal position of the person shown in a moving image, and the behavior of the person is recognized based on the temporal change in the skeletal position. For this method, for example, Patent Document 1 above can be referred to. In this method, when attempting to recognize various behaviors, the processing is performed from the beginning of the moving image for each behavior, and the processing burden tends to increase.

[0014] Further, a method can be considered in which the behavior shown in a moving image is represented as graph data, and a relatively complex behavior is recognized based on the graph data. For this method, for example, Reference Document 1 below can be referred to. In this method, since graph data corresponding to the moving image is generated, the longer the time of the moving image, the larger the scale of the graph data tends to be, and the processing time and processing burden required for recognizing a relatively complex behavior tend to increase.

[0015] Reference Document 1: Vizcarra, Julio, Satoshi Nishimura, and Ken Fukuda. “Knowledge graph retrieval and analysis for the evaluation of customer service in video.” (2020): 07-01.

[0016] Therefore, in the present embodiment, an information processing method that can reduce the processing load involved in recognizing a target action will be described.

[0017] In FIG. 1, the target action is formed by, for example, a combination of two or more elemental actions. Specifically, the target action is defined by a combination of two or more elemental actions and an effective time. The effective time indicates, for example, an upper limit regarding the time interval between elemental actions. In the example of FIG. 1, the target action is formed by a combination of elemental action 1 and elemental action 2. Specifically, the target action is formed by a combination of elemental action 1 and elemental action 2 in which the time interval is within the effective time.

[0018] (1-1) The information processing apparatus 100 acquires data 110 indicating the relationship between elemental actions for a plurality of elemental actions during a target period. The data 110 is, for example, graph data. The plurality of elemental actions include, for example, the elemental actions forming the target action. The information processing apparatus 100 acquires, for example, by using a predetermined model to recognize a plurality of elemental actions during the target period based on a moving image regarding the target period and generating data 110 indicating the relationship between the elemental actions. The predetermined model is, for example, a DNN. In the example of FIG. 1, the information processing apparatus 100 acquires data 110 indicating the relationship between elemental action 1 and elemental action 2 during the target period. For example, action 1-1 and action 1-2 are elemental action 1. For example, action 2-1 and action 2-2 are elemental action 2.

[0019] (1-2) The information processing apparatus 100 acquires the effective time corresponding to the target action. The information processing apparatus 100 acquires, for example, by reading out from the storage unit the effective time corresponding to the target action that is set in advance by the user and stored in the storage unit. The information processing apparatus 100 may acquire the effective time corresponding to the target action by receiving an input of the effective time corresponding to the target action based on, for example, a user's operation input.

[0020] (1-3) The information processing apparatus 100 divides the target period according to the acquired effective time to set a plurality of divided sections. The divided sections may, for example, overlap with each other. The information processing apparatus 100, for example, divides the target period in time units longer than the effective time to set a plurality of divided sections. In the example of FIG. 1, the information processing apparatus 100 divides the target period to set a first divided section and a second divided section.

[0021] (1-4) The information processing apparatus 100 recognizes the target behavior by searching for combinations of two or more elemental behaviors that form the target behavior among a plurality of elemental behaviors for each set divided section based on the acquired data 110. The information processing apparatus 100, for example, generates divided data indicating the relationship between elemental behaviors for the elemental behaviors in the set divided section for each set divided section based on the acquired data 110. The information processing apparatus 100, for example, searches for combinations of two or more elemental behaviors that form the target behavior among the elemental behaviors in the divided section indicated by the generated divided data for each generated divided data.

[0022] Specifically, the information processing apparatus 100 searches for a combination of an elemental behavior 1 and an elemental behavior 2 that form the target behavior and whose time interval is within the effective time among the elemental behaviors in the divided section indicated by the divided data. In the example of FIG. 1, the information processing apparatus 100 more specifically searches for a combination of an action 1-1 and an elemental behavior 2-1 that form the target behavior and whose time interval is within the effective time among the elemental behaviors in the first divided section. Similarly, the information processing apparatus 100 more specifically searches for a combination of an action 1-2 and an elemental behavior 2-2 that form the target behavior and whose time interval is within the effective time among the elemental behaviors in the second divided section.

[0023] As a result, the information processing apparatus 100 can make it easier to recognize the target behavior. For example, the information processing apparatus 100 can make it easier to recognize a relatively complex target behavior. Since the information processing apparatus 100 does not need to learn a model capable of recognizing the target behavior, it is possible to suppress an increase in processing time and processing load. Even if the size of the data 110 is large, the information processing apparatus 100 can suppress an increase in processing time and processing load.

[0024] Here, the case where the information processing apparatus 100 generates the data 110 indicating the relationship between the elemental behaviors has been described, but it is not limited to this. For example, the information processing apparatus 100 may obtain the data 110 indicating the relationship between the elemental behaviors by receiving it from another computer. Another computer may, for example, use a predetermined model to recognize a plurality of elemental behaviors in the target period based on a moving image regarding the target period, and generate the data 110 indicating the relationship between the elemental behaviors.

[0025] Here, the case where the information processing apparatus 100 operates alone has been described, but it is not limited to this. For example, the information processing apparatus 100 may cooperate with another computer. Also, for example, a plurality of computers may realize the functions as the information processing apparatus 100 in a distributed manner. An example of the case where the information processing apparatus 100 cooperates with another computer will be specifically described later with reference to FIG. 2.

[0026] (An example of the information processing system 200) Next, with reference to FIG. 2, an example of an information processing system 200 to which the information processing apparatus 100 shown in FIG. 1 is applied will be described.

[0027] FIG. 2 is an explanatory diagram showing an example of the information processing system 200. In FIG. 2, the information processing system 200 includes an information processing apparatus 100, an elemental behavior recognition apparatus 201, and a client apparatus 202.

[0028] In the information processing system 200, the information processing apparatus 100 and the element behavior recognition apparatus 201 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like. Also, in the information processing system 200, the information processing apparatus 100 and the client apparatus 202 are connected via a wired or wireless network 210.

[0029] The information processing apparatus 100 is a computer for facilitating the recognition of the target behavior. The information processing apparatus 100 stores, for example, in association, a combination of two or more element behaviors that form the target behavior and a valid time corresponding to the target behavior. Specifically, the information processing apparatus 100 receives and stores, from the client apparatus 202, a combination of two or more element behaviors that form the target behavior and a valid time corresponding to the target behavior.

[0030] The information processing apparatus 100 acquires, for example, data indicating the relationship between element behaviors for a plurality of element behaviors in a target period by receiving it from the element behavior recognition apparatus 201. The information processing apparatus 100 acquires, for example, by reading out the valid time corresponding to the stored target behavior. The information processing apparatus 100 sets, for example, a plurality of divided sections by dividing the target period according to the acquired valid time. The information processing apparatus 100 recognizes the target behavior, for example, by searching for a combination of two or more element behaviors that form the target behavior among the plurality of element behaviors for each of the set divided sections based on the acquired data.

[0031] The information processing apparatus 100 outputs, for example, the result of recognizing the target behavior so that it can be referred to by the system user. The information processing apparatus 100 transmits, for example, the result of recognizing the target behavior to the client apparatus 202. The information processing apparatus 100 is, for example, a server, a PC, or the like.

[0032] The elemental action recognition device 201 is a computer for recognizing elemental actions. The elemental action recognition device 201, for example, acquires a moving image regarding a target period. Specifically, the elemental action recognition device 201 acquires a moving image by receiving an input of the moving image. Specifically, the elemental action recognition device 201 may have a camera device and acquire a moving image by the camera device. Specifically, the elemental action recognition device 201 may acquire a moving image by receiving it from another computer. The other computer is, for example, the client device 202.

[0033] The elemental action recognition device 201, for example, recognizes elemental actions based on the acquired moving image. Specifically, the elemental action recognition device 201 uses a predetermined model to recognize a plurality of elemental actions in the target period based on the acquired moving image. The elemental action recognition device 201 may, for example, further recognize other elemental actions obtained by combining the recognized elemental actions. The elemental action recognition device 201, for example, generates data indicating the relationship between elemental actions for the recognized elemental actions and transmits it to the information processing device 100. The elemental action recognition device 201 is, for example, a server or a PC, etc.

[0034] The client device 202 is a computer used by a system user. The client device 202, for example, transmits to the information processing device 100 a combination of two or more elemental actions forming a target action and an effective time corresponding to the target action based on an operation input of the system user. The client device 202, for example, receives the result of recognizing the target action from the information processing device 100. The client device 202, for example, outputs the result of recognizing the target action in a manner that can be referred to by the system user. The client device 202 is, for example, a PC, a tablet terminal, or a smartphone, etc.

[0035] Here, the case where the information processing apparatus 100 is a different apparatus from the element behavior recognition apparatus 201 has been described, but it is not limited to this. For example, the information processing apparatus 100 may have the function as the element behavior recognition apparatus 201 and may also operate as the element behavior recognition apparatus 201. Here, the case where the information processing apparatus 100 is a different apparatus from the client apparatus 202 has been described, but it is not limited to this. For example, the information processing apparatus 100 may have the function as the client apparatus 202 and may also operate as the client apparatus 202.

[0036] (Hardware configuration example of information processing apparatus 100) Next, a hardware configuration example of the information processing apparatus 100 will be described with reference to FIG. 3.

[0037] FIG. 3 is a block diagram showing a hardware configuration example of the information processing apparatus 100. In FIG. 3, the information processing apparatus 100 includes a processor 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, a recording medium 305, and a camera device 306. Also, each component is connected by a bus 300.

[0038] Here, the processor 301 controls the entire information processing apparatus 100. The processor is a CPU (Central Processing Unit), or a GPU (Graphics Processing Unit), etc. The GPU is, for example, an arithmetic unit specialized for image processing.

[0039] The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash ROM, etc. Specifically, for example, the flash ROM and the ROM store various programs, and the RAM is used as the work area of the processor 301. The programs stored in the memory 302 are loaded into the processor 301 to cause the processor 301 to execute the coded processing.

[0040] The network I / F 303 is connected to the network 210 through a communication line and is connected to other computers via the network 210. Then, the network I / F 303 manages the interface with the network 210 and controls the input / output of data from other computers. The network I / F 303 is, for example, a modem, a LAN adapter, or the like.

[0041] The recording medium I / F 304 controls the read / write of data to / from the recording medium 305 according to the control of the processor 301. The recording medium I / F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like. The recording medium 305 is a non-volatile memory that stores the data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, or the like. The recording medium 305 may be detachable from the information processing apparatus 100. The camera device 306 has an imaging element and generates a moving image based on the signal of the imaging element.

[0042] In addition to the components described above, the information processing apparatus 100 may have, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, and the like. Also, the information processing apparatus 100 may have a plurality of recording medium I / Fs 304 and recording media 305. Further, the information processing apparatus 100 may not have the recording medium I / F 304 and the recording medium 305. The information processing apparatus 100 may not have the camera device 306.

[0043] (Hardware configuration example of the element behavior recognition device 201) The hardware configuration example of the element behavior recognition device 201 is the same as the hardware configuration example of the information processing apparatus 100 shown in FIG. 3, and thus the description is omitted.

[0044] (Hardware configuration example of the client device 202) Since the hardware configuration example of the client device 202 is the same as the hardware configuration example of the information processing device 100 shown in FIG. 3, the description thereof is omitted. The client device 202 may not have, for example, a GPU.

[0045] (Functional configuration example of the information processing device 100) Next, a functional configuration example of the information processing device 100 will be described with reference to FIG. 4.

[0046] FIG. 4 is a block diagram showing a functional configuration example of the information processing device 100. As shown in FIG. 4, the information processing device 100 includes, for example, a storage unit 400, an acquisition unit 401, a generation unit 402, a search unit 403, and an output unit 404.

[0047] The storage unit 400 is realized by a storage area such as the memory 302 and the recording medium 305 shown in FIG. 3, for example. Hereinafter, the case where the storage unit 400 is included in the information processing device 100 will be described, but it is not limited thereto. For example, the storage unit 400 may be included in a device different from the information processing device 100, and the stored content of the storage unit 400 may be referable from the information processing device 100.

[0048] The acquisition unit 401 to the output unit 404 function as an example of a control unit. Specifically, the acquisition unit 401 to the output unit 404 realize their functions by causing the processor 301 to execute a program stored in a storage area such as the memory 302 and the recording medium 305 shown in FIG. 3, or by the network I / F 303. The processing results of each functional unit are stored in a storage area such as the memory 302 and the recording medium 305 shown in FIG. 3, for example.

[0049] The storage unit 400 stores various information that is referred to or updated in the processing of each functional unit. The storage unit 400 stores, for example, a moving image during a target period. The moving image is acquired by the acquisition unit 401, for example. The moving image includes a plurality of frames, for example.

[0050] The memory unit 400 stores, for example, the types of actions treated as elemental actions. The actions treated as elemental actions are, for example, the types of actions that can be detected using a predetermined model. The predetermined model is, for example, a DNN. The actions treated as elemental actions may be, for example, actions formed by combinations of two or more elemental actions. The types of actions treated as elemental actions are, for example, set in advance by the user. The types of actions treated as elemental actions may be, for example, acquired by the acquisition unit 401.

[0051] The memory unit 400 stores, for example, a predetermined model. The predetermined model is, for example, a model for making it possible to detect actions treated as elemental actions. Specifically, the predetermined model is a model for making it possible to recognize a person, a skeleton, an object, etc. in order to make it possible to detect actions treated as elemental actions. More specifically, the predetermined model is a model for making it possible to recognize the skeletal position of a person. The predetermined model is, for example, a DNN. The predetermined model is, for example, set in advance by the user. The predetermined model may be, for example, acquired by the acquisition unit 401.

[0052] The memory unit 400 stores, for example, a first recognition rule for making it possible to recognize elemental actions based on the results recognized by a predetermined model. The first recognition rule is, for example, set in advance by the user. The first recognition rule may be, for example, acquired by the acquisition unit 401. The memory unit 400 stores, for example, a second recognition rule for making it possible to recognize combinations of two or more elemental actions that form actions treated as elemental actions. The second recognition rule is, for example, set in advance by the user. The second recognition rule may be, for example, acquired by the acquisition unit 401.

[0053] The memory unit 400 stores, for example, the types of actions to be treated as target actions. The actions to be treated as target actions are, for example, actions formed by combinations of two or more elemental actions. Specifically, the actions to be treated as target actions are defined by an effective time and a combination of two or more elemental actions. More specifically, the actions to be treated as target actions are actions formed by combinations of two or more elemental actions in which the time interval between at least any two elemental actions is within the effective time. The actions to be treated as target actions are, for example, types of actions that cannot be detected using a predetermined model. The types of actions to be treated as target actions are, for example, set in advance by the user. The types of actions to be treated as target actions may be, for example, acquired by the acquisition unit 401.

[0054] The memory unit 400 stores, for example, the effective time corresponding to the target action. The effective time indicates, for example, the upper limit of the time interval between elemental actions. The effective time is, for example, set in advance by the user. The effective time may be, for example, acquired by the acquisition unit 401.

[0055] The memory unit 400 stores, for example, a third recognition rule that enables the recognition of target actions. Specifically, the memory unit 400 stores a third recognition rule that enables the recognition of combinations of two or more elemental actions that form an action to be treated as a target action in which the time interval between at least any two elemental actions is within the effective time. The third recognition rule is, for example, set in advance by the user. The third recognition rule may be, for example, acquired by the acquisition unit 401.

[0056] The memory unit 400 stores, for example, relational data indicating the relationships between elemental actions for a plurality of elemental actions during a target period. The relational data indicates, for example, the attribute information of each elemental action of the plurality of elemental actions, the order relationship between elemental actions, the inclusion relationship between elemental actions, and the like. The attribute information indicates, for example, the person who performed the elemental action or the time when the elemental action was performed. The relational data is, for example, graph data indicating a graph structure formed by nodes corresponding to elemental actions.

[0057] Specifically, the graph structure is formed by nodes that associate element actions with the time at which the element actions are performed, and edges that indicate the inclusion relationship between an element action and other element actions formed by combinations of two or more element actions including the element action. The relational data may not be graph data, for example. The relational data is acquired by the acquisition unit 401, for example. The relational data may not be acquired by the acquisition unit 401, but may be generated by the generation unit 402, for example.

[0058] The acquisition unit 401 acquires various types of information used for the processing of each functional unit. The acquisition unit 401 stores the acquired various types of information in the storage unit 400 or outputs them to each functional unit. Further, the acquisition unit 401 may output the various types of information stored in the storage unit 400 to each functional unit. The acquisition unit 401 acquires various types of information based on a user's operation input, for example. The acquisition unit 401 may receive various types of information from a device different from the information processing apparatus 100, for example.

[0059] The acquisition unit 401 acquires, for example, the type of action to be treated as an element action. Specifically, the acquisition unit 401 acquires the type of action to be treated as an element action by receiving an input of the type of action to be treated as an element action based on a user's operation input. Specifically, the acquisition unit 401 may acquire the type of action to be treated as an element action by receiving it from another computer. The other computer is, for example, the client device 202.

[0060] The acquisition unit 401 acquires, for example, a predetermined model that enables the detection of an action to be treated as an element action. Specifically, the acquisition unit 401 acquires a predetermined model by receiving an input of the predetermined model based on a user's operation input. Specifically, the acquisition unit 401 may acquire a predetermined model by receiving it from another computer. The other computer is, for example, the client device 202.

[0061] The acquisition unit 401 acquires, for example, a first recognition rule. Specifically, the acquisition unit 401 acquires the first recognition rule by receiving an input of the first recognition rule based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the first recognition rule by receiving it from another computer. The other computer is, for example, the client device 202.

[0062] The acquisition unit 401 acquires, for example, a second recognition rule that enables recognition of two or more elemental actions that form an action to be treated as an elemental action. Specifically, the acquisition unit 401 acquires the second recognition rule by receiving an input of the second recognition rule based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the second recognition rule by receiving it from another computer. The other computer is, for example, the client device 202.

[0063] The acquisition unit 401 acquires, for example, a third recognition rule. Specifically, the acquisition unit 401 acquires the third recognition rule by receiving an input of the third recognition rule based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the third recognition rule by receiving it from another computer. The other computer is, for example, the client device 202.

[0064] The acquisition unit 401 acquires, for example, the type of action to be treated as the target action. Specifically, the acquisition unit 401 acquires the type of action to be treated as the target action by receiving an input of the type of action to be treated as the target action based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the type of action to be treated as the target action by receiving it from another computer. The other computer is, for example, the client device 202.

[0065] The acquisition unit 401 acquires, for example, the valid time corresponding to the target action. Specifically, the acquisition unit 401 acquires the valid time corresponding to the target action by receiving an input of the valid time corresponding to the target action based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the valid time corresponding to the target action by receiving it from another computer. The other computer is, for example, the client device 202.

[0066] The acquisition unit 401 acquires a moving image during the target period. Specifically, the acquisition unit 401 acquires the moving image during the target period by receiving an input of the moving image during the target period based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the moving image during the target period by receiving it from another computer. The other computer is, for example, the client device 202. At this time, when the acquisition unit 401 acquires the relationship data instead of the generation unit 402 generating the relationship data, the acquisition unit 401 does not necessarily have to acquire the moving image during the target period.

[0067] The acquisition unit 401 acquires, for example, relationship data indicating the relationship between element actions for a plurality of element actions during the target period. Specifically, the acquisition unit 401 acquires the relationship data by receiving an input of the relationship data based on a user's operation input. Specifically, the acquisition unit 401 may also acquire the relationship data by receiving it from another computer. The other computer is, for example, the client device 202. At this time, when the generation unit 402 generates the relationship data, the acquisition unit 401 does not necessarily have to acquire the relationship data.

[0068] The acquisition unit 401 may receive a start trigger for starting the processing of any functional unit. The start trigger may be, for example, that there has been a predetermined operation input by the user. The start trigger may be, for example, that predetermined information has been received from another computer. The start trigger may be, for example, that any functional unit has output predetermined information. The acquisition unit 401 may receive, as a start trigger for starting the processing of the generation unit 402, for example, that a moving image has been acquired. The acquisition unit 401 may receive, as a start trigger for starting the processing of the search unit 403, for example, that relational data has been acquired.

[0069] The generation unit 402 detects elemental actions. The generation unit 402 detects, for example, elemental actions related to a thing based on the result of recognizing the thing shown in the moving image during the target period acquired by the acquisition unit 401 and a first recognition rule using a predetermined model. Specifically, the generation unit 402 detects elemental actions related to a person based on the result of recognizing the skeletal position of the person shown in the moving image based on the moving image using a predetermined model. The generation unit 402 may further detect, for example, other elemental actions obtained by combining the detected elemental actions. Specifically, the generation unit 402 detects other elemental actions obtained by combining the detected elemental actions based on a second recognition rule.

[0070] Based on the detected elemental actions, the generation unit 402 generates relational data indicating the relationship between the elemental actions for a plurality of elemental actions during the target period. The generation unit 402 includes, for example, the detected elemental actions in association with the time when the elemental actions were performed, and generates relational data indicating the order relationship and inclusion relationship between the elemental actions. Thereby, the generation unit 402 can make the target action recognizable.

[0071] Based on the valid time acquired by the acquisition unit 401, the search unit 403 divides the target period and sets a plurality of divided intervals. The divided intervals may overlap with each other, for example. The search unit 403 divides the target period into time units longer than the valid time and sets a plurality of divided intervals, for example. Thereby, the search unit 403 can set a plurality of divided intervals obtained by dividing the target period so as to reduce the processing load involved in recognizing the target behavior.

[0072] The search unit 403 may, for example, divide the target period and set a plurality of divided intervals such that the divided intervals overlap with each other by at least the acquired valid time. Thereby, the search unit 403 can set a plurality of divided intervals obtained by dividing the target period so as to reduce the processing load involved in recognizing the target behavior. In addition, the search unit 403 can facilitate the recognition of the target behavior that occurs across the start or end time of the divided interval.

[0073] The search unit 403 recognizes the target behavior by searching for combinations of two or more elemental behaviors that form the target behavior among the plurality of elemental behaviors for each set divided interval based on the relationship data acquired by the acquisition unit 401. The search unit 403 searches for, for example, combinations of two or more elemental behaviors that form the target behavior for each set divided interval, and in which the time interval between at least any two of the elemental behaviors in the combination is equal to or less than the acquired valid time.

[0074] Specifically, the search unit 403 extracts partial data corresponding to the divided interval from the relationship data for each divided interval. Specifically, the search unit 403 refers to the third recognition rule and searches for combinations of two or more elemental behaviors that form the target behavior based on the extracted partial data for each divided interval. More specifically, the search unit 403 searches for combinations of two or more elemental behaviors that form the target behavior and in which the time interval between the elemental behaviors is equal to or less than the acquired valid time based on the extracted partial data for each divided interval. Thereby, the search unit 403 can recognize the target behavior while reducing the processing load involved in recognizing the target behavior.

[0075] The search unit 403 may, for example, determine whether or not there are some of the elemental actions included in the combination of two or more elemental actions that form the target action in the first divided section among the divided sections set by dividing the target period. For example, if there are some of the elemental actions included in the combination of two or more elemental actions that form the target action, the search unit 403 searches for the remaining elemental actions included in the combination in the second divided section after the first divided section.

[0076] Specifically, the search unit 403 may determine whether or not the time interval between the elemental actions is equal to or less than the acquired effective time in the combination of some of the elemental actions in the first divided section and the remaining elemental actions in the second divided section. Specifically, the search unit 403 specifies the combination of some of the elemental actions in the first divided section and the remaining elemental actions in the second divided section, which is determined to be equal to or less than the effective time, as the combination of two or more elemental actions that form the target action. Thereby, the search unit 403 can recognize the combination of two or more elemental actions that form the target action and straddle the end of the first divided section.

[0077] The output unit 404 outputs the processing result of at least any one of the functional units. The output format is, for example, display on a display, print output to a printer, transmission to an external device via the network I / F 303, or storage in a storage area such as the memory 302 or the recording medium 305. Thereby, the output unit 404 can notify the user of the processing result of at least any one of the functional units and improve the convenience of the information processing apparatus 100.

[0078] The output unit 404 outputs the result searched by the search unit 403. The output unit 404 outputs, for example, the target action recognized as a result of the search by the search unit 403 so that the user can refer to it. Specifically, the output unit 404 outputs the target action recognized by the search unit 403 so that the user can refer to it together with information that can specify the start or end point of the target action.

[0079] The output unit 404, more specifically, displays on the display the target action recognized by the search unit 403 together with information that can specify the start or end point of the target action. The output unit 404, more specifically, may transmit the target action recognized by the search unit 403 to another computer together with information that can specify the start or end point of the target action. The other computer is, for example, the client device 202 or the like. Thereby, the output unit 404 can make the result of recognizing the target action available to the user.

[0080] Here, the case where the information processing apparatus 100 includes the generation unit 402 has been described, but it is not limited thereto. For example, the information processing apparatus 100 may not include the generation unit 402. In this case, it is preferable that the information processing apparatus 100 can communicate with another computer having, for example, the generation unit 402. The other computer is, for example, the elemental action recognition apparatus 201 or the like.

[0081] (Operation Example 1 of Information Processing Apparatus 100) Next, with reference to FIGS. 5 to 7, operation example 1 of the information processing apparatus 100 will be described.

[0082] FIGS. 5 to 7 are explanatory diagrams showing operation example 1 of the information processing apparatus 100. In FIG. 5, (5-1) the information processing apparatus 100 acquires a moving image 500. The information processing apparatus 100 has a DNN that enables recognition of a person, a skeleton, an object, or the like shown in the moving image 500. The information processing apparatus 100 uses the DNN to recognize a person, a skeleton, an object, or the like shown in the moving image 500.

[0083] (5-2) The information processing device 100 has component action recognition rules that enable recognition of component actions captured in the video 500 based on the output of the DNN. The information processing device 100 refers to the component action recognition rules and recognizes the component actions captured in the video 500 based on the results of recognizing people, skeletons, objects, or the like captured in the video 500. The information processing device 100 recognizes component actions such as "walking," "putting a hand forward," "looking at one's hands," or "bumping into someone," and identifies the actor who performed the recognized action and the time when the component action was performed.

[0084] (5-3) The information processing device 100 has a combination action recognition rule that enables the recognition of other element actions by combining element actions. The combination action recognition rule is, for example, rule 521. Rule 521 is, for example, a rule for recognizing an element action "walking while using a smartphone" based on a combination of an element action "walking", an element action "putting a hand forward", and an element action "looking at hands" that are performed by the same actor.

[0085] Specifically, rule 521 indicates sub-rule 1, sub-rule 2, and behavior rule 1 as conditions for determining the presence of the element behavior "walking while using smartphone." Sub-rule 1 indicates, for example, that the same person performs the element behavior [walking] and the element behavior [putting hand forward] at the same time. Sub-rule 2 indicates, for example, that the same person performs the element behavior [walking] and the element behavior [looking at hands] at the same time. Behavior rule 1 indicates, for example, that sub-rule 1 and sub-rule 2 are satisfied at the same time for the same person.

[0086] The information processing device 100 refers to the combination behavior recognition rule and recognizes a new component behavior based on the recognized component behavior. For example, the information processing device 100 recognizes an component behavior such as "walking while using a smartphone" and identifies the actor who performed the recognized behavior and the time when the component behavior was performed.

[0087] As a result, the information processing apparatus 100 can recognize the element action group indicated by the code 510. The information processing apparatus 100 stores the result of recognizing the element action in a graph format. The information processing apparatus 100 generates and stores relational data representing a graph 520 that shows the relationship between the recognized element action, the element action recognition rule, and the combined action recognition rule. The relational data is, for example, graph data.

[0088] (5-4) The information processing apparatus 100 has a target action recognition rule that enables the recognition of a target action. The target action recognition rule is, for example, rule 522 or the like. Rule 522 is, for example, a rule for recognizing the target action "walking while using a smartphone and colliding with a person" based on the combination of the element actions "colliding with a person" and "walking while using a smartphone" by the same actor within a valid time interval between the element actions.

[0089] Specifically, rule 522 indicates sub-rule 3 and action rule 2 as the conditions for recognizing the existence of the element action "walking while using a smartphone and colliding with a person". Sub-rule 3 indicates, for example, that different persons have simultaneously performed the element action [colliding with a person]. Action rule 2 indicates, for example, that for the same person, action rule 1 and sub-rule 3 are satisfied, and the time interval between the element actions is within the valid time.

[0090] The information processing apparatus 100 recognizes a target action such as "walking while using a smartphone and colliding with a person", and identifies the actor who performed the target action, the time when the target action was performed, and so on. The information processing apparatus 100 outputs the result 501 of recognizing the element action and the target action.

[0091] As a result, the information processing apparatus 100 can recognize the target action. The information processing apparatus 100 stores the result of recognizing the target action in a graph format. The information processing apparatus 100 updates the relational data representing the graph 520 so as to show the relationship between the recognized element action and the target action recognition rule. Next, we will move on to the description of FIG. 6 and explain a specific example in which the information processing apparatus 100 recognizes a target action.

[0092] In FIG. 6, it is assumed that the target behavior is formed by a combination of element behavior 1, element behavior 2, and element behavior 3 in which the time interval between each element behavior is within the effective time. Similar to FIG. 5, it is assumed that the information processing apparatus 100 generates and stores graph data representing a graph structure showing the relationship between a plurality of element behaviors in the target period as shown by reference numeral 600 as a result of recognizing the element behaviors. The plurality of element behaviors include action 1-i that becomes element behavior 1, action 2-j that becomes element behavior 2, and action 3-k that becomes element behavior 3. i is a positive integer. j is a positive integer. k is a positive integer.

[0093] The information processing apparatus 100 divides the target period according to the effective time and sets a plurality of divided sections. For example, the information processing apparatus 100 sets each part obtained by dividing the target period in a time unit longer than the effective time as a divided section. The information processing apparatus 100 extracts a plurality of partial data representing divided graph structures corresponding to different divided sections from the graph structure represented by the relational data. In the example of FIG. 6, the information processing apparatus 100 extracts partial data representing divided graph structure 1 including action 1-1, action 2-1, action 3-1, and action 1-2. In the example of FIG. 6, the information processing apparatus 100 extracts partial data representing divided graph structure 2 including action 3-1, action 1-2, action 2-2, and action 3-2.

[0094] The information processing apparatus 100 recognizes the target behavior for each divided section based on the partial data corresponding to the divided section. In the example of FIG. 6, the information processing apparatus 100 recognizes the target behavior formed by the combination of action 1-1, action 2-1, and action 3-1 based on the partial data representing divided graph structure 1. The information processing apparatus 100 recognizes the target behavior formed by the combination of action 1-2, action 2-2, and action 3-2 based on the partial data representing divided graph structure 2. Thus, the information processing apparatus 100 can recognize the target behavior for each partial data. Next, the description will move to FIG. 7.

[0095] As shown in FIG. 7, for a plurality of element actions during a target period, a graph structure 701 showing the relationships between the element actions becomes relatively large in scale. Therefore, in the prior art, when attempting to recognize various target actions based on the graph structure 701, it is necessary to repeatedly inspect the entire graph structure 701, which is likely to lead to an increase in the processing load. For example, in the prior art, when attempting to recognize various target actions based on the graph structure 701, it is necessary to inspect a total of 8 combination patterns formed by combining 2 element actions 1, 2 element actions 2, and 2 element actions 3 respectively.

[0096] On the other hand, a graph structure 711 obtained by dividing the graph structure 701 becomes relatively small in scale. The graph structure 711 corresponds to, for example, the divided graph structure 1 shown in FIG. 6. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 711, it can suppress an increase in the processing load. For example, the information processing apparatus 100 inspects a total of 2 combination patterns formed by combining 2 element actions 1, 1 element action 2, and 1 element action 3 respectively.

[0097] Similarly, a graph structure 712 obtained by dividing the graph structure 701 becomes relatively small in scale. The graph structure 712 corresponds to, for example, the divided graph structure 2 shown in FIG. 6. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 712, it can suppress an increase in the processing load. For example, the information processing apparatus 100 inspects a total of 2 combination patterns formed by combining 1 element action 1, 1 element action 2, and 2 element actions 3 respectively.

[0098] In this way, the information processing apparatus 100 can reduce the processing time and processing load required for recognizing the target behavior. For example, the information processing apparatus 100 can suppress the number of combination patterns to be inspected to 4 compared with the prior art. Specifically, if the moving image is relatively long and the graph structure 701 is 100 times larger, the number of combination patterns to be inspected can be suppressed from about 8 million to about 400.

[0099] More specifically, in the prior art, when the graph structure 701 becomes 100 times larger, a total of 8 million combination patterns formed by combining 200 element actions 1, 200 element actions 2, and 200 element actions 3 respectively need to be inspected. On the other hand, even when the graph structure 701 becomes 100 times larger, the information processing apparatus 100 can divide the target period into 200, recognize the target behavior from the graph structure corresponding to each divided section, and only needs to inspect about 400 combination patterns, and can obtain inspection results equivalent to those of the prior art. Therefore, the information processing apparatus 100 can suppress the number of combination patterns to be inspected to about 1 / 20,000, and can reduce the processing time and processing load required for recognizing the target behavior.

[0100] (Generation processing procedure in operation example 1) Next, an example of the generation processing procedure in operation example 1 executed by the information processing apparatus 100 will be described with reference to FIG. 8. The generation processing is realized, for example, by the processor 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0101] FIG. 8 is a flowchart showing an example of the generation processing procedure in operation example 1. In FIG. 8, the information processing apparatus 100 reads a moving image in the target period (step S801).

[0102] Next, the information processing apparatus 100 uses a DNN to recognize a person, skeleton, object, etc. shown in a moving image (step S802). Then, the information processing apparatus 100 refers to the elemental action recognition rules and recognizes the elemental actions during the target period based on the result of recognizing a person, skeleton, object, etc. (step S803).

[0103] Next, the information processing apparatus 100 refers to the combined action recognition rules and recognizes other elemental actions obtained by combining the recognized elemental actions (step S804). Then, the information processing apparatus 100 associates the recognized elemental actions with the time of the elemental actions, generates graph data indicating the relationship between the recognized elemental actions, and stores it (step S805). After that, the information processing apparatus 100 ends the generation process.

[0104] (Recognition processing procedure in operation example 1) Next, with reference to FIG. 9, an example of the recognition processing procedure in operation example 1 executed by the information processing apparatus 100 will be described. The recognition processing in operation example 1 is realized by, for example, the processor 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0105] FIG. 9 is a flowchart showing an example of the recognition processing procedure in operation example 1. In FIG. 9, the information processing apparatus 100 reads graph data indicating the relationship between elemental actions (step S901).

[0106] Next, the information processing apparatus 100 refers to the recognition rules that enable recognition of any target action and acquires the valid time set for the target action (step S902). Then, the information processing apparatus 100 divides the graph data along the time axis based on the valid time set for the target action and generates a plurality of partial graph data (step S903).

[0107] Next, the information processing apparatus 100 refers to the target action recognition rule that enables recognition of any target action, and recognizes the target action for each piece of sub-graph data (step S904). Then, the information processing apparatus 100 generates integrated data obtained by integrating the results of recognizing the target action this time (step S905).

[0108] Next, the information processing apparatus 100 determines whether to end the recognition process (step S906). For example, when the information processing apparatus 100 has finished recognizing each of a plurality of preset target actions, it determines to end the recognition process. Here, when the recognition process is not ended (step S906: No), the information processing apparatus 100 proceeds to the process of step S907. On the other hand, when the recognition process is ended (step S906: Yes), the information processing apparatus 100 proceeds to the process of step S908.

[0109] In step S907, the information processing apparatus 100 changes the target action recognition rule to be referred to so as to refer to the target action recognition rule that enables recognition of other target actions (step S907). Then, the information processing apparatus 100 returns to the process of step S902.

[0110] In step S908, the information processing apparatus 100 stores the integrated data (step S908). Then, the information processing apparatus 100 ends the recognition process. Thereby, the information processing apparatus 100 can make it easier to recognize the target action.

[0111] (Operation Example 2 of Information Processing Apparatus 100) Next, with reference to FIGS. 10 and 11, operation example 2 of the information processing apparatus 100 will be described. Operation example 1 corresponds to the case where the information processing apparatus 100 sets a plurality of divided sections without overlapping the divided sections with each other. In contrast, operation example 2 corresponds to the case where the information processing apparatus 100 sets a plurality of divided sections with overlapping divided sections.

[0112] FIG. 10 and FIG. 11 are explanatory diagrams showing Operation Example 2 of the information processing apparatus 100. In FIG. 10, it is assumed that the target action is formed by a combination of Element Action 1 and Element Action 2 in which the time interval between respective element actions is within the valid time.

[0113] Similar to FIG. 5, it is assumed that the information processing apparatus 100 generates and stores graph data representing a graph structure showing the relationship between a plurality of element actions in the target period, as indicated by reference numeral 1000, as a result of recognizing the element actions. The plurality of element actions include, for example, Action 1-i that becomes Element Action 1 and Action 2-j that becomes Element Action 2. i is a positive integer. j is a positive integer.

[0114] The information processing apparatus 100 divides the target period according to the valid time and sets a plurality of divided sections. Here, unlike Operation Example 1, the information processing apparatus 100 sets a plurality of divided sections with the divided sections overlapping each other. For example, the information processing apparatus 100 sets each part obtained by dividing the target period so that the divided sections overlap by an overlap time or more in a time unit longer than twice the valid time. The overlap time is set to, for example, a time longer than the valid time.

[0115] The information processing apparatus 100 extracts a plurality of partial data representing divided graph structures corresponding to different divided sections from the graph structure represented by the relational data. In the example of FIG. 10, the information processing apparatus 100 extracts partial data representing a divided graph structure 1 including Action 1-1, Action 2-1, Action 1-2, and Action 1-3. In the example of FIG. 10, the information processing apparatus 100 extracts partial data representing a divided graph structure 2 including Action 2-1, Action 1-2, Action 1-3, and Action 2-2. In the example of FIG. 10, the information processing apparatus 100 extracts partial data representing a divided graph structure 3 including Action 2-2 and Action 2-3.

[0116] The information processing apparatus 100 recognizes a target action for each divided section based on partial data corresponding to the divided section. In the example of FIG. 10, the information processing apparatus 100 recognizes a target action formed by a combination of Action 1-1 and Action 2-1 based on the partial data representing the divided graph structure 1. The information processing apparatus 100 recognizes a target action formed by a combination of Action 1-2 and Action 2-2 based on the partial data representing the divided graph structure 2. The information processing apparatus 100 determines that no target action exists based on the partial data representing the divided graph structure 3.

[0117] Thereby, the information processing apparatus 100 can dispense with inspecting the entire target period by using the partial data representing the divided graph structure instead of the relational data. For this reason, the information processing apparatus 100 can dispense with inspecting a combination of Action 1-3 and Action 2-3 whose valid time has expired. The information processing apparatus 100 can inspect only combinations of elemental actions with a relatively high probability of not having exceeded the valid time. As a result, the information processing apparatus 100 can reduce the processing time and processing load required for recognizing the target action. Further, since the information processing apparatus 100 can make the divided section and the overlap time longer than the valid time respectively, it is possible to reduce the probability of failing to recognize the target action.

[0118] The information processing apparatus 100 can recognize a target action for each piece of partial data. The information processing apparatus 100 can overlap the divided sections. For this reason, the information processing apparatus 100 can facilitate recognizing a target action formed by a combination of two or more elemental actions existing across the head or tail of any of the divided sections, such as a combination of Action 1-2 and Action 2-2. As a result, the information processing apparatus 100 can improve the accuracy of recognizing the target action. Next, the description will proceed to FIG. 11.

[0119] As shown in FIG. 11, for a plurality of element actions during the target period, the graph structure 1101 showing the relationship between the element actions becomes relatively large in scale. Therefore, in the prior art, when trying to recognize various target actions based on the graph structure 1101, the entire graph structure 1101 needs to be repeatedly inspected, which is likely to cause an increase in the processing burden.

[0120] On the other hand, the graph structure 1111 obtained by dividing the graph structure 1101 becomes relatively small in scale. The graph structure 1111 corresponds to, for example, the divided graph structure 1 shown in FIG. 10. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 1111, it can suppress the increase in the processing burden.

[0121] Similarly, the graph structure 1112 obtained by dividing the graph structure 1101 becomes relatively small in scale. The graph structure 1112 corresponds to, for example, the divided graph structure 2 shown in FIG. 10. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 1112, it can suppress the increase in the processing burden.

[0122] Similarly, the graph structure 1113 obtained by dividing the graph structure 1101 becomes relatively small in scale. The graph structure 1113 corresponds to, for example, the divided graph structure 3 shown in FIG. 10. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 1113, it can suppress the increase in the processing burden. In this way, the information processing apparatus 100 can reduce the processing time and processing burden required for recognizing the target actions.

[0123] (Generation processing procedure in operation example 2) An example of the generation processing procedure in operation example 2 executed by the information processing apparatus 100 is specifically the same as an example of the generation processing procedure in operation example 1 shown in FIG. 8, so the description is omitted.

[0124] (Recognition processing procedure in operation example 2) Next, with reference to FIG. 12, an example of the recognition processing procedure in Operation Example 2 executed by the information processing apparatus 100 will be described. The recognition processing in Operation Example 2 is realized by, for example, the processor 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0125] FIG. 12 is a flowchart showing an example of the recognition processing procedure in Operation Example 2. In FIG. 12, the information processing apparatus 100 reads graph data indicating the relationship between element actions (step S1201).

[0126] Next, the information processing apparatus 100 refers to a recognition rule that enables recognition of any target action, and acquires the effective time set for the target action (step S1202). Then, the information processing apparatus 100 sets an overlap time Po that exceeds the effective time (step S1203).

[0127] Next, the information processing apparatus 100 divides the graph data along the time axis based on the effective time set for the target action and the overlap time Po, and generates a plurality of partial graph data (step S1204). Then, the information processing apparatus 100 refers to a target action recognition rule that enables recognition of any target action, and recognizes the target action for each partial graph data (step S1205).

[0128] Next, the information processing apparatus 100 generates integrated data obtained by integrating the results of recognizing the target action (step S1206). Then, the information processing apparatus 100 determines whether to end the recognition processing (step S1207). The information processing apparatus 100 determines to end the recognition processing, for example, when it has finished recognizing each of a plurality of preset target actions. Here, if the recognition processing is not ended (step S1207: No), the information processing apparatus 100 proceeds to the processing of step S1208. On the other hand, if the recognition processing is ended (step S1207: Yes), the information processing apparatus 100 proceeds to the processing of step S1209.

[0129] In step S1208, the information processing apparatus 100 changes the target action recognition rule to be referred to so as to refer to the target action recognition rule that enables recognition of other target actions (step S1208). Then, the information processing apparatus 100 returns to the process of step S1202.

[0130] In step S1209, the information processing apparatus 100 stores the integrated data (step S1209). Then, the information processing apparatus 100 ends the recognition process. Thereby, the information processing apparatus 100 can accurately recognize the target action.

[0131] (Operation example 3 of the information processing apparatus 100) Next, with reference to FIGS. 13 and 14, operation example 3 of the information processing apparatus 100 will be described. Operation example 1 corresponds to the case where the information processing apparatus 100 does not consider combinations of three or more elemental actions that straddle the head or tail of a divided section. In contrast, operation example 3 corresponds to the case where the information processing apparatus 100 considers combinations of three or more elemental actions that straddle the head or tail of a divided section.

[0132] FIGS. 13 and 14 are explanatory diagrams showing operation example 3 of the information processing apparatus 100. In FIG. 13, it is assumed that the target action is formed by a combination of elemental action 1, elemental action 2, and elemental action 3 in which the time interval between each elemental action is within the valid time.

[0133] Similar to FIG. 5, it is assumed that as a result of recognizing the elemental actions, the information processing apparatus 100 generates and stores graph data representing a graph structure showing the relationship between the plurality of elemental actions in the target period as shown by reference numeral 1300. The plurality of elemental actions include action 1-i that becomes elemental action 1, action 2-j that becomes elemental action 2, and action 3-k that becomes elemental action 3. i is a positive integer. j is a positive integer. k is a positive integer.

[0134] The information processing apparatus 100 divides a target period according to the effective time and sets a plurality of divided sections. Here, unlike in Operation Example 1, the information processing apparatus 100 sets a plurality of divided sections with the divided sections overlapping each other. For example, the information processing apparatus 100 sets each of the divided sections by dividing the target period such that the divided sections overlap by an overlap time or more in a time unit longer than twice the effective time. The overlap time is set to a time longer than the effective time, for example.

[0135] The information processing apparatus 100 extracts a plurality of sub-data representing divided graph structures corresponding to different divided sections from among the graph structures represented by the relational data. In the example of FIG. 13, the information processing apparatus 100 extracts sub-data representing a divided graph structure 1 including Action 1-1, Action 2-1, Action 1-2, and Action 3-1. In the example of FIG. 13, the information processing apparatus 100 extracts sub-data representing a divided graph structure 2 including Action 2-1, Action 1-2, Action 3-1, and Action 2-2. In the example of FIG. 13, the information processing apparatus 100 extracts sub-data representing a divided graph structure 3 including Action 3-1, Action 2-2, and Action 3-2.

[0136] The information processing apparatus 100 recognizes a target action in order from the first divided section based on the sub-data corresponding to the divided section. In the example of FIG. 13, the information processing apparatus 100 recognizes a target action formed by the combination of Action 1-1, Action 2-1, and Action 3-1 based on the sub-data representing the divided graph structure 1.

[0137] The information processing apparatus 100 recognizes that no target action exists in the divided section corresponding to the divided graph structure 2 based on the sub-data representing the divided graph structure 2. The information processing apparatus 100 detects the combination of Action 1-2 and Action 2-2 that forms the first half of the target action in the divided section corresponding to the divided graph structure 2 based on the sub-data representing the divided graph structure 2. In this case, the information processing apparatus 100 will use the combination of Action 1-2 and Action 2-2 that forms the first half of the detected target action when recognizing the target action in the divided section corresponding to the divided graph structure 3.

[0138] The information processing apparatus 100 recognizes a target behavior based on a combination of action 1-2 and action 2-2 that form the first half of the detected target behavior, and partial data representing the divided graph structure 3. For example, the information processing apparatus 100 recognizes a target behavior formed by a combination of action 1-2, action 2-2, and action 3-2 that exists across a plurality of divided sections.

[0139] As a result, the information processing apparatus 100 can dispense with inspecting the entire target period by using partial data representing the divided graph structure instead of relational data. As a result, the information processing apparatus 100 can reduce the processing time and processing load required for recognizing the target behavior.

[0140] When one or more element actions that form the first half of the target behavior exist in any of the divided sections, the information processing apparatus 100 can use the one or more element actions in subsequent divided sections when recognizing the target behavior. For this reason, the information processing apparatus 100 can make it easier to recognize a target behavior formed by a combination of two or more element actions that exist across the head or tail of any of the divided sections. As a result, the information processing apparatus 100 can improve the accuracy of recognizing the target behavior. Next, we will move on to the description of FIG. 14.

[0141] As shown in FIG. 14, the graph structure 1401 showing the relationship between element actions for a plurality of element actions in the target period becomes relatively large in scale. For this reason, in the prior art, when trying to recognize various target behaviors based on the graph structure 1401, the entire graph structure 1401 is repeatedly inspected, which tends to increase the processing load.

[0142] On the other hand, the graph structure 1411 obtained by dividing the graph structure 1401 becomes relatively small in scale. The graph structure 1411 corresponds to, for example, the divided graph structure 1 shown in FIG. 13. For this reason, when the information processing apparatus 100 recognizes various target behaviors based on the graph structure 1411, it can suppress an increase in the processing load.

[0143] Similarly, the graph structure 1412 obtained by dividing the graph structure 1401 has a relatively small scale. The graph structure 1412 corresponds to, for example, the divided graph structure 2 shown in FIG. 13. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 1412, it can suppress an increase in the processing load.

[0144] Similarly, the graph structure 1413 obtained by dividing the graph structure 1401 has a relatively small scale. The graph structure 1413 corresponds to, for example, the divided graph structure 3 shown in FIG. 13 with the element action 1-2 forming the first half of the target action added thereto. Therefore, when the information processing apparatus 100 recognizes various target actions based on the graph structure 1413, it can suppress an increase in the processing load. Thus, the information processing apparatus 100 can reduce the processing time and processing load required for recognizing the target action.

[0145] (Recognition processing procedure in operation example 3) Next, with reference to FIG. 15, an example of the recognition processing procedure in operation example 3 executed by the information processing apparatus 100 will be described. The recognition processing in operation example 3 is realized by, for example, the processor 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0146] FIG. 15 is a flowchart showing an example of the recognition processing procedure in operation example 3. In FIG. 15, the information processing apparatus 100 reads graph data indicating the relationship between element actions (step S1501).

[0147] Next, the information processing apparatus 100 refers to a recognition rule that enables recognition of any target action, and acquires the valid time set for the target action (step S1502). Then, the information processing apparatus 100 sets an overlap time Po that exceeds the valid time (step S1503).

[0148] Next, the information processing apparatus 100 divides the graph data along the time axis based on the effective time set for the target action and the overlap time Po, and generates a plurality of partial graph data (step S1504). Then, the information processing apparatus 100 refers to the target action recognition rule that enables recognition of any target action, and performs the detailed processing described later with reference to FIG. 16, thereby recognizing the target action for each partial graph data (step S1505).

[0149] Next, the information processing apparatus 100 generates integrated data obtained by integrating the results of recognizing the target actions (step S1506). Then, the information processing apparatus 100 determines whether to end the recognition process (step S1507). For example, when the information processing apparatus 100 has finished recognizing each of the plurality of preset target actions, it determines to end the recognition process. Here, if the recognition process is not ended (step S1507: No), the information processing apparatus 100 proceeds to the process of step S1508. On the other hand, when the recognition process is ended (step S1507: Yes), the information processing apparatus 100 proceeds to the process of step S1509.

[0150] In step S1508, the information processing apparatus 100 changes the target action recognition rule to be referred to so as to refer to the target action recognition rule that enables recognition of other target actions (step S1508). Then, the information processing apparatus 100 returns to the process of step S1502.

[0151] In step S1509, the information processing apparatus 100 stores the integrated data (step S1509). Then, the information processing apparatus 100 ends the recognition process. Thereby, the information processing apparatus 100 can accurately recognize the target action.

[0152] (Detailed processing procedure in operation example 3) Next, with reference to FIG. 16, an example of the detailed processing procedure in operation example 3 executed by the information processing apparatus 100 will be described. The detailed processing in operation example 3 is realized by, for example, the processor 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0153] FIG. 16 is a flowchart showing an example of the detailed processing procedure in operation example 3. In FIG. 16, the information processing apparatus 100 assigns indexes to each partial graph data in chronological order (step S1601). The indexes are, for example, 1, 2, ··· N - 1, N.

[0154] Next, the information processing apparatus 100 sets i = 1 (step S1602). Then, the information processing apparatus 100 recognizes the target action by searching for combinations of a plurality of elemental actions that form the target action in the i-th partial graph data with reference to the target action recognition rule that enables recognition of any target action (step S1603).

[0155] Next, the information processing apparatus 100 determines whether or not it has searched for combinations of a plurality of elemental actions that form the target action in the last partial graph data (step S1604). Here, when searching for combinations of a plurality of elemental actions (step S1604: Yes), the information processing apparatus 100 ends the detailed processing. On the other hand, when not searching for combinations of a plurality of elemental actions (step S1604: No), the information processing apparatus 100 proceeds to the processing of step S1605.

[0156] In step S1605, the information processing apparatus 100 determines whether a combination of a plurality of elemental actions that form the target action can be established across the i-th partial graph data to the (i + 1)-th partial graph data (step S1605). Here, if a combination of a plurality of elemental actions cannot be established (step S1605: No), the information processing apparatus 100 returns to the process of step S1603. On the other hand, if a combination of a plurality of elemental actions can be established (step S1605: Yes), the information processing apparatus 100 proceeds to the process of step S1606.

[0157] In step S1606, the information processing apparatus 100 adds the first half of the elemental actions included in the i-th partial graph data to the (i + 1)-th partial graph data among the combinations of a plurality of elemental actions that can be established (step S1606). Next, the information processing apparatus 100 sets i = i + 1 (step S1607). Then, the information processing apparatus 100 returns to the process of step S1603. Thereby, the information processing apparatus 100 can improve the accuracy of recognizing the target action.

[0158] As described above, according to the information processing apparatus 100, data indicating the relationship between elemental actions can be acquired for a plurality of elemental actions during the target period. According to the information processing apparatus 100, the effective time corresponding to the target action can be acquired. According to the information processing apparatus 100, based on the acquired data, for each divided section set by dividing the target period according to the acquired effective time, among a plurality of elemental actions, a combination of two or more elemental actions that form the target action can be searched. Thereby, the information processing apparatus 100 can make it easier to recognize the target action. The information processing apparatus 100 can, for example, reduce the processing time and processing load required for recognizing the target action.

[0159] According to the information processing apparatus 100, among a plurality of elemental actions, it is possible to search for a combination of two or more elemental actions that form a target action, and in which the time interval between at least any two of the elemental actions in the combination is equal to or less than the acquired effective time. Thereby, the information processing apparatus 100 can recognize the target action. The information processing apparatus 100 can recognize a relatively complex target action based on the effective time.

[0160] According to the information processing apparatus 100, using a predetermined model, based on the result of recognizing the things shown in the moving image during the target period, it is possible to detect the elemental actions related to the things. According to the information processing apparatus 100, for each detected elemental action, it is possible to generate data including the time when the detected elemental action was performed in association therewith. Thereby, the information processing apparatus 100 can use the elemental actions that can be detected by the predetermined model when recognizing the target action. The information processing apparatus 100 can acquire data without collaborating with other computers.

[0161] According to the information processing apparatus 100, furthermore, it is possible to detect other elemental actions obtained by combining the detected elemental actions. Thereby, the information processing apparatus 100 can use other elemental actions obtained by combining two or more elemental actions when recognizing the target action. The information processing apparatus 100 can acquire data without collaborating with other computers.

[0162] According to the information processing apparatus 100, it is possible to set a plurality of divided sections by dividing the target period so that the divided sections overlap by at least the acquired effective time or more. Thereby, the information processing apparatus 100 can make it difficult to overlook a combination of two or more elemental actions that form a target action and that exists across the head or tail of any of the divided sections.

[0163] According to the information processing apparatus 100, it is possible to determine whether or not there are some of the elemental actions included in the combination of two or more elemental actions that form the target action in the first divided section among the divided sections set by dividing the target period. According to the information processing apparatus 100, if there are some elemental actions, it is possible to search for the remaining elemental actions included in the combination in the second divided section after the first divided section. Thereby, the information processing apparatus 100 can reduce the likelihood of overlooking a combination of two or more elemental actions that form the target action and that exists across the head or the end of any of the divided sections.

[0164] According to the information processing apparatus 100, the elemental actions can adopt actions of types that can be detected using a predetermined model. According to the information processing apparatus 100, the target action can adopt actions of types that cannot be detected using a predetermined model. Thereby, the information processing apparatus 100 can dispense with learning a model capable of detecting the target action. For this reason, the information processing apparatus 100 can reduce the processing time and processing load involved in recognizing the target action.

[0165] According to the information processing apparatus 100, the searched result can be output. Thereby, the information processing apparatus 100 can make the searched result available.

[0166] According to the information processing apparatus 100, a predetermined model that enables recognition of the skeletal position of a person can be used. According to the information processing apparatus 100, on the basis of the result of recognizing the skeletal position of the person shown in the moving image using the predetermined model and based on the moving image, the elemental actions related to the person are detected. Thereby, the information processing apparatus 100 can accurately recognize the elemental actions related to the person.

[0167] According to the information processing apparatus 100, data indicating a graph structure formed by nodes that associate an element action with the time when the element action is performed, and edges that indicate an inclusion relationship between an element action and other element actions obtained by combining two or more element actions can be generated. Thereby, the information processing apparatus 100 can acquire data without collaborating with other computers.

[0168] Note that the information processing method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a PC or a workstation. The information processing program described in this embodiment is recorded on a computer-readable recording medium and is executed by being read from the recording medium by a computer. The recording medium is a hard disk, a flexible disk, a CD (Compact Disc)-ROM, an MO (Magneto Optical disc), a DVD (Digital Versatile Disc), or the like. Further, the information processing program described in this embodiment may be distributed via a network such as the Internet.

[0169] Regarding the above-described embodiment, the following additional remarks are disclosed.

[0170] (Supplementary Note 1) Obtaining data indicating the relationship between a plurality of element actions during a target period, obtaining the effective time corresponding to the target action, Based on the obtained data, for each divided section set by dividing the target period according to the obtained effective time, searching for a combination of two or more element actions that form the target action among the plurality of element actions, An information processing program characterized by causing a computer to execute the processing.

[0171] (Supplementary Note 2) The searching process is Among the plurality of element actions, search for a combination of two or more element actions that form the target action, wherein the time interval between at least any two of the element actions in the combination is equal to or less than the acquired effective time. The information processing program according to appended note 1, characterized by this.

[0172] (Appended note 3) Using a predetermined model, based on the result of recognizing the things shown in the moving image during the target period, detect the element actions related to the things. Cause the computer to execute the processing. The processing for acquiring the data is as follows. For each of the detected element actions, generate data including the time when the detected element action was performed in association therewith. The information processing program according to appended note 1 or 2, characterized by this.

[0173] (Appended note 4) The processing for detecting is as follows. Furthermore, detect other element actions formed by combining the detected element actions. The information processing program according to appended note 3, characterized by this.

[0174] (Appended note 5) Set a plurality of divided intervals by dividing the target period so that the divided intervals overlap at least by the acquired effective time or more. Cause the computer to execute the processing. The information processing program according to any one of appended notes 1 to 4, characterized by this.

[0175] (Appended note 6) The processing for searching is as follows. Among the divided intervals set by dividing the target period, if there are some element actions included in a combination of two or more element actions that form the target action in the first divided interval, then in the second divided interval after the first divided interval, search for the remaining element actions included in the combination. The information processing program according to any one of appended notes 1 to 5, characterized by this.

[0176] (Appended note 7) The element actions are actions of types that can be detected using a predetermined model. The information processing program according to any one of Appendices 1 to 6, wherein the target action is an action of a type that cannot be detected using the predetermined model.

[0177] (Appendix 8) Outputting the search result The information processing program according to any one of Appendices 1 to 7, characterized in that the computer is caused to execute the processing.

[0178] (Appendix 9) The predetermined model is a model that enables recognition of the skeletal positions of a person. The processing of detecting The information processing program according to Appendix 3 or 4, characterized in that, using the predetermined model, based on the moving image, based on the result of recognizing the skeletal positions of the persons shown in the moving image, the elemental actions regarding the persons are detected.

[0179] (Appendix 10) The processing of acquiring the data The information processing program according to Appendix 4, characterized in that data indicating a graph structure formed by nodes showing the detected elemental actions in association with the time when the detected elemental actions were performed, and edges showing the inclusion relationship between the detected elemental actions and other elemental actions obtained by combining two or more detected elemental actions is generated.

[0180] (Appendix 11) Acquiring data indicating the relationship between elemental actions for a plurality of elemental actions during a target period, Acquiring the effective time corresponding to the target action, Based on the acquired data, for each divided section set by dividing the target period according to the acquired effective time, searching for combinations of two or more elemental actions that form the target action among the plurality of elemental actions. An information processing method characterized in that a computer executes the processing.

[0181] (Appendix 12) Acquiring data indicating the relationship between elemental actions for a plurality of elemental actions during a target period, Acquiring the effective time corresponding to the target action, Based on the obtained data, for each divided section set by dividing the target period according to the obtained effective time, search for combinations of two or more elemental actions that form the target action among the plurality of elemental actions. An information processing apparatus characterized by having a control unit.

Explanation of Signs

[0182] 100 Information processing apparatus 110 Data 200 Information processing system 201 Elemental action recognition apparatus 202 Client apparatus 210 Network 300 Bus 301 Processor 302 Memory 303 Network I / F 304 Recording medium I / F 305 Recording medium 306 Camera apparatus 400 Storage unit 401 Acquisition unit 402 Generation unit 403 Search unit 404 Output unit 500 Moving image 501 Result 510, 600, 1000, 1300 Signs 520 Graph 521, 522 Rules 701, 711, 712, 1101, 1111, 1112, 1113, 1401, 1411, 1412, 1413 Graph structures

Claims

1. Obtain data indicating the relationship between a plurality of element actions during a target period, Obtain the effective time corresponding to the target action, Based on the obtained data, for each divided section set by dividing the target period according to the obtained effective time, search for combinations of two or more element actions that form the target action among the plurality of element actions. An information processing program characterized by causing a computer to execute the processing.

2. The searching process is Among the plurality of element actions, search for combinations of two or more element actions that form the target action, and in which the time interval between at least any two of the element actions in the combination is equal to or less than the obtained effective time. The information processing program according to claim 1, characterized by this.

3. Using a predetermined model, based on the result of recognizing the things shown in the moving image during the target period, detect the element actions related to the things. Cause the computer to execute the processing, The process of obtaining the data is For each detected element action, generate data including the time when the detected element action was performed in association therewith. The information processing program according to claim 1 or 2, characterized by this.

4. The detecting process is Furthermore, detect other element actions formed by combining the detected element actions. The information processing program according to claim 3, characterized by this.

5. Cause the computer to execute the process of setting a plurality of divided sections by dividing the target period so that the divided sections overlap by at least the obtained effective time or more. The information processing program according to any one of claims 1 to 4, characterized by this.

6. The searching process is Among the divided sections set by dividing the target period, if there are some element actions included in the combination of two or more element actions that form the target action in the first divided section, then in the second divided section after the first divided section, search for the remaining element actions included in the combination. The information processing program according to any one of claims 1 to 5, characterized by this.

7. The element actions are actions of a type that can be detected using a predetermined model, The target action is an action of a type that cannot be detected using the predetermined model. The information processing program according to any one of claims 1 to 6, characterized by this.

8. Obtain data indicating the relationship between element actions for a plurality of element actions during a target period, Obtain the effective time corresponding to the target action, Based on the obtained data, for each divided section set by dividing the target period according to the obtained effective time, search for combinations of two or more element actions that form the target action among the plurality of element actions, An information processing method characterized in that a computer executes the processing.

9. Obtain data indicating the relationship between element actions for a plurality of element actions during a target period, Obtain the effective time corresponding to the target action, Based on the obtained data, for each divided section set by dividing the target period according to the obtained effective time, search for combinations of two or more element actions that form the target action among the plurality of element actions, An information processing apparatus characterized by having a control unit.

Images