Information processing device, output method, and output program
The information processing device addresses the challenge of creating recognition models for human behaviors by detecting basic motions and evaluating similarity to identify candidate rules, reducing the need for extensive video data and manual effort.
Patent Information
- Application Number
- JP2021001876
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-01-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-01-08
AI Technical Summary
Generating recognition models for human behaviors using deep learning requires large amounts of video data, which can be time-consuming and costly, and it is challenging to create rules for recognizing complex behaviors without extensive know-how and experience.
An information processing device that detects basic motions from videos, evaluates similarity using time series information, and identifies candidate rules from existing rule databases to recognize similar actions without needing extensive video data or manual effort.
Efficiently identifies rules that recognize actions similar to those shown in videos, reducing the need for large datasets and manual rule creation, thereby lowering costs and time requirements.
Smart Images

Figure 0007589553000001 
Figure 0007589553000002 
Figure 0007589553000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an output method, and an output program. [Background technology]
[0002] In recent years, the demand for analyzing video data using artificial intelligence (AI) has been rapidly increasing in various business fields. For example, human behavior is recognized from videos using recognition models obtained by machine learning such as deep learning.
[0003] In this regard, techniques related to the analysis of images such as videos are known (for example, Patent Documents 1 to 3). Also, techniques related to similar video search using a video as input are known (for example, Non-Patent Document 1). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-176423 A [Patent Document 2] JP 2015-116308 A [Patent Document 3] JP 2005-228274 A [Non-patent literature]
[0005] [Non-Patent Document 1] Shogo Kimura et al., "Construction of a Similar Video Search System with a Function to Present Reasons for Similarity," 49th Programming Symposium, January 2008, pp. 97-106 Summary of the Invention [Problem to be solved by the invention]
[0006] For example, when generating a recognition model that detects the behavior of a recognition target using deep learning or the like, a large amount of video data for learning is required for each behavior to be recognized. In addition, for example, it may take a long time or be difficult to collect video data for learning, making it difficult to generate a recognition model that recognizes the behavior of the recognition target.
[0007] Incidentally, human behavior is generated from a combination of basic human actions such as walking, shaking the head, and reaching out. For this reason, it is possible to detect behavior by creating a recognition model that recognizes various basic human actions in advance and describing rules for recognizing complex human behavior such as suspicious behavior and purchasing behavior for combinations of the basic actions. By defining rules for combinations of basic actions in this way, it becomes possible to recognize the behavior of the recognition target using the rules without having to prepare a large number of videos in which the behavior of the recognition target is captured.
[0008] However, know-how and experience are required to generate rules by combining basic actions to recognize the target behavior. Furthermore, generating rules for each of the various target behaviors requires manpower and costs. Therefore, when creating new rules for behavior recognition, it is preferable to be able to reuse rules that have been created in the past.
[0009] In one aspect, the present invention aims to identify, from among existing rules, rules that recognize behaviors similar to those captured in a video. [Means for solving the problem]
[0010] An information processing device according to one embodiment of the present invention includes a detection unit that detects at least one basic action of a target from a video, and a plurality of rules that define patterns of the basic actions for detecting the target's behavior, In multiple rules that are classified into multiple groups according to the type of target behavioran evaluation unit that evaluates a similarity between a rule and a video for each of the multiple rules, the rule being associated with first time series information that represents in time series the detection timings of multiple basic actions included in a basic action pattern, based on the first time series information associated with the rule and second time series information that represents in time series the detection timings of at least one basic action detected from the video; A first candidate rule of the plurality of rules and a second candidate rule from a first group of the plurality of groups. Based on similarity The first candidate rule identified and , multiple group Of The second group from The second one was identified based on similarity. Candidate Rules and and an output unit that outputs the signal. Effect of the Invention
[0011] It is possible to identify rules from among existing rules that recognize behavior similar to that captured in a video. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram illustrating a behavior recognition system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a block configuration of an information processing device according to an embodiment. [Diagram 3] FIG. 11 is a diagram showing link relationships of rule information according to the embodiment. [Figure 4] FIG. 13 is a diagram illustrating an example of class definitions of basic action recognition results. [Diagram 5] FIG. 13 illustrates an example of a rule class definition. [Figure 6] FIG. 13 is a diagram illustrating an example of a class definition of a behavior detection period. [Figure 7] 13 is a diagram illustrating an example of a definition of a predicate in a graph structure of rule information. FIG. [Figure 8] 11 is a diagram illustrating an example of an operation flow of a candidate rule output process according to the embodiment. FIG. [Figure 9] 11 is a diagram illustrating an example of rules for detecting an action of picking up a product from a shelf and information associated with the rules. [Figure 10] FIG. 11 is a diagram illustrating an example of information related to input moving images. [Figure 11] FIG. 1 illustrates an exemplary application of dynamic time warping. [Figure 12] 13 is a diagram illustrating an example of a graph structure when a rule according to a modified example of an embodiment is applied to a plurality of moving images. [Figure 13] FIG. 11 is a diagram illustrating an example of an operational flow of a candidate rule output process according to a modified example of the embodiment. [Figure 14] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer for implementing the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, several embodiments of the present invention will be described in detail with reference to the drawings. Note that the same reference numerals are used to designate corresponding elements in the various drawings.
[0014] FIG. 1 is a diagram illustrating a behavior recognition system 100 according to an embodiment. The behavior recognition system 100 includes, for example, an information processing device 101 and a photographing device 102. The information processing device 101 may be, for example, a computer equipped with a function for processing moving images, such as a server computer, a personal computer (PC), a mobile PC, and a tablet terminal. The photographing device 102 is, for example, a device for photographing moving images, such as a camera. The information processing device 101 may be, for example, communicably connected to the photographing device 102, and receives moving image data photographed by the photographing device 102. In another embodiment, the information processing device 101 may receive moving image data photographed by the photographing device 102 from another device, or may be manufactured integrally with the photographing device 102.
[0015] When the information processing device 101 receives video data, it analyzes the received video and detects a recognition target ((1) in FIG. 1). In one example, the information processing device 101 may detect people from the video. In the example of FIG. 1, the information processing device 101 detects two people, recognition targets 1 and 2, from the video.
[0016] Next, the information processing device 101 recognizes basic actions from the recognition target appearing in the video ((2) in FIG. 1). The basic actions may be, for example, basic actions taken by the target, and in one example, may include the movements of each part of the target's body divided into parts for each joint. The basic actions may also include actions that the target often performs in various situations, such as walking, running, throwing, grabbing, kicking, jumping, and eating. For example, the information processing device 101 may detect multiple basic actions from the video using a recognition model that detects various basic actions. The example in FIG. 1 shows an example in which the information processing device 101 detects four basic actions, "running," "stopping," "walking," and "shaking head," from the person of the recognition target 1 appearing in the video.
[0017] Next, the information processing device 101 recognizes whether the recognition target has behaved in accordance with a rule based on whether the basic behavior of the recognition target detected from the video conforms to the rule ((3) in FIG. 1). The rule may be expressed, for example, as a pattern of basic behavior. FIG. 1 shows an example in which a rule for detecting suspicious behavior is applied to the recognition target 1 in a video, and an example in which the information processing device 101 has detected suspicious behavior because a basic behavior pattern of "run → stop → shake head" has been detected.
[0018] In this way, by defining rules for detecting the behavior of a recognition target using patterns of basic actions, it is possible to detect the behavior of a recognition target using the rules without having to prepare a large number of learning videos in which the behavior of the recognition target is recorded. Therefore, for example, even when introducing a system for detecting the behavior of a recognition target, the system can be easily introduced without the trouble of preparing learning data.
[0019] However, the generation of such rules requires know-how and experience. Furthermore, generating rules for each of the various behaviors to be recognized requires manpower and costs. Therefore, when creating new rules for detecting behaviors, it is preferable to be able to reuse rules that have been created in the past. For example, if it is possible to reuse rules that have been created in the past to recognize behaviors similar to the behavior to be recognized, the effort involved in generating rules can be reduced.
[0020] One example of a method for identifying rules that detect behavior similar to the behavior of the recognition target from among previously generated rules is to perform a keyword search on previously generated rules. For example, assume that metadata such as a name attached to rule data contains a keyword related to the behavior that the rule is to detect. In this case, by performing a search using a character string that represents the behavior to be recognized, it may be possible to identify a rule that detects behavior similar to the behavior of the recognition target.
[0021] However, in reality, the information registered in the metadata or the like may vary from person to person. For example, a rule that recognizes the same action may be titled "Screw fastening" or "Step 1-A." Or, even if the action is the same "screw fastening," the basic action that characterizes the action may actually differ depending on the fixing position of the screw. For this reason, it may be difficult to identify a rule suitable for reuse by searching by keywords.
[0022] As another method, for example, a video similar to the video in which the behavior to be recognized is captured may be searched for, and rules created from the hit videos may be extracted and reused. However, as described in Non-Patent Document 1, the search for similar videos is currently a technique for searching for similar videos using colors or the number of people appearing in the videos, and it is difficult to expect that videos in which similar behaviors are captured will be appropriately searched for. Therefore, there is a demand for a further technique for identifying rules for recognizing behaviors similar to the behaviors appearing in the video to be recognized from existing rules.
[0023] In the embodiment described below, the information processing device 101 stores a plurality of rules that define a pattern of a basic action for detecting a target action in a storage device in association with first time-series information that represents the detection timing of a plurality of basic actions included in the pattern of the basic action in a time series. Each rule may be, for example, a rule that detects a different action. When a video for which a new rule is to be created is input, the information processing device 101 executes detection of a basic action in the video to detect at least one basic action. Next, for each of the plurality of rules, the information processing device 101 evaluates the similarity between the rule and the video based on the first time-series information associated with the rule and the second time-series information that represents the detection timing of at least one basic action in the video in a time series. Then, the information processing device 101 outputs a candidate rule that is a candidate for reuse from among the plurality of rules based on the similarity.
[0024] In this way, by evaluating the similarity between the rule and the video based on the first time-series information associated with the rule and the second time-series information of at least one basic action detected from the video, it is possible to efficiently identify similar rules that are likely to be applicable. Hereinafter, the embodiment will be described in more detail.
[0025] FIG. 2 is a diagram illustrating a block configuration of an information processing device 101 according to an embodiment. The information processing device 101 includes, for example, a control unit 201, a storage unit 202, and a communication unit 203. The control unit 201 includes, for example, a detection unit 211, an evaluation unit 212, and an output unit 213, and may also include other functional units. The storage unit 202 of the information processing device 101 stores, for example, information such as a basic action recognition result 301, a rule 302, and a behavior detection period 303 of rule information 300 described later. The communication unit 203 communicates with other devices according to, for example, an instruction from the control unit 201. For example, the communication unit 203 may connect to the image capture device 102 and receive video data captured by the image capture device 102. Details of each of these units and details of the information stored in the storage unit 202 will be described later.
[0026] As described above, in the embodiment, a rule created in the past is linked to a recognition result of a basic action of a behavior recognized by the rule and stored in the rule information 300. Hereinafter, the rule information 300 that stores information related to rules will be described using a graph database (DB: Database) having a graph structure as an example.
[0027] Fig. 3 is a diagram showing link relationships of rule information 300 according to an embodiment. Three classes are shown in Fig. 3: basic action recognition result 301, rule 302, and behavior detection period 303. The classes are linked by predicates such as reference, generation, and source.
[0028] FIG. 4 is a diagram illustrating an example of a definition of a class of the basic action recognition result 301. The class of the basic action recognition result 301 may include properties such as a uniform resource identifier (URI), a video, a recognition model, and a body. The URI is, for example, an identifier for identifying an instance of the basic action recognition result 301. The video is, for example, a URI of video data used to generate the basic action recognition result 301. The recognition model is, for example, a URI of a recognition model of the basic action used to generate the basic action recognition result 301. The body may include, for example, data of the recognition result of the basic action obtained by recognizing the basic action by the recognition model for video data shown in the video of the basic action recognition result 301. The recognition result of the basic action stored in the body of the basic action recognition result 301 may be called, for example, first time-series information.
[0029] FIG. 5 is a diagram illustrating an example of a definition of a class of the rule 302. The rule 302 may include properties such as a URI, a version, an author, and a body. The URI is, for example, an identifier for identifying an instance of the rule 302. The version is, for example, information indicating a version of the rule defined in the rule 302. The author is, for example, information indicating the author of the rule defined in the rule 302. The body is, for example, information indicating the rule defined in the rule 302. Note that the rule may be expressed by information indicating a pattern of a basic action for recognizing a behavior to be detected, for example. The pattern of the basic action may include information indicating a combination of the basic actions, for example. The pattern of the basic action may also include information indicating, for example, the detection order of the basic actions.
[0030] FIG. 6 is a diagram illustrating an example of a class definition of the behavior detection period 303. The behavior detection period 303 may include properties such as a URI, a start, an end, and a target ID (Identifier). The URI is, for example, an identifier for identifying an instance of the behavior detection period 303. The start is, for example, information indicating a start frame of a behavior detected from a video. The end is, for example, information indicating an end frame of a behavior detected from a video. The target ID is, for example, an identifier for identifying an actor of a detected behavior. In the behavior detection period 303, for example, when a rule for detecting a certain behavior is applied to a certain video and the behavior is detected, information indicating a period from the start to the end of the detection of the behavior may be registered.
[0031] Note that the definitions of the classes of the basic action recognition result 301 shown in Fig. 4, the rule 302 shown in Fig. 5, and the behavior detection period 303 shown in Fig. 6 are merely examples, and the embodiment is not limited thereto. For example, in another embodiment, the classes of the basic action recognition result 301, the rule 302, and the behavior detection period 303 may include other properties, or some properties may be deleted or replaced.
[0032] 7 is a diagram illustrating definitions of predicates of a graph structure of the rule information 300. The rule information 300 according to the embodiment may include, for example, reference, creation, and source predicates.
[0033] 7, the reference indicates that "S was created with reference to O," where S is the rule 302 and O is the basic action recognition result 301. Therefore, in the rule information 300, a triple connected by a reference edge from a rule 302 to a basic action recognition result 301 indicates that the rule 302 was created with reference to the basic action recognition result 301 connected by the reference edge.
[0034] 7, generation indicates that “S generates O”, where S is a rule 302 and O is a behavior detection period 303. Therefore, in the rule information 300, a triple connected by a generation edge from a rule 302 to a behavior detection period 303 indicates that the rule 302 generated the behavior detection period 303 connected by the generation edge.
[0035] 7, the source indicates that "S is information indicating a part of O," where S is the behavior detection period 303 and O is the basic action recognition result 301. Therefore, in the rule information 300, a triple connected by a source edge from the behavior detection period 303 to the basic action recognition result 301 indicates that the behavior detection period 303 is information indicating a part of the basic action recognition result 301 connected by the source edge.
[0036] Note that the definitions of the predicates shown in Fig. 7 are merely examples, and the embodiment is not limited thereto. For example, in another embodiment, other predicates may be included, or some of the predicates in Fig. 7 may be deleted or replaced.
[0037] As described above, in one embodiment, a graph structure is used to associate existing rules 302 with the recognition results of basic actions included in the behaviors recognized by the rules, and the rules are stored as rule information 300.
[0038] Next, a description will be given of the specification of a candidate rule from existing rules for a video in which the behavior of a recognition target according to the embodiment is shown. Fig. 8 is a diagram illustrating an example of an operation flow of a candidate rule output process according to the embodiment. For example, the control unit 201 of the information processing device 101 may start the operation flow of Fig. 8 when an instruction to execute the candidate rule output process is input.
[0039] In step 801 (hereinafter, step will be abbreviated as "S", for example, S801), the control unit 201 of the information processing device 101 accepts input of video data showing the behavior for which a rule is to be created.
[0040] In S802, the control unit 201 executes recognition of basic movements for the input video. For example, the control unit 201 may recognize basic movements from the video using a recognition model that has been machine-learned by deep learning or the like to recognize basic movements of the target. As described above, the basic movements may be, for example, basic movements taken by the target, and in one example, may include movements of each part obtained by dividing the target's body into parts for each joint. In addition, the basic movements may include movements that the target often performs in various situations, such as walking, running, throwing, grabbing, kicking, flying, and eating. Note that the recognition result obtained by executing recognition of basic movements for the input video may be called, for example, second time-series information.
[0041] In S803, the control unit 201 selects one unprocessed rule 302 from among the rules 302 in the rule information 300.
[0042] In S804 , the control unit 201 acquires the basic action recognition result 301 associated with the selected rule 302 from the rule information 300 .
[0043] In S805, the control unit 201 evaluates the similarity between the selected rule 302 and the basic action of the input video. For example, the control unit 201 may evaluate the similarity between the basic action recognition result 301 associated with the selected rule 302 and the recognition result of the basic action detected from the input video. Hereinafter, an example of evaluation of the similarity of the basic actions according to one embodiment will be described with reference to Figs. 9 to 11.
[0044] [Example of evaluation of similarity of basic actions] FIG. 9 shows a rule 302 for detecting a behavior of a customer picking up an item from a shelf in a food section or the like, and information on a basic action recognition result 301 and a behavior detection period 303 linked to the rule 302 .
[0045] Information on the basic action of rule 302 for detecting the action of picking up a product from a shelf can be acquired, for example, from the body property of rule 302. In one example, rule 302 for detecting the action of picking up a product from a shelf can be defined as a rule that detects the basic action: pointing the right hand forward after the basic action: walking. Note that the definition of the rule is merely an example, and rule 302 for detecting the action of picking up a product from a shelf may be defined by other basic action patterns.
[0046] 9, the horizontal axis indicates the frame number in the video used to generate the basic action recognition result 301. In FIG. 9, the basic action recognition results 301 detected by the recognition model for the basic action: walking and the basic action: pointing right hand forward are vertically arranged. In FIG. 9, the points 901 shown in each row of the basic action: walking and the basic action: pointing right hand forward indicate the frames in which the basic action was detected when the basic action was detected by the recognition model for the video. Also, frames without points 901 indicate that the basic action was not detected by the recognition model. Information on the recognition results of the basic actions used in these rules 302 can be obtained, for example, from the body of the basic action recognition result 301.
[0047] 9 further illustrates, with an arrow, a behavior detection period 303 during which the behavior of picking up a product from a shelf was detected. Information on the behavior detection period 303 during which the behavior of picking up a product from a shelf was detected can be acquired from, for example, the start and end properties of the behavior detection period 303.
[0048] As described above, for example, the control unit 201 can acquire the information shown in FIG. 9 from the selected rule 302 and the basic action recognition result 301 and behavior detection period 303 linked to the rule 302.
[0049] Next, with reference to FIG. 10, the acquisition of information for evaluating the similarity of an input video will be described. In FIG. 10, the horizontal axis indicates frame numbers in the input video. FIG. 10 shows the recognition results of the basic actions executed in S802 for the input video. In the example of FIG. 10, the detection results of the basic actions of walking and pointing the right hand forward are shown as the basic actions, as in FIG. 9. For example, in FIG. 10, the points 1001 shown in each row of the basic action: walking and the basic action: pointing the right hand forward indicate frames in which the basic actions were detected when detection was executed for the video using the recognition model of each basic action. In addition, frames without the point 1001 indicate that the basic action was not detected by the recognition model. In addition, the recognition results of the basic actions may include the recognition results of other basic actions that are predetermined as basic actions to be detected.
[0050] 10, a target behavior period 1002 in which a behavior to be generated as a rule appears is indicated by an arrow. The target behavior period 1002 may be specified by, for example, a user. For example, when creating a new rule for detecting a behavior in a video, the user often recognizes the behavior to be detected, and can specify which section of the video contains the behavior to be detected by the rule by watching the video. Therefore, in one example, the user may specify a section in the video in which the behavior to be detected appears, and input the video to the information processing device 101. The control unit 201 of the information processing device 101 may use the specified section as the target behavior period 1002 for creating the rule.
[0051] As described above, for example, the control unit 201 can obtain the information shown in FIG. 10 for the input moving image.
[0052] Then, using the information shown in Fig. 9 and Fig. 10, the control unit 201 evaluates the similarity between the selected rule 302 and the input video. In one example, the control unit 201 evaluates the similarity between the recognition results of the corresponding basic actions between the selected rule 302 and the input video. For example, in the example of Fig. 9 and Fig. 10, both the selected rule and the input video include "basic action: walking" and "basic action: pointing right hand forward." Therefore, the control unit 201 may evaluate the similarity between the basic actions of "basic action: walking" and "basic action: pointing right hand forward," for example.
[0053] The length of the period during which the behavior to be detected may vary depending on the video. For example, in the selected rule 302 in FIG. 9, the behavior detection period 303 during which the behavior to be detected is detected is set to 100 to 230 frames, and the length is 130 frames. On the other hand, in the input video in FIG. 10, the target behavior period 1002 during which the behavior to be detected is captured is specified to be a period from 50 to 150 frames, and the length is 100 frames. Therefore, in one embodiment, the control unit 201 may use a method such as dynamic time warping (DTW) to generate a corresponding series by associating two pieces of time-series information of the behaviors to be compared.
[0054] Fig. 11 is a diagram showing an exemplary application of dynamic time warping. In Fig. 11(a), the upper part shows a recognition result of basic action 1 (e.g., walking) derived from a selected rule in a behavior detection period 303 as original sequence 1. In addition, the lower part of Fig. 11(a) shows a recognition result of basic action 1 (e.g., walking) derived from an input video in a target behavior period 1002 as original sequence 2. In addition, in original sequence 1 and original sequence 2, 0 represents a frame in which basic action 1 was not detected, and 1 represents a frame in which basic action 1 was detected.
[0055] In this case, a corresponding sequence of the same length can be obtained by using dynamic time warping. Dynamic time warping is an algorithm that, for example, finds the distance between each point of two time series in a brute force manner, and after finding all the distances, finds the path that makes the two time series the shortest. In the obtained corresponding sequence, all data is associated with the selected rule and the video.
[0056] Then, the control unit 201 calculates the similarity of the corresponding series. For example, the control unit 201 may use the Jaccard coefficient of the corresponding series as the similarity. The Jaccard coefficient can be calculated, for example, by the following formula. Jaccard coefficient = number of frames where both are 1 / number of frames where at least one is 1
[0057] 11(b), in the corresponding sequence, the number of frames in which at least one is 1 is four, and the number of frames in which both are 1 is three, so that the Jaccard coefficient can be calculated as 3 / 4. In one example, the control unit 201 may use this Jaccard coefficient as the similarity between the basic actions.
[0058] In addition, the similarity according to the embodiment is not limited to the Jaccard coefficient, and may be other similarity. For example, in another embodiment, the Dice coefficient or the Simpton coefficient may be used. In addition, for example, when the basic action recognition result 301 is expressed by a vector, the cosine similarity may be adopted.
[0059] For example, as described above, the control unit 201 can evaluate the similarity between the recognition results of the corresponding basic actions between the selected rule 302 and the input video.
[0060] In S806, the control unit 201 evaluates the similarity for the rule 302. For example, when similarities are obtained for multiple corresponding basic actions between the rule 302 and the input video in S805, the control unit 201 may further obtain a representative similarity that represents the similarity between the multiple corresponding basic actions. For example, the rule 302 shown in FIG. 9 corresponds to two basic actions, "walk" and "point right hand forward," in the video shown in FIG. 10. Therefore, the control unit 201 executes the process of S805 for these two basic actions, and obtains the similarity for each basic action. Then, in S806, the control unit 201 may obtain a representative similarity that represents the two obtained similarities.
[0061] In one example, the control unit 201 may use the average value of the similarities obtained for each recognition result of the corresponding basic action as the representative similarity. For example, it is assumed that the similarity between the recognition result of the basic action: walking associated with the rule 302 in FIG. 9 and the recognition result of the basic action: walking detected from the video in FIG. 10 is 0.9417. It is also assumed that the similarity between the recognition result of the basic action: turning right hand forward associated with the rule 302 in FIG. 9 and the recognition result of the basic action: turning right hand forward detected from the video in FIG. 10 is 0.7018. In this case, the control unit 201 may use 0.8218 as the representative similarity, since (0.9417+0.7018) / 2=0.8218. Note that the representative similarity representing the multiple similarities according to the embodiment is not limited to the average value, and may be other values. For example, in another embodiment, the representative similarity may be other statistical values representing the multiple similarities, such as the median, maximum value, and minimum value.
[0062] In another embodiment, the representative similarity may be obtained using a weighted average. For example, weighting may be performed according to the frequency of occurrence of the basic actions in the rule information 300.
[0063] For example, it is assumed that 100 rules 302 are registered in the rule information 300. It is assumed that, among these 100 rules 302, the number of rules 302 in which "walking" is registered as a basic action used in detecting an action is 50. On the other hand, it is assumed that, among the 100 rules 302, the number of rules 302 in which "pointing right hand forward" is registered as a basic action used in detecting an action is 10. In this case, it is understood that "pointing right hand forward" has a lower occurrence frequency than "walking" and is a rare basic action in the rule information 300. It is possible that a basic action that appears less frequently and is rare in the rule information 300 is more important in detecting an action by the rule 302 or strongly characterizes the rule 302 than a basic action that appears frequently. Therefore, in one embodiment, the control unit 201 may reflect the similarity of the recognition result of the basic action more strongly in the representative similarity as the occurrence frequency of the target basic action in the rule information 300 is lower.
[0064] For example, in the above example, the rule information 300 includes 100 rules 302, and 50 of the rules 302 include walking as a basic action of interest. Therefore, a weighting coefficient of 2 can be obtained by 100 / 50=2. Similarly, the rule information 300 includes 100 rules 302, and 10 of the rules 302 include pointing the right hand forward as a basic action of interest. Therefore, a weighting coefficient of 10 can be obtained by 100 / 10=10. Then, the control unit 201 may use the obtained weighting coefficients to calculate a weighted average, such as (2*0.9417+10*0.7808) / (2+10)=0.8076, to obtain the representative similarity.
[0065] Then, in the process of S806, the control unit 201 may use the obtained representative similarity as the similarity between the selected rule 302 and the input video.
[0066] In S807, the control unit 201 determines whether or not there is an unprocessed rule 302 in the rule information 300. If there is an unprocessed rule 302 in the rule information 300 (YES in S807), the flow returns to S803, and the control unit 201 selects the unprocessed rule 302 and repeats the process. On the other hand, if there is no unprocessed rule 302 in the rule information 300 (NO in S807), the flow proceeds to S808.
[0067] In S808, the control unit 201 identifies and outputs candidate rules based on the similarity. For example, the control unit 201 may rearrange the rules 302 in the rule information 300 so that rules 302 with high similarity are placed higher than rules 302 with low similarity, and output information indicating the rules 302 as candidate rules. In another example, the control unit 201 may output information indicating a predetermined number of rules 302 with high similarity as candidate rules.
[0068] Furthermore, when outputting a candidate rule, the control unit 201 may output a video designated by a video property of the basic action recognition result 301 corresponding to the candidate rule. This allows the user to view the video corresponding to the candidate rule, and to easily check whether the output candidate rule is suitable for reuse.
[0069] Also, for example, the rules 302 stored in the rule information 300 may be classified into a plurality of groups in advance according to the type of behavior to be detected. In this case, the control unit 201 may output a predetermined number of rules 302 with the highest similarity for each group in S808. The grouping of the rules 302 may be performed based on the similarity, for example. In one example, the control unit 201 evaluates the similarity between the rules 302 included in the rule information 300. Then, the control unit 201 may classify the rules 302 of the rule information 300 into a plurality of groups by grouping the rules 302 with a similarity equal to or higher than a predetermined similarity. Alternatively, the grouping of the rules 302 may be performed so that the rules 302 that are similar to the user in advance are grouped into the same group.
[0070] By grouping in this manner and outputting the rules 302 for each group, it is possible to prevent multiple, nearly identical rules 302 from being identified as candidate rules. For example, when searching for rules 302 that detect behavior similar to that captured in a video, it may be desirable to identify highly similar rules 302 from among the rules 302 that detect a variety of behaviors, rather than just similar behaviors. By grouping and identifying candidate rules from the rules 302 for each group, it is possible to identify rules 302 that target a variety of behaviors as candidate rules.
[0071] As described above, according to the embodiment, if a video showing a behavior to be recognized is prepared, the rule 302 focusing on a basic action that characterizes the behavior can be specified as a candidate rule. Then, in one example, the control unit 201 can start detecting the behavior of the recognition target from the video by using the candidate rule if the control unit 201 adjusts errors due to shooting conditions such as the angle at which a subject included in the video to be recognized is captured and the image quality of the shooting device by parameter fitting. Alternatively, the user may edit the candidate rule to generate a new rule 302 suitable for the video. In this case, too, by reusing the candidate rule, a new rule 302 can be created based on the rule 302 in which the basic action to be focused on is specified, thereby reducing the cost of creating the rule 302.
[0072] (Modification) Next, a modified example will be described. For example, one rule 302 may be applied to multiple videos. In this case, for example, the basic action recognition result 301 and the behavior detection period 303 may be obtained from each video and registered in the rule information 300.
[0073] Fig. 12 is a diagram illustrating a graph structure when a rule 302 according to a modified example of the embodiment is applied to multiple videos. As shown in Fig. 12, the rule 302 is applied to basic action recognition results 301 (basic action recognition results a to c) of multiple videos to generate multiple behavior detection periods 303 (behavior detection period a to behavior detection period d). Note that two behaviors that are targets of detection by the rule 302 are detected from the basic action recognition result c, and two behavior detection periods 303, behavior detection period c and behavior detection period d, are generated.
[0074] In this manner, it is assumed that one rule 302 is applied to basic action recognition results 301 of multiple videos to generate multiple behavior detection periods 303. In this case as well, the control unit 201 can evaluate the similarity between each basic action recognition result 301 and the recognition result of the basic action of the input video, and obtain a representative similarity that represents the multiple similarities, thereby evaluating the similarity between the input video and the rule 302.
[0075] 13 is a diagram illustrating an example of an operation flow of a candidate rule output process according to a modified example of the embodiment. For example, the control unit 201 of the information processing device 101 may start the operation flow of FIG. 13 when an instruction to execute the candidate rule output process is input.
[0076] The subsequent processes from S1301 to S1305 may correspond to, for example, the processes from S801 to S805, and the control unit 201 may execute the same processes as the processes from S801 to S805.
[0077] In S1306, the control unit 201 determines whether there is an unprocessed basic movement recognition result 301 linked to the selected rule 302. If there is an unprocessed basic movement recognition result 301 (YES in S1306), the flow returns to S1304 and repeats the process for the unprocessed basic movement recognition result 301. On the other hand, if there is no unprocessed basic movement recognition result 301 (NO in S1306), the flow proceeds to S1307.
[0078] In S1307, the control unit 201 evaluates the similarity of the selected rule 302. For example, if there is one basic action recognition result 301 associated with the selected rule 302, the control unit 201 may obtain a representative similarity that represents the similarity obtained for each corresponding basic action, and use the representative similarity as the similarity of the rule 302. On the other hand, if there are multiple basic action recognition results 301 associated with the selected rule 302, a similarity is obtained for each basic action for each basic action recognition result 301. In this case, the control unit 201 obtains a representative similarity that represents the similarity of the corresponding basic action for each basic action recognition result 301. Then, the control unit 201 may obtain a representative similarity that further represents the representative similarity for each obtained basic action recognition result 301, and use the representative similarity as the similarity between the video and the rule 302. Note that the representative similarity may be, for example, a similarity that represents a plurality of similarities, and may be a statistical value such as an average value, a median value, a minimum value, or a maximum value.
[0079] The subsequent processes of S1308 and S1309 may correspond to, for example, the processes of S807 to S808, and the control unit 201 may execute the same processes as the processes of S807 to S808.
[0080] As described above, for example, it is assumed that the rule 302 is applied to the basic action recognition results 301 of multiple videos. In this case, too, it is possible to evaluate the similarity between the rule and the videos based on multiple pieces of time-series information of the rule and second time-series information corresponding to the basic actions detected from the videos, and output candidate rules.
[0081] As described in the modified example, it is possible to identify a wide range of rules 302 as candidate rules by evaluating the similarity with multiple basic action recognition results 301. For example, assume that a video showing an action of walking with the right hand facing forward is input. In this case, the similarity of the rule 302 including the action of walking with the right hand facing forward is highly evaluated.
[0082] Furthermore, for example, it is assumed that the rule information 300 includes a rule 302 of walking with one hand facing forward. In this rule 302, the hand facing forward may be the right hand or the left hand, and if an action of walking with one hand facing forward is captured, this rule 302 is satisfied. However, for example, it is assumed that only the basic action recognition result 301 of a video showing an action of walking with the left hand facing forward exists in the rule information 300 as the basic action recognition result 301 associated with this rule 302. In this case, since the input video is a video showing the basic action of facing the right hand forward, the similarity of the basic action of facing the left hand forward is evaluated as low. As a result, the similarity between the input video and the rule 302 of walking with one hand facing forward is also evaluated as low.
[0083] On the other hand, for example, as basic action recognition result 301 linked to rule 302, basic action recognition result 301 of a video showing the basic action of walking with the left hand facing forward and basic action recognition result 301 of a video showing the basic action of walking with the right hand facing forward are linked. As a result, basic action recognition result 301 of a video showing the basic action of walking with the right hand facing forward is highly evaluated for similarity with the input video, so that the representative similarity representing the multiple basic action recognition results 301 can also be highly evaluated. As a result, the similarity of rule 302 of walking with one hand facing forward can be highly evaluated with the input video, and it becomes possible to identify rule 302 of walking with one hand facing forward as a candidate rule.
[0084] In this way, the rule 302 may be described to allow multiple basic actions, such as pointing one hand forward. By linking multiple basic action recognition results 301 to the rule 302 so as to cover these various descriptions, if the rule 302 to be evaluated matches any of the basic action recognition results 301, the rule 302 can be highly evaluated. As a result, it becomes possible to identify a wide range of rules 302 corresponding to the input video based on the similarity. Note that in another embodiment, for basic actions described in parallel in the rule 302, the control unit 201 may use the maximum similarity among the similarities of the multiple basic actions described in parallel as a representative similarity representing the multiple basic actions described in parallel.
[0085] Although the embodiment has been exemplified above, the embodiment is not limited thereto. For example, the above-mentioned operation flow is an example, and the embodiment is not limited thereto. If possible, the operation flow may be executed by changing the order of processing, may include additional processing, or may omit some processing. For example, in the past execution of the operation flow of FIG. 8 and FIG. 13, if basic operation recognition has already been executed for the input video, the processing of S802 and S1302 may not be executed.
[0086] In the above embodiment, the recognition result recorded in the basic action recognition result 301 linked to the rule 302 may be, for example, only information on the recognition result for the basic action used in the pattern of the basic action defined in the rule 302. This can reduce the storage capacity required for accumulating the basic action recognition result 301. However, the embodiment is not limited to this, and the basic action recognition result 301 may include information on the recognition result for other basic actions. In addition, the process of evaluating the similarity of the basic actions in S805 and S1305 may also be performed only for the basic actions included in the rule 302. That is, the control unit 201 may evaluate the similarity between, for example, a part of the second time-series information corresponding to at least one basic action detected from the video, the multiple basic actions of the rule, and the first time-series information associated with the rule. In addition, the detection of the basic action from the input video may be performed only for the basic actions registered in the rule 302 of the rule information 300. This can reduce the amount of processing.
[0087] Also, a basic action focused on in rule 302 may not be detected in the input video. In this case, control unit 201 may use a recognition result in which the basic action is not detected to evaluate the similarity with the basic action of rule 302. Alternatively, control unit 201 may not evaluate the similarity of a basic action included in rule 302 that is not detected in the input video, but may evaluate the similarity between rule 302 and the input video using the similarity evaluated for the other basic actions.
[0088] In the above embodiment, three classes, the basic action recognition result 301, the rule 302, and the behavior detection period 303, are defined as classes of the rule information 300, but the embodiment is not limited thereto. For example, in another embodiment, the behavior detection period 303 may not be included. Instead, the information of the behavior detection period 303 may be generated appropriately by the control unit 201 applying the rule 302 to the basic action recognition result 301. For example, the control unit 201 may specify a section in which the target basic action is detected at a predetermined frequency or more as a section in which the basic action is detected. Then, the control unit 201 may integrate a section in which a plurality of basic actions included in the pattern of the basic action defined by the rule 302 are detected, and use the integrated section as the behavior detection period. Alternatively, in another embodiment, the basic action recognition result 301 may be recorded in the rule information 300 so that the range of the video of the basic action recognition result 301 becomes the behavior detection period 303.
[0089] In the above-described embodiment, for example, in the processes from S801 to S802 and from S1301 to S1302, the control unit 201 of the information processing device 101 operates as the detection unit 211. In addition, in the processes from S806 and S1307, the control unit 201 of the information processing device 101 operates as, for example, the evaluation unit 212. In the processes from S808 to S1309, the control unit 201 of the information processing device 101 operates as, for example, the output unit 213.
[0090] Fig. 14 is a diagram illustrating a hardware configuration of a computer 1400 for realizing an information processing device 101 according to an embodiment. The hardware configuration in Fig. 14 includes, for example, a processor 1401, a memory 1402, a storage device 1403, a reader 1404, a communication interface 1406, and an input / output interface 1407. The processor 1401, the memory 1402, the storage device 1403, the reader 1404, the communication interface 1406, and the input / output interface 1407 are connected to each other, for example, via a bus 1408.
[0091] The processor 1401 may be, for example, a single processor, or a multi-processor or multi-core. The processor 1401 provides some or all of the functions of the control unit 201 described above by using the memory 1402 to execute a program in which the procedures of the above-mentioned operation flow are described. For example, the processor 1401 of the information processing device 101 operates as the detection unit 211, the evaluation unit 212, and the output unit 213 by reading and executing a program stored in the storage device 1403.
[0092] The memory 1402 is, for example, a semiconductor memory, and may include a RAM area and a ROM area. The storage device 1403 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device. Note that RAM is an abbreviation for Random Access Memory. Also, ROM is an abbreviation for Read Only Memory.
[0093] The reader 1404 accesses the removable storage medium 1405 according to an instruction from the processor 1401. The removable storage medium 1405 is realized by, for example, a semiconductor device, a medium in which information is input and output by magnetic action, or a medium in which information is input and output by optical action. The semiconductor device is, for example, a Universal Serial Bus (USB) memory. The medium in which information is input and output by magnetic action is, for example, a magnetic disk. The medium in which information is input and output by optical action is, for example, a CD-ROM, DVD, Blu-ray Disc, etc. (Blu-ray is a registered trademark). CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk.
[0094] The above-mentioned storage unit 202 includes, for example, a memory 1402, a storage device 1403, and a removable storage medium 1405. For example, the storage device 1403 of the information processing device 101 stores the basic action recognition result 301, the rule 302, and the behavior detection period 303 of the rule information 300.
[0095] The communication interface 1406 communicates with other devices, for example, in accordance with an instruction from the processor 1401. For example, the information processing device 101 may receive video data from the image capturing device 102 via the communication interface 1406. The communication interface 1406 is an example of the communication unit 203 described above.
[0096] The input / output interface 1407 is, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard, a mouse, or a touch panel that receives instructions from a user. The output device is, for example, a display device such as a display, and an audio device such as a speaker.
[0097] Each program according to the embodiment is provided to the information processing device 101 in the following form, for example. (1) Pre-installed in the storage device 1403. (2) Provided by a removable storage medium 1405. (3) Provided from a server such as a program server.
[0098] Note that the hardware configuration of the computer 1400 for realizing the information processing device 101 described with reference to FIG. 14 is an example, and the embodiment is not limited thereto. For example, a part of the above-mentioned configuration may be deleted, or a new configuration may be added. Also, in another embodiment, for example, a part or all of the functions of the above-mentioned control unit 201 may be implemented as hardware using FPGA, SoC, ASIC, PLD, or the like. Note that FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip. ASIC is an abbreviation for Application Specific Integrated Circuit. PLD is an abbreviation for Programmable Logic Device.
[0099] In the above, several embodiments are described. However, the embodiments are not limited to the above embodiments, and should be understood as including various modified and alternative forms of the above-mentioned embodiments. For example, it will be understood that the various embodiments can be embodied by modifying the components without departing from the spirit and scope of the embodiments. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-mentioned embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments. [Explanation of symbols]
[0100] 100 Behavior Recognition System 101 Information processing device 102 Imaging Device 201 Control section 202 Storage section 203 Communications Department 211 Detection unit 212 Evaluation Department 213 Output section 300 Rule Information 301 Basic motion recognition results 302 Rules 303 Behavior Detection Period 1002 Target Action Period 1400 Computer 1401 Processor 1402 Memory 1403 Storage device 1404 Reading device 1405 Removable storage media 1406 Communication Interface 1407 Input / Output Interface 1408 Bus
Claims
1. A detection unit that detects at least one basic motion of a target from a video; an evaluation unit that, for each of a plurality of rules defining a pattern of basic actions for detecting the behavior of the target, the plurality of rules being classified into a plurality of groups according to a type of the behavior of the target, is associated with first time series information that represents in time series a detection timing of a plurality of basic actions included in the pattern of the basic actions, and that evaluates a similarity between the rule and the video based on the first time series information associated with the rule and second time series information that represents in time series a detection timing of the at least one basic action detected from the video; an output unit that outputs a first candidate rule and a second candidate rule among the plurality of rules, the first candidate rule being identified from a first group among the plurality of groups based on the similarity, and the second candidate rule being identified from a second group among the plurality of groups based on the similarity; An information processing device comprising:
2. 2. The information processing device according to claim 1, wherein the evaluation unit evaluates the similarity between a portion of the second time series information corresponding to the at least one basic action detected from the video, the portion of the second time series information corresponding to the multiple basic actions of the rule, and the first time series information associated with the rule.
3. 3. The information processing device according to claim 1, wherein the evaluation unit evaluates a similarity between the first time-series information associated with the rule and the second time-series information corresponding to the at least one basic action detected from the video, the corresponding series generated by a dynamic time warping method.
4. a rule among the plurality of rules is associated with a plurality of pieces of time series information including the first time series information, 4. The information processing device according to claim 1, wherein the evaluation unit evaluates a similarity between the certain rule and the video based on the plurality of time series information of the certain rule and the second time series information corresponding to the at least one basic action detected from the video.
5. 1. A computer-implemented output method, comprising: The computer, Detecting at least one basic motion of a subject from the video; a plurality of rules that define a pattern of basic actions for detecting the behavior of the target, the plurality of rules being classified into a plurality of groups according to a type of the behavior of the target, the plurality of rules being associated with first time series information that represents, in a time series, detection timings of a plurality of basic actions included in the pattern of the basic actions; evaluating a similarity between the rule and the video, for each of the plurality of rules, based on the first time series information associated with the rule and second time series information that represents, in a time series, detection timings of the at least one basic action detected from the video; outputting a first candidate rule and a second candidate rule from the plurality of rules, the first candidate rule being identified from a first group of the plurality of groups based on the similarity, and the second candidate rule being identified from a second group of the plurality of groups based on the similarity; 4. An output method comprising:
6. Detecting at least one basic motion of a subject from the video; a plurality of rules that define a pattern of basic actions for detecting the behavior of the target, the plurality of rules being classified into a plurality of groups according to a type of the behavior of the target, the plurality of rules being associated with first time series information that represents, in a time series, detection timings of a plurality of basic actions included in the pattern of the basic actions; evaluating a similarity between the rule and the video, for each of the plurality of rules, based on the first time series information associated with the rule and second time series information that represents, in a time series, detection timings of the at least one basic action detected from the video; outputting a first candidate rule and a second candidate rule from the plurality of rules, the first candidate rule being identified from a first group of the plurality of groups based on the similarity, and the second candidate rule being identified from a second group of the plurality of groups based on the similarity; An output program that causes a computer to carry out processing.
Citation Information
Patent Citations
Image analyzing apparatus, image analysis method and computer program
JP2005228274A
Operation detector and operation detection program, and operation basic model generator and operation basic model generation program
JP2009009413A
Gesture registration device
JP2015116308A
Information processing apparatus and method, computer program, and monitoring system
JP2019176423A
Motion recognition method and device
JP2020038440A
