Program, apparatus, and method for generating behavioral definition rules

Abstract

To improve the efficiency of generating a behavior definition rule for retrieving a given behavior from a video.SOLUTION: A behavior definition rule generator includes the steps of: referring to a storage part containing a plurality of first behavior definition rules indicating combinations of basic motions of a person input as a video search query and extracting a first basic motion from the plurality of behavior definition rules based on the frequency of each basic motion included in the plurality of first behavior definition rules; extracting a second basic motion related to the extracted first basic motion from the plurality of behavior definition rules based on co-occurrence with the first basic motion; and generating a second behavior definition rule composed of the extracted first and second basic motions.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The disclosed technology relates to an action definition rule generation program, an action definition rule generation device, and an action definition rule generation method.

Background Art

[0002] Conventionally, a predetermined video has been searched from a large amount of accumulated video archives. In such video searches, keyword-based searches are common. Also, there is a need to search for a predetermined video regardless of keywords. For example, there are techniques for similar video searches in which a video is input as a query and similar videos to the query video are searched, and for searches in which information defining a predetermined action is input and a video in which the predetermined action is recorded is searched by action recognition.

[0003] Regarding the technology of video search by action recognition, for example, an action recognition system has been proposed that recognizes daily actions such as "eating" and "drinking" in which the order and number are different each time, rather than following a certain pattern. This system has a two-step configuration of a first step and a second step for the recognition process when processing the captured image data of a person to recognize the action. In the first step of this system, a certain action pattern performed in a short period is recognized based on the time-series change of the movement information of the person detected by processing the image data. Also, in the second step of this system, the entire action over a long period such as "eating" and "drinking" is recognized based on the ratio of a certain action pattern included in the period to be recognized.

[0004] Furthermore, an object recognition device has been proposed that generates training images representing the shape of a target object whose actions are to be recognized, accepts an input image of the target object, and recognizes the actions indicated by the received image based on the training images. This device generates images representing multiple mutually distinct shapes in a first action, which is a series of actions included in a predetermined action of the target object, for each action state of the series of actions other than the first action included in the predetermined action. This device also uses the generated images as training images and projects each of the training images as a feature point in an eigenspace for recognizing the actions indicated by the received image. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2005-215927 [Patent Document 2] Japanese Patent Publication No. 2010-267004 [Overview of the project] [Problems that the invention aims to solve]

[0006] Gesture recognition is a well-known technique for recognizing actions in videos. Gesture recognition can recognize predetermined gestures using techniques such as deep learning. However, gesture recognition cannot be applied to recognizing various actions defined by the user. Similarly, in the case of similar video search, it cannot cover variations of actions other than those entered as a query, and therefore cannot be applied to recognizing various actions defined by the user. For example, if you want to search for videos of people picking up a product, and your query in similar video search is videos of people picking up the product with their right hand, it will not be able to find videos of people picking up the product with their left hand.

[0007] Therefore, one approach is to input action definition rules that combine basic actions, and then search for videos in which a specific action is performed based on these action definition rules. However, since the number of combinations of basic actions is enormous, there is a problem in that generating action definition rules is time-consuming.

[0008] One aspect of disclosure technology is that it aims to streamline the generation of behavior definition rules for searching for specific actions from videos. [Means for solving the problem]

[0009] In one embodiment, the disclosure technology refers to a memory unit that stores multiple first behavior definition rules, which represent combinations of basic movements of a person, entered as a video search query. The disclosure technology then extracts a first basic movement from the multiple behavior definition rules based on the frequency of each basic movement included in the multiple first behavior definition rules. The disclosure technology also extracts a second basic movement related to the extracted first basic movement from the multiple behavior definition rules based on its co-occurrence with the extracted first basic movement. The disclosure technology then generates a second behavior definition rule composed of the extracted first and second basic movements. [Effects of the Invention]

[0010] One aspect of this approach is that it can streamline the generation of behavior definition rules for searching for specific actions within videos. [Brief explanation of the drawing]

[0011] [Figure 1] This is a functional block diagram of the video search system. [Figure 2] This figure shows an example of a video search database. [Figure 3] This is a diagram to explain the rules for defining actions. [Figure 4] This is a diagram illustrating a concrete example of an action definition rule. [Figure 5]It is a diagram showing an example of extraction of recognition results of basic operations for a search query. [Figure 6] It is a diagram showing an example of an action definition rule DB. [Figure 7] It is a block diagram showing a schematic configuration of a computer functioning as an action definition rule generation device. [Figure 8] It is a flowchart showing an example of reception processing. [Figure 9] It is a flowchart showing an example of generation registration processing. [Figure 10] It is a diagram for explaining the effects of the present embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, an example of an embodiment according to the disclosed technology will be described with reference to the drawings.

[0013] As shown in FIG. 1, a video search system 100 according to the present embodiment includes an action definition rule generation device 10 and a video search device 20.

[0014] As shown in FIG. 1, the video search device 20 functionally includes a recognition unit 21 and a search unit 22. Also, a basic operation recognition rule DB (database) 23 and a video search DB 24 are stored in a predetermined storage area of the video search device 20.

[0015] The recognition unit 21 acquires a video to be searched, which is input to the video search device 20. The recognition unit 21 recognizes the basic operations included in the acquired video based on the basic operation recognition rules stored in the basic operation recognition rule DB 23, and stores the recognition results in the video search DB 24.

[0016] Specifically, the basic operation recognition rule DB23 stores basic operation recognition rules for recognizing each of a plurality of predefined basic operations from a video. The basic operations are basic operations such as the movements of each part of the human body and the movement of the whole body, which are the constituent elements of various actions. The basic operations are, for example, "walking", "standing upright", "looking at an object", "looking at a person", "extending the right hand forward", "extending the left hand forward", "touching an object", etc.

[0017] The basic operation recognition rule may be, for example, a machine learning model such as a deep neural network generated by performing machine learning on the association between the time series information of features for recognizing the basic operation and the correct answer of the basic operation represented by the video. The features for recognizing the basic operation may be, for example, time series information such as skeleton information of the human body recognized from each frame of the video. Also, the basic operation recognition rule may be a rule-based description of features such as the time series change of the skeleton information of the human body. In the basic operation recognition rule DB23, for each basic operation, the defined name of the basic operation and the basic operation recognition rule are stored in association with each other.

[0018] Specifically, the recognition unit 21 extracts regions indicating a person from each frame of the acquired video, assigns an identifier, the operator ID, to that person, and tracks the same person across consecutive frames. The recognition unit 21 also estimates features for recognizing basic actions, such as skeletal information, for each frame of the acquired video of the tracked person. The recognition unit 21 recognizes basic actions by comparing the time-series changes of the estimated features with each of the basic action recognition rules stored in the basic action recognition rule DB 23. Furthermore, if the recognition unit 21 recognizes any basic action from the acquired video, it obtains the frame numbers of the start and end frames as the frame interval in which the basic action was recognized. Then, as shown in Figure 2, the recognition unit 21 associates the basic action recognition result with the video ID, which is the identifier of the target video, and stores it in the video search DB 24. The basic action recognition result includes the definition name of the recognized basic action, the operator ID assigned to the person performing that basic action, and the frame numbers of the start and end frames. Furthermore, the recognition results of the basic operation may include other information, such as identification information of the object being operated on.

[0019] The video search device 20 receives behavior definition rules as search queries instead of keywords, which are the input for a typical search system. The search unit 22 receives the input behavior definition rules, searches the video search DB 24 for videos that match the behavior definition rules, and outputs information about the found videos as search results.

[0020] Here, we will explain the action definition rules. An action definition rule is represented by a combination of basic actions that satisfy a given condition. For example, when using the AC (ActionChain) class, the action definition rule describes the conditions for creating a new event instance, using two event instances as elements. An example of an action definition rule using the AC class is shown below.

[0021] AC1:AC(EventA, #Class ID of Event 1) EventB, #Class ID of Event 2 The operator of EventA and EventB is the same & #Condition 1 Event B starts within 10 seconds after Event A ends, #Condition 2 )

[0022] As shown in Figure 3, an event instance with the class ID AC1 is created when a pair of event instances (EventA and EventB) that satisfy the conditions of the above action definition rule exists.

[0023] Let's explain some more specific examples of behavior definition rules. For example, suppose the behavior definition rule for "theft of unattended items" is defined as shown below. In "AC(X1,X2,X3)" below, X1 and X2 are basic actions, and X3 represents the conditions for recognizing a new basic action from basic actions X1 and X2.

[0024] Sneaking up: AC (placing an object, approaching an object, different agents & same object) Steal: AC (Sneak up, take an object, same agent & same object) To escape: AC (Steal, Run, Same agent) Pickpocketing: AC (Sneak up, steal, flee, same agent & same object)

[0025] Let's assume that the predefined basic actions are "placing an object," "approaching an object," "holding an object," and "running." In this case, as shown in Figure 4, the basic action "creeping" is recognized when the basic actions "placing an object" and "approaching an object" meet the conditions. Also, the basic action "stealing" is recognized from the basic action "holding an object" and the newly recognized basic action "creeping." Similarly, the basic action "fleeing" is newly recognized from the basic actions "stealing" and "running." And, when the agent and the object of the basic actions "creeping," "stealing," and "fleeing" are the same, the action of "bag snatching" is recognized. Note that in Figure 4, the basic actions represented by white nodes are predefined basic actions, and the basic actions represented by shaded nodes are basic actions newly recognized based on the action definition rules. Also, the parentheses within the nodes represent (agent ID, object ID).

[0026] The following describes video search performed by the search unit 22 using the action definition rules described above. For the sake of simplicity, each action definition rule is assumed to include the common conditions that the agent is the same and that there is a common part in the interval (frame interval) in which the basic action occurs. In the description of the action definition rule, the description of the conditions may be omitted.

[0027] The search unit 22 extracts the recognition results for each basic action included in the action definition rule entered as a search query from the video search DB 24. Here, it is assumed that the action definition rule includes basic actions X and Y, and the set of recognition results for basic action X is called set X, and the set of recognition results for basic action Y is called set Y. Figure 5 shows an example of the extraction results when basic action X is "extend right hand forward" and basic action Y is "stand upright". The search unit 22 determines whether all combinations of elements of set X and elements of set Y satisfy the conditions described in the action definition rule. For combinations that satisfy the conditions, the search unit 22 outputs search results that include the video ID of the corresponding video and information indicating the common part of the frame interval between basic action X and basic action Y.

[0028] As mentioned above, basic movements are the movements of individual body parts and the entire body, and since there are a great many of them, it is time-consuming for users to write action definition rules to input as search queries. Furthermore, when searching for videos, it is necessary to determine whether the combination of basic movements included in the action definition rule matches the conditions, which results in a very long search processing time. For example, in the case of action definition rules based on the AC class mentioned above, the number of times the condition is determined to be satisfied is on the order of the square of the number of basic movements included in the action definition rule. In particular, the search processing time becomes very long when the condition is determined to be satisfied multiple times, as in the example in Figure 4.

[0029] Therefore, in this embodiment, we focus on the fact that there are not many patterns of combinations of basic actions included in the action definition rules described as search queries, and that there are frequently occurring patterns. For example, combination patterns that accommodate left and right variations, such as "extend right hand forward" or "extend left hand forward," are easily described as action definition rules. Also, patterns that combine frequently occurring basic actions with other basic actions, such as "stand upright" and "extend right hand forward," which represent the action of "stopping and doing something," are also easily described as action definition rules. The action definition rule generation device 10 according to this embodiment efficiently generates action definition rules for video search based on such frequently occurring patterns.

[0030] Functionally, the behavior definition rule generation device 10 includes a reception unit 11, a first extraction unit 12, a second extraction unit 13, a generation unit 14, and a registration unit 15, as shown in Figure 1. Furthermore, a behavior definition rule DB 16 is stored in a predetermined storage area of ​​the behavior definition rule generation device 10.

[0031] The reception unit 11 receives behavior definition rules entered as video search queries by multiple users into the video search device 20. The reception unit 11 stores the received behavior definition rules in the behavior definition rule DB 16. Figure 6 shows an example of the behavior definition rule DB 16. The reception unit 11 treats a search query entered in a single search operation to the video search device 20 as one behavior definition rule, and even if the search query includes multiple basic operations, it receives and stores it as one behavior definition rule including those multiple basic operations. In other words, if 1000 search operations are performed, 1000 behavior definition rules will be stored in the behavior definition rule DB 16. The behavior definition rules received by the reception unit 11 and stored in the behavior definition rule DB 16 are examples of the "first behavior definition rules" of the disclosed technology.

[0032] The first extraction unit 12 refers to the behavior definition rule DB 16 and extracts high-frequency basic actions from among the multiple behavior definition rules based on the frequency of each basic action included in the multiple behavior definition rules. High-frequency basic actions are examples of the "first basic actions" of the disclosed technology.

[0033] Specifically, the first extraction unit 12 aggregates the frequency of occurrence of each basic action included in the action definition rule DB 16, and extracts basic actions whose frequency of occurrence is above a predetermined value, or up to the top predetermined number of occurrences, as high-frequency basic actions. For example, when extracting the basic action with the highest frequency of occurrence as a high-frequency basic action, if the basic action "standing upright" has the highest frequency of occurrence in the aggregated result of the frequency of occurrence of each basic action, the first extraction unit 12 extracts "standing upright" as a high-frequency basic action.

[0034] The second extraction unit 13 extracts co-occurring basic actions related to high-frequency basic actions from among multiple action definition rules stored in the action definition rule DB 16, based on their co-occurrence with high-frequency basic actions extracted by the first extraction unit 12. Co-occurring basic actions are an example of the "second basic actions" of the disclosed technology. Specifically, the second extraction unit 13 extracts as co-occurring basic actions basic actions from among the basic actions included in the action definition rule DB 16 those basic actions whose evaluation value indicating co-occurrence, i.e., the degree to which they appear in the same action definition rules as high-frequency basic actions, is above a predetermined value, or up to the top predetermined number of evaluation values.

[0035] For example, the second extraction unit 13 calculates the Jaccard coefficient and the like shown below as evaluation values ​​for the co-occurrence between basic movements for each of the high-frequency basic movements and other basic movements. Evaluation value (Jaccard coefficient) of the co-occurrence between basic movement X and basic movement Y = Number of behavior definition rules that have both X and Y as components. Number of behavior definition rules that consist of / X or Y

[0036] For example, the second extraction unit 13 calculates evaluation values ​​for the high-frequency basic action "standing upright" and other basic actions. Then, when extracting the basic action with the highest evaluation value as a co-occurring basic action, suppose the evaluation value for "extending the right hand forward" is the highest. In this case, the second extraction unit 13 extracts "extending the right hand forward" as a co-occurring basic action. In this example, the combination of the basic actions "standing upright" and "extending the right hand forward" is described as an action definition rule when recognizing various actions such as "picking up an item," "shaking hands," and "pointing and confirming." Therefore, the evaluation value of the co-occurrence of "extending the right hand forward" with respect to "standing upright" is high.

[0037] The generation unit 14 generates an action definition rule whose constituent elements are the high-frequency basic actions extracted by the first extraction unit 12 and the co-occurring basic actions extracted by the second extraction unit 13. The action definition rule generated by the generation unit 14 is an example of the "second action definition rule" of the disclosed technology. Specifically, the generation unit 14 generates an action definition rule that describes the high-frequency basic actions, the co-occurring basic actions, and the above-mentioned conditions in a format such as the example using the AC class described above.

[0038] Furthermore, the generation unit 14 may generate a definition name for the generated action definition rule based on the definition name of the high-frequency basic action and the definition name of the co-occurring basic action. Specifically, the generation unit 14 may generate a definition name for the action definition rule by adding the terminal form of the basic action name of the co-occurring basic action to the conjunctive form of the basic action name of the high-frequency basic action. For example, if the high-frequency basic action is "standing upright" and the co-occurring basic action is "extending the right hand forward", the generation unit 14 will generate a definition name of "standing upright and extending the right hand forward". Note that the definition name for the action definition rule is not limited to being automatically generated as described above; the extracted basic actions may be presented to the user, and a definition name may be accepted from the user. Alternatively, the definition names of both basic actions may be simply combined, such as "standing upright" and "extending the right hand forward", to form the definition name for the action definition rule.

[0039] The registration unit 15 registers the action definition rules generated by the generation unit 14 as basic action recognition rules in the basic action recognition rule DB 23 of the video search device 20. As a result, the recognition unit 21 of the video search device 20 recognizes the actions indicated by the action definition rules generated by the generation unit 14 as basic actions from the video and stores the recognition results in the video search DB 24. This allows subsequent search operations in the video search device 20 to use, for example, action definition rules that include "standing upright and extending the right hand forward" as a basic action as search queries.

[0040] The behavior definition rule generation device 10 may be implemented, for example, by a computer 40 as shown in Figure 7. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage device 43. The computer 40 also includes input / output devices 44 such as input devices and display devices, and an R / W (Read / Write) device 45 that controls the reading and writing of data to and from the storage medium 49. The computer 40 also includes a communication interface 46 that connects to a network such as the Internet. The CPU 41, memory 42, storage device 43, input / output devices 44, R / W device 45, and communication interface 46 are connected to each other via a bus 47.

[0041] The storage device 43 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. The storage device 43 stores an action definition rule generation program 50 that causes the computer 40 to function as an action definition rule generation device 10. The action definition rule generation program 50 includes an acceptance process control command 51, a first extraction process control command 52, a second extraction process control command 53, a generation process control command 54, and a registration process control command 55. The storage device 43 also has an information storage area 60 in which information constituting the action definition rule DB 16 is stored.

[0042] The CPU 41 reads the behavior definition rule generation program 50 from the storage device 43, expands it into memory 42, and sequentially executes the control instructions contained in the behavior definition rule generation program 50. The CPU 41 operates as the reception unit 11 shown in Figure 1 by executing the reception process control instruction 51. The CPU 41 also operates as the first extraction unit 12 shown in Figure 1 by executing the first extraction process control instruction 52. The CPU 41 also operates as the second extraction unit 13 shown in Figure 1 by executing the second extraction process control instruction 53. The CPU 41 also operates as the generation unit 14 shown in Figure 1 by executing the generation process control instruction 54. The CPU 41 also operates as the registration unit 15 shown in Figure 1 by executing the registration process control instruction 55. The CPU 41 also reads information from the information storage area 60 and expands the behavior definition rule DB 16 into memory 42. As a result, the computer 40 that executed the behavior definition rule generation program 50 functions as a behavior definition rule generation device 10. Note that CPU41, which executes the program, is hardware.

[0043] The functions implemented by the action definition rule generation program 50 may be implemented, for example, by a semiconductor integrated circuit, or more specifically, by an ASIC (Application Specific Integrated Circuit).

[0044] Next, the operation of the behavior definition rule generation device 10 according to this embodiment will be described. Each time a search operation is performed on the video search device 20, the behavior definition rule generation device 10 executes the reception process shown in Figure 8. In addition, the behavior definition rule generation device 10 executes the generation registration process shown in Figure 9 at a predetermined timing. The reception process and the generation registration process will be described in detail below. Note that the reception process and the generation registration process are examples of the behavior definition rule generation method of the disclosed technology.

[0045] First, let's explain the reception process shown in Figure 8. In step S11, the reception unit 11 receives the action definition rule to be entered as a search query into the video search device 20. Next, in step S12, the reception unit 11 stores the received action definition rule in the action definition rule DB 16, and the reception process ends.

[0046] Next, we will explain the generation and registration process shown in Figure 9. In step S21, the first extraction unit 12 aggregates the frequency of occurrence of each basic action included in the action definition rule DB 16, and extracts basic actions whose frequency of occurrence is above a predetermined value, or up to the top predetermined number of basic actions, as high-frequency basic actions.

[0047] Next, in step S22, the second extraction unit 13 extracts basic actions included in the action definition rule DB 16 that have an evaluation value of co-occurrence with high-frequency basic actions equal to or greater than a predetermined value, or up to the top predetermined number of basic actions with an evaluation value, as co-occurring basic actions.

[0048] Next, in step S23, the generation unit 14 generates an action definition rule whose constituent elements are the high-frequency basic actions extracted in step S21 and the co-occurring basic actions extracted in step S22. Next, in step S24, the registration unit 15 registers the action definition rule generated in step S23 as a basic action recognition rule in the basic action recognition rule DB23 of the video search device 20, and the generation and registration process ends.

[0049] As described above, the behavior definition rule generation device according to this embodiment receives behavior definition rules that indicate combinations of basic actions of a person, which are input as a query for video search, and stores them in the behavior definition rule DB. The behavior definition rule generation device also extracts high-frequency basic actions based on the frequency of each basic action included in the multiple behavior definition rules stored in the behavior definition rule DB, and extracts co-occurring basic actions based on their co-occurrence with the extracted high-frequency basic actions. The behavior definition rule generation device then generates behavior definition rules that consist of the extracted high-frequency basic actions and co-occurring basic actions. This streamlines the generation of behavior definition rules for searching for a predetermined action from a video and reduces the effort required for users to generate behavior definition rules.

[0050] Furthermore, the behavior definition rule generation device registers the generated behavior definition rules as basic action recognition rules in the video search device. This simplifies the description of behavior definition rules to be entered as search queries and shortens the search processing time for video searches using behavior definition rules.

[0051] For example, suppose you want to search for videos of the action "standing upright and extending the right hand forward," and the basic action recognition rules define the basic actions "standing upright" and "extending the right hand forward," but not the basic action "standing upright and extending the right hand forward." In this case, you would use an action definition rule that includes both basic actions, "standing upright" and "extending the right hand forward," as your search query, and then extract the recognition results for each basic action from the video search database for this search query, as shown in Figure 5. Then, for each combination of recognition results, you would need to determine whether the conditions are met that the performer is the same and that there are common frame intervals.

[0052] On the other hand, in this embodiment, the generated action definition rule (for example, "stand upright and extend the right hand forward") is registered as a basic action recognition rule, and the recognition result for the basic action "stand upright and extend the right hand forward" is stored in the video search DB. Therefore, it is not necessary to specify an action definition rule that includes the basic actions "standing upright" and "extending the right hand forward" as a search query, and the action definition rule "stand upright and extend the right hand" can be specified directly. Also, as shown by the dashed line in Figure 10, the recognition result for the basic action "stand upright and extend the right hand," that is, the recognition result with the conditions of the basic actions "standing upright" and "extending the right hand forward" already determined, is extracted from the video search DB. In other words, in this embodiment, there is no need to extract recognition results for each basic action and to perform conditional judgments between the recognition results of basic actions, as shown outside the dashed line in Figure 10, and the search processing time is shortened. In the effect verification experiment, the search processing time for the above example was reduced to approximately 1 / 4.

[0053] In the above embodiment, the behavior definition rule generation program is pre-stored (installed) in the storage device, but this is not limited to that. The program relating to the disclosed technology may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, or USB memory.

[0054] The following additional information is disclosed regarding the embodiments described above.

[0055] (Note 1) The system references a memory unit that stores multiple first action definition rules, each representing a combination of basic human actions, which are entered as a video search query. Based on the frequency of each basic action included in the multiple first action definition rules, the system extracts a first basic action from among the multiple action definition rules. Based on the co-occurrence with the extracted first basic action, a second basic action related to the first basic action is extracted from the plurality of action definition rules. A second action definition rule is generated, consisting of the extracted first basic action and the second basic action as constituent elements. A program that generates action definition rules to cause a computer to perform a process that includes the following.

[0056] (Note 2) An action definition rule generation program as described in Appendix 1, which causes the computer to perform a process that includes recognizing the basic actions included in the action definition rules entered as queries from each of several videos, and registering the second action definition rule as a recognition rule for basic actions to search for videos corresponding to the query.

[0057] (Note 3) An action definition rule generation program according to Appendix 1 or Appendix 2, which extracts basic actions from among a plurality of the above-mentioned first action definition rules whose frequency of occurrence is equal to or greater than a predetermined value, or whose frequency of occurrence is up to the top predetermined number, as the first basic action.

[0058] (Note 4) An action definition rule generation program according to any one of the appendices 1 to 3, wherein, among a plurality of the first action definition rules, an evaluation value indicating the degree to which an action appears in the same first action definition rule as the first basic action is equal to or greater than a predetermined value, or basic actions up to the top predetermined number of such evaluation values, are extracted as second basic actions.

[0059] (Note 5) An action definition rule generation program according to any one of the appendices 1 to 4, which causes the computer to perform a process that includes receiving the first action definition rules entered as video search queries by multiple users, and storing the received multiple first action definition rules in the storage unit.

[0060] (Note 6) An action definition rule generation program according to any one of the appendices 1 to 5, which generates the definition name of the second action definition rule based on the definition name of the first basic operation and the definition name of the second basic operation.

[0061] (Note 7) A first extraction unit refers to a memory unit that stores multiple first action definition rules, each representing a combination of basic human actions, which are entered as a video search query, and extracts a first basic action from the multiple first action definition rules based on the frequency of each basic action included in the multiple first action definition rules. A second extraction unit extracts a second basic action related to the first basic action from among the plurality of action definition rules based on the co-occurrence with the extracted first basic action, A generation unit that generates a second action definition rule comprising the extracted first basic action and the second basic action as constituent elements, A behavior definition rule generator that includes this.

[0062] (Note 8) An action definition rule generation device according to Appendix 7, which includes a registration unit that recognizes basic actions included in an action definition rule input as a query from each of multiple videos, and registers the second action definition rule as a recognition rule for basic actions to search for a video corresponding to the query.

[0063] (Note 9) The first extraction unit is an action definition rule generating device according to Appendix 7 or Appendix 8, which extracts basic actions from a plurality of first action definition rules whose frequency of appearance is equal to or greater than a predetermined value, or whose frequency of appearance is up to the top predetermined number, as the first basic actions.

[0064] (Note 10) The action definition rule generating device according to any one of the appendices 7 to 9, wherein the second extraction unit extracts basic actions as second basic actions if the evaluation value indicating the degree to which a first action definition rule appears in the same first action definition rule as the first basic action is equal to or greater than a predetermined value, or if the evaluation value is up to a predetermined number of the highest values.

[0065] (Note 11) An action definition rule generation device according to any one of the appendices 7 to 10, including a receiving unit that receives the first action definition rules entered as video search queries by multiple users and stores the received multiple first action definition rules in the storage unit.

[0066] (Note 12) The action definition rule generating device according to any one of the appendices 7 to 11, wherein the generation unit generates the definition name of the second action definition rule based on the definition name of the first basic operation and the definition name of the second basic operation.

[0067] (Note 13) The system references a memory unit that stores multiple first action definition rules, each representing a combination of basic human actions, which are entered as a video search query. Based on the frequency of each basic action included in the multiple first action definition rules, the system extracts a first basic action from among the multiple action definition rules. Based on the co-occurrence with the extracted first basic action, a second basic action related to the first basic action is extracted from the plurality of action definition rules. A second action definition rule is generated, consisting of the extracted first basic action and the second basic action as constituent elements. A method for generating action definition rules to cause a computer to perform a process that includes the following.

[0068] (Note 14) A method for generating an action definition rule according to Appendix 13, which causes the computer to perform a process that includes recognizing the basic actions included in the action definition rule entered as a query from each of several videos, and registering the second action definition rule as a recognition rule for basic actions to search for videos corresponding to the query.

[0069] (Note 15) A method for generating behavior definition rules according to Appendix 13 or Appendix 14, wherein, among a plurality of the above-mentioned first behavior definition rules, basic actions whose frequency of appearance is equal to or greater than a predetermined value, or the top predetermined number of basic actions whose frequency of appearance is greater than or equal to the first basic action, are extracted as the first basic action.

[0070] (Note 16) A method for generating behavior definition rules according to any one of the appendices 13 to 15, wherein, among a plurality of the first behavior definition rules, basic actions whose evaluation value indicating the degree to which they appear in the same first behavior definition rule as the first basic action is equal to or greater than a predetermined value, or basic actions whose evaluation value is up to a predetermined number of the highest, are extracted as second basic actions.

[0071] (Note 17) A method for generating behavioral definition rules according to any one of the appendices 13 to 16, which causes the computer to perform a process that includes receiving the first behavioral definition rules entered as video search queries by multiple users, and storing the received multiple first behavioral definition rules in the storage unit.

[0072] (Note 18) A method for generating an action definition rule according to any one of the appendices 13 to 17, which generates a definition name for the second action definition rule based on the definition name for the first basic action and the definition name for the second basic action.

[0073] (Note 19) The system references a memory unit that stores multiple first action definition rules, each representing a combination of basic human actions, which are entered as a video search query. Based on the frequency of each basic action included in the multiple first action definition rules, the system extracts a first basic action from among the multiple action definition rules. Based on the co-occurrence with the extracted first basic action, a second basic action related to the first basic action is extracted from the plurality of action definition rules. A second action definition rule is generated, consisting of the extracted first basic action and the second basic action as constituent elements. A non-temporary storage medium that stores a program for generating action definition rules to cause a computer to perform a process that includes the following. [Explanation of symbols]

[0074] 100 Video Search System 10. Behavioral Definition Rule Generator 11 Reception Department 12 1st extraction part 13 Second extraction part 14 Generation part 15 Registration Department 16. Action Definition Rule Database 20 Video Search Device 21 Recognition part 22 Search Section 23 Basic Action Recognition Rule Database 24 Video Search Database 40 Computers 41 CPU 42 memory 43 Storage device 44 Input / Output Devices 45 R / W device 46 Communication I / F 47 Bus 49 Storage medium 50 Action Definition Rule Generator Program 51 Receiving process control command 52 First Extraction Process Control Command 53 Second Extraction Process Control Command 54 Generation process control instructions 55 Registered process control instruction 60 Information storage area

Claims

1. The system references a memory unit that stores multiple first action definition rules, each representing a combination of two or more basic actions of a person, which are entered as a video search query. Based on the frequency of each basic action included in the multiple first action definition rules, it extracts one or more first basic actions from the multiple first action definition rules. Based on the co-occurrence with each of the extracted one or more first basic actions, one or more second basic actions related to the one or more first basic actions are extracted from the plurality of first action definition rules. A second action definition rule is generated, comprising the extracted one or more first basic actions and one or more second basic actions as constituent elements. A program that generates action definition rules to cause a computer to perform a process that includes the following.

2. The action definition rule generation program according to claim 1, which causes the computer to perform a process that includes recognizing a basic action included in an action definition rule entered as a query from each of several videos, and registering the second action definition rule as a recognition rule for a basic action for searching for a video corresponding to the query.

3. The action definition rule generation program according to claim 1 or claim 2, wherein among the plurality of first action definition rules, basic actions whose occurrence frequency is greater than or equal to a predetermined value, or the top predetermined number of basic actions whose occurrence frequency is greater than or equal to the first basic action.

4. The action definition rule generation program according to claim 1 or claim 2, wherein, among the plurality of first action definition rules, basic actions whose evaluation value indicating the degree to which they appear in the same first action definition rule as the first basic action is equal to or greater than a predetermined value, or basic actions whose evaluation value is up to a predetermined number of the highest, are extracted as the second basic action.

5. An action definition rule generation program according to claim 1 or 2, which causes the computer to perform a process including receiving the first action definition rules entered as video search queries by multiple users, and storing the received plurality of first action definition rules in the storage unit.

6. An action definition rule generation program according to claim 1 or claim 2, which generates a definition name for a second action definition rule based on the definition name for the first basic operation and the definition name for the second basic operation.

7. A first extraction unit refers to a storage unit that stores multiple first action definition rules, each representing a combination of two or more basic actions of a person, which are entered as a query for video search, and extracts one or more first basic actions from the multiple first action definition rules based on the frequency of each basic action included in the multiple first action definition rules. A second extraction unit extracts one or more second basic actions related to the one or more first basic actions from the plurality of first action definition rules based on their co-occurrence with each of the extracted one or more first basic actions, A generation unit that generates a second action definition rule comprising the extracted one or more first basic actions and one or more second basic actions, A behavior definition rule generator that includes this.

8. The system references a memory unit that stores multiple first action definition rules, each representing a combination of two or more basic actions of a person, which are entered as a video search query. Based on the frequency of each basic action included in the multiple first action definition rules, it extracts one or more first basic actions from the multiple first action definition rules. Based on the co-occurrence with each of the extracted one or more first basic actions, one or more second basic actions related to the one or more first basic actions are extracted from the plurality of first action definition rules. A second action definition rule is generated, comprising the extracted one or more first basic actions and one or more second basic actions as constituent elements. A method for generating action definition rules that include the following processes for a computer to execute.

Images