Event detection device, event detection method, and program

Abstract

To provide an event detection device capable of monitoring an object on the basis of video data in which the object is captured, an event detection method, and a storage medium.SOLUTION: An event detection device 2000 is configured to: acquire one or more pieces of video data; generate object relationship information indicative of two or more operation association relationships between objects from the one or more pieces of video data; acquire event information indicative of a noteworthy event by using an operation association state sequence representing the state of a specific operation performed by at least one specific entity; and determine whether a noteworthy event has occurred on the basis of the object relationship information and the event information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure generally relates to an event detection device, an event detection method, and a storage medium. [Background technology]

[0002] There is a technology that uses video data to monitor an object. Patent Document 1 discloses a technology that detects the center of gravity of a person from each video frame and recognizes the behavior of the person based on the trajectory of the center of gravity of the person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-075802 A Summary of the Invention [Problem to be solved by the invention]

[0004] Information that can be used to monitor objects based on video data is not limited to the center of gravity of those objects. It is an object of the present disclosure to provide a novel technique for monitoring objects based on video data in which those objects are captured. [Means for solving the problem]

[0005] The event detection device of the present disclosure includes at least one memory configured to store instructions and at least one processor, which executes the instructions to obtain one or more video data, generate object relation information from the one or more video data indicative of two or more action-related relations between objects, obtain event information indicative of an event of interest by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject, and determine whether the event of interest has occurred based on the object relation information and the event information.

[0006] The event detection method of the present disclosure includes the steps of acquiring one or more video data, generating object relationship information indicating two or more action-related relationships between objects from the one or more video data, acquiring event information indicating an event of interest by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject, and determining whether the event of interest has occurred based on the object relationship information and the event information.

[0007] The storage medium of the present disclosure stores a program that causes a computer to execute the steps of acquiring one or more video data; generating object relationship information indicating two or more action-related relationships between objects from the one or more video data; acquiring event information indicating a target event by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; and determining whether the target event has occurred based on the object relationship information and the event information. Effect of the Invention

[0008] According to the present disclosure, a novel technique for monitoring an object based on video data in which the object is captured is provided. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an event detection device. [Diagram 2] 2 is a block diagram showing an example of a functional configuration of the event detection device; [Diagram 3] 1 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes an event detection device. [Figure 4] 5 is a flowchart illustrating an exemplary flow of a process executed by the event detection device. [Diagram 5] FIG. 11 is a diagram showing object-related information in a table format. [Figure 6] FIG. 11 is a diagram showing object information in a table format. [Figure 7] FIG. 2 illustrates an example of a scene graph. [Figure 8] 13 is a flowchart showing the flow of a process executed by a generation unit. [Figure 9] FIG. 11 is a diagram showing event information in a table format. [Figure 10] FIG. 13 is a diagram showing event information indicating the duration of each situation. [Figure 11] FIG. 13 illustrates event information indicating that a particular action is missing. [Figure 12] 5 is a flowchart illustrating an exemplary flow of a process performed by a decision unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. In addition, the same or corresponding elements in the drawings are indicated by the same reference numerals, and duplicated descriptions are omitted as appropriate to clarify the description. Unless otherwise specified, the predetermined information (e.g., a predetermined value or a predetermined threshold value) is pre-stored in a storage device accessible to a computer that uses the information, unless otherwise specified. Furthermore, unless otherwise specified, the storage unit is composed of one or more storage devices.

[0011] <Summary> Fig. 1 shows an overview of the event detection device 2000. Note that the overview shown in Fig. 1 shows an example of the operation of the event detection device 2000 in order to make the event detection device 2000 easier to understand, and is not intended to limit or narrow the range of operations that the event detection device 2000 can perform.

[0012] The event detection apparatus 2000 is used to detect motion-related events from one or more video data 10. The video data 10 is a sequence of video frames 20 generated by a camera 30. In some circumstances, more than one video data 10 is generated using more than one camera 30.

[0013] A motion-related event is any type of event that may be defined by two or more motion-related situations, each of which represents a situation in which a particular action is taken by a particular subject or a situation in which a particular action is not taken by a particular subject. Examples of motion-related events include purchasing, shoplifting, bicycle theft, luggage theft, loitering, stalking, or leaving luggage behind.

[0014] The action-related situation that defines an action-related event includes at least one situation in which a specific action is performed by a specific subject. The subject of the action may include any kind of object that can perform an action, such as a human being, an animal (e.g., a dog or a cat), a vehicle (e.g., a car, a bicycle, or an airplane), and a robot.

[0015] For example, the action-related event “Purchase” may be defined by the following three action-related circumstances: 1. A person picks up a product in a store. 2. The person stands in front of the register for a while. 3. The person passes through the exit.

[0016] To detect a particular motion-related event from the video data 10, the event detection apparatus 2000 may operate as follows: The event detection apparatus 2000 obtains the video data 10 and generates object relation information 40. The object relation information 40 represents two or more temporal motion-related relations between objects, each of which is a motion-related relation between objects that exists at a certain point in time or during a certain period of time.

[0017] Specifically, the object relation information 40 may include two or more combinations of 1) type of action, 2) subject of action, 3) object of action, and 4) time when the action was performed. Suppose there is a relationship in which person P1 picks up store product I1 from time T1 to T2. The object relation information 40 may represent this relationship with a combination of 1) type of action: pick up, 2) subject: person P1, 3) object: store product I1, and 4) time: from T1 to T2.

[0018] Moreover, the event detection device 2000 acquires event information 50, and the event information 50 represents a motion-related event detected by a sequence of motion-related situations. Hereinafter, the event to be detected is also referred to as a "notable event." It is assumed that the notable motion-related event is "purchase" as described above. In this case, the event information 50 may represent a sequence of the three motion-related situations described above.

[0019] The event detection device 2000 determines whether an event indicated by the event information 50 has occurred based on the object relation information 40. Specifically, the event detection device 2000 determines whether the object relation information 40 includes a sequence of action-related relations between objects that matches a sequence of action-related situations indicated by the event information 50.

[0020] When the object relation information 40 includes a sequence of action-related relations between objects that matches the sequence of action-related situations indicated by the event information 50, the event detection device 2000 determines that the action-related event indicated by the event information 50 has occurred. On the other hand, when the object relation information 40 does not include a sequence of action-related relations between objects that matches the sequence of action-related situations indicated by the event information 50, the event detection device 2000 determines that the action-related event indicated by the event information 50 has not occurred.

[0021] <Examples of effects> As described above, the event detection device 2000 uses the video data 10 to generate object relation information 40 indicating one or more temporal motion-related relations between objects. In addition, the event detection device 2000 obtains event information 50 indicating a motion-related event by a sequence of motion-related situations. The event detection device 2000 then determines whether or not a motion-related event indicated by the event information 50 has occurred based on the object relation information 40 and the event information 50. The event detection device 2000 provides a novel technique for monitoring objects based on video data in which the objects are captured. More specifically, a novel technique for detecting events related to the motions of objects from video data in which the objects are captured is provided.

[0022] Furthermore, Patent Document 1 focuses only on the movement of a single object, and does not disclose a technique for detecting a motion-related event in which two or more objects have some kind of relationship. On the other hand, the event detection device 2000 detects a motion-related relationship between objects from the video data 10, and is therefore capable of detecting a motion-related event involving some kind of relationship between objects. Thus, the event detection device 2000 can detect events that are more complex than events defined by the movement of a single object.

[0023] The event detection device 2000 will be described in more detail below.

[0024] <Example of functional configuration> 2 is a block diagram showing an example of a functional configuration of the event detection device 2000. The event detection device 2000 includes a first acquisition unit 2020, a generation unit 2040, a second acquisition unit 2060, and a determination unit 2080. The first acquisition unit 2020 acquires video data 10. The generation unit 2040 generates object-related information 40 from the video data 10. The second acquisition unit 2060 acquires event information 50. The determination unit 2080 determines, based on the object-related information 40, whether or not an action-related event indicated by the event information 50 has occurred.

[0025] <Example of hardware configuration> The event detection device 2000 may be realized by one or more computers. FIG. 3 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes the event detection device 2000. The computer 1000 may be any type of computer. For example, the computer 1000 is a stationary computer such as a personal computer (PC) or a server machine. In another example, the computer 1000 is a mobile computer such as a smartphone or a table terminal. In another example, the computer 1000 is an integrated circuit such as a system on chip (SoC). The computer 1000 may be a dedicated computer manufactured for realizing the event detection device 2000, or may be a general-purpose computer.

[0026] The event detection device 2000 may be realized by installing an application in the computer 1000. The application is realized by a program that causes the computer 1000 to function as the event detection device 2000. In other words, the program is an implementation form of the functional parts of the event detection device 2000.

[0027] There are various ways to obtain the program. For example, the program may be obtained from a storage medium (e.g., a DVD disk or a USB memory) on which the program is stored. In another example, the program may be downloaded from a server that manages the storage medium on which the program is stored.

[0028] 3, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage element such as a random access memory (RAM) or a read only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0029] The processor 1040 may be configured to load instructions of the above-mentioned programs from the storage device 1080 into the memory 1060 and execute those instructions to cause the computer 1000 to operate as the event detection apparatus 2000 .

[0030] The hardware configuration of the computer 1000 is not limited to that shown in Fig. 3. For example, as described above, the event detection device 2000 may be realized as a combination of multiple computers. In this case, the computers may be connected to each other via a network.

[0031] In some embodiments, the event detection device 2000 may be realized with two computers, where a first computer operates as a first acquisition unit 2020 and a generation unit 2040 to generate object-related information 40 from the video data 10, and a second computer operates as a second acquisition unit 2060 and a determination unit 2080 to determine, based on the object-related information 40, whether or not a motion-related event indicated by the event information 50 has occurred.

[0032] In this case, the event detection device 2000 can be regarded as an event detection system including a generating device and a detecting device. The generating device includes a first acquiring unit 2020 and a generating unit 2040. The detecting device includes a second acquiring unit 2060 and a determining unit 2080.

[0033] <Process flow> 4 is a flowchart showing an exemplary flow of processing executed by the event detection device 2000. The first acquisition unit 2020 acquires video data 10 (S102). The generation unit 2040 generates object-related information 40 from the video data 10 (S104). The second acquisition unit 2060 acquires event information 50 (S106). The determination unit 2080 determines, based on the object-related information 40, whether or not a motion-related event indicated by the event information 50 has occurred (S108).

[0034] <Getting video data 10: S102> The first acquisition unit 2020 acquires the video data 10 (S102). There are various methods for acquiring the video data 10. For example, the camera 30 is configured to transmit the video data 10 to the event detection device 2000. In this case, the first acquisition unit 2020 receives the video data 10 transmitted by the camera 30 and acquires the video data 10.

[0035] In another example, the camera 30 is configured to, upon generating the video frame 20, transmit the video frame 20 to the event detection device 2000. In this case, the first acquisition unit 2020 receives the video frame 20 transmitted by the camera 30, and generates the video data 10 from the received video frame 20.

[0036] In another example, the camera 30 is configured to store the video data 10 in a storage unit accessible to the event detection device 2000. In this case, the event detection device 2000 retrieves the video data 10 from this storage unit.

[0037] <About object-related information 40> As described above, the object relation information 40 represents two or more action-related relationships between objects with respect to actions. FIG. 5 shows the object relation information 40 in a table format. In FIG. 5, the object relation information 40 is represented as a table 100. The table 100 has columns named "Subject 102", "Object 104", "Action 106", and "Period 108". The action 106 indicates the type of action. The subject 102 indicates an identifier of an object that performs a corresponding action. The object 104 indicates an identifier of an object that performs a corresponding action by a corresponding subject. The period 108 indicates when the corresponding action-related relationship exists. Specifically, the period 108 is composed of two columns named "Start time 110" and "End time 112". The start time 110 indicates the time when the corresponding action-related relationship starts. The end time 112 indicates the time when the corresponding action-related relationship ends.

[0038] The subjects 102 and the objects 104 are represented by object identifiers. The identifiers of each object may be defined by other information, called "object information," that is also generated from the video data 10 by the event detection apparatus 2000. The object information may indicate the object identifier and the object type (e.g., a person, a store item, a bag, etc.) for each object detected from the video data 10.

[0039] The position of each object may change in the video data 10. Therefore, the object information preferably indicates a time and position pair for each object. In other words, the object information indicates a time sequence of positions for each object.

[0040] There are various ways to represent the position of an object. For example, the position of an object may be represented by the coordinates on the video frame 20 where the object is located. When the event detection device 2000 handles two or more video data 20, the position of an object may be represented by a pair of a camera identifier and the coordinates on the video frame 20 where the object is located.

[0041] In another example, the location of an object may be represented by coordinates on a map of the area imaged by one or more cameras 30. The map may be a two-dimensional map or a three-dimensional map.

[0042] In this case, the generating unit 2040 converts the coordinates on the video frame 20 where the object is located into coordinates on the map. By using the map, the positions of objects captured by different cameras 30 can be represented by coordinates on a unified coordinate space. In addition, by using this map, the event detection device 2000 can handle a camera 30 whose field of view is changeable (for example, a pan-tilt-zoom camera).

[0043] Fig. 6 shows object information in a table format. In Fig. 6, the object information is represented by a table 200. The table 200 includes columns named "identifier 202", "type 204", and "location 206". The identifier 202 indicates an identifier assigned to the corresponding object. The type 204 indicates the type of the corresponding object. The location 206 indicates a sequence of time and location pairs for the corresponding object.

[0044] The motion-related relationships between objects at a given moment can also be represented by a scene graph, where each object is represented by a node and the motion-related relationships between the objects are represented by edges. The object relationship information 40 can be said to represent a sequence of scene graphs. Thus, the event detection device 2000 can be used to search the sequence of scene graphs for motion-related events.

[0045] An example of a scene graph is shown in Figure 7. In the example shown in Figure 7, three objects have been detected: a person with identifier 001, a bag with identifier 002, and another person assigned identifier 003. Thus, the scene graph 60 includes three nodes representing person 001, bag 002, and person 003, respectively.

[0046] Person 001 and Bag 002 are connected to each other by an edge tagged with "Put Down" and pointing from Person 001 to Bag 002. This action association relationship represents Person 001 putting down Bag 002. Person 003 is not connected to anything, which means Person 003 does not perform any action.

[0047] <Generation of object-related information 40: S104> The generating unit 2040 generates object relation information 40 from the video data 10. For example, for each video frame 20 included in the video data 10, the generating unit 2040 performs object detection to generate object information, and then detects an action relation between the objects detected by the object detection.

[0048] 8 is a flowchart showing the flow of processing executed by the generating unit 2040. The generating unit 2040 initializes the object information and object relation information 40.

[0049] Steps S204 to S210 constitute a loop process L1 that is executed for each video frame 20 included in the video data 10. In step S204, the generation unit 2040 determines whether or not the loop process L1 has been executed for all video frames 20. If the loop process L1 has been executed for all video frames 20, the loop process L1 ends.

[0050] If the loop process L1 has not yet been performed on all video frames 20, the generation unit 2040 selects a video frame 20 for which the loop process L1 will be performed next. The video frame 20 selected here is a video frame 20 for which the loop process L1 has not yet been performed and is the video frame 20 for which the earliest generation time point has occurred (e.g., the video frame 20 with the smallest frame number). The video frame 20 selected here is referred to as "video frame i."

[0051] The generating unit 2040 performs object detection on the video frame i to detect an object from the video frame i, and updates the object information (S206). If an object detected from the video frame i has not been detected from the preceding video frame 20, the generating unit 2040 assigns a new identifier to the object and adds a new record for the object to the object information. If an object detected from the video frame i has been detected from the preceding video frame 20, the generating unit 2040 updates the record of the object in the object information by adding a time and position pair of the object to the record. The time of the pair represents the time when the video frame i was generated. The position of the pair represents the position of the object on the video frame i.

[0052] The generating unit 2040 performs detection of an action related relationship between objects detected from the video frame i, and updates the object relationship information 40 (S208). If an action related relationship between specific objects detected from the video frame i is also detected from the video frame (i-1), the generating unit 2040 updates the record of this action related relationship in the object relationship information 40 to increase the duration of this relationship. On the other hand, if an action related relationship between specific objects detected from the video frame i is not detected from the video frame (i-1), the generating unit 2040 generates a new record for this relationship and adds this record to the object relationship information 40.

[0053] Step 210 is the end of the loop process L1. Therefore, the generation unit 2040 next executes step S204.

[0054] <About Event Information 50> Event information 50 indicates a sequence of action-related circumstances that represent an action-related event of interest. Figure 9 shows event information 50 in tabular form. In Figure 9, event information 50 is represented as a table 300. Table 300 has columns named "Subject 302", "Object 304", and "Action 306".

[0055] Action 306 indicates a type of action. Subject 302 indicates a type of thing that performs the corresponding action. Object 304 indicates a type of thing on which the corresponding subject performs the corresponding action.

[0056] Note that the event information 50 may need to indicate different objects that belong to the same type, and therefore the subject 302 and the object 304 need to distinguishably indicate objects of the same type.

[0057] For example, table 300 shown in FIG. 9 shows an event "Luggage Theft." Luggage theft includes a victim, who is the owner of the luggage, and a criminal who steals the luggage. The first row shows an action-related situation in which the victim puts down the luggage. To represent this situation, the subject 302, object 304, and action 306 in the first row show "Person:1," "Luggage:1," and "Put Down," respectively. The value "Person:1" represents the first person involved in this event. The value "Luggage:1" represents the first luggage involved in this event.

[0058] The second line represents a situation where the criminal picks up the package. To represent this situation, the subject 302, object 304, and action 306 are shown as "Person:2", "Package:1", and "Pick up", respectively. The value "Person:2" represents a second person involved in this event, who is a person other than the first person "Person:1". The value "Package:1" represents the first package, which is the same as that shown in the first line.

[0059] The third row represents an action-related situation in which a criminal carries away a package. To represent this situation, the subject 302, object 304, and action 306 in the third row indicate "Person:2", "Package:1", and "Carry", respectively. The value "Person:2" represents a second person who is the same person as the person shown in the second row. The value "Package:1" represents a first package that is the same as the package shown in the first and second rows.

[0060] Table 300 may include additional information. For example, table 300 may include another column entitled "Duration" that indicates the minimum length of time that the corresponding condition lasts.

[0061] For example, suppose that the last situation of the luggage theft represented by the third row of table 300 shown in Fig. 9 lasted only one second. This means that the second person carries the luggage just a few meters away from where the first person places the luggage. In this case, the second person may not be trying to steal the luggage.

[0062] The "Duration" column can be used to avoid this type of false positive. Figure 10 shows event information 50 indicating the duration for each situation. In Figure 10, table 300 further includes a column named "Duration 308," which indicates the duration of the corresponding action that was performed.

[0063] The duration 308 in the third row of table 300 in Figure 10 shows 10 seconds, meaning that the second person carries the package for at least 10 seconds, effectively removing it from where the first person left it. In this case, it is highly likely that the second person is attempting to steal the package.

[0064] In another example, the event information 50 may indicate that a particular action is missing. An example of an action-related event that may be defined when a particular action is missing is "shoplifting." Shoplifting may be defined as follows: 1. A person picks up a product in a store. 2. The person does not stand in front of the cash register for a while. 3. The person passes through the exit.

[0065] When a person shoplifts an item from a store, that person does not pay for the item, and therefore does not stand in front of the cash register. Therefore, shoplifting is missing the action of the person standing in front of the cash register for a while. The second item in the shoplifting list above represents the missing action of "the person stands in front of the cash register for a while."

[0066] In order to enable the event information 50 to represent an action-related situation in which a specific action is missing, the table 300 may include a "missing flag." The missing flag represents whether or not the corresponding action has been performed. Specifically, a missing flag indicating "true" represents that the corresponding action has not been performed (i.e., the corresponding action is missing). On the other hand, a missing flag indicating "false" represents that the corresponding action has been performed (i.e., the corresponding action is not missing). Hereinafter, an action-related situation in which a specific action is not performed is also referred to as a "missing situation."

[0067] Figure 11 shows event information 50 indicating that a particular action is missing. The table 300 shown in Figure 11 further includes a column named "missing flag 310."

[0068] Table 300 shown in FIG. 11 represents an event "shoplifting". Specifically, the first row of table 300 shown in FIG. 11 represents a situation in which a first person picks up a first store item. To represent this situation, subject 302, object 304, action 306, duration 308, and absence flag 310 indicate "person: 1", "store item: 1", "pick up", "1 second", and "false", respectively. Since the action of picking up was performed, absence flag 310 indicates "false", i.e., the corresponding action is not absent.

[0069] 11 represents a situation where the first person has not stood in front of the cash register for 10 seconds or more. To represent this situation, the subject 302, the object 304, the action 306, the duration 308, and the absence flag 310 indicate "person: 1", "cash register: 1", "standing in front", "10 seconds", and "true", respectively. Since the action of standing in front of the cash register has not been performed, the absence flag 310 indicates "true", i.e., that the corresponding action is absent.

[0070] 11 represents a situation where the first person passes through the exit for at least 1 second. To represent this situation, the subject 302, the object 304, the action 306, the duration 308, and the missing flag 310 indicate "person: 1", "exit: 1", "pass", "1 second", and "false", respectively. Since the action of passing through the exit has been performed, the missing flag 310 indicates "false", i.e., the corresponding action is not missing.

[0071] <Getting event information 50: S106> The second acquisition unit 2060 acquires the event information 50 (S106). In some embodiments, the second acquisition unit 2060 receives a query aimed at detecting a particular event from the video data 10. In this case, the query includes the event information 50 indicating a motion-related event of interest. The query may be transmitted by a user terminal used by a user of the event detection device 2000.

[0072] In some embodiments, one or more action-related events of interest are specified in advance. In this case, for each event of interest, event information 50 representing the event of interest is stored in advance in a storage unit accessible from the event detection device 2000. In this case, the second acquisition unit 2060 acquires one or more pieces of event information 50 from the storage unit. Next, the determination unit 2080 performs detection of the event of interest represented by the event information 50 for each event information 50.

[0073] <Event detection: S108> The determination unit 2080 determines whether or not a noted action-related event has occurred based on the object relation information 40 and the event information 50 (S108). To this end, the determination unit 2080 determines whether or not the object relation information 40 includes a sequence of action-related relations that matches a sequence of action-related situations indicated by the event information 50.

[0074] Hereinafter, for the sake of easy understanding of the description, first, it is assumed that the event information 50 does not include operation-related situations where no specific operation is performed.

[0075] The object relationship information 40 includes a sequence of 20 relationships represented by R[] = {R[1], R[2],..., R

[20] }, and it is assumed that the event information 50 includes a sequence of three situations represented by S[] = {S[1], S[2], S[3]}. In this case, if R[] includes R[i] that matches S[1], R[j] that matches S[2], and R[k] that matches S[3], and i < j < k is satisfied, the determination unit 2080 determines that the object relationship information 40 includes a sequence {R[i], R[j], R[k]} that matches the sequence {S[1], S[2], S[3]} indicated by the event information 50. Therefore, in this case, the determination unit 2080 determines that the operation-related event of interest has occurred.

[0076] Hereinafter, the sequence of operation-related relationships indicated by the object relationship information 40 is represented by R[] = {R[1],..., R[Nr]}. In addition, the sequence of operation-related situations indicated by the event information 50 is represented by S[] = {S[1],..., S[Ns]}. Note that both Nr and Ns are integers greater than 1.

[0077] Overall, the determination unit 2080 attempts to sequentially detect each operation-related situation {S[1],..., S[Ns]} from each operation-related relationship {R[1],..., R[Nr]}. First, the determination unit 2080 searches {R[1],..., R[Nr]} for the first situation S[1] and detects R[i] as the relationship that matches S[1]. Next, the determination unit 2080 searches the partial sequence {R[i + 1],.., R[Nr]} for the second situation S[2] and detects R[j] as the relationship that matches S[2]. Then, the determination unit 2080 searches the partial sequence {R[j + 1],.., R[Nr]} for the third situation S[3] and detects R[k] as the relationship that matches S[3]. This process is repeatedly executed until the determination unit 2080 succeeds in finding the corresponding operation-related relationship for the last situation S[Ns], or until the determination unit 2080 fails to find an operation-related relationship for any situation.

[0078] In some cases, the object relationship information 40 includes two or more operation-related relationships that match the situation S[a] (1 <= a < Ns). In this case, the determination unit 2080 executes the subsequent search for the operation-related relationship that matches the situation S[a + 1] for each of the multiple operation-related relationships that match S[a].

[0079] Assume that the object relation information 40 includes {R[1],...,R

[20] }, the event information 50 includes {S[1],S[2]}, R[3] and R

[10] match S[1], and R[5] matches S[2]. In this case, the determination unit 2080 first detects R[3] and R

[10] by searching for relations that match S[1] for {R[1],...,R

[20] }. Next, the determination unit 2080 detects R[5] by searching for relations that match S[2] for {R[4],...,R

[20] }. In addition, the determination unit 2080 searches for relations that match S[2] for {R

[11] ,...,R

[20] } and does not detect anything. As a result, the determining unit 2080 determines that the object relation information 40 includes the sequence {R[3], R[5]} that matches the sequence {S[1], S[2]}.

[0080] To determine whether the action-related relationship and the action-related situation match, the determination unit 2080 compares corresponding elements with each other. For example, the determination unit 2080 determines whether the subject 102 of the action-related relationship matches the subject 302 of the action-related situation, whether the object 104 of the action-related relationship matches the object 304 of the action-related situation, whether the action 106 of the action-related relationship matches the action 306 of the action-related situation, and whether the length of the period 108 of the action-related relationship matches the duration 308 of the action-related situation.

[0081] If the subject 102, the object 104, the action 106, and the length of the period 108 of the action-related relationship match the subject 302, the object 304, the action 306, and the duration 308, respectively, the determination unit 2080 determines that the action-related relationship matches the action-related situation. On the other hand, if the subject 102 and the subject 302 do not match, if the object 104 and the object 304 do not match, if the action 106 and the action 306 do not match, or if the length of the period 108 and the duration 308 do not match, the determination unit 2080 determines that the action-related relationship matches the action-related situation.

[0082] The determination of whether an action-related relationship matches an action-related situation is made under the constraints generated by the previous matching. When matching is made for the first situation, there are no constraints yet. Therefore, subject 102 and subject 302 are determined to match if they are the same type. Similarly, object 104 and object 304 are determined to match if they are the same type.

[0083] Assume that the first action-related situation S[1] represents "Person 1 picks up store product 1". In addition, the action-related relationship R[i] represents "Object 003, whose type is person, picks up object 007, whose type is store product". In this example, the determination unit 2080 determines that R[1] matches S[1]. Next, the determination unit 2080 generates constraints such as "Person 1 is object 003", "Persons other than Person 1 are not object 003", "Store product 1 is object 007", and "Store products other than store product 1 are not object 007".

[0084] In the next matching of the situation {S[2],...,S[Ns]}, the determining unit 2080 performs matching taking these constraints into account. For example, only if the subject 102 indicates "object 003", it is determined that the subject 102 matches with the subject 302 indicating "person 1". In addition, only if the subject 102 indicates an object whose type is person and is not object 003, it is determined that the subject 102 matches with the subject 302 indicating a person other than person 1 (e.g., person 2).

[0085] The determination unit 2080 generates constraints cumulatively. For example, after matching for situation S[2], the determination unit 2080 generates additional constraints based on matching for situation S[2]. Then, the determination unit 2080 performs matching for situation S[3] by taking into account both the constraints generated based on matching for situation S[1] and the constraints generated based on matching for situation S[2].

[0086] Figure 12 is a flow chart illustrating an exemplary flow of the processing performed by the determining unit 2080. Specifically, Figure 12 illustrates the flow of a recursive procedure named "search()". The procedure search() takes two arguments: target_sit represents the action-related situation that is the target of the search, and detected_rel_sec represents the sequence of action-related relations that are determined to match the sequence of action-related situations preceding target_sit.

[0087] The determination unit 2080 first executes search() with target_sit as the first situation S[1] and detected_rel_sec as empty. In step S302, the determination unit 2080 detects one or more action-related relations that match target_sit from the sequence of relations after the last relation indicated by detecte_rel_sec. Assume that detected_rel_sec includes {R[3], R[5]}. In this case, the determination unit 2080 searches for target_sit in {R[6],...,R[Nr]}. Note that when detected_rel_sec is empty, the determination unit 2080 searches for target_sit in the entire sequence {R[1],...,R[Nr]}.

[0088] Steps S304 to S316 constitute a loop process L2 that is performed for each of the action related relations detected in S302. In step S304, the determination unit 2080 determines whether or not the loop process L2 has been performed for all the action related relations detected in step S302. If the loop process L2 has been performed for all the action related relations detected in step S302, the execution of the current procedure search() ends. On the other hand, if there is at least one action related relation for which the loop process L2 has not yet been performed among the action related relations detected in step S302, the determination unit 2080 selects an action related relation for which the loop process L2 has not yet been performed among the action related relations detected in step S302. The action related relation selected here is represented by rd. Note that if no action related relation is detected in S302, the determination unit 2080 does not perform the loop process L2.

[0089] In step S306, the determination unit 2080 adds a relation rd to detected_rel_sec to define a new sequence named "new_sec". The sequence new_sec represents one of the sequences of action-related relations that match the sequence of action-related situations from the first situation to the target_sit.

[0090] Next, the determination unit 2080 determines whether or not the target_sit is the last action-related situation in the sequence indicated by the event information 50 (S308). If the target_sit is the last action-related situation in the sequence indicated by the event information 50, the determination unit 2080 outputs new_sec as a sequence of action-related relationships that matches the entire sequence of action-related situations indicated by the event information 50.

[0091] On the other hand, if target_sit is not the last action-related situation in the sequence indicated by the event information 50, the determination unit 2080 performs the next matching for the action-related situation next to target_sit. To this end, the determination unit 2080 defines a variable "next_sit" to represent the action-related situation next to target_sit in the event information 50 (S312). Next, the determination unit 2080 sets the first argument target_sit to next_sit, sets the second argument detected_rel_sec to new_sec, and recursively executes the procedure search() (S314).

[0092] Step S316 is the end of the loop process L2. Therefore, when the determination unit 2080 reaches step S316, the determination unit 2080 next executes step S304.

[0093] Hereinafter, a method of handling the event information 50 including one or more missing situations will be described. In this case, the determination unit 2080 may first search the object relation information 40 for a sequence of action-related situations indicated by the event information 50 from which the missing situations have been removed. For example, assume that the event information 50 includes S[]={S[1], S[2], S[3]}, where S[2] is a missing situation. In this case, the determination unit 2080 first determines whether the object relation information 40 includes a sequence of action-related relationships that matches {S[1], S[3]}, which is a sequence of situations obtained by removing S[2] from S[].

[0094] Next, the determination unit 2080 determines whether the partial sequence obtained from the object relation information 40 for each missing situation includes a situation opposite to the missing situation. This partial sequence is a sequence of action-related relations from the action-related relation next to the action-related relation that matches the situation before the missing situation to the action-related relation before the missing situation that matches the situation after the missing situation. For example, the event information 50 includes S[]={S[1], S[2], S[3]}, where S[2] is a missing situation, and action-related relations R[3] and R

[10] match the action-related situations S[1] and S[3], respectively. In this case, the partial sequence is {R[4],..., R[9]} (R[4] is next to R[3] that matches S[1], and R[9] is before R

[10] that matches S[3]).

[0095] The opposite situation of an absence situation is a situation in which the action specified by the absence situation is performed by the subject specified by the absence situation. For example, if the absence situation represents a situation in which "Person 1 is not standing in front of Cash Register 1 for at least 10 seconds," the opposite situation of the absence situation is a situation in which "Person 1 is standing in front of Cash Register 1 for at least 10 seconds."

[0096] When it is determined that the partial sequence does not include an opposite situation to the missing situation for all missing situations, the determination unit 2080 determines that the object relation information 40 includes a sequence of action-related relations that matches the sequence of action-related situations indicated by the event information 50. On the other hand, when it is determined that the partial sequence includes an opposite situation to the missing situation for at least one missing situation, the determination unit 2080 determines that the object relation information 40 does not include a sequence of action-related relations that matches the sequence of action-related situations indicated by the event information 50.

[0097] For example, the event information 50 includes S[]={S[1], S[2], S[3]}, where S[2] is a missing situation, and the action-related relations R[3] and R

[10] are determined to match the action-related situations S[1] and S[3], respectively. In this case, the determination unit 2080 determines whether the sequence {R[4], ..., R[9]} includes an opposite situation to S[2]. If it is determined that the sequence {R[4], ..., R[9]} does not include an opposite situation to S[2], the determination unit 2080 determines that the object-related information 40 includes a sequence {R[3], R

[10] } that matches the sequence S[]. On the other hand, if it is determined that the sequence {R[4],...,R[9]} contains the opposite situation to S[2], the determination unit 2080 determines that the sequence {R[3],R

[10] } does not match the sequence S[].

[0098] <Result output> The event detection device 2000 may output information related to the result of the determination performed by the determining unit 2080 (hereinafter referred to as “output information”). When it is determined that the motion-related event indicated by the event information 50 has not occurred, the event detection device 2000 generates output information indicating that the video data 10 does not include the motion-related event indicated by the event information 50.

[0099] Upon determining that a motion-related event indicated by the event information 50 has occurred, the event detection apparatus 2000 generates output information including information about the event. For example, the output information may indicate a name of the detected event and a sequence of motion-related relationships that is determined to match the sequence of motion-related situations indicated by the event information 50. The name of the event may be defined in the event information 50.

[0100] For each action-related relationship determined to match a sequence of action-related circumstances indicated by the event information 50, the output information may include a sequence of video frames 20 (i.e., short clips extracted from the video data 10) in which the action-related relationship was detected. For example, suppose a package theft is detected in the video data 10. In this case, the output information may include a first short clip of a first person being filmed placing the package, a second short clip of a second person being filmed picking up the package, and a third short clip of a second person being filmed carrying the package.

[0101] Each event may have one or more objects of interest. For example, if a criminal event is detected, the criminal is the object of interest. The output information is preferably indicative of information about the criminal. The information about the object may include an image of the object and characteristics of the object. The image of the object may be extracted from the video frame 20 in which the object is detected.

[0102] The object characteristics depend on the type of object. Person characteristics may include age, gender, and clothing characteristics. Package characteristics may include color, shape, and brand.

[0103] There are various methods for outputting the output information. For example, the event detection device 2000 may store the output information in a storage unit. In another example, the event detection device 2000 may output the output information to a display device, thereby causing the contents of the output information to be displayed on the display device. In another example, the event detection device 2000 may transmit the output information to another device, such as a mobile device carried by a security guard or a PC used in a security room.

[0104] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be appropriately combined with other embodiments.

[0105] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but to one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create an embodiment not explicitly shown or described, for example. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0106] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0107] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. <Additional Notes> (Appendix 1) An event detection device, at least one memory configured to store instructions; Execute the command, Obtain one or more video data, generating object relation information from the one or more video data indicative of two or more motion-related relations between objects; obtaining event information indicating an event of interest according to a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information; and at least one processor configured to: (Appendix 2) 2. The event detection device of claim 1, wherein the event of interest is determined to have occurred if the object relationship information includes a sequence of the action-related relationships that match the sequence of the action-related situations indicated by the event information. (Appendix 3) The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action, The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; The event detection device described in Appendix 2, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively. (Appendix 4) the event information includes a first action-related situation and a second action-related situation in this order; The event detection device according to claim 2 or 3, wherein the event of interest is determined to have occurred when the object relationship information includes, in that order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation. (Appendix 5) the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; Each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; The second action-related situation represents a situation in which a specific action is not performed by a specific subject; The event detection device described in Appendix 2 or 3, wherein the event of interest is determined to have occurred when the object relationship information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and when the object relationship information does not include a action-related relationship that matches a action-related situation that is opposite to the second action-related situation. (Appendix 6) 4. The event detection device according to claim 1, wherein the event information distinguishably indicates two or more objects of the same type but different from each other. (Appendix 7) 4. An event detection apparatus according to any one of claims 1 to 3, wherein the object relationship information represents a sequence of scene graphs, each of which represents a relationship between objects existing at a certain point in time or over a certain period of time. (Appendix 8) acquiring one or more video data; generating object relation information indicative of two or more action-related relations between objects from the one or more video data; acquiring event information indicative of an event of interest according to a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information. (Appendix 9) 9. The event detection method of claim 8, wherein the event of interest is determined to have occurred if the object relationship information includes a sequence of the action-related relationships that match the sequence of the action-related situations indicated by the event information. (Appendix 10) The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action, The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; The event detection method of claim 9, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively. (Appendix 11) the event information includes a first action-related situation and a second action-related situation in this order; The event detection method according to claim 9 or 10, wherein the event of interest is determined to have occurred when the object relationship information includes, in this order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation. (Appendix 12) the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; Each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; The second action-related situation represents a situation in which a specific action is not performed by a specific subject; The event detection method described in Appendix 9 or 10, wherein the event of interest is determined to have occurred when the object relation information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and when the object relation information does not include a action-related relationship that matches a action-related situation that is opposite to the second action-related situation. (Appendix 13) 11. The event detection method according to any one of appendixes 8 to 10, wherein the event information distinguishably indicates two or more objects of the same type but different from each other. (Appendix 14) 11. The event detection method of any one of claims 8 to 10, wherein the object relationship information represents a sequence of scene graphs, each of which represents a relationship between objects existing at a certain point in time or over a certain period of time. (Appendix 15) acquiring one or more video data; generating object relation information indicative of two or more action-related relations between objects from the one or more video data; acquiring event information indicative of an event of interest according to a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; and determining whether the event of interest has occurred based on the object-related information and the event information. (Appendix 16) 16. A storage medium as described in Appendix 15, wherein the event of interest is determined to have occurred if the object relationship information includes a sequence of the action-related relationships that match the sequence of the action-related situations indicated by the event information. (Appendix 17) The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action, The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; A storage medium as described in Appendix 16, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively. (Appendix 18) the event information includes a first action-related situation and a second action-related situation in this order; A storage medium as described in Appendix 16 or 17, wherein the event of interest is determined to have occurred when the object relationship information includes, in that order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation. (Appendix 19) the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; Each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; The second action-related situation represents a situation in which a specific action is not performed by a specific subject; The storage medium of claim 16 or 17, wherein the event of interest is determined to have occurred when the object relationship information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and the object relationship information does not include a action-related relationship that matches a action-related situation that is opposite to the second action-related situation.

[0108] This application claims priority to Singapore Patent Application No. 10202302052W, filed on July 20, 2023, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0109] 10 Video Data 20 Video Frames 30 Camera 40 Object-related information 50 Event Information 60 Scene Graph 100 Tables 102 Subject 104 Object 106 Operation 108 period 110 Start time 112 End time 200 Tables 202 Identifier 204 Type 206 position 300 Tables 302 Subject 304 Object 306 operation 308 Duration 310 Missing Flag 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Event Detector 2020 1st Acquisition Division 2040 Generation part 2060 2nd Acquisition Department 2080 Judgment section

Claims

1. An event detection device, at least one memory configured to store instructions; Execute the command, Obtaining one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; obtaining event information indicative of an event of interest according to a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information; and at least one processor configured to:

2. The event detection apparatus of claim 1 , wherein the event of interest is determined to have occurred if the object relation information includes a sequence of the action-related relations that match the sequence of the action-related circumstances indicated by the event information.

3. The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action, The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; The event detection device of claim 2, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively.

4. the event information includes a first action-related situation and a second action-related situation in this order; The event detection device according to claim 2 or 3, wherein the event of interest is determined to have occurred when the object relationship information includes, in this order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation.

5. the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; The second action-related situation represents a situation in which a specific action is not performed by a specific subject; The event detection device of claim 2 or 3, wherein the event of interest is determined to have occurred when the object relationship information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and when the object relationship information does not include an action-related relationship that matches an action-related situation that is opposite to the second action-related situation.

6. The event detection device according to claim 1 , wherein the event information distinguishably indicates two or more objects of the same type but different from each other.

7. The event detection apparatus according to claim 1 , wherein the object relation information represents a sequence of scene graphs, each of which represents a relation between objects existing at a certain point in time or over a certain period of time.

8. acquiring one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; acquiring event information indicative of an event of interest according to a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information.

9. The method of claim 8 , wherein the event of interest is determined to have occurred if the object relation information includes a sequence of the action-related relations that match the sequence of the action-related circumstances indicated by the event information.

10. acquiring one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; acquiring event information indicative of an event of interest according to a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; and determining whether the event of interest has occurred based on the object-related information and the event information.

Images