Event detection device, event detection method, and program

Abstract

To provide an event detection device capable of monitoring an object on the basis of video data in which the object is captured, an event detection method, and a storage medium.SOLUTION: An event detection device 2000 is configured to: acquire one or more pieces of video data; generate object relationship information indicative of two or more operation association relationships between objects from the one or more pieces of video data; acquire event information indicative of a noteworthy event by using an operation association state sequence representing the state of a specific operation performed by at least one specific entity; and determine whether a noteworthy event has occurred on the basis of the object relationship information and the event information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure generally relates to an event detection device, an event detection method, and a storage medium. [Background technology]

[0002] There is a technology that uses video data to monitor an object. Patent Document 1 discloses a technology that detects the center of gravity of a person from each video frame and recognizes the behavior of the person based on the trajectory of the center of gravity of the person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-075802 Summary of the Invention [Problem to be solved by the invention]

[0004] The information that can be used to monitor objects based on video data is not limited to the centers of gravity of those objects. An object of the present disclosure is to provide a novel technique for monitoring objects based on video data in which those objects are captured. [Means for solving the problem]

[0005] The event detection device of the present disclosure includes at least one memory configured to store instructions and at least one processor, which executes the instructions to acquire one or more video data, generate object relation information from the one or more video data indicating two or more action-related relations between objects, acquire event information indicating an event of interest by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject, and determine whether the event of interest has occurred based on the object relation information and the event information.

[0006] The event detection method of the present disclosure includes the steps of acquiring one or more video data, generating object relation information indicating two or more action-related relations between objects from the one or more video data, acquiring event information indicating a target event by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject, and determining whether the target event has occurred based on the object relation information and the event information.

[0007] The storage medium of the present disclosure stores a program that causes a computer to execute the following steps: acquiring one or more video data; generating object relationship information indicating two or more action-related relationships between objects from the one or more video data; acquiring event information indicating a target event by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; and determining whether the target event has occurred based on the object relationship information and the event information. [Effects of the Invention]

[0008] According to the present disclosure, a novel technique for monitoring an object based on video data in which the object is captured is provided. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an event detection device. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of the event detection device. [Figure 3] FIG. 1 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes an event detection device. [Figure 4] 10 is a flowchart illustrating an exemplary flow of a process executed by the event detection device. [Figure 5] FIG. 10 is a diagram showing object-related information in a table format. [Figure 6] FIG. 10 is a diagram showing object information in a table format. [Figure 7] FIG. 2 illustrates an example of a scene graph. [Figure 8] 10 is a flowchart showing the flow of processing executed by a generation unit. [Figure 9] FIG. 10 is a diagram showing event information in a table format. [Figure 10] FIG. 10 is a diagram showing event information indicating the duration of each situation. [Figure 11] FIG. 10 illustrates event information indicating that a specific action is missing. [Figure 12] 10 is a flowchart illustrating an exemplary flow of processing performed by a decision unit. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding elements are designated by the same reference numerals, and redundant explanations are omitted as appropriate for clarity. Unless otherwise specified, predetermined information (e.g., predetermined values ​​or predetermined threshold values) is pre-stored in a storage device accessible by a computer that uses the information. Furthermore, unless otherwise specified, the storage unit is composed of one or more storage devices.

[0011] <Summary> Fig. 1 shows an overview of the event detection device 2000. Note that the overview shown in Fig. 1 shows an example of the operation of the event detection device 2000 in order to make the event detection device 2000 easier to understand, and is not intended to limit or narrow the range of operations that the event detection device 2000 can perform.

[0012] The event detection apparatus 2000 is used to detect motion-related events from one or more video data 10. The video data 10 is a sequence of video frames 20 generated by a camera 30. In some situations, two or more video data 10 are generated using two or more cameras 30.

[0013] A motion-related event is any type of event that can be defined by two or more motion-related situations, where each motion-related situation represents a situation in which a specific action is taken by a specific subject or a situation in which a specific action is not taken by a specific subject. Examples of motion-related events include purchasing, shoplifting, bicycle theft, luggage theft, loitering, stalking, or leaving luggage behind.

[0014] The action-related situation that defines an action-related event includes at least one situation in which a specific action is performed by a specific subject. The subject of the action may include any kind of object that can perform the action, such as a human, an animal (e.g., a dog or a cat), a vehicle (e.g., a car, a bicycle, or an aircraft), and a robot.

[0015] For example, the action-related event "purchase" can be defined by the following three action-related situations: 1. A person picks up a product from a store. 2. The person stands in front of the register for a while. 3. The person passes through the exit.

[0016] To detect a specific motion-related event from the video data 10, the event detection device 2000 may operate as follows: The event detection device 2000 acquires the video data 10 and generates object relation information 40. The object relation information 40 represents two or more temporal motion-related relations between objects, each of which is a motion-related relation between objects that exists at a certain point in time or during a certain period of time.

[0017] Specifically, the object relationship information 40 may include two or more combinations of 1) the type of action, 2) the subject of the action, 3) the object of the action, and 4) the time when the action was performed. Suppose there is a relationship in which person P1 picks up store product I1 from time T1 to time T2. The object relationship information 40 may represent this relationship as a combination of 1) the type of action: picking up, 2) the subject: person P1, 3) the object: store product I1, and 4) the time: from T1 to T2.

[0018] The event detection device 2000 also acquires event information 50, which represents a motion-related event detected based on a sequence of motion-related situations. Hereinafter, the event to be detected will also be referred to as a "notable event." Assume that the notable motion-related event is "purchase" as described above. In this case, the event information 50 may represent a sequence of the three motion-related situations described above.

[0019] The event detection device 2000 determines whether an event indicated by the event information 50 has occurred, based on the object relation information 40. Specifically, the event detection device 2000 determines whether the object relation information 40 includes a sequence of action-related relations between objects that matches the sequence of action-related situations indicated by the event information 50.

[0020] If the object relation information 40 includes a sequence of action-related relations between objects that matches the sequence of action-related situations indicated by the event information 50, the event detection device 2000 determines that the action-related event indicated by the event information 50 has occurred. On the other hand, if the object relation information 40 does not include a sequence of action-related relations between objects that matches the sequence of action-related situations indicated by the event information 50, the event detection device 2000 determines that the action-related event indicated by the event information 50 has not occurred.

[0021] <Examples of effects> As described above, the event detection device 2000 uses the video data 10 to generate object relation information 40 indicating one or more temporal action-related relations between objects. In addition, the event detection device 2000 acquires event information 50 indicating an action-related event by a sequence of action-related situations. Then, based on the object relation information 40 and the event information 50, the event detection device 2000 determines whether or not the action-related event indicated by the event information 50 has occurred. The event detection device 2000 provides a novel technique for monitoring objects based on video data in which the objects are captured. More specifically, a novel technique for detecting events related to the actions of objects from video data in which the objects are captured is provided.

[0022] Furthermore, Patent Document 1 focuses only on the movement of a single object and does not disclose a technique for detecting a motion-related event involving some kind of relationship between two or more objects. On the other hand, event detection device 2000 detects a motion-related relationship between objects from video data 10, and is therefore capable of detecting a motion-related event involving some kind of relationship between the objects. Therefore, event detection device 2000 can detect events that are more complex than events defined by the movement of a single object.

[0023] The event detection device 2000 will be described in more detail below.

[0024] <Example of functional configuration> 2 is a block diagram showing an example of the functional configuration of the event detection device 2000. The event detection device 2000 includes a first acquisition unit 2020, a generation unit 2040, a second acquisition unit 2060, and a determination unit 2080. The first acquisition unit 2020 acquires video data 10. The generation unit 2040 generates object-related information 40 from the video data 10. The second acquisition unit 2060 acquires event information 50. The determination unit 2080 determines, based on the object-related information 40, whether or not an action-related event indicated by the event information 50 has occurred.

[0025] <Example of hardware configuration> The event detection device 2000 may be realized by one or more computers. FIG. 3 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes the event detection device 2000. The computer 1000 may be any type of computer. For example, the computer 1000 is a desktop computer such as a personal computer (PC) or a server machine. In another example, the computer 1000 is a smartphone and tablet The computer 1000 may be a mobile computer such as a terminal. In another example, the computer 1000 may be an integrated circuit such as a system on chip (SoC). The computer 1000 may be a dedicated computer manufactured for implementing the event detection device 2000, or may be a general-purpose computer.

[0026] The event detection device 2000 may be realized by installing an application on the computer 1000. The application is realized by a program that causes the computer 1000 to function as the event detection device 2000. In other words, the program is an implementation of the functional units of the event detection device 2000.

[0027] There are various ways to acquire the program. For example, the program may be acquired from a storage medium (e.g., a DVD disk or a USB memory) on which the program is stored. In another example, the program may be downloaded from a server that manages the storage medium on which the program is stored.

[0028] 3, a computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main memory element such as a random access memory (RAM) or a read-only memory (ROM). The storage device 1080 is an auxiliary memory element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0029] The processor 1040 may be configured to load instructions of the above-mentioned programs from the storage device 1080 into the memory 1060 and execute those instructions to cause the computer 1000 to operate as the event detection device 2000 .

[0030] The hardware configuration of the computer 1000 is not limited to that shown in Fig. 3. For example, as described above, the event detection device 2000 may be realized as a combination of multiple computers. In this case, the computers may be connected to each other via a network.

[0031] In some embodiments, the event detection device 2000 may be realized by two computers, where a first computer operates as a first acquisition unit 2020 and a generation unit 2040 to generate object-related information 40 from the video data 10, and a second computer operates as a second acquisition unit 2060 and a determination unit 2080 to determine, based on the object-related information 40, whether or not an action-related event indicated by the event information 50 has occurred.

[0032] In this case, the event detection device 2000 can also be regarded as an event detection system including a generating device and a detecting device. The generating device includes a first acquiring unit 2020 and a generating unit 2040. The detecting device includes a second acquiring unit 2060 and a determining unit 2080.

[0033] <Process flow> 4 is a flowchart showing an exemplary flow of processing executed by the event detection device 2000. The first acquisition unit 2020 acquires video data 10 (S102). The generation unit 2040 generates object-related information 40 from the video data 10 (S104). The second acquisition unit 2060 acquires event information 50 (S106). The determination unit 2080 determines, based on the object-related information 40, whether or not an action-related event indicated by the event information 50 has occurred (S108).

[0034] <Acquisition of video data 10: S102> The first acquisition unit 2020 acquires the video data 10 (S102). There are various methods for acquiring the video data 10. For example, the camera 30 is configured to transmit the video data 10 to the event detection device 2000. In this case, the first acquisition unit 2020 receives the video data 10 transmitted by the camera 30 and acquires the video data 10.

[0035] In another example, the camera 30 is configured to, upon generating the video frame 20, transmit the video frame 20 to the event detection device 2000. In this case, the first acquisition unit 2020 receives the video frame 20 transmitted by the camera 30 and generates video data 10 from the received video frame 20.

[0036] In another example, the camera 30 is configured to store the video data 10 in a storage unit that can be accessed by the event detection device 2000. In this case, the event detection device 2000 retrieves the video data 10 from this storage unit.

[0037] <About object-related information 40> As described above, the object relation information 40 represents two or more action-related relationships between objects in terms of actions. FIG. 5 shows the object relation information 40 in table format. In FIG. 5, the object relation information 40 is represented as a table 100. The table 100 has columns named "Subject 102," "Object 104," "Action 106," and "Period 108." The action 106 indicates the type of action. The subject 102 indicates the identifier of the object performing the corresponding action. The object 104 indicates the identifier of the object performing the corresponding action. The period 108 indicates the start and end of the corresponding action-related relationship. Specifically, the period 108 is composed of two columns named "Start Time 110" and "End Time 112." The start time 110 indicates the start time of the corresponding action-related relationship. The end time 112 indicates the end time of the corresponding action-related relationship.

[0038] The subjects 102 and the objects 104 are represented by object identifiers. The identifiers of each object may be defined by other information called “object information” that is also generated from the video data 10 by the event detection device 2000. The object information may indicate the object identifier and object type (e.g., person, store item, bag, etc.) for each object detected from the video data 10.

[0039] The position of each object may change in the video data 10. Therefore, the object information preferably indicates a pair of time and position for each object. In other words, the object information indicates a time sequence of positions for each object.

[0040] There are various ways to represent the position of an object. For example, the position of an object may be represented by the coordinates on the video frame 20 where the object is located. 1 When dealing with 0, the position of an object can be represented by a pair of a camera identifier and the coordinates on the video frame 20 where the object is located.

[0041] In another example, the location of an object may be represented by coordinates on a map of the area imaged by one or more cameras 30. The map may be a two-dimensional map or a three-dimensional map.

[0042] In this case, the generation unit 2040 converts the coordinates of the object on the video frame 20 into coordinates on the map. By using the map, the positions of objects captured by different cameras 30 can be represented by coordinates in a unified coordinate space. In addition, by using this map, the event detection device 2000 can handle cameras 30 with a changeable field of view (for example, a pan-tilt-zoom camera).

[0043] Fig. 6 shows object information in a table format. In Fig. 6, the object information is represented by table 200. Table 200 includes columns named "identifier 202," "type 204," and "location 206." Identifier 202 indicates an identifier assigned to the corresponding object. Type 204 indicates the type of the corresponding object. Location 206 indicates a sequence of time-location pairs for the corresponding object.

[0044] The motion-related relationships between objects at a given moment can also be represented by a scene graph, where each object is represented by a node and the motion-related relationships between the objects are represented by edges. The object relationship information 40 can be said to represent a sequence of scene graphs. Therefore, the event detection device 2000 can be used to search the sequence of scene graphs for motion-related events.

[0045] An example of a scene graph is shown in Figure 7. In the example shown in Figure 7, three objects have been detected: a person with identifier 001, a bag with identifier 002, and another person assigned identifier 003. Thus, scene graph 60 includes three nodes representing person 001, bag 002, and person 003, respectively.

[0046] Person 001 and Bag 002 are connected to each other by an edge tagged with "Put Down" and pointing from Person 001 to Bag 002. This action association relationship represents Person 001 putting Down Bag 002. Person 003 is not connected to anything, which means Person 003 does not perform any action.

[0047] <Generation of object-related information 40: S104> The generation unit 2040 generates object relation information 40 from the video data 10. For example, for each video frame 20 included in the video data 10, the generation unit 2040 performs object detection to generate object information, and then detects action relation relationships of the objects detected by the object detection.

[0048] 8 is a flowchart showing the flow of processing executed by the generating unit 2040. The generating unit 2040 initializes the object information and object-relation information 40.

[0049] Steps S204 to S210 constitute a loop process L1 that is executed for each video frame 20 included in the video data 10. In step S204, the generation unit 2040 determines whether the loop process L1 has been executed for all video frames 20. If the loop process L1 has been executed for all video frames 20, the loop process L1 ends.

[0050] If the loop process L1 has not yet been performed on all video frames 20, the generation unit 2040 selects the video frame 20 on which the loop process L1 will be performed next. The video frame 20 selected here is the video frame 20 on which the loop process L1 has not yet been performed and is the video frame 20 that was generated earliest (for example, the video frame 20 with the smallest frame number). The video frame 20 selected here is called "video frame i."

[0051] The generation unit 2040 performs object detection on the video frame i to detect an object in the video frame i, and updates the object information (S206). If an object detected in the video frame i has not been detected in the previous video frame 20, the generation unit 2040 assigns a new identifier to the object and adds a new record for the object to the object information. If an object detected in the video frame i has been detected in the previous video frame 20, the generation unit 2040 updates the record of the object in the object information by adding a time and position pair of the object to the record. The time of the pair represents the time when the video frame i was generated. The position of the pair represents the position of the object on the video frame i.

[0052] The generation unit 2040 detects an action relation relationship between objects detected from video frame i and updates the object relation information 40 (S208). If an action relation relationship between specific objects detected from video frame i is also detected from video frame (i-1), the generation unit 2040 updates the record of this action relation relationship in the object relation information 40 and increases the duration of this relationship. On the other hand, if an action relation relationship between specific objects detected from video frame i is not detected from video frame (i-1), the generation unit 2040 generates a new record for this relationship and adds this record to the object relation information 40.

[0053] Step 210 marks the end of loop processing L1, so the generation unit 2040 then executes step S204.

[0054] <About Event Information 50> Event information 50 shows a sequence of action-related situations that represent an action-related event of interest. Figure 9 shows event information 50 in tabular form. In Figure 9, event information 50 is represented by table 300. Table 300 has columns named "Subject 302," "Object 304," and "Action 306."

[0055] Action 306 indicates the type of action, Subject 302 indicates the type of object that performs the corresponding action, and Object 304 indicates the type of object that the corresponding subject performs the corresponding action on.

[0056] Note that the event information 50 may need to indicate different objects that belong to the same type, and therefore the subject 302 and the object 304 need to distinguishably indicate objects of the same type.

[0057] For example, table 300 shown in FIG. 9 shows an event "Luggage Theft." Luggage theft involves a victim, who is the owner of the luggage, and a criminal who steals the luggage. The first row represents an action-related situation in which the victim places the luggage. To represent this situation, the subject 302, object 304, and action 306 in the first row are "Person:1," "Luggage:1," and "Place," respectively. The value "Person:1" represents the first person involved in this event. The value "Luggage:1" represents the first luggage involved in this event.

[0058] The second row represents the situation where the criminal picks up the package. To represent this situation, the subject 302, object 304, and action 306 are "Person:2", "Package:1", and "Pick up", respectively. The value "Person:2" represents the second person involved in this event, who is someone other than the first person, "Person:1". The value "Package:1" represents the first package, which is the same as the one shown in the first row.

[0059] The third row represents an action-related situation in which a criminal carries away a package. To represent this situation, the subject 302, object 304, and action 306 in the third row represent "Person:2," "Package:1," and "Carry," respectively. The value "Person:2" represents a second person who is the same person as the person shown in the second row. The value "Package:1" represents a first package that is the same as the package shown in the first and second rows.

[0060] Table 300 may include additional information. For example, table 300 may include another column entitled "Duration" that indicates the minimum length of time that the corresponding condition lasts.

[0061] For example, suppose the last situation of the luggage theft represented by the third row of table 300 shown in Figure 9 lasted only one second. This means that the second person carried the luggage just a few meters away from where the first person placed it. In this case, the second person may not have been trying to steal the luggage.

[0062] The "Duration" column can be used to avoid this type of false positive. Figure 10 shows event information 50 indicating the duration for each situation. In Figure 10, table 300 further includes a column titled "Duration 308," which indicates the duration of the corresponding action that was performed.

[0063] The duration 308 in the third row of the table 300 in Figure 10 indicates 10 seconds. This means that the second person carries the package for at least 10 seconds, effectively removing it from the location where the first person left it. In this case, it is highly likely that the second person is attempting to steal the package.

[0064] In another example, the event information 50 may indicate that a particular action is missing. An example of an action-related event that can be defined when a particular action is missing is "shoplifting." Shoplifting can be defined as follows: 1. A person picks up a product from a store. 2. The person does not stand in front of the cash register for a while. 3. The person passes through the exit.

[0065] If a person shoplifts an item from a store, that person does not pay for the item and therefore does not stand in front of the cash register. Therefore, shoplifting lacks the action of the person standing in front of the cash register for a while. The second item in the shoplifting list above represents the lack of the action "the person stands in front of the cash register for a while."

[0066] In order for the event information 50 to represent an action-related situation in which a specific action is missing, the table 300 may include a "missing flag." The missing flag indicates whether or not the corresponding action has been performed. Specifically, a missing flag indicating "true" indicates that the corresponding action has not been performed (i.e., the corresponding action is missing). On the other hand, a missing flag indicating "false" indicates that the corresponding action has been performed (i.e., the corresponding action is not missing). Hereinafter, an action-related situation in which a specific action is not performed will also be referred to as a "missing situation."

[0067] Figure 11 shows event information 50 indicating that a particular action is missing. The table 300 shown in Figure 11 further includes a column named "missing flag 310."

[0068] Table 300 shown in FIG. 11 represents the event "shoplifting." Specifically, the first row of table 300 shown in FIG. 11 represents a situation in which a first person picks up a first store item. To represent this situation, subject 302, object 304, action 306, duration 308, and absence flag 310 indicate "person: 1," "store item: 1," "pick up," "1 second," and "false," respectively. Because the action of picking up was performed, absence flag 310 indicates "false," i.e., the corresponding action was not absent.

[0069] 11 represents a situation in which the first person has not stood in front of the cash register for 10 seconds or more. To represent this situation, the subject 302, object 304, action 306, duration 308, and absence flag 310 indicate "person: 1," "cash register: 1," "stand in front," "10 seconds," and "true," respectively. Because the action of standing in front of the cash register has not been performed, the absence flag 310 indicates "true," i.e., that the corresponding action is absent.

[0070] 11 represents a situation in which the first person passes through an exit for at least one second. To represent this situation, the subject 302, object 304, action 306, duration 308, and missing flag 310 indicate "Person: 1," "Exit: 1," "Pass," "1 second," and "False," respectively. Because the action of passing through the exit has occurred, the missing flag 310 indicates "False," i.e., the corresponding action is not missing.

[0071] <Getting event information 50: S106> The second acquisition unit 2060 acquires the event information 50 (S106). In some embodiments, the second acquisition unit 2060 receives a query aimed at detecting a specific event from the video data 10. In this case, the query includes the event information 50 indicating a motion-related event of interest. The query may be transmitted by a user terminal used by a user of the event detection device 2000.

[0072] In some embodiments, one or more action-related events of interest are specified in advance. In this case, for each event of interest, event information 50 representing the event of interest is stored in advance in a storage unit accessible by the event detection device 2000. In this case, the second acquisition unit 2060 acquires one or more pieces of event information 50 from the storage unit. Next, the determination unit 2080 performs detection of the event of interest represented by the event information 50 for each piece of event information 50.

[0073] <Event detection: S108> The determination unit 2080 determines whether or not a noted action-related event has occurred based on the object-relationship information 40 and the event information 50 (S108). To this end, the determination unit 2080 determines whether or not the object-relationship information 40 includes a sequence of action-related relations that matches the sequence of action-related situations indicated by the event information 50.

[0074] Hereinafter, to make the explanation easier to understand, first, it is assumed that the event information 50 does not include operation-related situations where a specific operation is not performed.

[0075] The object relationship information 40 includes a sequence of 20 relationships represented by R[] = {R[1], R[2],..., R

[20] }, and it is assumed that the event information 50 includes a sequence of three situations represented by S[] = {S[1], S[2], S[3]}. In this case, if R[] includes R[i] that matches S[1], R[j] that matches S[2], and R[k] that matches S[3], and i < j < k is satisfied, the determination unit 2080 determines that the object relationship information 40 includes a sequence {R[i], R[j], R[k]} that matches the sequence {S[1], S[2], S[3]} indicated by the event information 50. Therefore, in this case, the determination unit 2080 determines that the operation-related event of interest has occurred.

[0076] Hereinafter, the sequence of operation-related relationships indicated by the object relationship information 40 is represented by R[] = {R[1],..., R[Nr]}. In addition, the sequence of operation-related situations indicated by the event information 50 is represented by S[] = {S[1],..., S[Ns]}. Note that both Nr and Ns are integers greater than 1.

[0077] As a whole, the determination unit 2080 attempts to sequentially detect each operation-related situation {S[1],..., S[Ns]} from each operation-related relationship {R[1],..., R[Nr]}. First, the determination unit 2080 searches {R[1],..., R[Nr]} for the first situation S[1] to detect R[i] as the relationship that matches S[1]. Next, the determination unit 2080 searches the partial sequence {R[i + 1],.., R[Nr]} for the second situation S[2] to detect R[j] as the relationship that matches S[2]. Then, the determination unit 2080 searches the partial sequence {R[j + 1],.., R[Nr]} for the third situation S[3] to detect R[k] as the relationship that matches S[3]. This process is repeatedly executed until the determination unit 2080 succeeds in finding the corresponding operation-related relationship for the last situation S[Ns], or until the determination unit 2080 fails to find an operation-related relationship for any situation.

[0078] In some cases, the object relationship information 40 includes two or more operation-related relationships that match the situation S[a] (1 <= a < Ns). In this case, the determination unit 2080 executes the subsequent search for the operation-related relationship that matches the situation S[a + 1] for each of the multiple operation-related relationships that match S[a].

[0079] Assume that the object relation information 40 includes {R[1],...,R

[20] }, the event information 50 includes {S[1],S[2]}, R[3] and R

[10] match S[1], and R[5] matches S[2]. In this case, the determination unit 2080 first detects R[3] and R

[10] by searching for a relation that matches S[1] for {R[1],...,R

[20] }. Next, the determination unit 2080 detects R[5] by searching for a relation that matches S[2] for {R[4],...,R

[20] }. In addition, the determination unit 2080 searches for a relation that matches S[2] for {R

[11] ,...,R

[20] } and does not find anything. As a result, the determining unit 2080 determines that the object relation information 40 includes the sequence {R[3], R[5]} that matches the sequence {S[1], S[2]}.

[0080] To determine whether the action-related relationship and the action-related situation match, the determination unit 2080 compares corresponding elements with each other. For example, the determination unit 2080 determines whether the subject 102 of the action-related relationship matches the subject 302 of the action-related situation, whether the object 104 of the action-related relationship matches the object 304 of the action-related situation, whether the action 106 of the action-related relationship matches the action 306 of the action-related situation, and whether the length of the period 108 of the action-related relationship matches the duration 308 of the action-related situation.

[0081] If the subject 102, object 104, action 106, and the length of the period 108 of the action-related relationship match the subject 302, object 304, action 306, and duration 308, respectively, the determination unit 2080 determines that the action-related relationship matches the action-related situation. On the other hand, if the subject 102 and the subject 302 do not match, if the object 104 and the object 304 do not match, if the action 106 and the action 306 do not match, or if the length of the period 108 and the duration 308 do not match, the determination unit 2080 determines that the action-related relationship and the action-related situation do not match. do not It is determined that:

[0082] The determination of whether an action-related relationship matches an action-related situation is made under the constraints generated by the previous match. When matching is performed for the first situation, there are no constraints yet. Therefore, subject 102 and subject 302 are determined to match if they are the same type. Similarly, object 104 and object 304 are determined to match if they are the same type.

[0083] Assume that the first action-related situation S[1] represents "Person 1 picks up Store Item 1." In addition, the action-related relationship R[i] represents "Object 003, whose type is person, picks up Object 007, whose type is store item." In this example, the determination unit 2080 determines that R[1] matches S[1]. Next, the determination unit 2080 generates the constraints "Person 1 is Object 003," "Persons other than Person 1 are not Object 003," "Store Item 1 is Object 007," and "Store items other than Store Item 1 are not Object 007."

[0084] In the next matching of the situation {S[2],...,S[Ns]}, the determination unit 2080 performs matching taking these constraints into consideration. For example, only if the subject 102 indicates "object 003", it is determined that the subject 102 matches with the subject 302 indicating "person 1". In addition, only if the subject 102 indicates an object whose type is person and is not object 003, it is determined that the subject 102 matches with the subject 302 indicating a person other than person 1 (e.g., person 2).

[0085] The determination unit 2080 generates constraints cumulatively. For example, after matching for situation S[2], the determination unit 2080 generates additional constraints based on the matching for situation S[2]. Then, the determination unit 2080 performs matching for situation S[3] by taking into account both the constraints generated based on the matching for situation S[1] and the constraints generated based on the matching for situation S[2].

[0086] 12 is a flowchart illustrating an exemplary flow of processing performed by the determination unit 2080. Specifically, FIG. 12 illustrates the flow of a recursive procedure named "search()." The procedure search() takes two arguments: target_sit, which represents the action-related situation that is the target of the search, and detected_rel_sec, which represents the sequence of action-related relations that are determined to match the sequence of action-related situations preceding target_sit.

[0087] The determination unit 2080 first executes search() with target_sit set to the first situation S[1] and detected_rel_sec set to null. In step S302, the determination unit 2080 detects one or more action-related relations that match target_sit from the sequence of relations after the last relation indicated by detected_rel_sec. Assume that detected_rel_sec includes {R[3], R[5]}. In this case, the determination unit 2080 searches for target_sit in {R[6],...,R[Nr]}. Note that if detected_rel_sec is null, the determination unit 2080 searches for target_sit in the entire sequence {R[1],...,R[Nr]}.

[0088] Steps S304 to S316 constitute a loop process L2 that is performed for each action relationship detected in S302. In step S304, the determination unit 2080 determines whether loop process L2 has been performed for all action relationship relationships detected in step S302. If loop process L2 has been performed for all action relationship relationships detected in step S302, the execution of the current procedure search() ends. On the other hand, if there is at least one action relationship among the action relationship relationships detected in step S302 for which loop process L2 has not yet been performed, the determination unit 2080 selects an action relationship among the action relationship relationships detected in step S302 for which loop process L2 has not yet been performed. The action relationship selected here is represented by rd. Note that if no action relationship is detected in S302, the determination unit 2080 does not perform loop process L2.

[0089] In step S306, the determination unit 2080 adds a relation rd to detected_rel_sec to define a new sequence named "new_sec." The sequence new_sec represents one of the sequences of action-related relations that match the sequence of action-related situations from the first situation to target_sit.

[0090] Next, the determination unit 2080 determines whether or not target_sit is the last action-related situation in the sequence indicated by the event information 50 (S308). If target_sit is the last action-related situation in the sequence indicated by the event information 50, the determination unit 2080 outputs new_sec as a sequence of action-related relationships that matches the entire sequence of action-related situations indicated by the event information 50.

[0091] On the other hand, if target_sit is not the last action-related situation in the sequence indicated by the event information 50, the determination unit 2080 performs the next matching on the action-related situation next to target_sit. To do so, the determination unit 2080 defines a variable "next_sit" to represent the action-related situation next to target_sit in the event information 50 (S312). Next, the determination unit 2080 sets the first argument target_sit to next_sit, sets the second argument detected_rel_sec to new_sec, and recursively executes the procedure search() (S314).

[0092] Step S316 is the end of loop processing L2. Therefore, when the determination unit 2080 reaches step S316, the determination unit 2080 next executes step S304.

[0093] The following describes how to handle event information 50 that includes one or more missing situations. In this case, the determination unit 2080 may first search the object relation information 40 for a sequence of action-related situations indicated by the event information 50 from which the missing situations have been removed. For example, assume that the event information 50 includes S[]={S[1], S[2], S[3]}, where S[2] is a missing situation. In this case, the determination unit 2080 first determines whether the object relation information 40 includes a sequence of action-related relationships that matches {S[1], S[3]}, which is a sequence of situations obtained by removing S[2] from S[].

[0094] Next, the determination unit 2080 determines whether the partial sequence obtained from the object relation information 40 for each missing situation includes a situation opposite to the missing situation. This partial sequence is a sequence of action-related relationships from the action-related relationship next to the action-related relationship that matches the situation before the missing situation to the action-related relationship immediately before the action-related relationship that matches the situation after the missing situation. For example, the event information 50 includes S[]={S[1], S[2], S[3]}, where S[2] is a missing situation, and action-related relationships R[3] and R

[10] match the action-related situations S[1] and S[3], respectively. In this case, the partial sequence is {R[4],...,R[9]} (R[4] is next to R[3], which matches S[1], and R[9] is immediately before R

[10] , which matches S[3]).

[0095] The opposite situation of a missing situation is a situation in which the action specified by the missing situation is performed by the subject specified by the missing situation. For example, if the missing situation represents a situation in which "Person 1 is not standing in front of Cashier 1 for at least 10 seconds," the opposite situation of the missing situation is a situation in which "Person 1 is standing in front of Cashier 1 for at least 10 seconds."

[0096] When it is determined that the corresponding partial sequence does not include an opposite situation to the missing situation for all missing situations, the determination unit 2080 determines that the object relation information 40 includes a sequence of action-related relations that matches the sequence of action-related situations indicated by the event information 50. On the other hand, when it is determined that the partial sequence includes an opposite situation to the missing situation for at least one missing situation, the determination unit 2080 determines that the object relation information 40 does not include a sequence of action-related relations that matches the sequence of action-related situations indicated by the event information 50.

[0097] For example, the event information 50 includes S[]={S[1],S[2],S[3]}, where S[2] is a missing situation, and it is determined that the action-related relations R[3] and R

[10] match the action-related situations S[1] and S[3], respectively. In this case, the determination unit 2080 determines whether the sequence {R[4],...,R[9]} includes a situation opposite to that of S[2]. If it is determined that the sequence {R[4],...,R[9]} does not include a situation opposite to that of S[2], the determination unit 2080 determines that the object-related information 40 includes a sequence {R[3],R

[10] } that matches the sequence S[]. On the other hand, if it is determined that the sequence {R[4],...,R[9]} contains the opposite situation to S[2], the determination unit 2080 determines that the sequence {R[3],R

[10] } does not match the sequence S[].

[0098] <Result output> The event detection device 2000 may output information (hereinafter referred to as “output information”) relating to the result of the determination performed by the determination unit 2080. When it is determined that the motion-related event indicated by the event information 50 has not occurred, the event detection device 2000 generates output information indicating that the video data 10 does not include the motion-related event indicated by the event information 50.

[0099] When it is determined that a motion-related event indicated by the event information 50 has occurred, the event detection device 2000 generates output information including information about the event. For example, the output information indicates the name of the detected event and the sequence of motion-related relationships that is determined to match the sequence of motion-related situations indicated by the event information 50. The name of the event may be defined in the event information 50.

[0100] For each action-related relationship determined to match a sequence of action-related circumstances indicated by the event information 50, the output information may include a sequence of video frames 20 (i.e., short clips extracted from the video data 10) in which the action-related relationship was detected. For example, suppose a package theft is detected in the video data 10. In this case, the output information may include a first short clip of a first person placing the package, a second short clip of a second person picking up the package, and a third short clip of a second person carrying the package.

[0101] Each event may have one or more objects of interest. For example, if a criminal event is detected, the criminal is the object of interest. The output information preferably indicates information about the criminal. The information about the object may include an image of the object and characteristics of the object. The image of the object may be extracted from the video frame 20 in which the object is detected.

[0102] The object characteristics depend on the type of object. Person characteristics may include age, gender, and clothing characteristics. Package characteristics may include color, shape, and brand.

[0103] There are various methods for outputting the output information. For example, the event detection device 2000 may store the output information in a storage unit. In another example, the event detection device 2000 may output the output information to a display device, thereby displaying the contents of the output information on the display device. In another example, the event detection device 2000 may send the output information to another device, such as a mobile device carried by a security guard or a PC used in a security room.

[0104] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0105] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0106] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0107] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. <Additional Notes> (Appendix 1) An event detection device, at least one memory configured to store instructions; Execute the command, Obtain one or more video data generating object relation information indicating two or more motion-related relations between objects from the one or more video data; obtaining event information indicating an event of interest by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information; and at least one processor configured to: (Appendix 2) 2. The event detection device of claim 1, wherein the event of interest is determined to have occurred if the object-related information includes a sequence of action-related relationships that matches the sequence of action-related situations indicated by the event information. (Appendix 3) The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action; The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; An event detection device as described in Appendix 2, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively. (Appendix 4) the event information includes a first action-related situation and a second action-related situation in this order; The event detection device according to claim 2 or 3, wherein the event of interest is determined to have occurred when the object relationship information includes, in this order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation. (Appendix 5) the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; the second action-related situation represents a situation in which a specific action is not performed by a specific subject; The event detection device described in Appendix 2 or 3, wherein the event of interest is determined to have occurred when the object relationship information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and the object relationship information does not include an action-related relationship that matches an action-related situation that is opposite to the second action-related situation. (Appendix 6) 4. The event detection device according to claim 1, wherein the event information distinguishably indicates two or more objects of the same type but different from each other. (Appendix 7) 4. An event detection device according to any one of claims 1 to 3, wherein the object relationship information represents a sequence of scene graphs, each of which represents a relationship between objects that exists at a certain point in time or over a certain period of time. (Appendix 8) acquiring one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; acquiring event information indicating an event of interest by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information. (Appendix 9) 9. The event detection method of claim 8, wherein the event of interest is determined to have occurred if the object-related information includes a sequence of action-related relations that matches the sequence of action-related situations indicated by the event information. (Appendix 10) The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action; The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; 10. The event detection method of claim 9, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively. (Appendix 11) the event information includes a first action-related situation and a second action-related situation in this order; An event detection method according to claim 9 or 10, wherein the event of interest is determined to have occurred when the object relationship information includes, in this order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation. (Appendix 12) the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; the second action-related situation represents a situation in which a specific action is not performed by a specific subject; The event detection method described in Appendix 9 or 10, wherein the event of interest is determined to have occurred when the object relationship information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and the object relationship information does not include an action-related relationship that matches an action-related situation that is opposite to the second action-related situation. (Appendix 13) 11. The event detection method according to any one of appendices 8 to 10, wherein the event information distinguishably indicates two or more objects of the same type but different from each other. (Appendix 14) 11. The event detection method of any one of Supplementary Notes 8 to 10, wherein the object relationship information represents a sequence of scene graphs, each of which represents a relationship between objects that exists at a certain point in time or over a certain period of time. (Appendix 15) acquiring one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; acquiring event information indicating an event of interest by a sequence of action-related situations, at least one of which represents a situation in which a specific action is performed by a specific subject; and determining whether the event of interest has occurred based on the object-related information and the event information. (Appendix 16) 16. A storage medium as described in Appendix 15, wherein the event of interest is determined to have occurred if the object-related information includes a sequence of action-related relationships that matches the sequence of action-related situations indicated by the event information. (Appendix 17) The temporal action relation indicates a combination of an action, a subject of the action, and an object of the action; The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; A storage medium as described in Appendix 16, wherein the temporal action-related relationship is determined to match the action-related situation when the action, the subject of the action, and the object of the action indicated by the temporal action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively. (Appendix 18) the event information includes a first action-related situation and a second action-related situation in this order; A storage medium as described in Appendix 16 or 17, wherein the event of interest is determined to have occurred when the object relationship information includes, in this order, a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation. (Appendix 19) the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; the second action-related situation represents a situation in which a specific action is not performed by a specific subject; A storage medium as described in Appendix 16 or 17, wherein the event of interest is determined to have occurred when the object relationship information includes a first action-related relationship that matches the first action-related situation and a second action-related relationship that matches the second action-related situation, in that order, and the object relationship information does not include an action-related relationship that matches an action-related situation that is opposite to the second action-related situation.

[0108] This application claims priority from Singapore Patent Application No. 10202302052W, filed July 20, 2023, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0109] 10 Video Data 20 video frames 30 Camera 40 Object-related information 50 Event Information 60 Scene Graph 100 tables 102 Subject 104 Object 106 Operation 108 period 110 Starting point 112 End time 200 tables 202 Identifier 204 Type 206 position 300 tables 302 Subject 304 Object 306 operation 308 Duration 310 Missing Flag 1000 computers 1020 Bus 1040 processor 1060 memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Event Detector 2020 1st Acquisition Division 2040 Generation part 2060 2nd Acquisition Department 2080 Judgment section

Claims

1. a first acquisition unit that acquires one or more pieces of video data; a generating unit that generates object relation information indicating two or more motion relation relationships between objects from the one or more video data; a second acquisition unit that acquires event information indicating a noteworthy event based on a sequence of action-related situations; At least one of the action-related situations represents a situation in which a specific action is performed by a specific subject; a determination unit that determines whether the event of interest has occurred based on the object-related information and the event information; the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; the second action-related situation represents a situation in which a specific action is not being performed by a specific subject; The determination unit determines that the event of interest has occurred when the object relationship information contains, in that order, a first action-related relationship that matches the first action-related situation and a third action-related relationship that matches the third action-related situation, and when the object relationship information does not contain an action-related relationship that matches an action-related situation that represents a situation opposite to the second action-related situation.

2. The action relation indicates a combination of an action, a subject of the action, and an object of the action, The action-related situation indicates a combination of an action, a subject of the action, and an object of the action; The event detection device of claim 1, wherein the determination unit determines that the action-related relationship matches the action-related situation when the action, the subject of the action, and the object of the action indicated by the action-related relationship match the action, the subject of the action, and the object of the action indicated by the action-related situation, respectively.

3. The event detection device according to claim 1 , wherein the event information distinguishably indicates two or more objects of the same type but different from each other.

4. the object relation information represents a sequence of scene graphs; The event detection device according to claim 1 or 2, wherein the scene graph represents relationships between objects that exist at a certain point in time or during a certain period of time.

5. acquiring one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; acquiring event information indicating an event of interest according to a sequence of action-related situations; At least one of the action-related situations represents a situation in which a specific action is performed by a specific subject; determining whether the event of interest has occurred based on the object-related information and the event information; the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; the second action-related situation represents a situation in which a specific action is not being performed by a specific subject; An event detection method executed by a computer, in which, in the determination step, it is determined that the event of interest has occurred if the object relationship information contains a first action-related relationship that matches the first action-related situation and a third action-related relationship that matches the third action-related situation in that order, and the object relationship information does not contain an action-related relationship that matches an action-related situation that represents a situation opposite to the second action-related situation.

6. acquiring one or more video data; generating object relation information indicative of two or more motion-related relations between objects from the one or more video data; acquiring event information indicating an event of interest according to a sequence of action-related situations; At least one of the action-related situations represents a situation in which a specific action is performed by a specific subject; causing a computer to execute a step of determining whether the event of interest has occurred based on the object-related information and the event information; the event information includes a first action-related situation, a second action-related situation, and a third action-related situation; each of the first action-related situation and the third action-related situation represents a situation in which a specific action is performed by a specific subject; the second action-related situation represents a situation in which a specific action is not being performed by a specific subject; A program that, in the determination step, determines that the event of interest has occurred if the object relationship information contains a first action-related relationship that matches the first action-related situation and a third action-related relationship that matches the third action-related situation in that order, and if the object relationship information does not contain an action-related relationship that matches an action-related situation that represents a situation opposite to the second action-related situation.

Images