Relation extraction device, relation extraction method, and program

Abstract

To provide novel technology for providing information related to a notable event.SOLUTION: A relation extraction device acquires event information indicative of the characteristics of a notable event, determines an object period based on one or more characteristics of the notable event, and extracts one or more operation-related relation existing in the object period from matter-related information. The matter-related information indicates two or more operation-related relations between matters, in association with a time point or a period where the operation-related relation exists.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure generally relates to a relationship extraction device, a relationship extraction method, and a storage medium. [Background technology]

[0002] There are techniques for providing information on noteworthy events. Patent Document 1 discloses a technique for detecting important scenes that are part of an event from a source video and extracting key frames for the scenes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Patent Publication No. 2001 / 033863 Summary of the Invention [Problem to be solved by the invention]

[0004] The key frames provided by Patent Document 1 represent a portion of an event. Therefore, Patent Document 1 does not provide information that is not included in the event. The objective of the present disclosure is to provide a novel technique for providing information related to a featured event. [Means for solving the problem]

[0005] The present disclosure provides a relationship extraction device, comprising at least one memory configured to store instructions and at least one processor, configured to execute instructions to obtain event information indicating one or more characteristics of an event of interest, determine a target period based on the one or more characteristics of the event of interest, the target period including, as a part thereof, an event time which is a time or period when the event of interest occurs, extract one or more action-related relations existing during the target period from object relationship information, the object relationship information indicating two or more relations between objects in association with the time or period when the action-related relations exist.

[0006] The present disclosure provides a relationship extraction method, the relationship extraction method including: acquiring event information indicating one or more characteristics of an event of interest; determining a target period based on the one or more characteristics of the event of interest, the target period including, as part thereof, an event time which is a time or period when the event of interest occurs; and extracting one or more action-related relationships that exist during the target period from object relationship information, the object relationship information indicating two or more relationships between objects in correspondence with the time or period when the action-related relationships exist.

[0007] The present disclosure provides a storage medium on which a program is stored, the program causing a computer to execute the steps of: acquiring event information indicating one or more characteristics of an event of interest; determining a target period based on the one or more characteristics of the event of interest, the target period including, as part of the target period, an event time which is a time or period when the event of interest occurs; and extracting one or more action-related relationships that exist during the target period from object relationship information, the object relationship information indicating two or more relationships between objects in correspondence with the time or period when the action-related relationships exist. Effect of the Invention

[0008] SUMMARY In accordance with the present disclosure, novel techniques are provided for providing information related to events of interest. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of a relationship extraction device. [Diagram 2] 2 is a block diagram showing an example of a functional configuration of a relationship extraction device; [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a computer that realizes the relationship extraction device. [Figure 4] 10 is a flowchart illustrating an exemplary flow of a process executed by the relationship extraction device. [Diagram 5] FIG. 2 illustrates an exemplary structure of object-related information in table form. [Figure 6] FIG. 2 illustrates an exemplary structure of object information in table form. [Figure 7] FIG. 2 illustrates an example of a scene graph. [Figure 8] 13 is a flowchart showing a flow of processing executed by the object relationship information generating device. [Figure 9] FIG. 13 illustrates an exemplary structure of period information in table form. [Figure 10] FIG. 1 illustrates a time period of interest that is determined based on a predefined length of time associated with a type of event. [Figure 11] FIG. 2 illustrates an example of a functional configuration of a relationship extraction device 2000 including an output unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding elements are indicated by the same reference numerals, and duplicated descriptions are omitted as necessary to clarify the description. Unless otherwise specified, predetermined information (e.g., a predetermined value or a predetermined threshold value) is pre-stored in a storage device accessible to a computer that uses the information. Furthermore, unless otherwise specified, the storage unit is composed of one or more storage devices.

[0011] <Summary> 1 shows an overview of a relationship extraction device 2000. The overview shown in FIG. 1 shows an example of the operation of the relationship extraction device 2000 in order to facilitate understanding of the relationship extraction device 2000, and is not intended to limit or reduce the scope of possible operations of the relationship extraction device 2000.

[0012] The relation extraction device 2000 is used to extract one or more time-dependent motion relation relations between objects that are predicted to be related to a target event from the object relation information 20. The object relation information 20 represents a sequence of two or more time-dependent motion relation relations between objects, each of which is a motion relation relation between objects present at a certain time or during a certain period of time detected from one or more video frames.

[0013] A temporal action-related relationship can be represented by a combination of 1) action type, 2) subject of the action, 3) object of the action, and 4) time when the action was performed. Suppose there is an action-related relationship in which person P1 picks up store product I1 from time T1 to T2. The object relationship information 20 can represent this relationship by a combination of 1) action type: pick up, 2) subject: person P1, 3) object: store product I1, and 4) time: T1 to T2.

[0014] An event of interest is a motion-related event, which is any type of event that involves one or more actions taken by an object. Types of motion-related events can include crime events (e.g., shoplifting or theft of a package), accidents (e.g., a car accident or a left-behind package), sporting events (e.g., a goal event in a soccer game or a home run in a baseball game), and customer events (e.g., a purchase).

[0015] An action-associated relationship is predicted to be associated with an event of interest when the action-associated relationship exists at least during a time period associated with the event of interest, hereinafter the time period associated with the event of interest being referred to as the "time period of interest."

[0016] In order to extract an action association relation related to the event of interest, the relation extraction device 2000 determines a target period by acquiring the event information 10. The event information 10 includes information capable of identifying the characteristics of the event of interest.

[0017] The characteristics of the event of interest include a time when the event of interest occurred. Hereinafter, the time when the event of interest occurred is also referred to as an "event time." The event time may be represented by a specific point in time or a specific period of time. The characteristics of the event of interest also include a type of the event of interest, such as a purchase, shoplifting, etc.

[0018] The relationship extraction device 2000 determines a target period based on the event time and the type of the event of interest. The target period includes the event time as a part thereof. For example, the length of the period may be predefined for each type of event. In this case, the relationship extraction device 2000 determines the length of the predefined period corresponding to the type of the event of interest. Next, the relationship extraction device 2000 determines, as the target period, a period that includes the event time and whose length is determined based on the length of the predefined period corresponding to the type of the event of interest.

[0019] Based on the target period, the relationship extraction device 2000 extracts action-related relationships related to the event of interest by searching for them from the object relationship information 20. Specifically, action-related relationships that exist during the target period are extracted from the object relationship information 20.

[0020] <Examples of effects> As described above, the relationship extraction device 2000 determines a target period including an event time as a part thereof based on the characteristics of the target event indicated by the event information 10. Then, the relationship extraction device 2000 extracts, from the object relationship information 20, action related relations that exist during the target period as action related relations related to the target event.

[0021] According to the above-described operation of the relationship extraction device 2000, a novel technique for providing information related to an event of interest is provided. Specifically, the action-related relationships existing in a target period include information about what happened before the event of interest, what happened after the event of interest, or what happened both. Thus, the relationship extraction device 2000 facilitates understanding of what happened before the event of interest, what happened after the event of interest, or what happened both, thereby facilitating detailed understanding of the event of interest.

[0022] Furthermore, the relationship extraction device 2000 determines the target period based on the characteristics of the event of interest, such as the type of the event of interest, thereby allowing the relationship extraction device 2000 to take the characteristics of the event of interest into account in determining what information the relationship extraction device 2000 should provide.

[0023] A more detailed description of the relationship extraction device 2000 is provided below.

[0024] <Example of functional configuration> 2 is a block diagram showing an example of a functional configuration of the relationship extraction device 2000. The relationship extraction device 2000 includes a determination unit 2020 and an extraction unit 2040. The determination unit 2020 acquires event information 10 and determines a target period based on the type and event time of a noteworthy event indicated by the event information 10. The extraction unit 2040 extracts an action association relationship that exists during the target period as an action association relationship related to the noteworthy event.

[0025] <Example of hardware configuration> The relationship extraction device 2000 may be realized by one or more computers. FIG. 3 is a block diagram showing an example of a hardware configuration of a computer 1000 that realizes the relationship extraction device 2000. The computer 1000 may be any type of computer. For example, the computer 1000 is a stationary computer such as a personal computer (PC) or a server machine. In another example, the computer 1000 is a mobile computer such as a smartphone or a table terminal. In another example, the computer 1000 is an integrated circuit such as a System on Chip (SoC). The computer 1000 may be a dedicated computer manufactured for implementing the relationship extraction device 2000, or may be a general-purpose computer.

[0026] The relationship extraction device 2000 can be realized by installing an application in the computer 1000. The application is implemented as a program that causes the computer 1000 to function as the relationship extraction device 2000. In other words, the program is an implementation form of the functional components of the relationship extraction device 2000.

[0027] There are various ways to obtain the program. For example, the program may be obtained from a storage medium (e.g., a DVD disk or a USB memory) on which the program is stored. In another example, the program may be downloaded from a server that manages the storage medium on which the program is stored.

[0028] In FIG. 3, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120. The bus 1020 is a data transmission channel for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage element such as a random access memory (RAM) or a read only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network can be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0029] The processor 1040 may be configured to load instructions of the above-mentioned programs from the storage device 1080 into the memory 1060 and execute those instructions to cause the computer 1000 to operate as the relationship extraction device 2000 .

[0030] The hardware configuration of the computer 1000 is not limited to that shown in Fig. 3. For example, as described above, the relationship extraction device 2000 may be realized as a combination of multiple computers. In this case, the computers may be connected to each other via a network.

[0031] <Processing flow> 4 is a flowchart showing an exemplary flow of a process executed by the relationship extraction device 2000. The determination unit 2020 acquires the event information 10 (S102). The determination unit 2020 determines a target period based on the characteristics of the event of interest (S104). The extraction unit 2040 extracts an action association relationship that exists in the target period (S106).

[0032] <About object-related information 20> As described above, the object relationship information 20 represents two or more action-related relationships between objects. The object relationship information 20 may be generated by the relationship extraction device 2000 or another device. Hereinafter, the device that generates the object relationship information 20 is referred to as an "object relationship information generating device."

[0033] The object relation information generating device generates the object relation information 20 based on one or more sequences of video frames (in other words, one or more video data), each generated by a camera. Specifically, the object relation information generating device analyzes a scene captured in the video frames and detects motion-related relationships between objects captured in the video frames, thereby generating the object relation information 20.

[0034] There may be various ways to obtain the video data, for example, the camera may be configured to transmit the video data to the object relation information generating device, in which case the object relation information generating device receives the video data transmitted by the camera to obtain the video data.

[0035] In another example, the camera is configured to, upon generating a video frame, transmit the video frame to the object related information generating device, where the object related information generating device receives the video frames transmitted by the camera and generates video data from the received video frames.

[0036] In another example, the camera is configured to store the video data in a storage unit accessible by the object relationship information generating device, in which case the object relationship information generating device retrieves the video data from the storage unit.

[0037] FIG. 5 shows an exemplary structure of the object relation information 20 in table form. In FIG. 5, the object relation information 20 is represented by a table 100. The table 100 has columns named "Subject 102", "Object 104", "Action 106", and "Period 108". The action 106 indicates the type of action. The subject 102 indicates the identifier of the object performing the corresponding action. The object 104 indicates the identifier of the object for which the corresponding subject takes the corresponding action. The period 108 indicates the time period during which the corresponding action-related relationship exists. Specifically, the period 108 is composed of two columns named "Start Time 110" and "End Time 112". The start time 110 indicates the time when the corresponding action-related relationship begins. The end time 112 indicates the time when the corresponding action-related relationship ends.

[0038] The subject 102 and the object 104 are represented by object identifiers. The identifiers of each object may be defined by other information called "object information" that is also generated from the video data by an object relation information generating device. The object information may indicate the object identifier and the object type (e.g., person, store item, bag, etc.) for each object detected from the video data.

[0039] In the video data, the position of each object may change. Therefore, the object information preferably indicates a pair of time and position for each object. In other words, the object information indicates a time sequence of positions for each object.

[0040] There are various ways to represent the position of an object. For example, the position of an object may be represented by the coordinates on a video frame where the object is located. When the object relation information generating device handles two or more video data, the position of an object may be represented by a pair of a camera identifier and the coordinates on a video frame where the object is located.

[0041] In another example, the location of an object may be represented by coordinates on a map of the area imaged by one or more cameras, which may be a two-dimensional or three-dimensional map.

[0042] In this case, the object relation information generating device converts the coordinates in the video frame where the object is located into coordinates on the map. By using the map, the positions of objects captured by different cameras can be represented by coordinates in a unified coordinate space. In addition, by using the map, the object relation information generating device can handle cameras that can change their field of view (e.g., pan-tilt-zoom cameras).

[0043] Fig. 6 illustrates an exemplary structure of object information in table form. In Fig. 6, the object information is represented by a table 200. The table 200 includes columns named "identifier 202", "type 204", and "location 206". The identifier 202 indicates an identifier assigned to the corresponding object. The type 204 indicates the type of the corresponding object. The location 206 indicates a sequence of time and location pairs for the corresponding object.

[0044] The action-related relationships between objects at a certain time can also be represented by a scene graph, in which each object is represented by a node and the action-related relationships between the objects are represented by edges. It can be said that the object relationship information 20 represents a sequence of the scene graph. Therefore, the relationship extraction device 2000 can be used to search for a sequence of action-related relationships related to a target event from the scene graph.

[0045] An example of a scene graph is shown in Figure 7. In the example shown in Figure 7, three objects are detected: a person assigned identifier 001, a bag assigned identifier 002, and another person assigned identifier 003. Thus, the scene graph 60 includes three nodes representing person 001, bag 002, and person 003, respectively.

[0046] Person 001 and Bag 002 are tagged with "Put" and are connected to each other by an edge going from Person 001 to Bag 002. This action association relationship represents Person 001 putting Bag 002. Person 003 is not connected to anything, which means Person 003 does not perform any action.

[0047] The object relation information generating device generates object relation information 20 from the video data. For example, for each video frame included in the video data, the object relation information generating device performs object detection to generate object information, and then detects action-related relations for the objects detected through the object detection.

[0048] 8 is a flowchart showing a flow of processing executed by the object relationship information generating device. The object relationship information generating device initializes the object information and the object relationship information 20.

[0049] Steps S204 to S210 constitute a loop process L1 that is executed for each video frame included in the video data. In step S204, the object relationship information generating device determines whether the loop process L1 has been executed for all video frames. If the loop process L1 has been executed for all video frames, the loop process L1 ends.

[0050] If the loop process L1 has not yet been performed on all video frames, the object-related information generating device selects a video frame on which the loop process L1 should be performed next. The video frame selected here is the video frame with the earliest generation time (e.g., the smallest frame number) among the video frames on which the loop process L1 has not yet been performed. The video frame selected here is denoted as "video frame i."

[0051] The object relation information generating device performs object detection on the video frame i to detect an object from the video frame i and update the object information (S206). When an object detected from the video frame i has not been detected from the preceding video frame, the object relation information generating device assigns a new identifier to the object and adds a new record for the object to the object information. When an object detected from the video frame i has been detected from the preceding video frame, the object relation information generating device updates the record of the object in the object information by adding a pair of time and position of the object to the record. The time of the pair represents the time when the video frame i was generated. The position of the pair represents the position of the object on the video frame i.

[0052] The object relation information generating device performs detection of an action related relationship between objects detected from the video frame i to update the object relation information 20 (S208). If an action related relationship between specific objects detected from the video frame i is also detected from the video frame (i-1), the object relation information generating device updates the record of this action related relationship in the object relation information 20 so as to increase the period of this relationship. On the other hand, if an action related relationship between specific objects detected from the video frame i is not detected from the video frame (i-1), the object relation information generating device generates a new record for this relationship and adds this record to the object relation information 20.

[0053] Step 210 is the end of the loop process L1. Therefore, the object relationship information generation device next executes step S204.

[0054] <About Event Information 10> The event information 10 indicates information that can identify the characteristics of the event of interest. As described above, the characteristics of the event of interest include the type of the event of interest and the event time, which is the time when the event of interest occurs. In this manner, the event information 10 indicates the type of the event of interest and the event time.

[0055] In addition, the event information 10 can indicate an object related to the event of interest. The object related to the event of interest includes a subject of an action involved in the event of interest and an object of an action involved in the event of interest. Assume that the event of interest is a purchase in which "at time t1, object 001 (person) purchased object 002 (store product)." In this case, the objects related to the event of interest are object 001 and object 002. The event information 10 can indicate 1) event type = purchase, 2) event time = t1, 3) purchaser = object 001, and 4) purchased item = object 002.

[0056] In some implementations, the event of interest is represented by a sequence of action-related situations, each of which is a situation in which a specific subject performs a specific action on a specific object. In this case, the event information 10 may indicate the subject of the action involved in the event of interest and the object of the action involved in the event of interest as objects related to the event of interest.

[0057] For example, when the event of interest is a purchase, the event may be represented as follows: 1. Object 001 (person) picks up object 002 (store item) at time t1. 2. Object 001 (person) stands in front of Object 003 (cash register) from time t2 to t3. 3. Object 001 (person) passes through object 004 (store exit) at time t4.

[0058] In this case, the objects of interest may include object 001, object 002, object 003, and object 004. The event information 10 may indicate: 1) event type = purchase, 2) event time = t1 to t4, 3) purchaser = object 001, 4) purchase item = object 002, 5) cash register = object 003, and 6) exit = object 004.

[0059] Here, the event information 10 does not necessarily indicate all objects related to the event of interest. Specifically, the event information 10 may indicate only a specific type of object related to the event of interest. For example, with respect to the purchase described above, the event information 10 may indicate only the purchaser and one or more purchased items, and may not indicate the cash register or the exit. The type of object to be indicated by the event information 10 may be predefined for each type of event.

[0060] <Getting event information 10: S102> The determining unit 2020 acquires the event information 10 (S102). There may be various methods for the determining unit 2020 to acquire the event information 10. For example, the event information 10 is stored in advance in a storage unit accessible from the relationship extraction device 2000. In this case, the determining unit 2020 reads the event information 10 from this storage unit to acquire the event information 10. In another example, another device transmits the event information 10 to the relationship extraction device 2000. In this case, the determining unit 2020 receives the event information 10 and acquires the event information 10.

[0061] There may be various triggers for the determining unit 2020 to acquire the event information 10 (in other words, to start the execution of the process illustrated by FIG. 4). For example, the relationship extraction device 2000 receives a request to provide information about an action-associated relationship related to a target event, and acquires the event information 10 in response to the request. In this case, the request includes information capable of identifying the event information 10 to be acquired.

[0062] For example, the request includes the event information 10. In this case, the determining unit 2020 obtains the event information 10 by extracting the event information 10 from the request.

[0063] In another example, the request specifies an identifier of the event information 10 to be acquired. For example, it is assumed that the event information 10 is stored as a file in the storage unit. In this case, the request specifies a file name of the event information 10 to be acquired. The determination unit 2020 acquires, from the storage unit, a file having the specified name as the event information 10.

[0064] There may be various ways to provide a request to the relationship extraction device 2000. For example, a user of the relationship extraction device 2000 inputs a request by operating an input device attached to the relationship extraction device 2000. In another example, the user operates another device, such as a client terminal, and sends a request from that device to the relationship extraction device 2000.

[0065] The request is not necessarily generated based on user input. In some implementations, there is a device called an "event detector" configured to detect a particular event from video data. In this case, the event detected by the event detector is treated as the event of interest. The event detector generates a request to provide information about the action-related relations related to the event detected by the event detector and sends the request to the relation extraction device 2000.

[0066] <Determining the target period: S104> The determination unit 2020 determines the target period based on the event information 10 (S104). In some embodiments, the length of the period may be predefined for each type of event, allowing the relationship extraction device 2000 to determine what information to provide, taking into account the type of the event of interest.

[0067] In this case, there is information called "period information" in which each type of event is associated with data indicating the length of the period. FIG. 9 shows an exemplary structure of the period information in table form. In FIG. 9, the period information is represented by a table 300. The table 300 includes columns of "type 302" and "period 304." Type 302 indicates the type of event. Period 304 indicates data indicating the length of the period corresponding to the type of event.

[0068] The time period 304 includes two columns: a "previous time period 306" and a "following time period 308." The previous time period 306 shows data representing the length of a period included in the time period of interest that precedes the event time. The following time period 308 shows data representing the length of a period included in the time period of interest that follows the event time.

[0069] The preceding period 306, the following period 308, or both may indicate the length of the period in terms of typical time units such as seconds, minutes, or hours. For example, the first row of the table 300 illustrated by FIG. 9 indicates "Preceding Period: 10 minutes" and "Following Period: 5 minutes."

[0070] 10 illustrates a target period determined based on a predefined length of time associated with an event type. Event information 10 indicates that 1) the type of event of interest is shoplifting, and 2) the event time is from "2023 / 6 / 20 15:20" to "2023 / 6 / 20 15:13." Period information associates "Type: Shoplifting" with "Preceding Period: 10 minutes" and "Subsequent Period: 5 minutes."

[0071] Since the start time of the event of interest is "2023 / 6 / 20 15:10" and the length of the preceding period is 10 minutes, the start time of the target period is determined to be "2023 / 6 / 20 15:10". In addition, since the end time of the event of interest is "2023 / 6 / 20 15:13" and the length of the succeeding period is 5 minutes, the end time of the target period is determined to be "2023 / 6 / 20 15:18". As a result, the determination unit 2020 determines the target period to be from "2023 / 6 / 20 15:10" to "2023 / 6 / 20 15:18".

[0072] In another example, the preceding time period 306 can represent a length of time using conditions that can identify behavior-related relationships that represent the start of the time period of interest. Similarly, the following time period 308 can represent a length of time using conditions that can identify behavior-related relationships that represent the end of the time period of interest.

[0073] Assume that the type of the event of interest is a "goal event in a soccer game." In this case, when a goal is scored, a user of the relationship extraction device 2000 (e.g., a viewer of a video of a soccer game) may be interested in some play before the goal. Thus, the preceding period 306 may indicate one or more plays to be included in the target period. For example, the second row of the table 300 illustrated by FIG. 9 indicates "Preceding period: 10 passes before the goal." In this case, the determination unit 2020 may identify that the action-related relationship representing the 10th to the last pass before the goal is the action-related relationship representing the start of the target period.

[0074] For example, the second row of the table 300 shown in FIG. 9 indicates "Subsequent Period: Goal Celebration". By this definition, the goal celebration should be included in the target period. The determining unit 2020 can identify that the action related relationship representing the end of the goal celebration is the action related relationship representing the end of the target period.

[0075] The preceding period 306 may directly indicate a particular behavior-related relationship that should occur at the beginning of the period of interest. Similarly, the following period 308 may directly indicate a particular behavior-related relationship that should occur at the end of the period of interest.

[0076] For example, the third row of table 300 shown in FIG. 9 indicates "preceding period: purchaser enters store" and "subsequent period: purchaser leaves store" in association with "type: purchase". In this case, the determination unit 2020 can identify the action related relationship representing the purchaser entering the store as the action related relationship representing the start of the target period. Similarly, the determination unit 2020 can identify the action related relationship representing the purchaser leaving the store as the action related relationship representing the end of the target period.

[0077] <<Adjustment of period>> In some implementations, the length of the target period may be adjusted. For example, the relationship extraction device 2000 adjusts the length of the target period based on user input. This allows the relationship extraction device 2000 to determine the target period taking into account the user's preferences.

[0078] For example, a user of the relationship extraction device 2000 can specify an ambiguity parameter, which is a parameter that represents how ambiguous an event of interest is to the user. Conceptually, the more ambiguous an event of interest is to the user, the more information the user needs to understand the event of interest, so the target period needs to be set longer. Suppose the event of interest is shoplifting and the user of the relationship extraction device 2000 is a store clerk at a store where the shoplifting occurred. If the clerk knows the shoplifter well (e.g., the shoplifter often comes to the store), the event of interest may not be very ambiguous to the clerk. Therefore, the clerk specifies a lower value of the ambiguity parameter.

[0079] On the other hand, if the store clerk is unfamiliar with the shoplifter (e.g., this is the first time this shoplifter has been to the store), the event of interest may be ambiguous to the store clerk, and thus the store clerk may specify a higher value for the ambiguity parameter.

[0080] In another example, assume that the event of interest is a goal event in a soccer match, and the user of the relationship extraction device 2000 is a viewer of the match. If the user is watching the match intently, the goal event is not ambiguous to the user. Therefore, the user specifies a lower value of the ambiguity parameter. On the other hand, if the user is talking with a friend while watching the match, the goal event may be ambiguous to the user, for example, a goal is scored when the user is not watching the match but is watching the friend. In this case, the user specifies a higher value of the ambiguity parameter.

[0081] The determining unit 2020 may adjust the length of the preceding period, the length of the following period, or both, based on the ambiguity parameter. When adjusting the length of the preceding period, the determining unit 2020 may determine an adjustment factor based on the ambiguity parameter and multiply the adjustment parameter by the length of time represented by the preceding period 306. The determining unit 2020 uses the adjusted length of the preceding period as the length of the period from the start of the period of interest to the event time. The adjustment factor is a positive real value within a predefined range.

[0082] Similarly, when adjusting the length of the subsequent period, the decision unit 2020 multiplies the adjustment factor by the length of time represented by the preceding period 306. The decision unit 2020 then uses the adjusted length of the subsequent period as the length of the period from the event time to the end of the period of interest.

[0083] The adjustment factor may be the ambiguity parameter itself or a value obtained by transforming the ambiguity parameter using a predefined function. In the latter case, the determination unit 2020 inputs the ambiguity parameter into the predefined function, thereby obtaining the adjustment factor.

[0084] Suppose the lead period 306 indicates 10 minutes, the predefined range of the ambiguity parameter is from 0.5 to 1.5, and the ambiguity parameter is used as the adjustment factor as is. A user who wants to extend the lead period specifies an ambiguity parameter greater than 1. For example, a user can extend the lead period to 15 minutes by specifying an ambiguity parameter of 1.5.

[0085] On the other hand, a user who wishes to shorten the advance period specifies an ambiguity parameter less than 1. For example, a user can shorten the advance period to 5 minutes by specifying an ambiguity parameter of 0.5.

[0086] The adjustment factors for adjusting the length of the preceding period and the adjustment factors for adjusting the length of the following period may be the same or different from each other, e.g., the function that converts the ambiguity parameter to the adjustment factor used for the preceding period and the function that converts the ambiguity parameter to the adjustment factor used for the following period are predefined separately.

[0087] It is not necessarily the ambiguity parameter that adjusts the length of the target period. For example, the determination unit 2020 allows the user to specify an importance parameter that indicates how important the event of interest is to the user. In this case, the adjustment coefficient is set to be larger as the importance increases.

[0088] In another example, the decision unit 2020 allows the user to specify a curiosity parameter that represents how much the user wants to know about the featured event, in which case the adjustment coefficient is set to be larger as the curiosity parameter is larger.

[0089] In some implementations, the relationship extractor 2000 adjusts the length of the period of interest based on the temporal concentration of action-related relationships that include an object related to the event of interest. Hereinafter, the object related to the event of interest is referred to as an "object of interest."

[0090] If the period around the event of interest including the action-related relations involving the object of interest is long (i.e., the temporal concentration of the action-related relations involving the object of interest is small), the information required by the user of the relationship extraction device 2000 may be widely spread in time, so the target period should be long. On the other hand, if the period around the event of interest including the action-related relations involving the object of interest is short (i.e., the temporal concentration of the action-related relations involving the object of interest is large), the information required by the user of the relationship extraction device 2000 may be concentrated in time, so the target period can be short.

[0091] Specifically, the determining unit 2020 determines the adjustment factor based on the temporal concentration of the action-related relationships involved in the object of interest. For example, the determining unit 2020 determines the length of a period that includes the event of interest and includes at least one of the objects of interest. The determining unit 2020 then calculates a ratio between the calculated length of the period and a predefined standard length, and determines the adjustment factor based on the ratio. For example, the ratio may be used as the adjustment factor as is. In another example, a specific function is used to convert the ratio into the adjustment factor.

[0092] Here, the object of interest is one or more objects related to the event of interest. A more detailed description of the object of interest will be provided later.

[0093] <Relationship Extraction: S106> The extraction unit 2040 extracts an action association relationship related to the event of interest based on the target period (S106). Specifically, the extraction unit 2040 extracts an action association relationship that satisfies the condition that "the action association relationship exists during the target period." Hereinafter, this condition will be referred to as the "first condition."

[0094] In order to extract an action-related relation related to the event of interest, the extraction unit 2040 searches for an action-related relation that satisfies a first condition from the object relation information 20. When the start and end of a target period are expressed by date and time, the extraction unit 2040 determines whether an action-related relation exists during the target period by comparing the period 108 with the target period. Specifically, an action-related relation whose period 108 overlaps with the target period is determined to satisfy the first condition. On the other hand, an action-related relation whose period 108 does not overlap with the target period is determined not to satisfy the first condition.

[0095] As described above, the start and end of the target period may be represented by a specific action-related relationship. For example, the start of the target period may be represented by an action-related relationship representing the 10th pass before the goal to the last pass. In this case, the extraction unit 2040 identifies all action-related relationships included between the action-related relationship representing the start of the target period and the action-related relationship representing the end of the target period in the object relation information 20 as action-related relationships that satisfy the first condition.

[0096] For example, it is assumed that the i-th action related relationship included in the object relation information 20 represents the start of a target period, and the j-th action related relationship included in the object relation information 20 represents the end of the target period. In this case, all action related relationships between the i-th and j-th actions in the object relation information 20 are determined to satisfy the first condition.

[0097] In some implementations, an additional condition is utilized to determine whether the action-related relationship is related to the event of interest. One example of the additional condition is that the action-related relationship involves an object of interest. More specifically, the additional condition may be that "the subject, the object, or both of the action-related relationship are objects of interest." Hereinafter, this condition is referred to as the "second condition."

[0098] The object of interest is one or more objects related to the event of interest. In some embodiments, all the subjects and objects of the actions included in the event of interest are treated as the object of interest. In other embodiments, some of all the subjects and objects of the actions included in the event of interest are treated as the object of interest.

[0099] In the latter case, the extraction unit 2040 may treat one or more specific types of objects as the object of interest. For example, assume that a user of the relationship extraction device 2000 is interested in human behavior related to the event of interest. In this case, it is preferable to treat each person who is the subject or object of the action included in the event of interest as the object of interest.

[0100] Alternatively, the extraction unit 2040 can treat one or more specific objects as the object of interest. For example, assume that the event of interest is a crime event such as shoplifting. In this case, the user of the relationship extraction device 2000 may be interested in the behavior of the perpetrator of the event of interest (e.g., a shoplifter). Therefore, it may be preferable to treat the perpetrator of the event of interest as the object of interest.

[0101] When the second condition is also used, the extraction unit 2040 extracts an action related relationship that satisfies both the first condition and the second condition as an action related relationship related to the event of interest. For example, the extraction unit 2040 extracts an action related relationship that satisfies the first condition (i.e., exists during the target period) from the object relation information 20. Next, the extraction unit 2040 extracts an action related relationship that satisfies the second condition (i.e., involves the object of interest) from the extracted action related relationships.

[0102] The extraction unit 2040 can determine whether the action-related relationship satisfies the second condition based on the subject 102 and the object 104 of the action-related relationship. Specifically, the action-related relationship in which the subject 102 or the object 104 indicates the target object is determined to satisfy the second condition. On the other hand, the action-related relationship in which neither the subject 102 nor the object 104 indicates the target object is determined to not satisfy the second condition.

[0103] <Result output> The relationship extraction device 2000 may output information (hereinafter, "output information") related to the extraction result of the action association relationship related to the event of interest. A functional configuration unit that generates the output information is called an "output unit." FIG. 11 shows an example of the functional configuration of the relationship extraction device 2000 including the output unit 2060.

[0104] The output information may include various information, for example, the output unit 2060 generates output information including all the action-associated relations extracted by the extraction unit 2040 as being related to the event of interest.

[0105] In addition to or instead of the action-related relationship related to the event of interest, the output information may include one or more of the video frames in which the action-related relationship related to the event of interest is detected. By providing those video frames, the user of the relation extraction device 2000 can visually and therefore easily understand the scene related to the event of interest. For example, if the event of interest is shoplifting, the user of the relation extraction device 2000 can view video frames capturing the behavior of the shoplifter before, during, and after the shoplifting.

[0106] The output information may include all video frames in which motion related relationships associated with the event of interest are detected, or may include a subset of them, in which case the output information may include a certain number (e.g., one) of video frames for each motion related relationship associated with the event of interest.

[0107] For example, suppose there is an action-related relationship in which a shoplifter picks up a store item. If this relationship exists for 2 seconds, there will be dozens of video frames in which this relationship is detected (e.g., 60 video frames when the camera frame rate is 30 frames / second). In this case, it may be enough for a user to see one or some of the video frames to understand the behavior of the shoplifter picking up the store item. In this way, the output unit 2060 includes a certain number of video frames as a representative action-related relationship for each action-related relationship related to the event of interest.

[0108] There may be various ways to output the output information. For example, the relationship extraction device 2000 may store the output information in a storage device. In another example, the relationship extraction device 2000 may output the output information to a display device, thereby causing the contents of the output information to be displayed on the display device. In another example, the relationship extraction device 2000 may send the output information to another device, for example, a client terminal from which the request described above was sent to the relationship extraction device 2000.

[0109] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be appropriately combined with other embodiments.

[0110] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but to one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create an embodiment not explicitly shown or described, for example. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0111] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0112] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) obtaining event information indicative of one or more characteristics of the event of interest; determining a time period of interest based on the one or more characteristics of the event of interest, the time period of interest including, in part, an event time that is a time or period during which the event of interest occurs; A relationship extraction device that extracts one or more action-related relationships that exist during the target period from object relationship information, and the object relationship information indicates two or more relationships between objects in correspondence with a time or period when the action-related relationships exist. (Appendix 2) the one or more characteristics of the event of interest include a type of the event of interest and an event time; The determination of the time period in question comprises: determining a start time of the period of interest by determining a length of time from the start time of the period of interest to the event time based on the type of the event of interest; and determining an end point of the target period by determining a length of time from the event time to the end point of the target period based on the type of the event of interest. (Appendix 3) The determination of the time period in question comprises: obtaining duration information for each of two or more types of events that indicates a correspondence between the type of event and a duration of the event; determining the start point of the target period and the end point of the target period based on the length of the period indicated by the period information in association with the type of the event of interest. (Appendix 4) The determination of the time period in question comprises: obtaining an adjustment factor, the adjustment factor being a positive real value; and adjusting, based on the adjustment coefficient, the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both. (Appendix 5) The relationship extraction device of claim 4, wherein the adjustment coefficient is determined by a parameter representing how vague the event of interest is to a user, how important the event of interest is to a user, or how interested the user is in the event of interest. (Appendix 6) The relationship extraction device according to any one of appendix 1 to 4, wherein the extraction of the action-related relationship includes extracting from the object relationship information one or more action-related relationships that include objects that exist during the target period and are related to the event of interest. (Appendix 7) the object relation information is generated by detecting each of the motion related relations from one or more video frames; The at least one processor executes the instructions to further: and for each of the extracted motion association relationships, output information including one or more video frames in which the extracted motion association relationship was detected. 5. The relationship extraction device according to claim 1, configured as follows: (Appendix 8) 8. The relationship extraction device of claim 7, wherein the output information includes, for each of the extracted behavioral relationship relations, one or more predetermined number of video frames in which the extracted behavioral relationship is detected. (Appendix 9) obtaining event information indicative of one or more characteristics of the event of interest; determining a time period of interest based on the one or more characteristics of the event of interest, the time period of interest including, in part, an event time, which is a time or period when the event of interest occurs; A relationship extraction method comprising extracting one or more action-related relationships that exist during the target period from object relationship information, the object relationship information indicating two or more relationships between objects in correspondence with a time or period when the action-related relationships exist. (Appendix 10) the one or more characteristics of the event of interest include a type of the event of interest and an event time; The determination of the time period in question comprises: determining a start time of the period of interest by determining a length of time from the start time of the period of interest to the event time based on the type of the event of interest; and determining an end point of the period of interest by determining a length of time from the event time to the end point of the period of interest based on the type of the event of interest. (Appendix 11) The determination of the time period in question comprises: obtaining duration information for each of two or more types of events that indicates a correspondence between the type of event and a duration of the event; determining the start point of the target period and the end point of the target period based on the length of the period indicated by the period information in association with the type of the event of interest. (Appendix 12) The determination of the time period in question comprises: obtaining an adjustment factor, the adjustment factor being a positive real value; and adjusting, based on the adjustment coefficient, the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both. (Appendix 13) The relationship extraction method of claim 12, wherein the adjustment coefficient is determined by a parameter representing how vague the event of interest is to a user, how important the event of interest is to a user, or how interested the user is in the event of interest. (Appendix 14) The relationship extraction method according to any one of appendixes 9 to 12, wherein the extraction of the action-related relationship includes extracting one or more action-related relationships from the object relationship information, the action-related relationships including objects that exist during the target period and are related to the event of interest. (Appendix 15) the object relation information is generated by detecting each of the motion related relations from one or more video frames; 13. The relationship extraction method of any one of appendixes 9 to 12, further comprising: outputting, for each of the extracted action related relationships, output information including one or more video frames in which the extracted action related relationship was detected. (Appendix 16) 16. The relationship extraction method of claim 15, wherein the output information includes, for each of the extracted action relationship relations, one or more predetermined number of video frames in which the extracted action relationship relation was detected. (Appendix 17) A program, On the computer, obtaining event information indicative of one or more characteristics of an event of interest; determining a time period of interest based on the one or more characteristics of the event of interest, the time period of interest including, in part, an event time, which is a time or period when the event of interest occurs; A program that executes the extraction of one or more behavioral association relationships that exist during the target period from object relationship information, the object relationship information indicating two or more relationships between objects in correspondence with a time or period when the behavioral association relationship exists. (Appendix 18) the one or more characteristics of the event of interest include a type of the event of interest and an event time; The determination of the time period in question comprises: determining a start time of the period of interest by determining a length of time from the start time of the period of interest to the event time based on the type of the event of interest; and determining an end point of the period of interest by determining a length of time from the event time to the end point of the period of interest based on the type of the event of interest. (Appendix 19) The determination of the time period in question comprises: obtaining duration information for each of two or more types of events that indicates a correspondence between the type of event and a duration of the event; and determining the start point of the target period and the end point of the target period based on a length of the period indicated by the period information in association with the type of the event of interest. (Appendix 20) The determination of the time period in question comprises: obtaining an adjustment factor, the adjustment factor being a positive real value; and adjusting, based on the adjustment coefficient, the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both. (Appendix 21) 21. The relationship extraction method of claim 20, wherein the adjustment coefficient is determined by a parameter representing how vague the event of interest is to a user, how important the event of interest is to a user, or how interested the user is in the event of interest. (Appendix 22) 21. The relationship extraction method according to any one of appendices 17 to 20, wherein the extraction of the action-related relationships includes extracting one or more action-related relationships from the object relationship information, the action-related relationships including objects that exist during the target period and are related to the event of interest. (Appendix 23) the object relation information is generated by detecting each of the motion related relations from one or more video frames; 21. The relationship extraction method of any one of appendixes 17 to 20, wherein the program further causes the computer to output, for each of the extracted action relationship relationships, output information including one or more video frames in which the extracted action relationship relationship was detected. (Appendix 24) 24. The relationship extraction method of claim 23, wherein the output information includes, for each of the extracted action relationship relations, one or more predetermined number of video frames in which the extracted action relationship relation was detected.

[0113] This application claims priority to Singapore Patent Application No. 10202302053T, filed on July 20, 2023, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0114] 10. Event Information 20 Object-related information 60 Scene Graph 100 Tables 102 Subject 104 Object 106 Operation 108 period 110 Start time 112 End time 200 Tables 202 Identifier 204 Type 206 position 300 Tables 302 Type 304 period 306 Leading Period 308 Subsequent Period 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Relationship Extraction Device 2020 Decision Section 2040 Extraction part 2060 Output section

Claims

1. obtaining event information indicative of one or more characteristics of the event of interest; determining a time period of interest based on the one or more characteristics of the event of interest, the time period of interest including, in part, an event time that is a time or period during which the event of interest occurs; A relationship extraction device that extracts one or more action-related relationships that exist during the target period from object relationship information, and the object relationship information indicates two or more relationships between objects in correspondence with a time or period when the action-related relationships exist.

2. the one or more characteristics of the event of interest include a type of the event of interest and an event time; The determination of the time period in question comprises: determining a start time of the period of interest by determining a length of time from the start time of the period of interest to the event time based on the type of the event of interest; and determining an end point of the period of interest by determining a length of time from the event time to the end point of the period of interest based on the type of the event of interest.

3. The determination of the time period in question comprises: obtaining duration information for each of two or more types of events that indicates a correspondence between the type of event and a duration of the event; The relationship extraction device of claim 2, further comprising: determining the start point of the target period and the end point of the target period based on the length of the period indicated by the period information in correspondence with the type of the event of interest.

4. The determination of the time period in question comprises: obtaining an adjustment factor, the adjustment factor being a positive real value; and adjusting, based on the adjustment coefficient, the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both.

5. The relationship extraction device according to claim 4 , wherein the adjustment coefficient is determined by a parameter representing how vague the event of interest is to a user, how important the event of interest is to a user, or how interested the user is in the event of interest.

6. The relationship extraction device according to claim 1 , wherein the extraction of the action-related relationships includes extracting, from the object relationship information, one or more action-related relationships that include objects that exist during the target period and are related to the event of interest.

7. the object relation information is generated by detecting each of the motion related relations from one or more video frames; The at least one processor executes the instructions and further comprises: and for each of the extracted action related relationships, output information including one or more video frames in which the extracted action related relationship was detected. The relationship extraction device according to claim 1 , configured to:

8. The apparatus of claim 7 , wherein the output information comprises, for each of the extracted action-related relations, one or more predetermined number of video frames in which the extracted action-related relation was detected.

9. obtaining event information indicative of one or more characteristics of the event of interest; determining a time period of interest based on the one or more characteristics of the event of interest, the time period of interest including, in part, an event time, which is a time or period when the event of interest occurs; A relationship extraction method comprising extracting one or more action-related relationships that exist during the target period from object relationship information, the object relationship information indicating two or more relationships between objects in correspondence with a time or period when the action-related relationships exist.

10. A program, On the computer, obtaining event information indicative of one or more characteristics of an event of interest; determining a time period of interest based on the one or more characteristics of the event of interest, the time period of interest including, in part, an event time, which is a time or period when the event of interest occurs; A program that executes the extraction of one or more behavioral association relationships that exist during the target period from object relationship information, the object relationship information indicating two or more relationships between objects in correspondence with a time or period when the behavioral association relationship exists.

Images