Relationship extraction device, relationship extraction method, and program

Abstract

To provide novel technology for providing information related to a notable event.SOLUTION: A relation extraction device acquires event information indicative of the characteristics of a notable event, determines an object period based on one or more characteristics of the notable event, and extracts one or more operation-related relation existing in the object period from matter-related information. The matter-related information indicates two or more operation-related relations between matters, in association with a time point or a period where the operation-related relation exists.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates generally to a relation extraction device, a relation extraction method, and a storage medium.

Background Art

[0002] There is a technology for providing information on a target event. Patent Document 1 discloses a technology for detecting an important scene that is part of an event from a source video and extracting a key frame for that scene.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The key frame provided by Patent Document 1 represents a part of an event. Therefore, Patent Document 1 does not provide information that is not included in the event. The object of the present disclosure is to provide a novel technology for providing information related to a target event.

Means for Solving the Problems

[0005] This disclosure provides a relationship extraction device comprising at least one memory configured to store instructions and at least one processor. The at least one processor is configured to execute instructions to obtain event information indicating one or more features of an event of interest, to determine a target period based on the one or more features of the event of interest, the target period including, as part thereof, an event time which is the time or period during which the event of interest occurs, to extract one or more operational relationship relationships existing during the target period from object relationship information, the object relationship information being configured to indicate two or more relationships between objects in association with the time or period during which the operational relationship relationships exist.

[0006] This disclosure provides a relationship extraction method, which includes obtaining event information that shows one or more features of an event of interest, determining a target period based on the one or more features of the event of interest, the target period including, as part thereof, an event time which is the time or period during which the event of interest occurs, and extracting one or more action-related relationships that exist during the target period from object-related information, the object-related information showing two or more relationships between objects in association with the time or period during which the action-related relationships exist.

[0007] This disclosure provides a storage medium on which a program is stored, the program causing a computer to perform the task of obtaining event information that shows one or more features of an event of interest, and determining a target period based on the one or more features of the event of interest, the target period including, as part thereof, an event time which is the time or period during which the event of interest occurs, and causing the computer to perform the task of extracting one or more behavioral relationships that exist during the target period from object relationship information, the object relationship information showing two or more relationships between objects in association with the time or period during which the behavioral relationships exist. [Effects of the Invention]

[0008] This disclosure provides a novel technology that provides information related to notable events. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the relationship extraction device. [Figure 2] This is a block diagram showing an example of the functional configuration of a relation extraction device. [Figure 3] This block diagram shows an example of a computer hardware configuration for implementing a relation extraction device. [Figure 4] This flowchart shows an exemplary flow of the process performed by the relation extraction device. [Figure 5] This diagram shows an example structure of object-related information in table format. [Figure 6] This diagram shows an example structure of object information in table format. [Figure 7] This is a diagram showing an example of a scene graph. [Figure 8] This is a flowchart showing the processing flow performed by the object relationship information generation device. [Figure 9] This diagram shows an example structure of period information in table format. [Figure 10] This figure shows the target period, which is determined based on a predefined length of time related to the type of event. [Figure 11] This figure shows an example of the functional configuration of the relation extraction device 2000, including the output unit. [Modes for carrying out the invention]

[0010] Illustrative embodiments of this disclosure are described in detail below with reference to the drawings. In the drawings, the same or corresponding elements are denoted by the same reference numerals, and redundant descriptions are omitted where necessary for clarity. Unless otherwise specified, predetermined information (e.g., predetermined values ​​or predetermined thresholds) is pre-stored in a storage device accessible to the computer using that information. Furthermore, unless otherwise specified, the storage unit consists of one or more storage devices.

[0011] <Summary> FIG. 1 shows an overview of the relationship extraction device 2000. The overview shown in FIG. 1 shows an example of the operation of the relationship extraction device 2000 for the purpose of facilitating understanding of the relationship extraction device 2000, and does not limit or narrow the possible range of operations of the relationship extraction device 2000.

[0012] The relationship extraction device 2000 is used to extract one or more time-series operation-related relationships between objects predicted to be related to the target event from the object relationship information 20. The object relationship information 20 represents a sequence of two or more time-series operation-related relationships between objects, and each of them is an operation-related relationship between objects that exist at a certain point in time or during a certain period detected from one or more video frames.

[0013] The time-series operation-related relationship can be represented by a combination of 1) the type of operation, 2) the subject of the operation, 3) the object of the operation, and 4) the time when the operation was performed. Suppose there is an operation-related relationship that a person P1 holds a store product I1 in his hand from time point T1 to T2. The object relationship information 20 can represent this relationship by a combination of 1) the type of operation: pick up, 2) the subject: person P1, 3) the object: store product I1, and 4) the time: from T1 to T2.

[0014] The target event is an operation-related event, which is any type of event involving one or more operations taken by an object. The types of operation-related events can include crime events (e.g., shoplifting, luggage theft), accidents (e.g., car accidents, luggage forgetting), sports events (e.g., goal events in a soccer game, home runs in a baseball game), and customer events (e.g., purchases).

[0015] The operation-related relationship is predicted to be related to the target event at least when the operation-related relationship exists during a certain period related to the target event. Hereinafter, the period related to the target event is called the "target period".

[0016] To extract operation-related relationships associated with a target event, the relationship extraction device 2000 acquires event information 10 and determines a target period. The event information 10 includes information that can identify the characteristics of the target event.

[0017] The characteristics of the target event include the time when the target event occurred. Hereinafter, the time when the target event occurred is also referred to as the "event time". The event time can be represented by a specific point in time or a specific period. The characteristics of the target event also include the type of the target event, such as purchase, shoplifting, etc.

[0018] The relationship extraction device 2000 determines the target period based on the event time and the type of the target event. The target period includes the event time as a part thereof. For example, the length of the period can be predefined for each type of event. In this case, the relationship extraction device 2000 determines the predefined length of the period corresponding to the type of the target event. Next, the relationship extraction device 2000 determines, as the target period, a period that includes the event time and whose length is determined based on the predefined length of the period corresponding to the type of the target event.

[0019] Based on the target period, the relationship extraction device 2000 extracts operation-related relationships associated with the target event by searching for them from the object relationship information 20. Specifically, the operation-related relationships existing during the target period are extracted from the object relationship information 20.

[0020] <Example of the effect> As described above, the relationship extraction device 2000 determines a target period that includes the event time as a part thereof based on the characteristics of the target event indicated by the event information 10. Next, the relationship extraction device 2000 extracts, from the object relationship information 20, the operation-related relationships existing during the target period as the operation-related relationships associated with the target event.

[0021] According to the above-described operation of the relationship extraction device 2000, a novel technique is provided for providing information related to the event of interest. Specifically, the operation-related relationships existing during the target period include information about events that occurred before the event of interest, after the event of interest, or both. Therefore, the relationship extraction device 2000 facilitates the understanding of events that occurred before the event of interest, after the event of interest, or both, thereby facilitating a detailed understanding of the event of interest.

[0022] Furthermore, the relationship extraction device 2000 determines the target period based on the characteristics of the event of interest, such as the type of event of interest. This allows the relationship extraction device 2000 to consider the characteristics of the event of interest in order to determine what information it provides.

[0023] A more detailed description of the relation extraction device 2000 is provided below.

[0024] <Example of functional configuration> Figure 2 is a block diagram showing an example of the functional configuration of the relationship extraction device 2000. The relationship extraction device 2000 includes a determination unit 2020 and an extraction unit 2040. The determination unit 2020 acquires event information 10 and determines the target period based on the type of event of interest and the event time indicated by the event information 10. The extraction unit 2040 extracts the operation relationship relationships that exist during the target period as operation relationship relationships related to the event of interest.

[0025] <Example hardware configuration> The relation extraction device 2000 can be implemented by one or more computers. Figure 3 is a block diagram showing an example of the hardware configuration of computer 1000 that implements the relation extraction device 2000. Computer 1000 can be any type of computer. For example, computer 1000 is a stationary computer such as a personal computer (PC) or a server machine. In another example, computer 1000 could be a smartphone or tablet This refers to a mobile computer such as a terminal. In another example, computer 1000 is an integrated circuit such as a SoC (System on Chip). Computer 1000 may be a dedicated computer manufactured to implement the relation extraction device 2000, or it may be a general-purpose computer.

[0026] The relation extraction device 2000 can be realized by installing an application on the computer 1000. The application is implemented as a program that makes the computer 1000 function as the relation extraction device 2000. In other words, the program is the implementation form of the functional components of the relation extraction device 2000.

[0027] There are various ways to obtain a program. For example, a program can be obtained from a storage medium on which it is stored (e.g., a DVD disc or USB memory stick). In another example, a program can be downloaded from a server that manages the storage medium on which it is stored.

[0028] In Figure 3, computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120. Bus 1020 is a data transmission channel for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. Processor 1040 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), or FPGA (Field-Programmable Gate Array). Memory 1060 is a main memory element such as RAM (Random Access Memory) or ROM (Read Only Memory). Storage device 1080 is an auxiliary storage element such as a hard disk, SSD (Solid State Drive), or memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, mouse, or display device. The network interface 1120 is an interface between the computer 1000 and a network. The network can be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0029] The processor 1040 may be configured to load the instructions of the program described above from the storage device 1080 into the memory 1060, execute those instructions, and operate the computer 1000 as a relation extraction device 2000.

[0030] The hardware configuration of computer 1000 is not limited to that shown in Figure 3. For example, as described above, the relation extraction device 2000 can be implemented as a combination of multiple computers. In this case, these computers can be connected to each other via a network.

[0031] <Processing flow> Figure 4 is a flowchart illustrating an exemplary flow of processing performed by the relationship extraction device 2000. The decision unit 2020 acquires event information 10 (S102). The decision unit 2020 determines the target period based on the characteristics of the event of interest (S104). The extraction unit 2040 extracts the operation-related relationships that exist during the target period (S106).

[0032] <Regarding object relationship information 20> As described above, object relationship information 20 represents two or more action relationship relationships between objects. Object relationship information 20 can be generated by the relationship extraction device 2000 or another device. Hereinafter, the device that generates object relationship information 20 will be referred to as the "object relationship information generation device".

[0033] Each object relationship information generation device generates object relationship information 20 based on one or more sequences of video frames (in other words, one or more video data) generated by a camera. Specifically, the object relationship information generation device analyzes the scene captured in the video frame, detects the motion relationship between objects captured in the video frame, and thereby generates object relationship information 20.

[0034] There are various ways to acquire video data. For example, a camera can be configured to transmit video data to an object-relationship information generation device. In this case, the object-relationship information generation device receives the video data transmitted by the camera and acquires the video data.

[0035] In another example, a camera is configured to generate video frames and then transmit them to an object-relationship information generator. In this case, the object-relationship information generator receives the video frames transmitted by the camera and generates video data from the received video frames.

[0036] In another example, the camera is configured to store video data in a memory unit accessible from the object-relationship information generation device. In this case, the object-relationship information generation device retrieves the video data from the memory unit.

[0037] Figure 5 shows an exemplary structure of object relationship information 20 in table format. In Figure 5, object relationship information 20 is represented by table 100. Table 100 has columns named "Subject 102", "Object 104", "Action 106", and "Period 108". Action 106 indicates the type of action. Subject 102 indicates the identifier of the object that performs the corresponding action. Object 104 indicates the identifier of the object that the corresponding subject is targeting when the corresponding action is performed. Period 108 indicates when the corresponding action relationship exists and when it lasts. Specifically, Period 108 consists of two columns named "Start Time 110" and "End Time 112". Start Time 110 indicates the time when the corresponding action relationship begins. End Time 112 indicates the time when the corresponding action relationship ends.

[0038] Subject 102 and object 104 are represented by object identifiers. Each object identifier may be defined by other information called "object information," which is similarly generated from video data by an object relationship information generation device. The object information may indicate, for each object detected from the video data, the object identifier and the type of object (e.g., person, store merchandise, bag, etc.).

[0039] In video data, the position of each object may change. Therefore, it is preferable that object information show a pair of time and position for each object. In other words, object information shows the time series of positions for each object.

[0040] There are various ways to represent the position of an object. For example, an object's position can be represented by its coordinates on the video frame in which it is located. When an object relationship information generator handles two or more video data sets, the position of an object can be represented by a pair of camera identifiers and the coordinates on the video frame in which the object is located.

[0041] In another example, the position of an object may be represented by coordinates on a map of an area captured by one or more cameras. The map may be a two-dimensional map or a three-dimensional map.

[0042] In this case, the object relationship information generation device converts the coordinates of the object on the video frame to coordinates on a map. By using a map, the positions of objects captured by different cameras can be represented by coordinates in a unified coordinate space. In addition, by using a map, the object relationship information generation device can handle cameras whose field of view can be changed (for example, pan-tilt-zoom cameras).

[0043] Figure 6 shows an exemplary structure of object information in table format. In Figure 6, object information is represented by Table 200. Table 200 includes columns named "Identifier 202", "Type 204", and "Location 206". Identifier 202 shows the identifier assigned to the corresponding object. Type 204 shows the type of the corresponding object. Location 206 shows a sequence of time and location pairs for the corresponding object.

[0044] The relationship between objects at a given point in time can also be represented by a scene graph, where each object is represented by a node and the relationship between objects is represented by an edge. The object relationship information 20 can be said to represent a sequence of the scene graph. Therefore, the relationship extraction device 2000 can be used to search for a sequence of relationship between objects related to an event of interest from the scene graph.

[0045] Figure 7 shows an example of a scene graph. In the example shown in Figure 7, three objects are detected: a person assigned identifier 001, a bag assigned identifier 002, and another person assigned identifier 003. Therefore, the scene graph 60 contains three nodes, each representing person 001, bag 002, and person 003.

[0046] Person 001 and bag 002 are tagged with "place" and are connected to each other by an edge from person 001 to bag 002. This action relationship represents person 001 placing bag 002. Person 003 is not connected to anything. This means that person 003 does not perform any action.

[0047] The object relationship information generation device generates object relationship information 20 from video data. For example, for each video frame contained in the video data, the object relationship information generation device performs object detection to generate object information, and then detects the behavioral relationship of the object detected through object detection.

[0048] Figure 8 is a flowchart showing the processing flow performed by the object relationship information generation device. The object relationship information generation device initializes the object information and object relationship information 20.

[0049] Steps S204 to S210 constitute a loop process L1 that is executed for each video frame included in the video data. In step S204, the object relationship information generation device determines whether or not the loop process L1 has been executed for all video frames. If the loop process L1 has been executed for all video frames, the loop process L1 terminates.

[0050] If loop processing L1 has not yet been performed on all video frames, the object relationship information generator selects the next video frame on which loop processing L1 should be performed. The video frame selected here is the one with the earliest generation time (e.g., the smallest frame number) among the video frames on which loop processing L1 has not yet been performed. The video frame selected here is denoted as "video frame i".

[0051] The object relationship information generator performs object detection on video frame i to detect an object from video frame i and update the object information (S206). If the object detected in video frame i was not detected in the preceding video frame, the object relationship information generator assigns a new identifier to this object and adds a new record about this object to the object information. If the object detected in video frame i was detected in the preceding video frame, the object relationship information generator updates the record for this object in the object information by adding a time and position pair for this object to the record. The time in this pair represents the time when video frame i was generated. The position in this pair represents the position of the object on video frame i.

[0052] The object relationship information generator performs detection of motion relationship relationships between objects detected from video frame i and updates the object relationship information 20 (S208). If a motion relationship between specific objects detected from video frame i is also detected from video frame (i-1), the object relationship information generator updates the record of this motion relationship in the object relationship information 20 to increase the duration of this relationship. On the other hand, if a motion relationship between specific objects detected from video frame i is not detected from video frame (i-1), the object relationship information generator generates a new record for this relationship and adds this record to the object relationship information 20.

[0053] Step 210 marks the end of loop processing L1. Therefore, the object relationship information generation device then executes step S204.

[0054] <Regarding Event Information 10> Event information 10 provides information that can identify the characteristics of a notable event. As mentioned above, the characteristics of a notable event include the type of the notable event and the event time, which is the time when the notable event occurred. Thus, event information 10 shows the type of the notable event and the event time.

[0055] In addition, event information 10 can indicate objects related to the event of interest. Objects related to the event of interest include the subject of the action involved in the event of interest and the object of the action involved in the event of interest. Suppose the event of interest is a purchase, where "at time t1, object 001 (person) purchased object 002 (store goods)." In this case, the objects related to the event of interest are object 001 and object 002. Event information 10 can indicate 1) event type = purchase, 2) event time = t1, 3) purchaser = object 001, and 4) purchased item = object 002.

[0056] In some embodiments, the event of interest is represented by a sequence of action-related situations, each of which is a situation in which a particular subject performs a particular action on a particular object. In this case, the event information 10 can indicate the subject of the action involved in the event of interest and the object of the action involved in the event of interest as objects related to the event of interest.

[0057] For example, when the event of interest is a purchase, this event can be represented as follows: 1. Object 001 (person) picks up object 002 (store merchandise) at time t1. 2. Object 001 (person) stands in front of object 003 (cash register) from time t2 to t3. 3. Object 001 (person) passes through object 004 (store exit) at time t4.

[0058] In this case, the objects of interest may include object 001, object 002, object 003, and object 004. The event information 10 may indicate 1) event type = purchase, 2) event time = t1 to t4, 3) buyer = object 001, 4) purchased items = object 002, 5) cash register = object 003, and 6) exit = object 004.

[0059] Here, event information 10 does not necessarily indicate all objects related to the event of interest. Specifically, event information 10 may indicate only certain types of objects related to the event of interest. For example, with respect to the purchase described above, event information 10 may indicate only the purchaser and one or more purchased items, and may not indicate the checkout or exit. The types of objects to be indicated by event information 10 may be predefined for each type of event.

[0060] <Retrieving event information 10: S102> The decision unit 2020 acquires the event information 10 (S102). There are various ways in which the decision unit 2020 can acquire the event information 10. For example, the event information 10 is pre-stored in a memory unit accessible from the relation extraction device 2000. In this case, the decision unit 2020 reads the event information 10 from this memory unit and acquires it. In another example, another device transmits the event information 10 to the relation extraction device 2000. In this case, the decision unit 2020 receives the event information 10 and acquires it.

[0061] There can be various triggers for the decision unit 2020 to acquire event information 10 (in other words, to start executing the process shown in Figure 4). For example, the relationship extraction device 2000 receives a request to provide information about operational relationships related to the event of interest and acquires event information 10 in response to the request. In this case, the request includes information that can identify the event information 10 to be acquired.

[0062] For example, the request includes event information 10. In this case, the decision unit 2020 obtains the event information 10 by extracting it from the request.

[0063] In another example, the request specifies the identifier of the event information 10 to be retrieved. For example, suppose the event information 10 is stored as a file in the storage unit. In this case, the request specifies the file name of the event information 10 to be retrieved. The decision unit 2020 retrieves the file with the specified name from the storage unit as the event information 10.

[0064] There are various ways to provide a request to the relation extraction device 2000. For example, a user of the relation extraction device 2000 can input a request by operating an input device attached to the relation extraction device 2000. In another example, the user can operate another device, such as a client terminal, and send a request from that device to the relation extraction device 2000.

[0065] Requests are not necessarily generated based on user input. In some embodiments, there is a device called an "event detection device" configured to detect specific events from video data. In this case, the events detected by the event detection device are treated as events of interest. The event detection device generates a request for information on the operational relationships related to the events detected by the event detection device and sends the request to the relationship extraction device 2000.

[0066] <Determination of the target period: S104> The decision unit 2020 determines the target period based on the event information 10 (S104). In some embodiments, the length of the period may be predefined for each type of event. This allows the relationship extraction device 2000 to determine what information to provide, taking into account the type of event of interest.

[0067] In this case, there is information called "duration information" in which each event type is associated with data indicating the length of the duration. Figure 9 shows an exemplary structure of duration information in table form. In Figure 9, duration information is represented by Table 300. Table 300 includes columns for "Type 302" and "Duration 304". Type 302 indicates the type of event. Duration 304 shows data representing the length of the duration corresponding to the event type.

[0068] Period 304 includes two columns: "Preceding Period 306" and "Subsequent Period 308". Preceding Period 306 shows data representing the length of the period included in the target period and preceding the event time. Subsequent Period 308 shows data representing the length of the period included in the target period and following the event time.

[0069] The preceding period 306, the succeeding period 308, or both, may indicate the length of the period in typical time units such as seconds, minutes, or hours. For example, the first row of Table 300 shown in Figure 9 shows "Preceding period: 10 minutes" and "Successing period: 5 minutes".

[0070] Figure 10 shows the target period, which is determined based on a predefined length of time related to the event type. Event information 10 indicates that 1) the type of event in question is shoplifting, and 2) the event time is from "2023 / 6 / 20 15:20" to "2023 / 6 / 20 15:13". The period information associates "Type: Shoplifting" with "Preceding period: 10 minutes" and "Successing period: 5 minutes".

[0071] The start date of the featured event is "2023 / 6 / 20 15: 2Since the time is 0 and the length of the preceding period is 10 minutes, the start time of the target period is determined to be "2023 / 6 / 20 15:10". In addition, since the end time of the featured event is "2023 / 6 / 20 15:13" and the length of the following period is 5 minutes, the end time of the target period is determined to be "2023 / 6 / 20 15:18". As a result, the decision unit 2020 determines that the target period is from "2023 / 6 / 20 15:10" to "2023 / 6 / 20 15:18".

[0072] In another example, the preceding period 306 can be expressed in terms of its length using a condition that identifies an action relationship that represents the start of the period in question. Similarly, the succeeding period 308 can be expressed in terms of its length using a condition that identifies an action relationship that represents the end of the period in question.

[0073] Suppose the type of event of interest is a "goal event in a soccer match." In this case, when a goal is scored, the user of the relation extraction device 2000 (for example, a viewer of a soccer match video) may be interested in some play that preceded the goal. Therefore, the preceding period 306 can indicate one or more plays that should be included in the target period. For example, the second row of the table 300 shown in Figure 9 shows "Preceding period: 10 passes before the goal." In this case, the decision unit 2020 can identify the action relation representing the 10th pass before the goal up to the last pass as the action relation representing the start of the target period.

[0074] For example, the second row of Table 300 shown in Figure 9 indicates "Subsequent Period: Goal Celebration". This definition means that the goal celebration should be included in the target period. The decision unit 2020 can identify that the action relationship representing the end of the goal celebration is the same as the action relationship representing the end of the target period.

[0075] The preceding period 306 may directly indicate a specific operational relationship that should occur at the start of the target period. Similarly, the succeeding period 308 may directly indicate a specific operational relationship that should occur at the end of the target period.

[0076] For example, the third row of Table 300 shown in Figure 9 shows "Preceding Period: Buyer enters the store" and "Subsequent Period: Buyer leaves the store" in relation to "Type: Purchase". In this case, the decision unit 2020 can identify the action relationship representing the buyer entering the store as the action relationship representing the start of the target period. Similarly, the decision unit 2020 can identify the action relationship representing the buyer leaving the store as the action relationship representing the end of the target period.

[0077] <<Adjustment of duration>> In some embodiments, the length of the target period can be adjusted. For example, the relationship extraction device 2000 adjusts the length of the target period based on user input. This allows the relationship extraction device 2000 to determine the target period while taking the user's preferences into account.

[0078] For example, a user of the relation extraction device 2000 can specify an ambiguity parameter, which represents how ambiguous the event of interest is to the user. Conceptually, the more ambiguous the event of interest is to the user, the more information the user needs to understand it, and therefore the target period should be set to be longer. Suppose the event of interest is shoplifting, and the user of the relation extraction device 2000 is an employee of the store where the shoplifting occurred. If the employee knows the shoplifter well (for example, this shoplifter often comes to the store), the event of interest may not be very ambiguous to the employee. Therefore, the employee will specify a lower value for the ambiguity parameter.

[0079] On the other hand, if the store clerk doesn't know the shoplifter well (for example, if this is the first time this shoplifter has been to the store), the event of interest may be ambiguous to the clerk. Therefore, the clerk will specify a higher ambiguity parameter value.

[0080] In another example, suppose the event of interest is a goal in a soccer match, and the user of the relation extraction device 2000 is a viewer of the match. If the user is intently watching the match, the goal event is not ambiguous to the user. Therefore, the user will specify a lower value for the ambiguity parameter. On the other hand, if the user is talking to a friend while watching the match, the goal event may be ambiguous to the user, for example, if a goal is scored while the user is not watching the match but is watching a friend. In this case, the user will specify a higher value for the ambiguity parameter.

[0081] The decision unit 2020 can adjust the length of the preceding period, the length of the succeeding period, or both, based on the ambiguity parameter. When adjusting the length of the preceding period, the decision unit 2020 can determine an adjustment factor based on the ambiguity parameter and multiply the adjustment factor by the length of time represented by the preceding period 306. The decision unit 2020 uses the adjusted length of the preceding period as the length of the period from the start of the target period to the event time. The adjustment factor is a positive real value within a predefined range.

[0082] Similarly, when adjusting the length of the subsequent period, the determination unit 2020 multiplies the adjustment coefficient by the length of time represented by the preceding period 306. The determination unit 2020 then uses the adjusted length of the subsequent period as the length of the period from the event time to the end of the target period.

[0083] The adjustment coefficient can be the ambiguity parameter itself, or a value obtained by transforming the ambiguity parameter using a predefined function. In the latter case, the decision unit 2020 inputs the ambiguity parameter into a predefined function, thereby obtaining the adjustment coefficient.

[0084] Assume that the lead period 306 is 10 minutes, the ambiguity parameter has a predefined range from 0.5 to 1.5, and the ambiguity parameter is used directly as the adjustment factor. A user who wants to extend the lead period specifies an ambiguity parameter greater than 1. For example, a user can extend the lead period to 15 minutes by specifying an ambiguity parameter of 1.5.

[0085] On the other hand, users who want to shorten the lead time can specify an ambiguity parameter less than 1. For example, a user can shorten the lead time to 5 minutes by specifying an ambiguity parameter of 0.5.

[0086] The adjustment coefficients for adjusting the length of the preceding period and the adjustment coefficients for adjusting the length of the succeeding period may be the same or different. In the latter case, for example, a function that converts the ambiguity parameter to the adjustment coefficient used for the preceding period and a function that converts the ambiguity parameter to the adjustment coefficient used for the succeeding period are defined separately in advance.

[0087] The length of the target period is not necessarily adjusted by the ambiguity parameter. For example, the decision unit 2020 allows the user to specify an importance parameter that represents how important a particular event is to the user. In this case, the adjustment coefficient is set to increase as importance increases.

[0088] In another example, the decision unit 2020 allows the user to specify a curiosity parameter, which represents how interested the user is in learning about a particular event. In this case, the adjustment coefficient is set to be larger the greater the curiosity parameter.

[0089] In some embodiments, the relationship extraction device 2000 adjusts the length of the target period based on the temporal concentration of action-related relationships, including objects associated with the event of interest. Hereinafter, objects associated with the event of interest will be referred to as "objects of interest."

[0090] If the period around the event of interest, which includes the action-related relationships involving the object of interest, is long (i.e., the temporal concentration of the action-related relationships involving the object of interest is small), the target period should be long because the information required by the user of the relationship extraction device 2000 may be widely dispersed over time. On the other hand, if the period around the event of interest, which includes the action-related relationships involving the object of interest, is short (i.e., the temporal concentration of the action-related relationships involving the object of interest is large), the target period may be short because the information required by the user of the relationship extraction device 2000 may be concentrated over time.

[0091] Specifically, the determination unit 2020 determines the adjustment coefficient based on the temporal concentration of action-related relationships involving the object of interest. For example, the determination unit 2020 determines the length of a period that includes the event of interest and includes at least one of the objects of interest. The determination unit 2020 then calculates the ratio of this calculated length to a predefined standard length and determines the adjustment coefficient based on this ratio. For example, this ratio may be used directly as the adjustment coefficient. In another example, a specific function is used to convert this ratio into an adjustment coefficient.

[0092] Here, the object of interest is one or more objects related to the event of interest. A more detailed explanation of the object of interest will be provided later.

[0093] <Relationship extraction: S106> The extraction unit 2040 extracts action-related relationships associated with the event of interest based on the target period (S106). Specifically, the extraction unit 2040 extracts action-related relationships that satisfy the condition that "the action-related relationship exists during the target period." Hereinafter, this condition will be referred to as the "first condition."

[0094] To extract action-related relationships associated with the event of interest, the extraction unit 2040 searches the object relationship information 20 for action-related relationships that satisfy the first condition. If the start and end of the target period are represented by date and time, the extraction unit 2040 compares the period 108 with the target period to determine whether or not an action-related relationship exists during the target period. Specifically, an action-related relationship in which the period 108 overlaps with the target period is determined to satisfy the first condition. On the other hand, an action-related relationship in which the period 108 does not overlap with the target period is determined not to satisfy the first condition.

[0095] As described above, the start and end of the target period can be represented by specific action relationship relationships. For example, the start of the target period may be represented by an action relationship relationship that represents the period from the 10th pass before the goal to the final pass. In this case, as an action relationship relationship that satisfies the first condition, the extraction unit 2040 identifies all action relationship relationships in the object relationship information 20 that are contained between the action relationship relationship representing the start of the target period and the action relationship relationship representing the end of the target period.

[0096] For example, suppose the i-th action relationship in object relationship information 20 represents the start of the target period, and the j-th action relationship in object relationship information 20 represents the end of the target period. In this case, all action relationships between the i-th and j-th objects in object relationship information 20 are determined to satisfy the first condition.

[0097] In some embodiments, additional conditions are used to determine whether an action relationship is related to an event of interest. One example of an additional condition is that an object of interest is involved in the action relationship. More specifically, the condition that "the subject, object, or both of the action relationship are the object of interest" can be used as an additional condition. Hereinafter, this condition will be referred to as the "second condition."

[0098] The object of interest is one or more objects associated with the event of interest. In some embodiments, all subjects and objects of the action included in the event of interest are treated as objects of interest. In other embodiments, some of all subjects and objects of the action included in the event of interest are treated as objects of interest.

[0099] In the latter case, the extraction unit 2040 may treat one or more specific types of objects as objects of interest. For example, suppose the user of the relation extraction device 2000 is interested in the human behavior associated with an event of interest. In this case, it is preferable to treat each person who is the subject or object of the action included in the event of interest as an object of interest.

[0100] Alternatively, the extraction unit 2040 can handle one or more specific objects as objects of interest. For example, suppose the event of interest is a criminal event such as shoplifting. In this case, the user of the relation extraction device 2000 may be interested in the behavior of the perpetrator of the event of interest (e.g., the shoplifter). Therefore, it may be preferable to treat the perpetrator of the event of interest as the object of interest.

[0101] If the second condition is also used, the extraction unit 2040 extracts action-related relationships that satisfy both the first and second conditions as action-related relationships related to the event of interest. For example, the extraction unit 2040 extracts action-related relationships that satisfy the first condition (i.e., exist during the target period) from the object relationship information 20. Then, the extraction unit 2040 extracts action-related relationships that satisfy the second condition (i.e., involve the object of interest) from these extracted action-related relationships.

[0102] The extraction unit 2040 can determine whether an action relationship satisfies the second condition based on the subject 102 and object 104 of the action relationship. Specifically, an action relationship in which either the subject 102 or object 104 points to the object of interest is determined to satisfy the second condition. On the other hand, an action relationship in which neither the subject 102 nor the object 104 points to the object of interest is determined not to satisfy the second condition.

[0103] <Output of results> The relationship extraction device 2000 can output information related to the extraction results of operation-related relationships associated with the event of interest (hereinafter referred to as "output information"). The functional component that generates the output information is called the "output unit". Figure 11 shows an example of the functional configuration of the relationship extraction device 2000, including the output unit 2060.

[0104] The output information may include various types of information. For example, the output unit 2060 generates output information that includes all operational relationships extracted by the extraction unit 2040 as being related to the event of interest.

[0105] In addition to, or instead of, the behavioral relationships related to the event of interest, the output information may include one or more video frames in which behavioral relationships related to the event of interest have been detected. By providing these video frames, the user of the relationship extraction device 2000 can visually and easily understand the scenes related to the event of interest. For example, if the event of interest is shoplifting, the user of the relationship extraction device 2000 can view video frames capturing the shoplifter's behavior before, during, and after the shoplifting.

[0106] The output information may include all video frames in which behavioral relationships related to the event of interest are detected, or only some of them. In the latter case, the output information may include a specific number (e.g., one) of video frames for each behavioral relationship related to the event of interest.

[0107] For example, suppose there is an action-related relationship where a shoplifter picks up an item in a store. If this relationship exists for 2 seconds, there will be dozens of video frames in which this relationship is detected (for example, 60 video frames if the camera's frame rate is 30 frames / second). In this case, it may be sufficient for the user to view only one or more of these video frames to understand the shoplifter's behavior of picking up an item in the store. Thus, the output unit 2060 includes a specific number of video frames as representative action-related relationships for each action-related relationship associated with the event of interest.

[0108] There are various ways to output output information. For example, the relation extraction device 2000 can store the output information in a memory device. In another example, the relation extraction device 2000 can output the output information to a display device, thereby displaying the contents of the output information on the display device. In yet another example, the relation extraction device 2000 can send the output information to another device, for example, a client terminal from which the aforementioned request was sent to the relation extraction device 2000.

[0109] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0110] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0111] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0112] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) Retrieve event information that shows one or more characteristics of a featured event, The target period is determined based on one or more of the aforementioned characteristics of the event of interest, and the target period includes, as part thereof, the event time, which is the time or period during which the event of interest occurs. A relationship extraction device that extracts one or more action-related relationships that exist during the aforementioned target period from object relationship information, wherein the object relationship information shows two or more relationships between objects, corresponding to the time or period during which the action-related relationships exist. (Note 2) The one or more features of the aforementioned event of interest include the type of the event of interest and the duration of the event, The aforementioned determination of the target period is The start time of the target period is determined by determining the length of time from the start time of the target period to the event time, based on the type of the event in question. The relationship extraction device according to Appendix 1, comprising determining the end time of the target period by determining the length of time from the event time to the end time of the target period based on the type of the event of interest. (Note 3) The aforementioned determination of the target period is For each of two or more types of events, obtain duration information that shows the correspondence between the type of event and the length of its duration, The relationship extraction device according to Appendix 2, comprising determining the start time and end time of the target period based on the length of the period indicated by the period information in correspondence with the type of the event of interest. (Note 4) The aforementioned determination of the target period is Obtaining an adjustment coefficient that is a positive real value, The relationship extraction device according to Appendix 3, which includes adjusting the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both, based on the adjustment coefficient. (Note 5) The relationship extraction device according to Appendix 4, wherein the adjustment coefficient is determined by parameters representing how ambiguous the event of interest is to the user, how important the event of interest is to the user, or how interested the user is in the event of interest. (Note 6) The relationship extraction device according to any one of the appendices 1 to 4, wherein the extraction of the aforementioned action-related relationships includes extracting one or more action-related relationships from the object relationship information, which include an object that exists during the target period and is related to the event of interest. (Note 7) The object relationship information is generated by detecting each of the motion relationship relationships from one or more video frames. The at least one processor executes the instruction and further, For each of the extracted operational relationships, output information is output that includes one or more video frames in which the extracted operational relationship was detected. A relation extraction device as described in any one of the appendices 1 to 4, configured as such. (Note 8) The relationship extraction device described in Appendix 7, wherein the output information includes, for each of the extracted relationship relationships, one or more predetermined video frames in which the extracted relationship relationship was detected. (Note 9) This includes obtaining event information that shows one or more characteristics of a featured event, This includes determining a target period based on one or more of the aforementioned characteristics of the aforementioned event of interest, wherein the target period includes, as part thereof, the event time, which is the time or period during which the aforementioned event of interest occurs. A relationship extraction method comprising extracting one or more action-related relationships that exist during the aforementioned target period from object relationship information, wherein the object relationship information shows two or more relationships between objects, associated with the time or period during which the action-related relationships exist. (Note 10) The one or more features of the aforementioned event of interest include the type of the event of interest and the duration of the event, The aforementioned determination of the target period is The start time of the target period is determined by determining the length of time from the start time of the target period to the event time, based on the type of the event in question. The relationship extraction method according to Appendix 9, which includes determining the end time of the target period by determining the length of time from the event time to the end time of the target period based on the type of the event of interest. (Note 11) The aforementioned determination of the target period is For each of two or more types of events, obtain duration information that shows the correspondence between the type of event and the length of its duration, The relationship extraction method according to Appendix 10, comprising determining the start time and end time of the target period based on the length of the period indicated by the period information in correspondence with the type of the event of interest. (Note 12) The aforementioned determination of the target period is Obtaining an adjustment coefficient that is a positive real value, The relationship extraction method according to Appendix 11, which includes adjusting the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both, based on the adjustment coefficient. (Note 13) The relationship extraction method described in Appendix 12, wherein the adjustment coefficient is determined by parameters representing how ambiguous the event of interest is to the user, how important the event of interest is to the user, or how interested the user is in the event of interest. (Note 14) The extraction of the aforementioned action-related relationships is a relationship extraction method according to any one of the appendices 9 to 12, comprising extracting one or more action-related relationships from the object relationship information that exist during the target period and include an object related to the event of interest. (Note 15) The object relationship information is generated by detecting each of the motion relationship relationships from one or more video frames. The relationship extraction method according to any one of the appendices 9 to 12, further comprising outputting output information for each of the extracted operational relationship relationships, which includes one or more video frames in which the extracted operational relationship relationship was detected. (Note 16) The relationship extraction method described in Appendix 15, wherein the output information includes, for each of the extracted operational relationship relationships, one or more predetermined video frames in which the extracted operational relationship relationship was detected. (Note 17) It is a program, On the computer, The computer is tasked with obtaining event information that shows one or more characteristics of a notable event. The computer is made to determine a target period based on one or more of the characteristics of the event of interest, and the target period includes, as part thereof, the event time, which is the time or period during which the event of interest occurs. A program that extracts one or more action-related relationships that exist during the aforementioned target period from object relationship information, wherein the object relationship information shows two or more relationships between objects, associated with the time or period during which the action-related relationships exist. (Note 18) The one or more features of the aforementioned event of interest include the type of the event of interest and the duration of the event, The aforementioned determination of the target period is The start time of the target period is determined by determining the length of time from the start time of the target period to the event time, based on the type of the event in question. The relationship extraction method according to Appendix 17, which includes determining the end time of the target period by determining the length of time from the event time to the end time of the target period based on the type of the event of interest. (Note 19) The aforementioned determination of the target period is For each of two or more types of events, obtain duration information that shows the correspondence between the type of event and the length of its duration, The relationship extraction method according to Appendix 18, comprising determining the start time and end time of the target period based on the length of the period indicated by the period information in correspondence with the type of the event of interest. (Note 20) The aforementioned determination of the target period is Obtaining an adjustment coefficient that is a positive real value, The relationship extraction method according to Appendix 19, comprising adjusting the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both, based on the adjustment coefficient. (Note 21) The relationship extraction method described in Appendix 20, wherein the adjustment coefficient is determined by parameters representing how ambiguous the event of interest is to the user, how important the event of interest is to the user, or how interested the user is in the event of interest. (Note 22) The extraction of the aforementioned action-related relationships is a relationship extraction method according to any one of the appendices 17 to 20, comprising extracting one or more action-related relationships from the object relationship information that exist during the target period and include an object related to the event of interest. (Note 23) The object relationship information is generated by detecting each of the motion relationship relationships from one or more video frames. The relationship extraction method according to any one of the appendices 17 to 20, further comprising the program causing the computer to output output information for each of the extracted relationship relationships, which includes one or more video frames in which the extracted relationship relationship was detected. (Note 24) The relationship extraction method described in Appendix 23, wherein the output information includes, for each of the extracted operational relationship relationships, one or more predetermined video frames in which the extracted operational relationship relationship was detected.

[0113] This application claims priority based on Singapore Patent Application No. 10202302053T, filed on 20 July 2023, and incorporates all of its disclosures herein. [Explanation of symbols]

[0114] 10 Event Information 20. Object-related information 60 Scene Graph 100 tables 102 Subject 104 Object 106 Operation 108 period 110 Starting point 112 End 200 tables 202 Identifier Type 204 206 position 300 tables Type 302 304 period 306 Pre-release period 308 Subsequent period 1000 computers 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface 2000 Related Extraction Device 2020 Decision Committee 2040 Extraction part 2060 Output Section

Claims

1. The system has a determination means that acquires event information indicating one or more characteristics of a notable event, and determines the target period based on the one or more characteristics of the notable event. The aforementioned period includes, as part of it, the event time, which is the time or period during which the aforementioned event of interest occurred. Includes an extraction means for extracting action-related relationship information from object-related information that indicates one or more actions performed during the target period, The aforementioned action-related relationship information indicates a combination of a subject, an object, and an action performed by the subject on the object. A relationship extraction device in which the object relationship information indicates two or more pieces of information that associate the action relationship information with the time or period during which the action indicated in the action relationship information was performed.

2. The one or more features of the aforementioned event of interest include the type of the event of interest and the duration of the event. The aforementioned determination of the target period is The start time of the target period is determined by determining the length of time from the start of the target period to the event time, based on the type of the event in question. The relationship extraction device according to claim 1, comprising determining the end time of the target period by determining the length of time from the event time to the end time of the target period based on the type of the event of interest.

3. The aforementioned determination of the target period is For each of two or more types of events, obtain duration information that shows the correspondence between the type of event and the length of its duration, The relationship extraction device according to claim 2, comprising determining the start time and end time of the target period based on the length of the period indicated by the period information in correspondence with the type of the event of interest.

4. The aforementioned determination of the target period is Obtaining an adjustment coefficient that is a positive real value, The relationship extraction device according to claim 3, comprising adjusting, based on the adjustment coefficient, the length of time from the start of the target period to the event time, the length of time from the event time to the end of the target period, or both.

5. The relationship extraction device according to claim 4, wherein the adjustment coefficient is determined by a parameter representing how ambiguous the event of interest is to the user, how important the event of interest is to the user, or how interested the user is in the event of interest.

6. The relationship extraction device according to any one of claims 1 to 4, wherein the extraction of the operation-related relationship information includes extracting one or more operation-related relationship information from the object relationship information, which includes an object that exists during the target period and is related to the event of interest.

7. The object relationship information is generated by detecting each of the motion relationship information from one or more video frames. The relationship extraction device according to any one of claims 1 to 4, further comprising output means for outputting output information including one or more video frames in which the extracted operation-related relationship information is detected, for each of the extracted operation-related relationship pieces.

8. The relationship extraction device according to claim 7, wherein the output information includes, for each of the extracted operation-related relationship pieces, one or more predetermined video frames in which the extracted operation-related relationship piece is detected.

9. A relationship extraction method performed by a computer, The process includes a decision step of obtaining event information that shows one or more characteristics of the featured event, and determining the target period based on the one or more characteristics of the featured event, The aforementioned period includes, as part of it, the event time, which is the time or period during which the aforementioned event of interest occurred. The extraction step includes extracting action-related relationship information from object-related information that indicates one or more actions performed during the target period, The aforementioned action-related relationship information indicates a combination of a subject, an object, and an action performed by the subject on the object. A relationship extraction method in which the object relationship information indicates two or more pieces of information that associate the action relationship information with the time or period during which the action indicated in the action relationship information was performed.

10. Retrieve event information that shows one or more characteristics of a featured event, The computer is made to perform a decision step of determining the target period based on one or more of the aforementioned characteristics of the event of interest, The aforementioned period includes, as part of it, the event time, which is the time or period during which the aforementioned event of interest occurred. The computer is instructed to perform an extraction step of extracting action-related relationship information from object-related information that indicates one or more actions performed during the target period. The aforementioned action-related relationship information indicates a combination of a subject, an object, and an action performed by the subject on the object. A program in which the object relationship information indicates two or more pieces of information that associate the action relationship information with the time or period during which the action indicated in the action relationship information was performed.

Images