Video query method and related apparatus

Abstract

This application discloses a video query method and related apparatus, belonging to the field of data processing technology. In this method, the structured index corresponding to the target video records the attribute information and spatiotemporal relationships of multiple video objects within the target video. Therefore, regardless of changes in the number of query objects and the type of attribute information in the query conditions, the video frame containing the video object whose attribute information and spatiotemporal relationships match those of the multiple query objects can be determined; that is, the target video frame that satisfies the query conditions. In other words, regardless of changes in the number of query objects and the type of attribute information of the query objects, or whether the query conditions involve spatiotemporal relationships between multiple query objects, the target video frame that satisfies the query conditions can be determined. This means that it can handle flexible and varied video query conditions and can meet diverse video query needs.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video query method and related apparatus. Background Technology

[0002] As users' demand for video queries increases, the search criteria they input are becoming more diverse. In related technologies, electronic devices typically input the user's search criteria into a pre-trained video query model. This model then analyzes and processes the video data to identify video segments that meet the search criteria.

[0003] However, since video query models are trained using labeled training samples, the types of searches supported by the video query model are closely related to the training samples and are limited by the type and number of training samples. This results in poor generalization ability of video query methods, which do not support flexible and varied query conditions and cannot meet diverse video query needs. Summary of the Invention

[0004] This application provides a video query method and related apparatus, capable of handling flexible and varied video query conditions, thereby fulfilling diverse video query needs. The technical solution is as follows:

[0005] Firstly, a video query method is provided, comprising: obtaining query conditions, the query conditions including attribute information of multiple objects to be queried and the spatiotemporal relationship between the multiple objects to be queried; filtering target video frames that meet the query conditions from the video frames included in the target video based on the structured index corresponding to the target video, the structured index being used to indicate the attribute information of multiple video objects in the target video and the spatiotemporal relationship between the multiple video objects; and outputting the video segment corresponding to the target video frame.

[0006] The structured index corresponding to the target video records attribute information of multiple video objects within the target video, as well as the spatiotemporal relationships between these video objects. Therefore, regardless of changes in the number of query objects or the types of their attribute information in the query conditions, the video frame containing the video object that matches both the attribute information and the spatiotemporal relationships between these query objects can be identified; this is the target video frame that satisfies the query conditions. In summary, regardless of changes in the number of query objects or the types of their attribute information in the query conditions, and regardless of whether the query conditions involve spatiotemporal relationships between multiple query objects, the target video frame that satisfies the query conditions can be identified. In other words, it can handle flexible and varied video query conditions, enabling diverse video query needs.

[0007] For any two objects among a plurality of query objects, the spatiotemporal relationship between them includes both spatial and temporal relationships. The spatial relationship includes the distance between the two objects and / or the angle between them (i.e., their relative spatial positions). The temporal relationship includes the duration for which this spatial relationship is maintained or occurs between the two objects. In other words, the spatiotemporal relationship between two query objects can be characterized by the distance and / or angle between them, as well as the time elapsed.

[0008] In one possible implementation, the target video includes multiple non-overlapping video windows, each video window includes multiple video frames, and the multiple video frames within the same video window have an association relationship. The structured index includes attribute information of the multiple video objects and a spatiotemporal relationship storage graph corresponding to the multiple video windows respectively. The spatiotemporal relationship storage graph indicates the spatial relationship between every two video objects in the corresponding video window and the video frames where every two video objects are located.

[0009] Based on the structured index corresponding to the target video, target video frames that meet the query conditions are filtered from the video frames included in the target video. This includes: filtering multiple target video objects from the multiple video objects based on the attribute information of the multiple objects to be queried, wherein the attribute information of the multiple target video objects is the same as that of the multiple objects to be queried; determining the target video window from the multiple video windows in which the multiple target video objects appear simultaneously; and filtering target video frames from the multiple video frames included in the target video window based on the spatiotemporal relationship between the multiple objects to be queried, through the spatiotemporal relationship storage graph corresponding to the target video window, wherein the spatiotemporal relationship between the multiple target video objects in the target video frame matches the spatiotemporal relationship between the multiple objects to be queried.

[0010] Since the structured index records the attribute information of multiple video objects included in the target video, the server can compare the attribute information of the multiple objects to be queried with the attribute information of the multiple video objects included in the target video, thereby filtering out multiple target video objects from the multiple video objects included in the target video that have the same attribute information as the multiple objects to be queried.

[0011] To improve query efficiency, this attribute index structure is used to store the attribute information of multiple video objects included in the target video in a structured manner. This allows the server to filter out the target video objects from among the multiple video objects included in the target video based on the attribute information of these multiple query objects, using the attribute index structure. Alternatively, it can be understood that the server uses the attribute information of the multiple query objects in the query conditions as pruning conditions, pruning video objects in the attribute index structure that do not meet the attribute information of these multiple query objects, thus obtaining all possible target video objects that may match the attribute information of these multiple query objects.

[0012] In one possible implementation, the structured index further includes an attribute index structure for storing attribute information of the plurality of video objects. The attribute index structure includes multiple layers of attribute information, each layer of attribute information corresponding to an attribute type. Each layer of attribute information includes multiple pairs of attribute information. The multiple layers of attribute information are associated according to the attribute information of the plurality of video objects. Each pair of attribute information in the last layer of attribute information corresponds to at least one pair of video objects among the plurality of video objects.

[0013] Based on the attribute information of the multiple query objects, multiple target video objects are selected from the multiple video objects, including: combining every two query objects to obtain at least one query object; based on the attribute information of the at least one query object, comparing the attribute information of the at least one query object with the multi-level attribute information layer by layer according to the order of the attribute types corresponding to the multi-level attribute information, so as to select at least one pair of target video objects whose attribute information is the same as that of the at least one query object.

[0014] In addition to multiple layers of attribute information, the attribute index structure also includes a layer of video object identifiers. This layer of video object identifiers comprises multiple pairs of video object identifiers within the target video, with each pair of video object identifiers associated with a pair of attribute information in the final layer of attribute information. Thus, the server compares the attribute information of at least one target object with the multi-layered attribute information layer by layer, according to the order of the attribute types corresponding to these multiple layers of attribute information, and can then filter out at least one pair of target video object identifiers. These at least one pair of target video object identifiers constitute the identifiers of the multiple target video objects.

[0015] As can be seen from the attribute index structure described above, each layer of attribute information corresponds to an attribute type. This attribute information is obtained by combining the attribute information of video objects in the target video. Only the last layer of attribute information corresponds to at least one pair of video objects. However, in general, the attribute information of different video objects may be partially or completely the same. Therefore, according to the attribute index structure described above, the same attribute information does not need to be stored repeatedly, which can save the storage space occupied by attribute information.

[0016] Since the target video comprises multiple non-overlapping video windows, each containing multiple video frames, and these frames within the same window are related, the relationships between video frames are typically reflected as relationships between video objects. Furthermore, multiple objects to be queried are usually also related and often located within the same video window. Therefore, the server determines the target video window from these multiple video windows where all multiple target video objects appear simultaneously. This can also be understood as the server using these multiple target video objects as a pruning condition, pruning video windows that do not simultaneously contain all multiple target video objects, thus obtaining the target video window that simultaneously contains all multiple target video objects.

[0017] Multiple video frames within the same video window are related, or similar. This similarity is reflected in the similarity of their content, which can be represented by the video objects within the frame. Therefore, the similarity between video frames is essentially a reflection of the similarity of the video objects within them.

[0018] Since the multiple query objects and the multiple target video objects share the same attribute information, it can be understood that there is a one-to-one correspondence between the multiple query objects and the multiple target video objects. Therefore, the server can determine the first spatiotemporal relationship between the multiple target video objects based on the spatiotemporal relationship between the multiple query objects. That is, by using the correspondence between the multiple query objects and the multiple target video objects, the spatiotemporal relationship between the multiple query objects is converted into the spatiotemporal relationship between the multiple target video objects. For ease of description, the spatiotemporal relationship between the multiple target video objects obtained after the conversion is called the first spatiotemporal relationship, which includes the first spatial relationship and the first temporal relationship. In this way, the server can search for the target video frame based on the first spatiotemporal relationship between the multiple target video objects and through the spatiotemporal relationship storage graph corresponding to the target video window.

[0019] In one possible implementation, the spatiotemporal relation storage graph includes multiple frequent sub-graphs. Figure 1A set of multiple spatiotemporal relationship tables are used to indicate the spatiotemporal relationship between every two video objects in the corresponding video window. These multiple frequent subgraphs are constructed according to the multiple video objects in the corresponding video window.

[0020] Based on the spatiotemporal relationships between the multiple objects to be queried, the target video frame is selected from the multiple video frames included in the target video window through the spatiotemporal relationship storage graph corresponding to the target video window. This includes: selecting a first spatiotemporal relationship table from the multiple spatiotemporal relationship tables, where the frequent subgraph corresponding to the first spatiotemporal relationship table includes the multiple target video objects; determining the first spatiotemporal relationship between the multiple target video objects based on the spatiotemporal relationships between the multiple objects to be queried; and determining the target video frame through the first spatiotemporal relationship table based on the first spatiotemporal relationship between the multiple target video objects.

[0021] In the first possible implementation, each spatiotemporal relationship table is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. Thus, after filtering out the first spatiotemporal relationship table, the target video frames that satisfy the first spatiotemporal relationship between the multiple target video objects are directly determined through the first spatiotemporal relationship table.

[0022] In the second possible implementation, the multiple frequent subgraphs have an inclusion relationship, meaning that one frequent subgraph may be a subset of another. In this case, it's unnecessary for each spatiotemporal relationship table to indicate the spatiotemporal relationship between every two video objects in its corresponding frequent subgraph; it's sufficient for the spatiotemporal relationship table corresponding to the frequent subgraph with the larger number of video objects to indicate the spatiotemporal relationship between newly added video objects relative to its subset. For example, frequent subgraphs... Figure 1 It is a frequent sub Figure 3 A subset of, then frequently Figure 3 The spatiotemporal relation table only needs to indicate frequent sub-tables. Figure 3 Compared to frequent sub Figure 1 The spatiotemporal relationship between at least one pair of newly added video objects is sufficient.

[0023] In this case, the spatiotemporal relation storage graph also includes a directed frequent structure graph, which includes the multiple frequent sub-graphs. Figure 1A set of multiple subgraph vertices are connected according to the inclusion relationship between the corresponding frequent subgraphs. The vertices of the frequent subgraph with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraph with more video objects are the ending points of the directed edges. The spatiotemporal relationship table corresponding to the frequent subgraph with the fewest video objects is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. The spatiotemporal relationship table corresponding to the first frequent subgraph is used to indicate the spatiotemporal relationship between at least one pair of video objects added to the first frequent subgraph compared to the subset of the first frequent subgraph. The first frequent subgraph refers to any frequent subgraph other than the frequent subgraph with the fewest video objects.

[0024] Based on the first spatiotemporal relationship between the multiple target video objects, the target video frame is determined through the first spatiotemporal relationship table, including: if the subgraph vertex corresponding to the first spatiotemporal relationship table is not the starting vertex of the directed frequent structure graph, then based on the first spatiotemporal relationship between the multiple target video objects, the target video frame is determined through the first spatiotemporal relationship table and at least one second spatiotemporal relationship table, wherein the at least one second spatiotemporal relationship table is the spatiotemporal relationship table corresponding to each subset contained in the frequent subgraph corresponding to the first spatiotemporal relationship table.

[0025] Based on the above description, if the first spatiotemporal relationship table is not the spatiotemporal relationship table corresponding to the starting vertex of the directed frequent structure graph, then the first spatiotemporal relationship table only records the spatiotemporal relationships between at least one pair of video objects newly added to the frequent subgraph corresponding to the first spatiotemporal relationship table compared to the subset of the frequent subgraph. The spatiotemporal relationships between at least one pair of video objects that are not newly added are recorded in the second spatiotemporal relationship table corresponding to the subset of the frequent subgraph. Although the frequent subgraph corresponding to the first spatiotemporal relationship table includes the multiple target video objects, the first spatiotemporal relationship table may not completely record the video frames where the multiple target video objects are located and the spatiotemporal relationships between the multiple target video objects. Therefore, it is necessary to search for target video frames that satisfy the first spatiotemporal relationships between the multiple target video objects from the first spatiotemporal relationship table and at least one second spatiotemporal relationship table.

[0026] If the first spatiotemporal relation table is the spatiotemporal relation table corresponding to the starting vertex of the directed frequent structure graph, then the first spatiotemporal relation table records the spatiotemporal relationships between all video objects in the frequent subgraph corresponding to the first spatiotemporal relation table. Therefore, once it is determined that the frequent subgraph corresponding to the first spatiotemporal relation table includes multiple target video objects, it means that the first spatiotemporal relation table also records the spatiotemporal relationships between these multiple target video objects. In this way, the target video frames that satisfy the first spatiotemporal relationships between these multiple target video objects can be directly searched from the first spatiotemporal relation table.

[0027] The above describes the video query process; the following describes the process of creating a structured index.

[0028] In one possible implementation, the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame of the target video, and the video frame in which each video object is located are determined; based on the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located, a structured index corresponding to the target video is constructed.

[0029] Object detection algorithms are used to detect video objects in each video frame of the target video, resulting in bounding boxes for multiple video objects within each frame. Based on these bounding boxes, object tracking algorithms are used to track video objects across all video frames in the target video, obtaining the identifier for each video object. Attribute information for each video object in the target video is then extracted.

[0030] For each video frame in the target video, the process of determining the spatial relationship between multiple video objects in that video frame includes: determining the coordinates of each video object in the video frame; and determining the spatial relationship between every two video objects in the video frame based on the coordinates of each video object in the video frame.

[0031] In one possible implementation, the structured index includes a spatiotemporal relationship storage graph corresponding to multiple video windows in the target video.

[0032] Based on the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located, a structured index corresponding to the target video is constructed, including: dividing the target video into multiple video windows, each video window including multiple video frames; for each video window, generating a spatiotemporal relationship storage graph corresponding to the video window based on the spatial relationship between multiple video objects in each video frame included in the video window and the video frame in which each video object is located in the video window.

[0033] In one possible implementation, dividing the target video into multiple video windows includes: using the first video frame in the target video as the starting frame of the first video window among the multiple video windows; determining the similarity between the second video frame in the target video and the starting frame of the first video window; if the similarity is less than a similarity threshold, using the second video frame as the starting frame of the second video window among the multiple video windows, and continuing to determine the similarity between the third video frame in the target video and the starting frame of the second video window; if the similarity is greater than or equal to the similarity threshold, assigning the second video frame to the first video window, and continuing to determine the similarity between the third video frame and the starting frame of the first video window.

[0034] In one possible implementation, the spatiotemporal relation storage graph includes multiple spatiotemporal relation tables;

[0035] Based on the spatial relationships between multiple video objects in each video frame included in the video window and the video frame in which each video object is located, a spatiotemporal relationship storage graph corresponding to the video window is generated. This includes: generating a complete graph corresponding to each of the multiple video frames included in the video window, wherein the complete graph includes multiple object vertices that correspond one-to-one with the multiple video objects in the corresponding video frames, and each pair of object vertices has a connection relationship; determining multiple frequent subgraphs based on the video objects in the multiple video frames included in the video window, wherein each frequent subgraph is a subset of the complete graph corresponding to each of the multiple video frames included in the video window; and determining a spatiotemporal relationship table corresponding to each of the multiple frequent subgraphs based on the spatial relationships between multiple video objects in each video frame included in the video window and the video frame in which each video object is located, thereby obtaining the multiple spatiotemporal relationship tables.

[0036] In one possible implementation, based on the video objects in the multiple video frames included in the video window, multiple frequent subgraphs are determined, including: determining the occurrence count of the complete graph corresponding to each of the multiple video frames, where the occurrence count of the complete graph refers to the number of times the complete graph appears in the complete graph corresponding to the multiple video frames; combining the object vertices corresponding to any i video objects in the multiple video frames to generate multiple derived subgraphs, where the multiple derived subgraphs are complete graphs composed of the object vertices corresponding to the i video objects, and i takes values ​​from 2 to N, where N is the number of video objects included in the multiple video frames; determining the occurrence count of the multiple derived subgraphs, where the occurrence count of the derived subgraph refers to the number of times the derived subgraph appears in the complete graph corresponding to the multiple video frames; and filtering the multiple derived subgraphs based on the occurrence counts of the complete graph corresponding to each of the multiple video frames and the occurrence counts of the multiple derived subgraphs to obtain the multiple frequent subgraphs.

[0037] The process of filtering these multiple derived subgraphs to obtain multiple frequent subgraphs includes: for any one of these derived subgraphs, if the occurrence count of the derived subgraph is less than the occurrence count threshold (also known as minimum support), then the derived subgraph is deleted. If the derived subgraph is a proper subset of the target complete graph, and the occurrence count of the derived subgraph is the same as the occurrence count of the target complete graph, then the derived subgraph is deleted. The target complete graph is the complete graph corresponding to any one of the multiple video frames. If at least two derived subgraphs are proper subsets of the same complete graph, then all derived subgraphs in these at least two derived subgraphs except for the derived subgraph with the most object vertices are deleted. Following this process, the remaining derived subgraphs after deletion are determined as the multiple frequent subgraphs.

[0038] In a first possible implementation, each spatiotemporal relationship table is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. Thus, for each of the multiple frequent subgraphs, a spatiotemporal relationship table corresponding to the frequent subgraph is constructed based on the spatial relationship between every two video objects in the frequent subgraph and the video frames in which every two video objects are located.

[0039] In the second possible implementation, the multiple frequent subgraphs have an inclusion relationship, meaning that one frequent subgraph may be a subset of another frequent subgraph. In this case, it is not necessary for each spatiotemporal relationship table to indicate the spatiotemporal relationship between every two video objects in its corresponding frequent subgraph; it is sufficient for the spatiotemporal relationship table corresponding to the frequent subgraph with the larger number of video objects to indicate the spatiotemporal relationship between video objects newly added relative to its subset.

[0040] In this case, the spatiotemporal relation storage graph also includes a directed frequent structure graph, which includes the multiple frequent sub-graphs. Figure 1A set of multiple subgraph vertices corresponding to each video window; based on the spatial relationships between multiple video objects in each video frame included in the multiple video windows and the video frame in which each video object is located in the video window, determine the spatiotemporal relationship table corresponding to each of the multiple frequent subgraphs, including: creating the directed frequent structure graph according to the inclusion relationship between the multiple frequent subgraphs, where the vertices of the frequent subgraphs with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraphs with more video objects are the ending points of the directed edges; for each frequent subgraph in the multiple frequent subgraphs, if there is no subset of the frequent subgraph in the multiple frequent subgraphs, then based on all object vertices included in the frequent subgraph... The spatiotemporal relationship table corresponding to the frequent subgraph is generated based on the spatial relationship between every two video objects in the corresponding video objects and the video frames to which each two video objects belong. If a subset of the frequent subgraph exists in the multiple frequent subgraphs, a spatiotemporal relationship table corresponding to the frequent subgraph is generated based on the spatial relationship between every two video objects in the video objects corresponding to all object vertices included in the frequent subgraph and the video frames to which each two video objects belong, as well as the subset of the frequent subgraph. The spatiotemporal relationship table corresponding to the frequent subgraph is used to store the spatial relationship between at least one pair of video objects newly added to the frequent subgraph compared to the subset of the frequent subgraph and the video frames to which the at least one pair of video objects belong.

[0041] Since the vertices of the frequent subgraphs with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraphs with more video objects are the ending points of the directed edges, and the number of object vertices in each frequent subgraph is equal to the number of video objects in that frequent subgraph, the frequent subgraphs can be sorted in ascending order of the number of object vertices in each frequent subgraph to obtain the sorted results. Then, according to the sorted results, the vertices of the frequent subgraphs can be connected to obtain the directed frequent structure graph.

[0042] Here, the frequent subgraphs are sorted in ascending order of the number of object vertices they contain. In other embodiments, frequent subgraphs with fewer object vertices typically appear more frequently, while frequent subgraphs with more object vertices typically appear less frequently. The frequency of a frequent subgraph refers to the number of times it appears in the multiple complete graphs included in the video window. Therefore, the frequent subgraphs can also be sorted in descending order of their frequency of occurrence.

[0043] It is understandable that the spatiotemporal relationship table corresponding to each generated frequent subgraph includes the identifiers of two video objects, the spatial relationship between the two video objects, and the identifiers of the video frames to which the two video objects belong. If the spatial relationship between the two video objects is the same in multiple video frames, the spatiotemporal relationship table records the spatial relationship between the two video objects in multiple video frames only once, thereby compressing the storage of the spatial relationship between the two video objects in multiple video frames and thus compressing the size of the spatiotemporal relationship storage graph.

[0044] Secondly, if a subset of a frequent subgraph exists in multiple frequent subgraphs, the spatiotemporal relation table corresponding to the frequent subgraph only includes at least one pair of video objects newly added to the frequent subgraph compared to the subset of the frequent subgraph, and does not include at least one pair of video objects that are the same as those contained in the frequent subgraph and the subset of the frequent subgraph. In other words, the spatiotemporal relation table corresponding to the frequent subgraph no longer stores at least one pair of video objects repeatedly in the spatiotemporal relation table corresponding to the subset of the frequent subgraph, which can reduce the amount of data recorded in the spatiotemporal relation table corresponding to the frequent subgraph, thereby compressing the size of the spatiotemporal relation storage graph.

[0045] In one possible implementation, the structured index further includes an attribute index structure, which includes multi-level attribute information, with each level of attribute information corresponding to an attribute type. The method further includes: combining every two video objects among the multiple video objects included in the target video to obtain multiple pairs of video objects; storing the attribute information of the multiple pairs of video objects into the corresponding level of attribute information according to the order of the attribute types corresponding to the multi-level attribute information, and establishing the association relationship of the attribute information of the same pair of video objects in the multi-level attribute information, as well as the association relationship between the last level of attribute information and at least one corresponding pair of video objects.

[0046] In the process of storing the attribute information of multiple pairs of video objects into their corresponding attribute information layers, if the attribute information of a pair of video objects has already been saved in the attribute index structure, the server will not save it again. The server only needs to establish the association between the attribute information of this pair of video objects in the multiple layers of attribute information, as well as the association between the last layer of attribute information and this pair of video objects. In other words, the same attribute information is not saved repeatedly in each layer of attribute information in the attribute index structure.

[0047] After obtaining the attribute index structure and the spatiotemporal relationship storage graph corresponding to each of the multiple video windows through the above content, an inverted index structure can be created for the identifiers of the multiple video windows. Then, the association between the identifiers of the video objects in the attribute index structure and the identifiers of the multiple video windows, as well as the association between the identifiers of the multiple video windows and the spatiotemporal relationship storage graph corresponding to the video windows, is established to obtain the structured index corresponding to the target video.

[0048] The attribute information of video objects and the spatiotemporal relationships between them are unstructured information. Using the method described above, this unstructured information can be converted into structured information for storage. This allows for fast and accurate retrieval based on query conditions using structured information, improving video query efficiency and offering greater flexibility and breadth. Furthermore, storing fine-grained spatiotemporal relationships enables precise queries of these relationships.

[0049] Secondly, a video query device is provided, the device having the function of implementing the method behavior described in the first aspect. The device includes at least one module for implementing the method provided in the first aspect.

[0050] Thirdly, an electronic device is provided, comprising a processor and a memory, the memory being used to store a computer program for performing the methods provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the methods described in the first aspect.

[0051] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0052] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0053] The technical effects achieved by the second to fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0055] Figure 2 This is a block diagram of a server performing a video query method provided in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0058] Figure 5 This is a flowchart of a video query method provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of an attribute index structure provided in an embodiment of this application;

[0060] Figure 7 This is a schematic diagram of a structured index provided in an embodiment of this application;

[0061] Figure 8 This is a schematic diagram of another structured index provided in an embodiment of this application;

[0062] Figure 9 This is a schematic diagram of yet another structured index provided in an embodiment of this application;

[0063] Figure 10 This is a flowchart illustrating a method for constructing a structured index provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram illustrating the detection of video objects in a video frame according to an embodiment of this application;

[0065] Figure 12 This is a schematic diagram of a complete diagram provided in an embodiment of this application;

[0066] Figure 13 This is a schematic diagram of multiple complete graphs corresponding to multiple video frames within a video window, provided in an embodiment of this application.

[0067] Figure 14 This is a schematic diagram illustrating the occurrence frequency of multiple frequent subgraphs provided in an embodiment of this application;

[0068] Figure 15 This is a schematic diagram illustrating the attribute information of a video object in a target video according to an embodiment of this application;

[0069] Figure 16 This is a schematic diagram of the structure of a video query device provided in an embodiment of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0071] Before providing a detailed explanation of the methods provided in the embodiments of this application, the terms, application scenarios, and implementation environments involved in the embodiments of this application will be introduced first.

[0072] First, the terminology used in the embodiments of this application will be introduced.

[0073] 1. Complete graph

[0074] A complete graph consists of multiple vertices, and every two vertices are connected by an edge. The complete graph corresponding to a video frame includes the vertices corresponding to all video objects in that video frame, as well as the edges connecting every two vertices. In other words, for a video frame, each video object in the video frame is treated as a vertex in the complete graph, and edges are connected between every two vertices corresponding to these video objects to obtain the complete graph corresponding to that video frame.

[0075] 2. Frequent subgraphs

[0076] In a graph dataset composed of multiple graphs, a subgraph is called a frequent subgraph if it appears in a sufficient number of graphs. In other words, a frequent subgraph is a subgraph that appears frequently in the graph dataset. The number of times a subgraph appears in a graph dataset can be called its support. A frequent subgraph is a subgraph whose support exceeds a support threshold.

[0077] In the embodiments of this application, frequent subgraphs include original subgraphs and / or derived subgraphs. An original subgraph is a complete graph, meaning every complete graph is an original subgraph. A derived subgraph is a proper subgraph of one or more complete graphs. A proper subgraph is also called a proper subset. For example, if all video objects in subgraph A belong to complete graph B, but at least one video object in complete graph B does not belong to subgraph A, then subgraph A is said to be a proper subgraph of complete graph B.

[0078] 3. Spatiotemporal Relationship Storage Diagram

[0079] The spatiotemporal relationship storage graph is used to store the spatiotemporal relationships between multiple video objects in multiple video frames. For two video objects, the spatiotemporal relationship between the two video objects includes the spatial relationship between the two video objects and the temporal relationship between the two video objects. The spatial relationship between the two video objects can be characterized by the distance and relative position between the two objects, and the temporal relationship between the two video objects can be characterized by the video frames in which the two video objects are located.

[0080] Since video has a frame rate, which refers to the frequency at which video frames are displayed, the frame rate determines the number of video frames that appear within a certain time period. Therefore, the temporal relationship between two video objects can be characterized by the video frames in which these two video objects appear.

[0081] Secondly, the application scenarios involved in the embodiments of this application will be introduced.

[0082] As video capture equipment becomes increasingly widely used, users have a growing need to search for specific video clips within target videos, and the search criteria for these clips are becoming more diverse. For example, users might need to find video clips of multiple men walking together at a certain location, or clips of a white car and a black car traveling less than 5 meters apart, or clips of a white car and a black car traveling less than 5 meters apart within 10 seconds. It's clear that user-inputted search criteria involve not only the attribute information of the objects being searched for but also the spatiotemporal relationships between multiple objects. These spatiotemporal relationships can include precisely quantifiable information such as distance, angle, and time between the objects; in other words, the search criteria involve precise spatiotemporal relationships between multiple objects.

[0083] To meet users' video query needs, a video query method is proposed. This method includes: first, extracting multiple key video frames from the target video; then, using a trained video query model to extract information from each key video frame, obtaining a semantic vector corresponding to each key video frame. The semantic vector for each key video frame includes attribute information of the video object within that key video frame; next, obtaining the user-input query conditions and converting them into a query vector; then, calculating the similarity between this query vector and the semantic vectors corresponding to the multiple key video frames; finally, selecting video segments composed of key video frames with high similarity as those meeting the query conditions.

[0084] The aforementioned video query model is used to extract semantic information (e.g., attribute information of video objects in the target video). However, it cannot accurately quantify the spatiotemporal relationships between multiple video objects in the target video. Therefore, this technology does not support query conditions involving precise spatiotemporal relationships between multiple query objects. Secondly, the aforementioned video query model is trained using labeled training samples. Therefore, the query conditions supported by the model are often limited by the number of video objects labeled in the training samples and the types of attribute information of those video objects, making them fixed. When the number of video objects in the user's input query conditions differs from the number labeled in the training samples, or when the types of attribute information of the video objects in the user's input query conditions differ from those labeled in the training samples, or when the user's input query conditions involve spatiotemporal relationships between multiple video objects that are not present in the training samples, the video query model cannot find video segments that match these query conditions. Therefore, this technology does not support flexible and varied query conditions. In summary, this technology does not support query conditions involving precise spatiotemporal relationships between multiple video objects, nor does it support flexible and varied query conditions, thus failing to meet diverse video query needs.

[0085] Based on this, this application provides a video query method. In this method, the structured index corresponding to the target video records the attribute information of multiple video objects in the target video. Therefore, regardless of the number of query objects or the type of attribute information of the query objects in the query conditions, this method can determine the video object that matches the attribute information of the multiple query objects in the query conditions based on the attribute information of the multiple video objects in the target video. In addition, the structured index also records the spatiotemporal relationship between the multiple video objects in the target video. Therefore, based on the attribute information of the multiple query objects and the spatiotemporal relationship between the multiple query objects, the video frame containing the video object that matches both the attribute information and the spatiotemporal relationship of the multiple query objects can be determined, i.e., the target video frame that satisfies the query conditions. In summary, regardless of the number of query objects or the type of attribute information of the query objects in the query conditions, and whether the query conditions involve the spatiotemporal relationship between multiple query objects, this method can determine the target video frame that satisfies the query conditions. That is to say, this method can handle flexible and varied video query conditions and can realize diverse video query needs.

[0086] The video query method provided in this application can be applied to various scenarios requiring video queries, such as the transportation and travel sector or the public management sector. In the transportation and travel sector, multiple cameras installed on a transportation network collect target videos recording vehicle movement. This method can obtain a structured index corresponding to the target video; then, it obtains query conditions related to the vehicle, for example, searching for video clips of a white sedan and a black sedan traveling less than 5 meters apart; and then, based on the structured index corresponding to the target video, it filters out video clips that meet the query conditions from the target video. In the public management sector, multiple cameras installed in public places collect target videos recording people traveling. This method can obtain a structured index corresponding to the target video; then, it obtains query conditions related to the people present, for example, searching for video clips of a boy and an adult walking together in a certain area; and then, based on the structured index corresponding to the target video, it filters out video clips that meet the query conditions from the target video.

[0087] The above is merely a simple example of the application scenarios of the video query method provided in this application embodiment. In actual applications, the video query method provided in this application embodiment can be applied to any scenario for video querying, and will not be listed one by one here.

[0088] Finally, the implementation environment involved in the embodiments of this application will be described.

[0089] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment according to an exemplary embodiment. The implementation environment includes at least one video acquisition device 101. Figure 1 (Illustrative illustration using two video capture devices), and at least one client 102 ( Figure 1 (Three clients are illustrated in the illustration) and server 103. The video acquisition device 101 and client 102 can all communicate with server 103. This communication connection can be wired or wireless, and this embodiment does not limit it.

[0090] The video acquisition device 101 is used to acquire target video in real time and send the acquired target video to the server 103.

[0091] Client 102 is used to obtain the query conditions input by the user and send the query conditions to server 103. The query conditions include the attribute information of multiple objects to be queried and the spatiotemporal relationship between the multiple objects to be queried.

[0092] Server 103 generates a structured index corresponding to the target video. This structured index indicates the attribute information of multiple video objects in the target video and the spatiotemporal relationship between these multiple video objects. Based on the structured index corresponding to the target video, server 103 selects target video frames that meet the query conditions from the video frames included in the target video and sends the video segments corresponding to the target video frames to client 102.

[0093] Client 102 is also used to output the video segment corresponding to the target video frame.

[0094] In some embodiments, such as Figure 2 As shown, server 103 includes extraction module 111, structured storage module 112, index module 113 and query module 114.

[0095] The extraction module 111 is used to acquire the target video captured by the video acquisition device 101, and extract the attribute information of video objects in each video frame of the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located. The structured storage module 112 is used to store the attribute information of each video object in the target video, and to divide the target video into multiple non-overlapping video windows, each video window including multiple video frames; then, based on the spatial relationship between multiple video objects in each video frame included in each video window and the video frame in which each video object is located, to construct a spatiotemporal relationship storage graph corresponding to each video window. The indexing module 113 is used to construct a structured index corresponding to the target video based on the spatiotemporal relationship storage graphs corresponding to the multiple video windows respectively, and the attribute information of all video objects in the target video. The query module 114 is used to receive the query conditions sent by the client 102, and based on the structured index corresponding to the target video, to filter out the target video frames that meet the query conditions from the video frames included in the target video.

[0096] Among them, the video acquisition device 101 can be any device capable of acquiring video, such as a camera or a pan-tilt unit.

[0097] Client 102 is a video query application or a terminal used to run a video query application. The terminal can be any electronic product that can interact with the user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer (PC), mobile phone, smartphone, personal digital assistant (PDA), wearable device, handheld PC (pocketPC), tablet computer, smart car system, smart TV, etc.

[0098] Server 103 can be a standalone server, a server cluster or distributed system composed of multiple physical servers, a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, a cloud computing service center, or a dedicated network element device. Dedicated network elements include intelligent network video recorders (NVRs) for storing and managing video, or AI edge devices for real-time video processing without relying on the cloud.

[0099] In some possible implementations, embodiments of this application also provide another implementation environment. (and) Figure 1 The implementation environment shown is different. This implementation environment includes at least one video acquisition device 101 and a client 102, but does not include a server 103. In this implementation environment, the video acquisition device 101 can send the target video to the client 102. The client 102 is a terminal used to run a video query application. The client 102 can generate a structured index corresponding to the target video based on the target video; after receiving the query conditions input by the user, it filters the target video frames that meet the query conditions from the video frames included in the target video based on the structured index corresponding to the target video, and then outputs the video segment corresponding to the target video frame to the user.

[0100] In some embodiments, client 102 includes an extraction module, a structured storage module, an indexing module, and a query module. Details of the extraction module, structured storage module, indexing module, and query module in client 102 can be found in the above description of the extraction module 111, structured storage module 112, indexing module 113, and query module 114 in server 103, and will not be repeated here.

[0101] The following is based on Figure 1 Taking the implementation environment shown as an example, the video query method provided in this application embodiment will be introduced.

[0102] Those skilled in the art should understand that the video capture devices, terminals, and servers described above are merely examples. Other existing or future video capture devices, terminals, or servers that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.

[0103] It should be noted that the application scenarios and implementation environments described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of implementation environments, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0104] Please refer to Figure 3 , Figure 3 This is a structural block diagram of a terminal provided in an embodiment of this application. The terminal can be a portable mobile terminal, such as a smartphone, tablet, laptop, or desktop computer. The terminal may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0105] Typically, a terminal includes a processor 301 and a memory 302.

[0106] Processor 301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 301 may be implemented using at least one hardware form selected from digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as the central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0107] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement the video query method provided in the embodiments of this application.

[0108] In some embodiments, the terminal may further include a peripheral device interface 303 and at least one peripheral device. The processor 301, memory 302, and peripheral device interface 303 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 304, a display screen 305, a camera assembly 306, an audio circuit 307, a positioning assembly 308, and a power supply 309.

[0109] Peripheral device interface 303 can be used to connect at least one input / output (I / O) related peripheral device to processor 301 and memory 302. In some embodiments, processor 301, memory 302 and peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 301, memory 302 and peripheral device interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0110] The radio frequency (RF) circuit 304 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 304 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or wireless fidelity (WiFi) networks. In some embodiments, the RF circuit 304 may also include circuitry related to near-field communication (NFC), which is not limited in this application embodiment.

[0111] Display screen 305 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When display screen 305 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 305, serving as the front panel of the terminal; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of the terminal or in a folded design; in still other embodiments, display screen 305 may be a flexible display screen, disposed on a curved or folded surface of the terminal. Furthermore, display screen 305 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 305 may be made of materials such as liquid crystal display (LCD) or organic light-emitting diode (OLED).

[0112] The camera assembly 306 is used to acquire images or videos. Optionally, the camera assembly 306 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, virtual reality (VR) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0113] The audio circuit 307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 301 for processing, or input to the radio frequency circuit 304 to achieve voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 301 or the radio frequency circuit 304 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 307 may also include a headphone jack.

[0114] The positioning component 308 is used to determine the current geographic location of the terminal in order to enable navigation or location-based services (LBS). The positioning component 308 can be a positioning component of the Global Positioning System (GPS) or the BeiDou system.

[0115] Power supply 309 is used to power the various components in the terminal. Power supply 309 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 309 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0116] In some embodiments, the terminal further includes one or more sensors 310. The one or more sensors 310 include, but are not limited to, an accelerometer, a gyroscope, a pressure sensor, a fingerprint sensor, an optical sensor, and a proximity sensor.

[0117] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0118] Please refer to Figure 4 , Figure 4This is a schematic diagram of a server structure provided in an embodiment of this application. The server includes a processor such as a central processing unit (CPU) 401, a system memory 404 including random access memory (RAM) 402 and read-only memory (ROM) 403, and a system bus 405 connecting the system memory 404 and the CPU 401. The server also includes a basic input / output system (I / O system) 406 that facilitates the transfer of information between various devices within the computer, and a mass storage device 407 for storing the operating system 413, application programs 414, and other program modules 415.

[0119] The basic input / output system 406 includes a display 408 for displaying information and an input device 409 for user input, such as a mouse or keyboard. Both the display 408 and the input device 409 are connected to the central processing unit 401 via an input / output controller 410 connected to the system bus 405. The basic input / output system 406 may also include the input / output controller 410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 410 also provides output to a display screen, printer, or other types of output devices.

[0120] Mass storage device 407 is connected to central processing unit 401 via a mass storage controller (not shown) connected to system bus 405. Mass storage device 407 and its associated computer-readable media provide non-volatile storage for the server. That is, mass storage device 407 may include computer-readable media (not shown) such as hard disks or compact discread-only memory (CD-ROM) drives.

[0121] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EEPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 404 and mass storage device 407 described above can be collectively referred to as memory.

[0122] According to various embodiments of this application, the server can also connect to a remote computer on a network, such as the Internet. That is, the server can connect to network 412 via network interface unit 411 connected to system bus 405, or it can use network interface unit 411 to connect to external devices, such as cameras and clients (not shown). The server can receive target video sent by the camera via network interface unit 411 and then save the target video to RAM 402. CPU 401 processes the target video to obtain a spatiotemporal relationship storage map corresponding to each video window in the target video. The server may also include GPU 416, which generates a structured index corresponding to the target video based on the spatiotemporal relationship storage map corresponding to multiple video windows, and saves the structured index corresponding to the target video to RAM 402 or external storage including solid state drive (SSD) 417. Then, the server receives the query conditions sent by the client through the network interface unit 411. Based on the structured index corresponding to the target video, the GPU 416 filters out the target video frames that meet the query conditions from the target video and then sends the video segments corresponding to the target video frames to the client through the network interface unit 411.

[0123] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the processor.

[0124] The methods provided in the embodiments of this application will now be explained in detail.

[0125] Figure 5This is a flowchart illustrating a video query method provided in an embodiment of this application. Please refer to it. Figure 5 The method includes the following steps.

[0126] Step 501: The client obtains the query conditions, which include the attribute information of multiple objects to be queried and the spatiotemporal relationship between these multiple objects to be queried.

[0127] The object to be queried can be a person, a vehicle, or an animal. The attribute information also differs depending on the type of object. For example, the attribute information for a person may include gender, height, and clothing color, while the attribute information for a vehicle may include color, vehicle type, and brand.

[0128] For any two of the multiple query objects, the spatiotemporal relationship between them includes both spatial and temporal relationships. The spatial relationship includes the distance between the two query objects and / or the angle between them (i.e., their relative spatial positions). The temporal relationship includes the duration for which this spatial relationship is maintained or occurs between the two query objects. In other words, the spatiotemporal relationship between two query objects can be characterized by the distance and / or angle between them, as well as the time elapsed. Of course, in some embodiments, other information can be used to characterize the relationship, such as the sequential positional relationship of the multiple query objects; this application does not limit this aspect.

[0129] In some embodiments, the client may receive a segment of natural language input by the user and then extract query criteria from that natural language. Alternatively, the client may receive a combination of multiple words input by the user and then determine the query criteria based on that combination of words.

[0130] For example, the natural language received by the client could be "The distance between a blue A-type sedan and a black B-type sedan and a white C-type truck is less than 5 meters during their travels." The client extracts query conditions from this natural language. The attribute information of multiple query objects in these conditions includes: blue A-type sedan, black B-type sedan, and white C-type truck. The spatial relationship between these multiple query objects includes: the travel distance between the blue A-type sedan and the white C-type truck is less than 5 meters, and the travel distance between the black B-type sedan and the white C-type truck is less than 5 meters. Alternatively, the natural language received by the client could be "The distance between a blue A-type sedan and a black B-type sedan and a white C-type truck is less than 5 meters within 10 seconds." In this case, the temporal relationship between these multiple query objects also includes: the blue A-type sedan and the white C-type truck maintain the above spatial relationship within 10 seconds, and the black B-type sedan and the white C-type truck maintain the above spatial relationship within 10 seconds.

[0131] In some embodiments, the client may display a video search interface, which includes an input box where the user can enter the aforementioned natural language or multiple words.

[0132] Step 502: The client sends a video query request to the server, which carries the query conditions.

[0133] Step 503: The server receives the video query request and, based on the structured index corresponding to the target video, filters out the target video frames that meet the query conditions from the video frames included in the target video. The structured index is used to indicate the attribute information of multiple video objects in the target video and the spatiotemporal relationship between the multiple video objects.

[0134] The server can obtain the query conditions through the above steps, and then filter out the target video frames that meet the query conditions from the video frames included in the target video.

[0135] The target video refers to the video that needs to be queried. The target video can be a video captured by a video capture device installed in the scene where the video needs to be queried. Of course, the target video can also be a video obtained through other means. This application embodiment does not limit this.

[0136] It should be noted that the details of the attribute information of the video object in the structured index corresponding to the target video can be found in the introduction of the attribute information of the object to be queried in the query conditions in step 501, and will not be repeated here.

[0137] Since the structured index corresponding to the target video records the attribute information of multiple video objects in the target video, as well as the spatiotemporal relationship between these multiple video objects, the server can determine multiple target video objects that match the attribute information and spatiotemporal relationship between the multiple query objects based on the structured index corresponding to the target video. Then, the video segment corresponding to the target video frame where these multiple target video objects are located is determined as the video segment that meets the query conditions, thereby satisfying the user's video query needs.

[0138] In some embodiments, the target video includes multiple non-overlapping video windows, each video window includes multiple video frames, and the multiple video frames within the same video window have an association relationship. The structured index includes attribute information of multiple video objects in the target video and a spatiotemporal relationship storage graph corresponding to each of the multiple video windows. The spatiotemporal relationship storage graph indicates the spatiotemporal relationship between every two video objects in the corresponding video window. The process by which the server filters out target video frames that meet the query conditions from the video frames included in the target video based on the structured index includes the following steps (1)-(3):

[0139] (1) Based on the attribute information of the multiple objects to be queried, multiple target video objects are selected from the multiple video objects included in the target video. The attribute information of the multiple target video objects is the same as the attribute information of the multiple objects to be queried.

[0140] Since the structured index records the attribute information of multiple video objects included in the target video, the server can compare the attribute information of the multiple objects to be queried with the attribute information of the multiple video objects included in the target video, thereby filtering out multiple target video objects from the multiple video objects included in the target video that have the same attribute information as the multiple objects to be queried.

[0141] In other embodiments, to improve query efficiency, the structured index structure further includes an attribute index structure. This attribute index structure stores attribute information of multiple video objects included in the target video in a structured manner. Thus, the server can filter out the multiple target video objects from the multiple video objects included in the target video based on the attribute information of these multiple query objects, using the attribute information of the multiple query objects as pruning conditions. Alternatively, the server can be understood as using the attribute information of the multiple query objects in the query conditions as pruning conditions to prune video objects in the attribute index structure that do not meet the attribute information of the multiple query objects, thereby obtaining all multiple target video objects that may meet the attribute information of the multiple query objects.

[0142] In one possible implementation, the attribute index structure includes multiple layers of attribute information. Each layer corresponds to an attribute type, and each layer includes multiple pairs of attribute information. These multiple layers of attribute information are linked according to the attribute information of multiple video objects included in the target video. The last layer of attribute information includes at least one pair of video objects among these multiple video objects. In other words, each layer of attribute information in the attribute index structure is structured and stored according to the attribute information of a pair of video objects. Therefore, the server can combine every two query objects among the multiple query objects to obtain at least one query object. Based on the attribute information of this at least one query object, and in the order of the attribute types corresponding to the multiple layers of attribute information, the attribute information of this at least one query object is compared layer by layer with the multiple layers of attribute information to filter out at least one pair of target video objects whose attribute information is identical to that of the at least one query object.

[0143] As an example, the attribute index structure includes not only multiple layers of attribute information but also a layer of video object identifiers. This layer of video object identifiers includes multiple pairs of video object identifiers within the target video, with each pair of video object identifiers associated with a pair of attribute information in the last layer of attribute information. Thus, the server, following the order of the attribute types corresponding to these multiple layers of attribute information, compares the attribute information of at least one target object to the multi-layered attribute information layer by layer, and can then filter out at least one pair of target video object identifiers. These at least one pair of target video object identifiers constitute the identifiers of the multiple target video objects.

[0144] For example, the attribute index structure corresponding to the target video is as follows: Figure 6 As shown, the attribute index structure includes three layers of attribute information (color attribute, brand attribute, and category attribute) and one layer of video object identifier, and there are relationships between these layers of information (such as...). Figure 6 (As indicated by the arrows in the image). The query criteria include multiple attribute information for the objects to be queried: a blue Type A sedan, a black Type B sedan, and a white Type C truck. The server combines this attribute information to obtain three sets of attribute information for each object to be queried. These three sets of attribute information are as follows: The first set includes {blue, black}, {Type A, Type B}, and {sedan, sedan}; the second set includes {blue, white}, {Type A, Type C}, and {sedan, truck}; and the third set includes {black, white}, {Type B, Type C}, and {sedan, truck}. Based on these three sets of attribute information, the server... Figure 6 In the attribute index structure shown, the identifiers of the first pair of target video objects whose attribute information is the same as that of the first target object to be queried include id1 and id2; the identifiers of the second pair of target video objects whose attribute information is the same as that of the second target object to be queried include id1 and id3; and the identifiers of the third pair of target video objects whose attribute information is the same as that of the third target object to be queried include id3 and id2. These three pairs of target video object identifiers constitute the identifiers of the plurality of target video objects, namely, the identifiers of the plurality of target video objects include id1, id2, and id3.

[0145] The identifier of a video object is used to uniquely identify the video object. The identifier of a video object can be set according to certain rules, as long as the identifiers of multiple video objects included in the target video do not overlap. This application embodiment does not limit the way the identifier of a video object is set.

[0146] As can be seen from the attribute index structure described above, each layer of attribute information corresponds to an attribute type. This attribute information is obtained by combining the attribute information of video objects in the target video. Only the last layer of attribute information corresponds to at least one pair of video objects. However, in general, the attribute information of different video objects may be partially or completely the same. Therefore, according to the attribute index structure described above, the same attribute information does not need to be stored repeatedly, which can save the storage space occupied by attribute information.

[0147] In addition, the aforementioned attribute information and video objects are stored in pairs (i.e., two). This is just an example. In actual applications, they can also be stored individually or in threes. This application embodiment does not limit this.

[0148] (2) Determine the target video window from the multiple video windows in which the multiple target video objects appear simultaneously.

[0149] Since the target video comprises multiple non-overlapping video windows, each containing multiple video frames, and these frames within the same window are related, the relationships between video frames are typically reflected as relationships between video objects. Furthermore, multiple objects to be queried are usually also related and often located within the same video window. Therefore, the server determines the target video window from these multiple video windows where all multiple target video objects appear simultaneously. This can also be understood as the server using these multiple target video objects as a pruning condition, pruning video windows that do not simultaneously contain all multiple target video objects, thus obtaining the target video window that simultaneously contains all multiple target video objects.

[0150] Multiple video frames within the same video window are related, or similar. This similarity is reflected in the similarity of their content, which can be represented by the video objects within the frame. Therefore, the similarity between video frames is essentially a reflection of the similarity of the video objects within them.

[0151] In some embodiments, the structured index includes a correspondence between video window identifiers and video object identifiers. That is, the correspondence stores the identifiers of the plurality of video windows and the identifiers of each video object within each video window. Thus, after the server obtains the identifiers of the plurality of target video objects, it determines, based on the identifiers of the plurality of target video objects, the identifier of the video window that simultaneously includes the plurality of target video objects from the correspondence, and identifies the video window corresponding to that identifier as the target video window.

[0152] It should be noted that the correspondence between video window identifiers and video object identifiers can be stored separately, or it can be stored as an extension of the attribute index structure. For example, using... Figure 6 For example, the attribute index structure includes a layer of video object identifiers. A layer of video window identifiers can be extended after this layer of video object identifiers to establish a relationship between the two, resulting in something like... Figure 7 The structure shown, Figure 7 W1 to W m This serves as an identifier for multiple video windows. The identifier for this video object and the identifiers for the multiple video windows can be stored using an inverted index structure, or other methods. Figure 7 Let's take the inverted index structure as an example.

[0153] Continuing with the example above, the multiple target video objects are identified as id1, id2, and id3. The server can obtain them from... Figure 7 The inverted index structure shown includes m video window identifiers. The identifier of the target video window that simultaneously contains id1, id2, and id3 is W1.

[0154] (3) Based on the spatiotemporal relationship between the multiple objects to be queried, the target video frame is selected from the multiple video frames included in the target video window through the spatiotemporal relationship storage graph corresponding to the target video window. The spatiotemporal relationship between the multiple target video objects in the target video frame matches the spatiotemporal relationship between the multiple objects to be queried.

[0155] Since the multiple query objects and the multiple target video objects share the same attribute information, it can be understood that there is a one-to-one correspondence between the multiple query objects and the multiple target video objects. Therefore, the server can determine the first spatiotemporal relationship between the multiple target video objects based on the spatiotemporal relationship between the multiple query objects. That is, by using the correspondence between the multiple query objects and the multiple target video objects, the spatiotemporal relationship between the multiple query objects is converted into the spatiotemporal relationship between the multiple target video objects. For ease of description, the spatiotemporal relationship between the multiple target video objects obtained after the conversion is called the first spatiotemporal relationship, which includes the first spatial relationship and the first temporal relationship. In this way, the server can search for the target video frame based on the first spatiotemporal relationship between the multiple target video objects and through the spatiotemporal relationship storage graph corresponding to the target video window.

[0156] In some embodiments, the spatiotemporal relationship storage graph corresponding to the target video window includes multiple frequent sub-graphs. Figure 1A set of multiple spatiotemporal relationship tables are used to indicate the spatiotemporal relationship between every two video objects in the target video window. These multiple frequent subgraphs are constructed based on the multiple video objects in the target video window. Thus, the server can filter out a first spatiotemporal relationship table from the multiple spatiotemporal relationship tables included in the spatiotemporal relationship storage graph corresponding to the target video window. The frequent subgraphs corresponding to the first spatiotemporal relationship table include the multiple target video objects. Based on the first spatiotemporal relationship between these multiple target video objects, the target video frame is determined through the first spatiotemporal relationship table.

[0157] In the first possible implementation, each spatiotemporal relationship table is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. Thus, after filtering out the first spatiotemporal relationship table, the target video frames that satisfy the first spatiotemporal relationship between the multiple target video objects are directly determined through the first spatiotemporal relationship table.

[0158] As an example, the spatiotemporal relationship table stores the spatial relationship between every two video objects in the corresponding frequent subgraphs, the identifier of the video frame containing each pair of video objects, and the identifier of each pair of video objects. Thus, the server combines every two target video objects from the plurality of target video objects to obtain at least one pair of target video objects. Based on the first spatiotemporal relationship between the plurality of target video objects, pairs of target video objects that do not have a first spatiotemporal relationship are filtered out. For any remaining pair of target video objects after filtering, based on the identifiers of this pair of target video objects, the spatial relationship between this pair of target video objects is obtained from the first spatiotemporal relationship table, and this obtained spatial relationship is used as the second spatial relationship between this pair of target video objects. If the first spatial relationship between this pair of target video objects is the same as the second spatial relationship between this pair of target video objects, the identifier of the video frame containing this pair of target video objects is obtained from the first spatiotemporal relationship table. After obtaining the identifiers of the video frames containing this pair of target video objects, video frames that satisfy the first spatiotemporal relationship between this pair of target video objects are determined from these video frames as candidate video frames. After identifying the candidate video frames corresponding to each pair of target video objects remaining after filtering, the intersection of the candidate video frames corresponding to each pair of target video objects remaining after filtering is determined as the target video frame.

[0159] Since the spatiotemporal relationship table stores the identifiers of every two video objects in the corresponding frequent subgraphs, the server can filter out the first spatiotemporal relationship table from the multiple spatiotemporal relationship tables included in the spatiotemporal relationship storage graph corresponding to the target video window based on the identifiers of these multiple target video objects, and the first spatiotemporal relationship table stores the identifiers of these multiple target video objects.

[0160] It should be noted that the spatiotemporal relationship storage graph corresponding to the target video window includes multiple spatiotemporal relationship tables. At least two of these tables may store the identifiers of the multiple target video objects. In this case, any one of these two tables can be designated as the first spatiotemporal relationship table. Alternatively, the spatiotemporal relationship table with the fewest data among these two tables can be designated as the first spatiotemporal relationship table. This reduces the number of data matching operations and improves efficiency.

[0161] The identifier for a video frame can be its frame number, which can be a unique identifier used to identify the video frame within the target video. Of course, the identifier for a video frame can also be other information that can uniquely identify the video frame.

[0162] In some embodiments, the structured index includes a correspondence between video window identifiers and a spatiotemporal relationship storage graph. That is, the correspondence stores the identifiers of the plurality of video windows and the spatiotemporal relationship storage graph corresponding to each video window. Thus, after obtaining the identifier of the target video window, the server determines the spatiotemporal relationship storage graph corresponding to the target video window from the correspondence based on the identifier of the target video window.

[0163] It should be noted that the correspondence between video window identifiers and the spatiotemporal relationship storage graph can be stored separately, or directly... Figure 7 The structure shown extends inwards. Figure 7 For example, a spatiotemporal relationship storage graph can be extended after the video window identifier to establish the relationship between the two, resulting in, for instance, a spatiotemporal relationship storage graph. Figure 8 The structure shown, Figure 8 This includes a spatiotemporal relationship storage graph corresponding to each of the multiple video windows, and the spatiotemporal relationship storage graph includes multiple spatiotemporal relationship tables.

[0164] For example, the spatiotemporal relationship between the multiple query objects in the query conditions includes: the driving distance between the blue A-type sedan and the white C-type truck is less than 5 meters within 10 seconds, and the driving distance between the black B-type sedan and the white C-type truck is less than 5 meters within 10 seconds. Based on the identifiers of the multiple target video objects corresponding to these multiple query objects (i.e., the identifier of the target video object corresponding to the blue A-type sedan is id1, the identifier of the target video object corresponding to the black B-type sedan is id2, and the identifier of the target video object corresponding to the white C-type truck is id3), the server determines that the first spatiotemporal relationship between the multiple target video objects includes: the driving distance between id1 and id3 is less than 5 meters within 10 seconds, and the driving distance between id2 and id3 is less than 5 meters within 10 seconds.

[0165] The spatiotemporal relationship storage graph corresponding to the target video window includes multiple spatiotemporal relationship tables, such as... Figure 8 As shown, the multiple target video objects are identified as id1, id2, and id3. The first spatiotemporal relationship table selected from these multiple spatiotemporal relationship tables is Table 3. Each pair of target video objects is combined to obtain at least one pair of target video objects, namely (id1, id2), (id1, id3), and (id2, id3). Since (id1, id3) and (id2, id3) have a first spatial relationship, while (id1, id2) does not have a first spatiotemporal relationship, (id1, id2) is filtered out. For (id1, id3) and (id2, id3), the spatial relationship between (id1, id3) is obtained from Table 3, resulting in a second spatial relationship between this pair of target video objects: θ3 and d3. θ3 is the angle between this pair of target video objects, and d3 is the distance between this pair of target video objects. Assuming d3 is less than 5, the video frames where this pair of target video objects reside are identified as f3 and f4 from Table 3. From Table 3, the spatial relationship between (id2, id3) is obtained, resulting in a second spatial relationship between the pair of target video objects: θ2 and d2, where θ2 is the angle between the pair of target video objects and d2 is the distance between them. Assuming d2 is less than 5, the video frames containing the pair of target video objects are identified from Table 3 as f2, f3, and f4.

[0166] Assume the video segment consisting of video frames f2, f3, and f4 has a duration of 10 seconds (this is just an example; in actual applications, there may be many video frames within 10 seconds). Furthermore, since the spatial relationships corresponding to video frames f3 and f4 both satisfy the condition that the distance between id1 and id3 is less than 5 meters, and the spatial relationships corresponding to video frames f2, f3, and f4 both satisfy the condition that the distance between id2 and id3 is less than 5 meters within 10 seconds, it can be concluded that the target video frames that simultaneously satisfy the conditions of the distance between id1 and id3 being less than 5 meters within 10 seconds, and the distance between id2 and id3 being less than 5 meters within 10 seconds, are video frames f3 and f4. In other words, the server identifies multiple target video frames as f3 and f4.

[0167] In the second possible implementation, the multiple frequent subgraphs have an inclusion relationship, meaning that one frequent subgraph may be a subset of another. In this case, it's unnecessary for each spatiotemporal relationship table to indicate the spatiotemporal relationship between every two video objects in its corresponding frequent subgraph; it's sufficient for the spatiotemporal relationship table corresponding to the frequent subgraph with the larger number of video objects to indicate the spatiotemporal relationship between newly added video objects relative to its subset. For example, frequent subgraphs... Figure 1 It is a frequent sub Figure 3 A subset of, then frequently Figure 3 The spatiotemporal relation table only needs to indicate frequent sub-tables. Figure 3 Compared to frequent sub Figure 1 The spatiotemporal relationship between at least one pair of newly added video objects is sufficient.

[0168] In this case, the spatiotemporal relationship storage graph corresponding to the target video window also includes a directed frequent structure graph, which includes the multiple frequent sub-graphs. Figure 1 A set of multiple subgraph vertices are connected according to the inclusion relationship between the corresponding frequent subgraphs. The vertices of the frequent subgraph with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraph with more video objects are the ending points of the directed edges. The spatiotemporal relationship table corresponding to the frequent subgraph with the fewest video objects is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. The spatiotemporal relationship table corresponding to the first frequent subgraph is used to indicate the spatiotemporal relationship between at least one pair of video objects newly added to the first frequent subgraph compared to its subset. The first frequent subgraph refers to any frequent subgraph other than the frequent subgraph with the fewest video objects. Thus, after filtering out the first spatiotemporal relationship table, if the subgraph vertex corresponding to the first spatiotemporal relationship table is not the starting vertex of the directed frequent structure graph, the target video frame is determined based on the first spatiotemporal relationship between the multiple target video objects, through the first spatiotemporal relationship table and at least one second spatiotemporal relationship table. The at least one second spatiotemporal relationship table is the spatiotemporal relationship table corresponding to each subset contained in the frequent subgraph corresponding to the first spatiotemporal relationship table. If the subgraph vertex corresponding to the first spatiotemporal relationship table is the starting vertex of a directed frequent structure graph, then the target video frame is determined through the first spatiotemporal relationship table based on the first spatiotemporal relationship between the multiple target video objects.

[0169] It should be noted that multiple spatiotemporal relationship tables in the spatiotemporal relationship storage graph corresponding to the target video window, as well as multiple subgraph vertices in the directed frequent structure graph, are all related to multiple frequent subgraphs. Figure 1 Since there is a one-to-one correspondence, the multiple spatiotemporal relationship tables also correspond one-to-one with the multiple subgraph vertices.

[0170] In some embodiments, each subgraph vertex in the directed frequent structure graph is labeled with the identifiers of all video objects included in the corresponding frequent subgraph. Based on this, the server can determine the subgraph vertices labeled with the identifiers of the plurality of target video objects, and determine the spatiotemporal relation table corresponding to the subgraph vertex as the first spatiotemporal relation table.

[0171] In a directed frequent structure graph, among the vertices of multiple subgraphs, there may be at least two subgraph vertices that are marked with the identifiers of the multiple target video objects. In this case, any one of the spatiotemporal relation tables corresponding to these at least two subgraph vertices can be determined as the first spatiotemporal relation table. Alternatively, the spatiotemporal relation table with the fewest data among the spatiotemporal relation tables corresponding to these at least two subgraph vertices can be determined as the first spatiotemporal relation table. This reduces the number of data matching attempts and improves efficiency.

[0172] Based on the above description, if the first spatiotemporal relationship table is not the spatiotemporal relationship table corresponding to the starting vertex of the directed frequent structure graph, then the first spatiotemporal relationship table only records the spatiotemporal relationships between at least one pair of video objects newly added to the frequent subgraph corresponding to the first spatiotemporal relationship table compared to the subset of the frequent subgraph. The spatiotemporal relationships between at least one pair of video objects that are not newly added are recorded in the second spatiotemporal relationship table corresponding to the subset of the frequent subgraph. Although the server determines that the frequent subgraph corresponding to the first spatiotemporal relationship table includes the multiple target video objects, the first spatiotemporal relationship table may not completely record the video frames where the multiple target video objects are located and the spatiotemporal relationships between the multiple target video objects. Therefore, it is necessary to search for target video frames that satisfy the first spatiotemporal relationships between the multiple target video objects from the first spatiotemporal relationship table and at least one second spatiotemporal relationship table.

[0173] If the first spatiotemporal relation table corresponds to the starting vertex of a directed frequent structure graph, then the first spatiotemporal relation table records the spatiotemporal relationships between all video objects in the frequent subgraph corresponding to the first spatiotemporal relation table. Therefore, once the server determines that the frequent subgraph corresponding to the first spatiotemporal relation table includes multiple target video objects, it means that the first spatiotemporal relation table also records the spatiotemporal relationships between these multiple target video objects. In this way, the server can directly search for target video frames that satisfy the first spatiotemporal relationships between these multiple target video objects from the first spatiotemporal relation table.

[0174] It should be noted that if the subgraph vertex corresponding to the first spatiotemporal relationship table is the starting vertex of a directed frequent structure graph, then the process of determining the target video frame based on the first spatiotemporal relationship between the multiple target video objects is similar to the implementation process in the first possible implementation method described above. For details, please refer to the description above, which will not be repeated here.

[0175] As an example, the spatiotemporal relationship table corresponding to the most frequent subgraph with the fewest video objects stores the spatial relationship between every two video objects in the corresponding frequent subgraph, the identifier of the video frame in which each video object is located, and the identifier of every two video objects. The spatiotemporal relationship table corresponding to the first frequent subgraph stores the spatial relationship between at least one pair of video objects newly added to the subset of the first frequent subgraph, the identifier of the video frame in which the newly added at least one pair of video objects is located, and the identifier of the newly added at least one pair of video objects. Thus, the above-described process of determining the target video frame based on the first spatiotemporal relationship between the multiple target video objects, through the first spatiotemporal relationship table and at least one second spatiotemporal relationship table, includes: combining every two target video objects among the multiple target video objects to obtain at least one pair of target video objects; and filtering out pairs of target video objects that do not have a first spatiotemporal relationship according to the first spatiotemporal relationship between the multiple target video objects. For any pair of target video objects remaining after filtering, based on the identifiers of these two pairs, the spatial relationship between them is obtained from the target spatiotemporal relationship table. This obtained spatial relationship is used as the second spatial relationship between the two pairs. The target spatiotemporal relationship table consists of a first spatiotemporal relationship table and at least one second spatiotemporal relationship table storing the identifiers of these two pairs of target video objects. If the first spatial relationship between these two pairs of target video objects is the same as the second spatial relationship, the identifier of the video frame containing these two pairs of target video objects is obtained from the target spatiotemporal relationship table. After obtaining the identifiers of the video frames containing these two pairs of target video objects, video frames that satisfy the first temporal relationship between these two pairs of target video objects are selected as candidate video frames. After determining the candidate video frames corresponding to each pair of target video objects remaining after filtering, the intersection of the candidate video frames corresponding to each pair of target video objects remaining after filtering is determined as the target video frame.

[0176] In some embodiments, the structured index includes a correspondence between video window identifiers and a spatiotemporal relationship storage graph. That is, the correspondence stores the identifiers of the plurality of video windows and the spatiotemporal relationship storage graph corresponding to each video window. Thus, after obtaining the identifier of the target video window, the server determines the spatiotemporal relationship storage graph corresponding to the target video window from the correspondence based on the identifier of the target video window.

[0177] It should be noted that the correspondence between video window identifiers and the spatiotemporal relationship storage graph can be stored separately, or directly... Figure 7 The structure shown extends inwards. Figure 7 For example, a spatiotemporal relationship storage graph can be extended after the video window identifier to establish the relationship between the two, resulting in, for instance, a spatiotemporal relationship storage graph. Figure 9 The structure shown, Figure 9 It includes a spatiotemporal relationship storage graph corresponding to each of the multiple video windows, which includes multiple spatiotemporal relationship tables and a directed frequent structure graph.

[0178] For example, the spatiotemporal relationship between the multiple query objects in the query conditions includes: the driving distance between the blue A-type sedan and the white C-type truck is less than 5 meters within 10 seconds, and the driving distance between the black B-type sedan and the white C-type truck is less than 5 meters within 10 seconds. Based on the identifiers of the multiple target video objects corresponding to these multiple query objects (i.e., the identifier of the target video object corresponding to the blue A-type sedan is id1, the identifier of the target video object corresponding to the black B-type sedan is id2, and the identifier of the target video object corresponding to the white C-type truck is id3), the server determines that the first spatiotemporal relationship between the multiple target video objects includes: the driving distance between id1 and id3 is less than 5 meters within 10 seconds, and the driving distance between id2 and id3 is less than 5 meters within 10 seconds.

[0179] like Figure 9 As shown, since subgraph vertex v3 in the directed frequent structure graph corresponding to video window W1 is marked with the identifiers of the multiple target video objects (i.e., id1, id2, and id3), the server selects the spatiotemporal relationship table corresponding to subgraph vertex v3 from the multiple spatiotemporal relationship tables included in the spatiotemporal relationship storage graph corresponding to video window W1 as the first spatiotemporal relationship table. Since the first spatiotemporal relationship table is not the starting vertex of the directed frequent structure graph in the spatiotemporal relationship storage graph corresponding to video window W1, the server can use subgraph vertex v3 as the starting point and search backwards for subgraph nodes located before subgraph vertex v3 and directly or indirectly connected to subgraph vertex v3 as subgraph vertex v1 and subgraph vertex v2. Therefore, the frequent subgraphs corresponding to subgraph vertex v1 and subgraph vertex v2 are determined to be subsets of the frequent subgraphs corresponding to subgraph vertex v3, that is, the spatiotemporal relationship tables corresponding to subgraph vertex v1 and subgraph vertex v2 are the second spatiotemporal relationship tables.

[0180] Each pair of target video objects is combined to obtain at least one pair of target video objects, namely (id1, id2), (id1, id3), and (id2, id3). Since (id1, id3) and (id2, id3) have a first spatiotemporal relationship, while (id1, id2) does not, (id1, id2) is filtered out. For (id1, id3) and (id2, id3), since the spatiotemporal relationship table corresponding to subgraph vertex v3 includes (id1, id2), the spatial relationship between this pair of target video objects is obtained from the spatiotemporal relationship table corresponding to subgraph vertex v3, resulting in the second spatial relationship between this pair of target video objects as θ3 and d3, where θ3 is the angle between this pair of target video objects and d3 is the distance between this pair of target video objects. Assuming d3 is less than 5, the video frames where this pair of target video objects is located are identified as f3 and f4 from the spatiotemporal relationship table corresponding to subgraph vertex v3. Since the spatiotemporal relation table corresponding to subgraph vertex v2 includes (id2, id3), the spatial relationship between this pair of target video objects can be obtained from the spatiotemporal relation table corresponding to subgraph vertex v2. This yields the second spatial relationship between these two pairs of target video objects as θ2 and d2, where θ2 is the angle between them and d2 is the distance between them. Assuming d2 is less than 5, the video frames containing these two pairs of target video objects are identified as f2, f3, and f4 from the spatiotemporal relation table corresponding to subgraph vertex v2.

[0181] Assume the video segment consisting of video frames f2, f3, and f4 has a duration of 10 seconds (this is just an example; in actual applications, there may be many video frames within 10 seconds). Furthermore, since the spatial relationships corresponding to video frames f3 and f4 both satisfy the condition that the distance between id1 and id3 is less than 5 meters, and the spatial relationships corresponding to video frames f2, f3, and f4 both satisfy the condition that the distance between id2 and id3 is less than 5 meters within 10 seconds, it can be concluded that the target video frames that simultaneously satisfy the conditions of the distance between id1 and id3 being less than 5 meters within 10 seconds, and the distance between id2 and id3 being less than 5 meters within 10 seconds, are video frames f3 and f4. In other words, the server identifies multiple target video frames as f3 and f4.

[0182] Step 504: The server sends the video segment corresponding to the target video frame to the client, and the client outputs the video segment corresponding to the target video frame.

[0183] After the server obtains the identifier of the target video frame from the spatiotemporal relationship table corresponding to the target video window, it can generate a video clip based on the target video frame and send it to the client. The client can then output the video clip, for example, by playing it on a display screen. For instance, the client plays a video clip including the second video frame f3 and the third video frame f4 on the display screen.

[0184] The server may determine multiple target video frames, which may or may not be in a consecutive order. If the target video frames are in a consecutive order, the corresponding video segments can be generated directly according to their sequential order. If the target video frames are not in a consecutive order, meaning there may be other video frames between two target video frames, then the consecutive target video frames can be used to generate corresponding video segments, thus obtaining at least two video segments.

[0185] The video clip can also be extracted directly from the target video according to the identifiers of multiple target video frames, and it is not necessarily generated in the order of the target video frames. This application does not limit this.

[0186] In the application embodiment, the structured index corresponding to the target video records attribute information of multiple video objects in the target video. Therefore, regardless of changes in the number of queried objects and the type of attribute information of the queried objects in the query conditions, the server can determine multiple target video objects that match the attribute information of the multiple queried objects in the query conditions based on the attribute information of the multiple video objects in the target video. The structured index also records the spatiotemporal relationships between the multiple video objects in the target video. After determining the multiple target video objects, the spatiotemporal relationships between the multiple queried objects are converted into spatiotemporal relationships between the multiple target video objects. Then, through these spatiotemporal relationships, the video frame containing the video object that matches both the attribute information and the spatiotemporal relationships between the multiple queried objects is determined; that is, the target video frame that satisfies the query conditions. In summary, regardless of changes in the number of queried objects and the type of attribute information of the queried objects in the query conditions, and whether the query conditions involve spatiotemporal relationships between multiple queried objects, the server can determine the target video frame that satisfies the query conditions. In other words, the server can handle flexible and varied video query conditions and can realize diverse video query needs.

[0187] The video query method has been introduced above. The following section introduces the method for constructing the structured index corresponding to the target video.

[0188] Figure 10This is a flowchart illustrating a method for constructing a structured index according to an embodiment of this application. Please refer to... Figure 10 The method includes the following steps.

[0189] Step 1001: The server determines the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame of the target video, and the video frame in which each video object is located.

[0190] In some embodiments, a target detection algorithm is used to detect video objects in each video frame of the target video, obtaining detection boxes to which multiple video objects belong in each video frame. Based on the detection boxes to which multiple video objects belong in each video frame, a target tracking algorithm is used to track video objects in all video frames included in the target video, obtaining the identifier of each video object in all video frames included in the target video. Attribute information of each video object in the target video is then extracted.

[0191] Target detection algorithms may include the You Only Look Once (YOLO) algorithm, and target tracking algorithms may include Deep Simple Online and Real-Time Tracking (DeepSORT), etc. Of course, other algorithms may also be used, and this application does not limit them.

[0192] In some embodiments, before using the object detection algorithm to detect video objects in the target video, the server may first perform a training process for the object detection algorithm. The training process for the object detection algorithm includes: first, obtaining labeled training samples from a database, or manually labeling the training samples; then, dividing the labeled training samples into a training set, a validation set, and a test set; and then, using the training set, validation set, and test set, training and optimizing the initial detection algorithm to obtain the object detection algorithm.

[0193] In some embodiments, for each video frame in the target video, the process of the server determining the spatial relationship between multiple video objects in the video frame includes: determining the coordinates of each video object in the video frame; and determining the spatial relationship between every two video objects in the video frame based on the coordinates of each video object in the video frame.

[0194] The coordinates of each video object are the coordinates of the center point of the detection box to which the video object belongs, in the target coordinate system. The target coordinate system is established based on the video frame to which the video object belongs. For example, the target coordinate system is established with the top-left corner of the video frame to which the video object belongs as the origin, the horizontal direction to the right as the positive direction of the X-axis, and the vertical direction downward of the video frame as the positive direction of the Y-axis.

[0195] for example, Figure 11 This is a schematic diagram illustrating the detection of video objects in a video frame, as provided in an embodiment of this application. Figure 11 As shown in (a), the server detects and tracks video objects in a certain video frame, determining that the video frame includes three video objects, namely video objects a, b, and c. Figure 11 As shown in (b), video object a is identified as id1, with coordinates (122.5, 262) and attribute information as truck; video object b is identified as id2, with coordinates (220.5, 209) and attribute information including sedan; and video object c is identified as id3, with coordinates (253.5, 319.5) and attribute information as sedan.

[0196] In some embodiments, the server may use the following formula (1) to calculate the distance d between every two video objects:

[0197]

[0198] The coordinates of the two video objects are (x1, y1) and (x2, y2), respectively.

[0199] In some embodiments, the server can calculate the angle θ between every two video objects using the following formula (2), with clockwise as the positive angle:

[0200]

[0201] Continuing with the example above, the server uses formulas (1) and (2) to obtain the spatial relationship between the three video objects in the video frame, including the angle θ between the two video objects represented by id1 and id3. 1-3 The distance is 156.3, and the distance is d. 1-3 The angle θ between the two video objects represented by id1 and id2 is 143.06. 1-2 The value is 28.41, and the distance is d. 1-2 The value is 111.41; the angle θ between the two video objects represented by id3 and id2 is 111.41. 2-3 The distance is 106.6, and the distance is d. 2-3 It is 115.32.

[0202] Step 1002: The server constructs a structured index corresponding to the target video based on the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located.

[0203] The structured index corresponding to the target video can include multiple components, which are used to store the attribute information of multiple video objects in the target video, the identifiers of the video frames to which the multiple video objects belong, and the spatial relationships between the multiple video objects in each video frame.

[0204] In some embodiments, the structured index corresponding to the target video includes a spatiotemporal relationship storage graph corresponding to multiple video windows in the target video. In this case, the process of the server constructing the spatiotemporal relationship storage graph corresponding to the multiple video windows includes the following steps (1)-(2):

[0205] (1) Divide the target video into multiple non-overlapping video windows, each video window including multiple video frames.

[0206] In some embodiments, the server divides multiple video frames in the target video with a similarity greater than a similarity threshold into a video window, and the multiple video frames within the same video window are consecutive, thereby obtaining multiple non-overlapping video windows.

[0207] As an example, the first video frame in the target video is taken as the starting frame of the first video window among the multiple video windows. The similarity between the second video frame in the target video and the starting frame of the first video window is determined. If the similarity between the second video frame and the starting frame of the first video window is less than the similarity threshold, the second video frame is taken as the starting frame of the second video window among the multiple video windows. The similarity between the third video frame in the target video and the starting frame of the second video window is then determined. If the similarity between the second video frame and the starting frame of the first video window is greater than or equal to the similarity threshold, the second video frame is assigned to the first video window. The similarity between the third video frame and the starting frame of the first video window is then determined.

[0208] For example, the target video includes a first video frame f1, a second video frame f2, a third video frame f3, and so on. The server uses the first video frame f1 as the starting frame of the first video window W1; it determines the similarity between the second video frame f2 and the starting frame of the first video window W1; if the similarity between the second video frame f2 and the starting frame of the first video window W1 is less than a similarity threshold, the second video frame f2 is used as the starting frame of the second video window W2, and the similarity between the third video frame f3 and the starting frame of the second video window W2 is determined; if the similarity between the second video frame f2 and the starting frame of the first video window W1 is greater than or equal to the similarity threshold, the second video frame f2 is assigned to the first video window W1, and the similarity between the third video frame f3 and the starting frame of the first video window W1 is determined. This process continues until the last video frame is assigned to a video window, resulting in multiple video windows.

[0209] There are several methods for a server to determine the similarity between two video frames. For example, it can determine the number of identical video objects in the two video frames, obtaining a first quantity; divide the first quantity by a first total quantity to obtain a first proportion; divide the first quantity by a second total quantity to obtain a second proportion. The first total quantity represents the total number of video objects in one video frame, and the second total quantity represents the total number of video objects in the other video frame. If the first proportion or the second proportion is greater than or equal to a proportion threshold, then the similarity between the two video frames is determined to be greater than or equal to the similarity threshold; otherwise, the similarity between the two video frames is determined to be less than the similarity threshold. Alternatively, it can extract feature vectors from the two video frames, determine the distance between the two feature vectors, and use the reciprocal of the distance between the two feature vectors as the similarity between the two video frames, thereby determining whether the similarity between the two video frames is greater than or equal to the similarity threshold. This application does not limit the method for determining the similarity between two video frames.

[0210] The ratio threshold and similarity threshold are preset, and these two thresholds can be adjusted under different circumstances.

[0211] (2) For each of the multiple video windows, a spatiotemporal relationship storage diagram corresponding to the video window is generated based on the spatial relationship between multiple video objects in each video frame included in the video window and the video frame in which each video object is located.

[0212] In some embodiments, the spatiotemporal relationship storage graph corresponding to each video window includes multiple frequent sub-windows. Figure 1One corresponding spatiotemporal relationship table. The multiple frequent subgraphs are constructed according to multiple video objects in each video window, and the multiple spatiotemporal relationship tables are used to indicate the spatiotemporal relationship between every two video objects in the corresponding video window. The process of the server generating the spatiotemporal relationship storage graph corresponding to each video window may include the following steps (21)-(25):

[0213] (21) Generate a complete graph corresponding to the multiple video frames included in the video window. The complete graph includes multiple object vertices that correspond one-to-one with the multiple video objects in the corresponding video frames. Each pair of object vertices has a connection relationship.

[0214] The server provides multiple object vertices that correspond one-to-one with multiple video objects in the corresponding video frame, and connects every two object vertices to obtain a complete graph consisting of these multiple object vertices and the connections between them, which is the complete graph corresponding to the video frame.

[0215] In some embodiments, the server may also mark the identifier of the corresponding video object at each object vertex in the complete graph corresponding to the video frame, and mark the spatial relationship between the two video objects at the line connecting each pair of object vertices. That is, the complete graph corresponding to the video frame may also include the identifiers of the video objects in the video frame and the spatial relationship between each pair of video objects in the video frame.

[0216] Continuing from the above Figure 11 For example, Figure 11 The video frame in the graph includes three video objects. The server provides three object vertices, and lines are connected between every two object vertices to obtain a complete graph consisting of the three object vertices and the lines connecting them. The server can also mark the corresponding video object identifiers at the three object vertices in this complete graph, and mark the spatial relationship between the two video objects at the lines connecting them, resulting in a graph like this: Figure 12 The complete diagram shown.

[0217] The line connecting any two object vertices can be called an edge, and the spatial relationship between the two video objects marked by each edge can be called an edge attribute. For example, the edge attributes can include the distance d between the two video objects and the angle θ between the two video objects.

[0218] It should be noted that in a complete graph, it is not necessary to mark video objects and their spatial relationships, as long as the video objects corresponding to the vertices of the objects in the complete graph can be determined. The following example illustrates this using a complete graph with marked video objects and their spatial relationships.

[0219] For example, video window W1 includes 5 video frames: video frame f1, video frame f2, video frame f3, video frame f4, and video frame f5. The complete graph corresponding to these 5 video frames is shown below. Figure 13 As shown, these are the complete... Figure 1 ,completely Figure 2 ,completely Figure 3 ,completely Figure 4 and complete Figure 5 .

[0220] Among them, complete Figure 1 This includes the spatial relationship between video object id1 and video object id2 (including θ1 and d1).

[0221] completely Figure 2 This includes the spatial relationship between video object id2 and video object id3 (including θ2 and d2).

[0222] completely Figure 3 This includes: the spatial relationship between video object id1 and video object id2 (including θ1 and d1), the spatial relationship between video object id1 and video object id3 (including θ3 and d3), and the spatial relationship between video object id2 and video object id3 (including θ2 and d2).

[0223] completely Figure 4 This includes: the spatial relationship between video object id1 and video object id2 (including θ4 and d4), the spatial relationship between video object id1 and video object id3 (including θ3 and d3), the spatial relationship between video object id1 and video object id4 (including θ5 and d5), the spatial relationship between video object id2 and video object id3 (including θ2 and d2), the spatial relationship between video object id2 and video object id4 (including θ6 and d6), and the spatial relationship between video object id3 and video object id4 (including θ7 and d7).

[0224] completely Figure 5This includes: the spatial relationship between video object id1 and video object id2 (including θ4 and d4), the spatial relationship between video object id1 and video object id3 (including θ8 and d8), the spatial relationship between video object id1 and video object id4 (including θ5 and d5), the spatial relationship between video object id1 and video object id5 (including θ10 and d10), the spatial relationship between video object id2 and video object id3 (including θ9 and d9), the spatial relationship between video object id2 and video object id4 (including θ6 and d6), the spatial relationship between video object id2 and video object id5 (including θ13 and d13), the spatial relationship between video object id3 and video object id4 (including θ7 and d7), the spatial relationship between video object id3 and video object id5 (including θ12 and d12), and the spatial relationship between video object id4 and video object id5 (including θ11 and d11).

[0225] (22) Based on the video objects in the multiple video frames included in the video window, determine multiple frequent subgraphs, each of which is a subset of the complete graph corresponding to the multiple video frames included in the video window.

[0226] In some embodiments, the server may use the Frequent Pattern Tree (FP-tree) algorithm to mine multiple complete graphs corresponding to multiple video frames within the video window to determine the multiple frequent subgraphs.

[0227] As an example, the occurrence count of the complete graph corresponding to each of the multiple video frames within the video window is determined. The occurrence count refers to the number of times the complete graph appears in the complete graph corresponding to each of the multiple video frames. The object vertices corresponding to any i video objects in the multiple video frames are combined to generate multiple derived subgraphs. These derived subgraphs are complete graphs composed of the object vertices corresponding to any i video objects, where i ranges from 2 to N, and N is the number of video objects included in the multiple video frames. The occurrence count of these derived subgraphs is determined, where the occurrence count refers to the number of times the derived subgraph appears in the complete graph corresponding to each of the multiple video frames. Based on the occurrence counts of the complete graphs corresponding to each of the multiple video frames and the occurrence counts of the derived subgraphs, these derived subgraphs are filtered to obtain multiple frequent subgraphs. Each frequent subgraph includes at least two object vertices.

[0228] When determining the number of occurrences of a complete graph and its derived subgraphs, the properties of the edges, i.e., spatial relationships, need not be considered.

[0229] The server filters the multiple derived subgraphs to obtain multiple frequent subgraphs. The process includes: for any given derived subgraph, if its occurrence count is less than a threshold (also called minimum support), it is deleted. If the derived subgraph is a proper subset of the target complete graph, and its occurrence count is the same as the target complete graph, it is also deleted. The target complete graph is the complete graph corresponding to any one of the multiple video frames. If at least two derived subgraphs are proper subsets of the same complete graph, all derived subgraphs except the one with the most object vertices are deleted. Following this process, the remaining derived subgraphs are determined as the multiple frequent subgraphs.

[0230] In some embodiments, each object vertex in each frequent subgraph can be labeled with the identifier of the corresponding video object. This occurrence threshold is preset, for example, to 1. Under different circumstances, this occurrence threshold can also be adjusted according to different needs.

[0231] Continuing from the above Figure 13 For example, video window W1 consists of 5 video frames, and the server determines the number of occurrences of the complete graph corresponding to each of these 5 video frames. Among these, the complete graph... Figure 1 The number of occurrences is 4, completely Figure 2 The number of times it appears is 4, completely Figure 3 The number of occurrences is 3, completely Figure 4 The number of occurrences is 2, completely Figure 5 The occurrence count is 1. These 5 video frames contain a total of 5 video objects. Combining the vertices corresponding to any i video objects (where i ranges from 2 to 5) generates a derived subgraph. That is, combining the vertices corresponding to any 2 video objects yields 10 derived subgraphs; combining the vertices corresponding to any 3 video objects yields 10 derived subgraphs; combining the vertices corresponding to any 4 video objects yields 5 derived subgraphs; and combining the vertices corresponding to any 5 video objects yields 1 derived subgraph. A total of 26 derived subgraphs are obtained. The occurrence count of these 26 derived subgraphs is determined, and then, based on the occurrence counts of the complete graph corresponding to each of the 5 video frames and the occurrence counts of these 26 derived subgraphs, these 26 derived subgraphs are filtered. Suppose the server receives a derived subgraph consisting of id1 and id3, and this derived subgraph is complete. Figure 4 A proper subset of, and the derived subgraph and complete Figure 4If the occurrence count of each term is 2, the server deletes the derived subgraph. With an occurrence count threshold of 1, filtering these 26 derived subgraphs using the above method yields 5 frequent subgraphs. These 5 frequent subgraphs are... Figure 13 The five complete graphs shown are identical.

[0232] (23) Based on the spatial relationship between multiple video objects in each video frame included in the video window and the video frame in which each video object is located, determine the spatiotemporal relationship tables corresponding to the multiple frequent subgraphs respectively, and obtain multiple spatiotemporal relationship tables.

[0233] In a first possible implementation, each spatiotemporal relationship table is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. Thus, for each of the multiple frequent subgraphs, a spatiotemporal relationship table corresponding to the frequent subgraph is constructed based on the spatial relationship between every two video objects in the frequent subgraph and the video frames in which every two video objects are located.

[0234] As an example, the spatiotemporal relationship table stores the spatial relationship between every two video objects in the corresponding frequent subgraph, the identifier of the video frame to which each two video objects belong, and the identifier of each two video objects. The complete graph corresponding to each video frame includes the identifier of each video object in that video frame and the spatial relationship between every two video objects in that video frame. Each frequent subgraph includes the identifier of the video object corresponding to each object vertex. Thus, based on the identifiers of every two video objects included in the frequent subgraph, the spatial relationship between each two video objects is obtained from the multiple complete graphs; the identifiers of every two video objects included in the frequent subgraph, the spatial relationship between each two video objects, and the identifier of the video frame to which each two video objects belong are mapped and stored to obtain the spatiotemporal relationship table corresponding to the frequent subgraph.

[0235] Continuing with the example above, video window W1 corresponds to 5 frequent subgraphs, and these 5 frequent subgraphs are related to... Figure 13 The five complete graphs shown are identical. Frequent sub-graphs Figure 1 Including video objects id1 and id2, frequent sub- Figure 2 Including video objects id2 and id3, frequent sub- Figure 3 Including video objects id1, id2, and id3, frequent sub- Figure 4 Including video objects id1, id2, id3, and id4, frequent sub- Figure 5 This includes video objects id1, id2, id3, id4, and id5. Based on frequent sub-... Figure 1The system includes two video objects, id1 and id2. The spatial relationships between each pair of video objects are obtained from the multiple complete graphs as (θ1, d1) and (θ4, d4). The video frames containing the spatial relationship (θ1, d1) are identified as f1 and f3, and the video frames containing the spatial relationship (θ4, d4) are identified as f4 and f5. Based on the identifiers of these two video objects, the spatial relationships between them, and the identifiers of the video frames to which they belong, a mapping and storage process is performed to obtain frequent sub-objects. Figure 1 The corresponding spatiotemporal relationship table is as follows Figure 8 Table 1 in the table. Similarly, frequent sub-sub ... Figure 2 The corresponding spatiotemporal relationship table is as follows Figure 8 Table 2 in the table yields frequent sub-sub ... Figure 3 The corresponding spatiotemporal relationship table is as follows Figure 8 Table 3 in the table yields frequent sub-sub ... Figure 4 The corresponding spatiotemporal relationship table is as follows Figure 8 Table 4 in the table yields frequent sub-sub ... Figure 5 The corresponding spatiotemporal relationship table is as follows Figure 8 Table 5 in the table.

[0236] In some embodiments, whether to use the first implementation method to generate the spatiotemporal relation table corresponding to each frequent subgraph can be determined by whether the frequent subgraph includes a subset. For example, for each frequent subgraph among the multiple frequent subgraphs, if there is no subset of the frequent subgraph among the multiple frequent subgraphs, then the spatiotemporal relation table corresponding to the frequent subgraph is generated according to the first possible implementation method.

[0237] For example, for frequent sub- Figure 1 Since these 5 frequent subgraphs do not include frequent subgraphs Figure 1 A subset of the set, the server can generate frequent subsets according to the first possible implementation. Figure 1 The corresponding spatiotemporal relationship table.

[0238] In the second possible implementation, the multiple frequent subgraphs have an inclusion relationship, meaning that one frequent subgraph may be a subset of another frequent subgraph. In this case, it is not necessary for each spatiotemporal relationship table to indicate the spatiotemporal relationship between every two video objects in its corresponding frequent subgraph; it is sufficient for the spatiotemporal relationship table corresponding to the frequent subgraph with the larger number of video objects to indicate the spatiotemporal relationship between video objects newly added relative to its subset.

[0239] In this context, the spatiotemporal relation storage graph also includes a directed frequent structure graph, which comprises graphs related to the plurality of frequent sub-graphs. Figure 1A set of multiple subgraph vertices is used. Based on the inclusion relationships between these frequent subgraphs, a directed frequent structure graph is created, where the vertices of the frequent subgraphs with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraphs with more video objects are the ending points of the directed edges. For each frequent subgraph, if there is no subset of the frequent subgraph, a spatiotemporal relationship table is generated based on the spatiotemporal relationships between every two video objects corresponding to all object vertices included in the frequent subgraph. If there is a subset of the frequent subgraph, a spatiotemporal relationship table is generated based on the spatiotemporal relationships between every two video objects corresponding to all subgraph vertices included in the frequent subgraph, and the subset of the frequent subgraph. This spatiotemporal relationship table stores at least one additional pair of video objects added to the frequent subgraph compared to its subset.

[0240] Since the vertices of the frequent subgraphs with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraphs with more video objects are the ending points of the directed edges, and the number of object vertices in each frequent subgraph is equal to the number of video objects in that frequent subgraph, the server can sort the frequent subgraphs based on the number of object vertices in each frequent subgraph from smallest to largest, obtain the sorted results, and then establish connection relationships between the vertices of the frequent subgraphs according to the sorted results to obtain a directed frequent structure graph.

[0241] Here, the frequent subgraphs are sorted in ascending order of the number of object vertices they contain. In other embodiments, frequent subgraphs with fewer object vertices typically appear more frequently, while frequent subgraphs with more object vertices typically appear less frequently. The frequency of a frequent subgraph refers to the number of times it appears in the multiple complete graphs included in the video window. Therefore, the frequent subgraphs can also be sorted in descending order of their frequency of occurrence.

[0242] In some embodiments, the server can label the vertices of each frequent subgraph in the directed frequent structure graph with identifiers of all video objects included in that frequent subgraph. Thus, by querying the identifiers of the video objects labeled at each subgraph vertex, the video objects included in the spatiotemporal relation table corresponding to that subgraph vertex can be determined.

[0243] Continuing with the example above, video window W1 corresponds to 5 frequent subgraphs, such as... Figure 14 As shown, frequent sub Figure 1 The number of occurrences is equal to 4, which is a frequent occurrence. Figure 2The number of occurrences is equal to 4, which is a frequent occurrence. Figure 3 The number of occurrences is equal to 3, frequent sub Figure 4 The occurrence count is equal to 2, and the frequent sub-items... Figure 5 The occurrence frequency of the term is equal to 1. Based on the descending order of the occurrence frequency of these frequent subgraphs, the frequent subgraphs are sorted as follows: Frequent Subgraphs Figure 1 and frequent sub Figure 2 Tied for first place, frequent sub-items Figure 3 Located in the second, frequent sub Figure 4 Located in the third, frequent sub Figure 5 Located fourth. Based on the sorting result, the five corresponding vertices of these five frequent subgraphs are determined: v1, v2, v3, v4, and v5. Connections are then established between these five vertices according to the sorting result, resulting in the following: Figure 9 The diagram shows a directed frequent structure graph. Furthermore, each of the five subgraph vertices is labeled with the identifier of all video objects in its corresponding frequent subgraph. The directed edges between these five subgraph vertices are also labeled with the vertices of newly added video objects; that is, the directed edge from v1 to v3 is labeled with the vertices of the frequently occurring video objects. Figure 3 Compared to frequent sub Figure 1 The newly added video object identifier id3, the directed edge from v2 to v3 is marked with frequent sub-elements. Figure 3 Compared to frequent sub Figure 2 The newly added video object identifier id1, the directed edge from v3 to v4 is marked with frequent sub-elements. Figure 4 Compared to frequent sub Figure 3 The newly added video object identifier id4, the directed edge from v4 to v5 is marked with frequent sub-elements. Figure 5 Compared to frequent sub Figure 4 The newly added video object's identifier is id5.

[0244] As can be seen, in a directed frequent structure graph, when the directed edge between a vertex of a frequent subgraph and a vertex of a subset of that frequent subgraph is marked with the identifiers of all video objects newly added to the frequent subgraph compared to the subset, the server can perform a reverse lookup of the directed frequent structure graph when querying the spatiotemporal relationships between all video objects in the frequent subgraph. That is, based on the identifiers of all video objects newly added to the frequent subgraph compared to the subset marked by the directed edge, at least one pair of video objects included in the subset of the frequent subgraph can be determined, and then the spatiotemporal relationships between at least one pair of video objects can be obtained from the spatiotemporal relationship table corresponding to the subset of the frequent subgraph.

[0245] Directed frequent structure graphs include multiple frequent sub-graphs. Figure 1A one-to-one correspondence between multiple vertices and the connections between these vertices. In some embodiments, the vertex corresponding to each frequent subgraph in a directed frequent structure graph can be called a subgraph vertex.

[0246] In some embodiments, whether to use the second implementation method to generate the spatiotemporal relation table corresponding to each frequent subgraph can be determined by whether the frequent subgraph includes a subset. For example, for each frequent subgraph among the multiple frequent subgraphs, if a subset of the frequent subgraph exists among the multiple frequent subgraphs, then the spatiotemporal relation table corresponding to the frequent subgraph is generated according to the second possible implementation method.

[0247] If there is no subset of the frequent subgraph among the multiple frequent subgraphs, the process of generating the spatiotemporal relation table corresponding to the frequent subgraph is similar to the first implementation method described above, and will not be repeated here.

[0248] As an example, the complete graph corresponding to each video frame includes the identifier of each video object in the video frame and the spatial relationship between every two video objects in the video frame; each frequent subgraph includes the identifier of the video object corresponding to each object vertex. In this case, the implementation process of generating the spatiotemporal relationship table corresponding to the frequent subgraph based on the spatial relationship between every two video objects corresponding to all subgraph vertices included in the frequent subgraph, the video frames in which every two video objects are located, and subsets of the frequent subgraph includes: filtering all subsets of the frequent subgraph, such that the filtered subsets do not contain each other; determining at least one pair of video object identifiers newly added to the frequent subgraph compared to all the subsets obtained after filtering; then, based on the identifiers of each newly added pair of video objects, obtaining the spatial relationship between each newly added pair of video objects from the multiple complete graphs; mapping and storing the identifiers of each newly added pair of video objects, the spatial relationship between each newly added pair of video objects, and the identifiers of the video frames to which each newly added pair of video objects belong, to obtain the spatiotemporal relationship table corresponding to the frequent subgraph.

[0249] The server's filtering process for subsets of the frequent subgraph includes: for each subset of the frequent subgraph, if the subset is contained within another subset of the frequent subgraph, then the subset is deleted. In other words, deleted subsets are contained within non-deleted subsets. The subsets obtained after filtering can indirectly determine the filtered subsets. Therefore, determining newly added video objects through the filtered subsets can avoid some redundant operations and improve efficiency.

[0250] Continuing with the example above, video window W1 corresponds to 5 frequent subgraphs, and these 5 frequent subgraphs are related to... Figure 13 The five complete graphs shown are identical. Frequent sub-graphs Figure 1 Including video objects id1 and id2, frequent sub- Figure 2Including video objects id2 and id3, frequent sub- Figure 3 Including video objects id1, id2, and id3, frequent sub- Figure 4 Including video objects id1, id2, id3, and id4, frequent sub- Figure 5 This includes video objects id1, id2, id3, id4, and id5. Frequent sub-categories. Figure 1 and frequent sub Figure 2 Since no subsets exist, frequent subsets are generated according to the first implementation method described above. Figure 1 The corresponding spatiotemporal relationship table is as follows: Figure 9 The spatiotemporal relation table corresponding to subgraph node v1 in the graph, frequent subgraphs Figure 2 The corresponding spatiotemporal relationship table is as follows: Figure 9 The spatiotemporal relation table corresponding to subgraph node v2 in the graph.

[0251] For frequent sub Figure 3 In other words, since these 5 frequent subgraphs include frequent subgraphs... Figure 3 A subset of, i.e., a frequent subset Figure 1 and frequent sub Figure 2 Due to frequent sub- Figure 1 and frequent sub Figure 2 Since they do not contain each other, the server directly determines the frequent sub-series. Figure 3 Compared to frequent sub Figure 1 and frequent sub Figure 2 A new pair of video objects is added, identified as id1 and id3. Then, based on the identifiers of this pair of video objects, the spatial relationship between them, and the identifiers of the video frames to which they belong, frequent sub-elements are obtained. Figure 3 The corresponding spatiotemporal relationship table is as follows: Figure 9 The spatiotemporal relation table corresponding to neutron graph node v3.

[0252] For frequent sub Figure 4 In other words, since these 5 frequent subgraphs include frequent subgraphs... Figure 4 A subset of, i.e., a frequent subset Figure 1 Frequent sub Figure 2 and frequent sub Figure 3 Then the server will first target frequent sub-processes. Figure 5 Filtering a subset yields a filtered subset that includes frequent subsets. Figure 3 Then, based on frequent sub-subs Figure 5 Compared to frequent sub Figure 3 The identifiers of at least one newly added pair of video objects, the spatial relationship between the at least one pair of video objects, and the identifiers of the video frames to which the at least one pair of video objects belong are used to obtain frequent sub-elements. Figure 4 The corresponding spatiotemporal relationship table is as follows: Figure 9 The spatiotemporal relation table corresponding to neutron graph node v4.

[0253] For frequent sub Figure 5 In other words, since these 5 frequent subgraphs include frequent subgraphs... Figure 5 A subset of, i.e., a frequent subset Figure 1 Frequent sub Figure 2 Frequent sub Figure 3 and frequent sub Figure 4 Then the server will first target frequent sub-processes. Figure 5 Filtering a subset yields a filtered subset that includes frequent subsets. Figure 4 Then, based on frequent sub-subs Figure 5 Compared to frequent sub Figure 4 The identifiers of at least one newly added pair of video objects, the spatial relationship between the at least one pair of video objects, and the identifiers of the video frames to which the at least one pair of video objects belong are used to obtain frequent sub-elements. Figure 5 The corresponding spatiotemporal relationship table is as follows: Figure 9 The spatiotemporal relation table corresponding to neutron graph node v5.

[0254] It should be noted that the server may also choose not to filter all subsets of each frequent subgraph, and directly determine the identifiers of at least one pair of video objects newly added to the frequent subgraph compared to its subsets. Specifically, the at least one pair of video objects newly added to the frequent subgraph compared to all filtered subsets is the same as the at least one pair of video objects newly added to the frequent subgraph compared to all subsets of the frequent subgraph.

[0255] Understandably, the spatiotemporal relationship table corresponding to each frequent subgraph generated by the server includes the identifiers of two video objects, the spatial relationship between these two video objects, and the identifiers of the video frames to which these two video objects belong. Therefore, if the spatial relationship between these two video objects is the same across multiple video frames, the spatiotemporal relationship table records this spatial relationship only once, thus compressing the storage of the spatial relationship between these two video objects across multiple video frames and consequently compressing the size of the spatiotemporal relationship storage graph. For example, frequent subgraphs... Figure 1 The corresponding spatiotemporal relationship table includes id1 and id2. These two video objects have the same spatial relationship in video frames f1 and f3, and the same spatial relationship in video frames f4 and f5. Frequent sub-objects... Figure 1 The corresponding spatiotemporal relationship table records the spatial relationship between the two video objects only once in video frames f1 and f3, and only once in video frames f4 and f5.

[0256] Secondly, if a subset of a frequent subgraph exists within multiple frequent subgraphs, the spatiotemporal relation table corresponding to that frequent subgraph only includes at least one pair of video objects newly added to the frequent subgraph compared to its subset. It does not include at least one pair of video objects that are identical to those in the frequent subgraph and its subset. In other words, the spatiotemporal relation table corresponding to that frequent subgraph does not repeatedly store at least one pair of video objects from the spatiotemporal relation table corresponding to the subset of the frequent subgraph. This reduces the amount of data recorded in the spatiotemporal relation table corresponding to the frequent subgraph, thereby compressing the size of the spatiotemporal relation storage graph. For example, frequent subgraphs... Figure 4 The corresponding spatiotemporal relation table only includes frequent sub-sub ... Figure 4 Compared to frequent sub Figure 1 and frequent sub Figure 2 The newly added multiple pairs of video objects do not include frequent sub-objects. Figure 4 and frequent sub Figure 1 It contains the same pair of video objects (i.e., the pair of video objects id1 and id2), and does not include frequent sub-videos. Figure 4 and frequent sub Figure 2 It contains the same pair of video objects (i.e., the pair of video objects id2 and id3).

[0257] The following section introduces the attribute information of the multiple video objects included in the target video in the structured index.

[0258] In one possible implementation, the attribute information of multiple video objects included in the target video can be directly stored in the structured index corresponding to the target video. For example, the mapping between the identifiers of video objects and their attribute information can be stored.

[0259] In another possible implementation, considering that the attribute information of video objects can be divided into multiple attribute types, the server can classify and save the attribute information of multiple video objects in the target video according to multiple attribute types. Also considering that the spatiotemporal relationship storage graph corresponding to each video window records the spatiotemporal relationship between two video objects, the server can divide multiple video objects in the target video into multiple pairs of video objects, and classify and save the attribute information of each pair of video objects according to multiple attribute types.

[0260] As an example, the structured index corresponding to the target video includes an attribute index structure, which stores attribute information of multiple video objects included in the target video. This attribute index structure includes multiple layers of attribute information, each layer corresponding to an attribute type. In this way, the server can combine every two video objects from the multiple video objects included in the target video to obtain multiple pairs of video objects. According to the order of the attribute types corresponding to the multiple layers of attribute information, the attribute information of these multiple pairs of video objects is stored in the corresponding layer of attribute information. Furthermore, the server establishes the association relationship between the attribute information of the same pair of video objects within these multiple layers of attribute information, as well as the association relationship between the last layer of attribute information and at least one corresponding pair of video objects.

[0261] In addition to multiple layers of attribute information, the attribute index structure also includes a layer of video object identifiers. This layer of video object identifiers comprises multiple pairs of video object identifiers within the target video, and each pair of video object identifiers is associated with a pair of attribute information in the final layer of attribute information. In this way, the server establishes an association between the final layer of attribute information and at least one corresponding pair of video object identifiers.

[0262] In some embodiments, the complete graph corresponding to each video frame in the target video includes the identifiers of video objects and the spatiotemporal relationship between every two video objects. The attribute information of the video objects is divided into multiple attribute types. The process of the server constructing an attribute index structure based on the attribute information of multiple video objects in the target video includes: starting from the complete graph corresponding to the first video frame in the target video, combining the identifiers of every two video objects in the complete graph to obtain at least one pair of video object identifiers; for each pair of video objects, based on the identifiers of this pair of video objects, obtaining the attribute information of this pair of video objects from the attribute information of all video objects included in the target video, where the attribute information of this pair of video objects includes attribute information of at least one attribute type; the server stores the attribute information of the at least one attribute type into the corresponding layer of attribute information according to the order of the multiple attribute types, and establishes the association relationship of the attribute information of the at least one attribute type in the multi-layer attribute information, as well as the association relationship between the attribute information of the last attribute type and the identifiers of this pair of video objects. Similarly, by traversing the complete graph corresponding to each video frame, the attribute index structure corresponding to the target video can be constructed.

[0263] In other embodiments, the complete graph corresponding to each video frame may include not only the identifiers of the video objects and the spatiotemporal relationships between every two video objects, but also the attribute information of the video objects. In this case, the attribute information of each pair of video objects can also be directly obtained from the complete graph corresponding to each video frame.

[0264] For example, taking a target video where all video objects are vehicles, the order of multiple attribute types can include color, brand, and category. For a pair of video objects, the server can save the color attribute information of the pair of video objects to the first-level color attribute information, the brand attribute information of the pair of video objects to the second-level brand attribute information, and the category attribute information of the pair of video objects to the third-level category attribute information, and establish an association between the color attribute information, brand attribute information, and category attribute information of the pair of video objects, and establish an association between the category attribute information of the pair of video objects and the identifier of the pair of video objects.

[0265] During the process of storing the attribute information of multiple pairs of video objects into their corresponding attribute information layers, if the attribute information of a pair of video objects has already been stored in the attribute index structure, the server will not save it again. The server only needs to establish the association between the attribute information of this pair of video objects in the multiple layers of attribute information, as well as the association between the last layer of attribute information and this pair of video objects. In other words, the same attribute information is not stored repeatedly in each layer of attribute information in the attribute index structure.

[0266] For example, taking a target video where all video objects are vehicles, the order of these multiple attribute types is color, brand, and category. The attribute information of these multiple video objects is as follows: Figure 15 As shown. The attribute index structure generated based on the attribute information of these multiple video objects is as follows. Figure 6 As shown.

[0267] After obtaining the attribute index structure and the spatiotemporal relationship storage graph corresponding to each of the multiple video windows through the above content, an inverted index structure can be created for the identifiers of the multiple video windows. Then, the association between the identifiers of the video objects in the attribute index structure and the identifiers of the multiple video windows, as well as the association between the identifiers of the multiple video windows and the spatiotemporal relationship storage graph corresponding to the video windows, is established to obtain the structured index corresponding to the target video.

[0268] It should be noted that the identifiers of all video objects in the attribute index structure and the identifiers of the multiple video windows form an inverted index structure. The dictionary of this inverted index structure includes the identifiers of all video objects in the attribute index structure, and the inverted list of this inverted index structure includes the identifiers of the multiple video windows.

[0269] For example, the target video includes m video windows, and the identifiers of the m video windows include W1, W2, W3, ..., W... mThe server indexes the identifier of each video object in the attribute index structure and the identifier of the video window to which that video object belongs, and also indexes the identifier of each video window in the m video windows and the corresponding spatiotemporal relationship storage graph, resulting in the following... Figure 8 or Figure 9 The structured index shown.

[0270] In this embodiment, the attribute information of video objects and the spatiotemporal relationships between video objects are unstructured information. The method described above can convert this unstructured information into structured information for storage. This allows for fast and accurate retrieval based on query conditions using structured information, improving video query efficiency and offering greater flexibility and breadth. Furthermore, storing fine-grained spatiotemporal relationships enables precise queries of these relationships.

[0271] Figure 16 This is a schematic diagram of a video query device provided in an embodiment of this application. The device can be implemented as part or all of an electronic device by software, hardware, or a combination of both. This electronic device can be the aforementioned server or client. See also... Figure 16 The device includes: a query condition acquisition module 1601, a video frame filtering module 1602, and a video segment output module 1603.

[0272] The query condition acquisition module 1601 is used to acquire query conditions, which include attribute information of multiple objects to be queried and the spatiotemporal relationship of the multiple objects to be queried.

[0273] The video frame filtering module 1602 is used to filter out target video frames that meet the query conditions from the video frames included in the target video based on the structured index corresponding to the target video. The structured index is used to indicate the attribute information of multiple video objects in the target video and the spatiotemporal relationship between the multiple video objects.

[0274] The video clip output module 1603 is used to output the video clip corresponding to the target video frame.

[0275] In one possible implementation, the target video includes multiple non-overlapping video windows, each video window includes multiple video frames, and the multiple video frames within the same video window have an association relationship. The structured index includes attribute information of the multiple video objects and a spatiotemporal relationship storage graph corresponding to each of the multiple video windows. The spatiotemporal relationship storage graph indicates the spatiotemporal relationship between every two video objects in the corresponding video window; the video frame filtering module 1602 includes:

[0276] The target object filtering submodule is used to filter out multiple target video objects from the multiple video objects based on the attribute information of the multiple objects to be queried. The attribute information of the multiple target video objects is the same as the attribute information of the multiple objects to be queried.

[0277] The video window determination submodule is used to determine the target video window from among the multiple video windows in which the multiple target video objects appear simultaneously.

[0278] The video frame filtering submodule is used to filter target video frames from multiple video frames included in the target video window based on the spatiotemporal relationship between the multiple objects to be queried, through the spatiotemporal relationship storage graph corresponding to the target video window. The spatiotemporal relationship between the multiple target video objects in the target video frame matches the spatiotemporal relationship between the multiple objects to be queried.

[0279] In one possible implementation, the structured index further includes an attribute index structure for storing attribute information of the multiple video objects. This attribute index structure includes multiple layers of attribute information, each layer corresponding to an attribute type. Each layer includes multiple pairs of attribute information. These multiple layers of attribute information are associated according to the attribute information of the multiple video objects. Each pair of attribute information in the last layer corresponds to at least one pair of video objects among the multiple video objects. The target object filtering submodule is specifically used for:

[0280] Combine every two of the multiple objects to be queried to obtain at least one object to be queried.

[0281] Based on the attribute information of the at least one object to be queried, the attribute information of the at least one object to be queried is compared with the multi-level attribute information layer by layer in the order of the attribute types corresponding to the multi-level attribute information, so as to filter out at least one pair of target video objects whose attribute information is the same as that of the at least one object to be queried.

[0282] In one possible implementation, the spatiotemporal relation storage graph includes multiple frequent sub-graphs. Figure 1 A set of multiple spatiotemporal relationship tables are used to indicate the spatiotemporal relationship between every two video objects in the corresponding video window. These multiple frequent subgraphs are constructed according to the multiple video objects in the corresponding video window. The video frame filtering submodule is specifically used for:

[0283] The first spatiotemporal relationship table is selected from the multiple spatiotemporal relationship tables. The frequent subgraph corresponding to the first spatiotemporal relationship table includes the multiple target video objects.

[0284] Based on the spatiotemporal relationship between the multiple objects to be queried, the first spatiotemporal relationship between the multiple target video objects is determined;

[0285] Based on the first spatiotemporal relationship between the multiple target video objects, the target video frame is determined through the first spatiotemporal relationship table.

[0286] In one possible implementation, the spatiotemporal relation storage graph further includes a directed frequent structure graph, which includes the multiple frequent sub-graphs. Figure 1 A set of multiple subgraph vertices are connected according to the inclusion relationship between the corresponding frequent subgraphs. The vertices of the frequent subgraph with fewer video objects are the starting points of directed edges, and the vertices of the frequent subgraph with more video objects are the ending points of directed edges. The spatiotemporal relationship table corresponding to the frequent subgraph with the fewest video objects indicates the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. The spatiotemporal relationship table corresponding to the first frequent subgraph indicates the spatiotemporal relationship between at least one additional pair of video objects added to the first frequent subgraph compared to a subset of the first frequent subgraph. The first frequent subgraph refers to any frequent subgraph other than the one with the fewest video objects. The video frame filtering submodule is also specifically used for:

[0287] If the vertex of the subgraph corresponding to the first spatiotemporal relation table is not the starting vertex of the directed frequent structure graph, then the target video frame is determined based on the first spatiotemporal relation between the multiple target video objects, through the first spatiotemporal relation table and at least one second spatiotemporal relation table, wherein the at least one second spatiotemporal relation table is the spatiotemporal relation table corresponding to each subset contained in the frequent subgraph corresponding to the first spatiotemporal relation table.

[0288] In one possible implementation, the device further includes:

[0289] The information determination module is used to determine the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame of the target video, and the video frame in which each video object is located.

[0290] The structured index building module is used to build a structured index for the target video based on the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located.

[0291] In one possible implementation, the structured index includes a spatiotemporal relationship storage graph corresponding to multiple video windows in the target video.

[0292] The structured index building module includes:

[0293] The video window partitioning submodule is used to divide the target video into multiple video windows, each video window including multiple video frames;

[0294] The spatiotemporal relationship construction submodule is used to generate a spatiotemporal relationship storage graph corresponding to each of the multiple video windows, based on the spatial relationship between multiple video objects in each video frame included in the video window and the video frame in which each video object is located.

[0295] In one possible implementation, the video window is divided into sub-modules specifically for:

[0296] Use the first video frame in the target video as the starting frame of the first video window among the multiple video windows;

[0297] Determine the similarity between the second video frame in the target video and the starting frame of the first video window;

[0298] If the similarity is less than the similarity threshold, the second video frame is taken as the starting frame of the second video window among the multiple video windows, and the similarity between the third video frame in the target video and the starting frame of the second video window is determined.

[0299] If the similarity is greater than or equal to the similarity threshold, the second video frame is assigned to the first video window, and the similarity between the third video frame and the starting frame of the first video window is determined.

[0300] In one possible implementation, the spatiotemporal relation storage graph includes multiple spatiotemporal relation tables;

[0301] The spatiotemporal relationship construction submodule includes:

[0302] The complete graph generation unit is used to generate complete graphs corresponding to the multiple video frames included in the video window. The complete graph includes multiple object vertices that correspond one-to-one with multiple video objects in the corresponding video frames, and each pair of object vertices has a connection relationship.

[0303] The frequent subgraph determination unit is used to determine multiple frequent subgraphs based on video objects in multiple video frames included in the video window. Each frequent subgraph is a subset of the complete graph corresponding to the multiple video frames included in the video window.

[0304] The spatiotemporal relationship construction unit is used to determine the spatiotemporal relationship tables corresponding to the multiple frequent subgraphs based on the spatial relationship between multiple video objects in each video frame included in the video window and the video frame in which each video object is located in the video window, thereby obtaining the multiple spatiotemporal relationship tables.

[0305] In one possible implementation, the frequent subgraph determination unit is specifically used for:

[0306] Determine the number of occurrences of the complete graph corresponding to each of the multiple video frames. The number of occurrences of the complete graph refers to the number of times the complete graph appears in the complete graph corresponding to the multiple video frames.

[0307] Combine the object vertices corresponding to any i video objects in the multiple video frames to generate multiple derived subgraphs. These multiple derived subgraphs are complete graphs composed of the object vertices corresponding to any i video objects, where i can be 2 to N, and N is the number of video objects included in the multiple video frames.

[0308] Determine the occurrence count of the multiple derived subgraphs, where the occurrence count refers to the number of times the derived subgraph appears in the complete graph corresponding to the multiple video frames;

[0309] Based on the occurrence counts of the complete graph corresponding to each of the multiple video frames and the occurrence counts of the multiple derived subgraphs, the multiple derived subgraphs are filtered to obtain the multiple frequent subgraphs.

[0310] In one possible implementation, the spatiotemporal relation storage graph further includes a directed frequent structure graph, which includes the multiple frequent sub-graphs. Figure 1 One-to-one correspondence of multiple subgraph vertices;

[0311] The spatiotemporal relationship construction unit is specifically used for:

[0312] Based on the inclusion relationship between these multiple frequent subgraphs, a directed frequent structure graph is created, where the vertices of the frequent subgraphs with fewer video objects are the starting points of the directed edges, and the vertices of the frequent subgraphs with more video objects are the ending points of the directed edges.

[0313] For each frequent subgraph in the multiple frequent subgraphs, if there is no subset of the frequent subgraph in the multiple frequent subgraphs, then a spatiotemporal relationship table corresponding to the frequent subgraph is generated based on the spatial relationship between every two video objects in the video objects corresponding to all object vertices included in the frequent subgraph and the video frames to which every two video objects belong.

[0314] If a subset of the frequent subgraph exists within the multiple frequent subgraphs, a spatiotemporal relationship table corresponding to the frequent subgraph is generated based on the spatial relationship between every two video objects corresponding to all object vertices included in the frequent subgraph, the video frames where every two video objects are located, and the subset of the frequent subgraph. The spatiotemporal relationship table corresponding to the frequent subgraph is used to store the spatial relationship between at least one pair of video objects newly added to the frequent subgraph compared to the subset of the frequent subgraph, and the video frames where the at least one pair of video objects are located.

[0315] In one possible implementation, the structured index further includes an attribute index structure, which includes multiple layers of attribute information, each layer corresponding to an attribute type; the device also includes:

[0316] The video object combination module is used to combine every two video objects from the multiple video objects included in the target video to obtain multiple pairs of video objects;

[0317] The attribute structure construction module is used to store the attribute information of multiple pairs of video objects into the corresponding first-level attribute information according to the order of the attribute types corresponding to the multi-level attribute information, and to establish the association relationship of the attribute information of the same pair of video objects in the multi-level attribute information, as well as the association relationship between the last level attribute information and at least one pair of video objects.

[0318] In this embodiment, the structured index corresponding to the target video records attribute information of multiple video objects in the target video. Therefore, regardless of changes in the number of queried objects and the type of attribute information of the queried objects in the query conditions, the server can determine multiple target video objects that match the attribute information of the multiple queried objects in the query conditions based on the attribute information of the multiple video objects in the target video. The structured index also records the spatiotemporal relationships between the multiple video objects in the target video. After determining the multiple target video objects, the spatiotemporal relationships between the multiple queried objects are converted into spatiotemporal relationships between the multiple target video objects. Then, through these spatiotemporal relationships, the video frame containing the video object that matches both the attribute information and the spatiotemporal relationships between the multiple queried objects is determined; that is, the target video frame that satisfies the query conditions. In summary, regardless of changes in the number of queried objects and the type of attribute information of the queried objects in the query conditions, and regardless of whether the query conditions involve spatiotemporal relationships between multiple queried objects, the server can determine the target video frame that satisfies the query conditions. In other words, the server can handle flexible and varied video query conditions and can realize diverse video query needs.

[0319] It should be noted that the video query device provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video query device and the video query method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0320] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the video query method described above.

[0321] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the video query method described above.

[0322] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0323] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0324] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0325] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A video query method, characterized in that, The method includes: Obtain query conditions, which include attribute information of multiple objects to be queried and the spatiotemporal relationship of the multiple objects to be queried; Based on the structured index corresponding to the target video, target video frames that meet the query conditions are filtered from the video frames included in the target video. The structured index is used to indicate the attribute information of multiple video objects in the target video, as well as the spatiotemporal relationship between the multiple video objects. Output the video segment corresponding to the target video frame.

2. The method as described in claim 1, characterized in that, The target video includes multiple non-overlapping video windows, each video window includes multiple video frames, and the multiple video frames within the same video window have an association relationship. The structured index includes attribute information of the multiple video objects and a spatiotemporal relationship storage graph corresponding to each of the multiple video windows. The spatiotemporal relationship storage graph indicates the spatiotemporal relationship between every two video objects in the corresponding video window. The step of filtering target video frames that meet the query conditions from the video frames included in the target video based on the structured index corresponding to the target video includes: Based on the attribute information of the multiple objects to be queried, multiple target video objects are selected from the multiple video objects, and the attribute information of the multiple target video objects is the same as the attribute information of the multiple objects to be queried; From the plurality of video windows, determine the target video window in which the plurality of target video objects appear simultaneously; Based on the spatiotemporal relationship between the multiple objects to be queried, the target video frame is selected from the multiple video frames included in the target video window through the spatiotemporal relationship storage graph corresponding to the target video window. The spatiotemporal relationship between the multiple target video objects in the target video frame matches the spatiotemporal relationship between the multiple objects to be queried.

3. The method as described in claim 2, characterized in that, The structured index also includes an attribute index structure for storing attribute information of the plurality of video objects. The attribute index structure includes multiple layers of attribute information, each layer of attribute information corresponding to an attribute type. Each layer of attribute information includes multiple pairs of attribute information. The multiple layers of attribute information are associated according to the attribute information of the plurality of video objects. Each pair of attribute information included in the last layer of attribute information corresponds to at least one pair of video objects among the plurality of video objects. The step of filtering multiple target video objects from the multiple video objects based on the attribute information of the multiple objects to be queried includes: Combine every two of the plurality of objects to be queried to obtain at least one object to be queried. Based on the attribute information of the at least one object to be queried, the attribute information of the at least one object to be queried is compared with the multi-level attribute information layer by layer in the order of the attribute types corresponding to the multi-level attribute information, so as to filter out at least one pair of target video objects whose attribute information is the same as that of the at least one object to be queried.

4. The method as described in claim 2 or 3, characterized in that, The spatiotemporal relationship storage graph includes multiple spatiotemporal relationship tables that correspond one-to-one with multiple frequent subgraphs. The multiple spatiotemporal relationship tables are used to indicate the spatiotemporal relationship between every two video objects in the corresponding video window. The multiple frequent subgraphs are constructed according to the multiple video objects in the corresponding video window. The step of filtering the target video frame from multiple video frames included in the target video window based on the spatiotemporal relationship between the multiple objects to be queried and through the spatiotemporal relationship storage graph corresponding to the target video window includes: A first spatiotemporal relationship table is selected from the plurality of spatiotemporal relationship tables, and the frequent subgraph corresponding to the first spatiotemporal relationship table includes the plurality of target video objects; Based on the spatiotemporal relationship between the plurality of objects to be queried, a first spatiotemporal relationship is determined between the plurality of target video objects; The target video frame is determined by the first spatiotemporal relationship table based on the first spatiotemporal relationship table among the plurality of target video objects.

5. The method as described in claim 4, characterized in that, The spatiotemporal relationship storage graph also includes a directed frequent structure graph, which includes multiple subgraph vertices corresponding one-to-one with the multiple frequent subgraphs. The multiple subgraph vertices are connected according to the inclusion relationship between the corresponding frequent subgraphs. The subgraph vertices corresponding to the frequent subgraphs with fewer video objects are the starting points of the directed edges, and the subgraph vertices corresponding to the frequent subgraphs with more video objects are the ending points of the directed edges. The spatiotemporal relationship table corresponding to the frequent subgraph with the fewest video objects is used to indicate the spatiotemporal relationship between every two video objects in the corresponding frequent subgraph. The spatiotemporal relationship table corresponding to the first frequent subgraph is used to indicate the spatiotemporal relationship between at least one pair of video objects added to the first frequent subgraph compared to the subset of the first frequent subgraph. The first frequent subgraph refers to any frequent subgraph other than the frequent subgraph with the fewest video objects. The step of determining the target video frame based on the first spatiotemporal relationship between the plurality of target video objects through the first spatiotemporal relationship table includes: If the subgraph vertex corresponding to the first spatiotemporal relation table is not the starting vertex of the directed frequent structure graph, then the target video frame is determined based on the first spatiotemporal relation between the multiple target video objects, through the first spatiotemporal relation table and at least one second spatiotemporal relation table, wherein the at least one second spatiotemporal relation table is the spatiotemporal relation table corresponding to each subset contained in the frequent subgraph corresponding to the first spatiotemporal relation table.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Determine the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame of the target video, and the video frame in which each video object is located; Based on the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located, a structured index corresponding to the target video is constructed.

7. The method as described in claim 6, characterized in that, The structured index includes a spatiotemporal relationship storage graph corresponding to multiple video windows in the target video; The step of constructing a structured index corresponding to the target video based on the attribute information of multiple video objects in the target video, the spatial relationship between multiple video objects in each video frame, and the video frame in which each video object is located includes: The target video is divided into multiple video windows, and each video window includes multiple video frames; For each of the plurality of video windows, a spatiotemporal relationship storage graph corresponding to the video window is generated based on the spatial relationship between multiple video objects in each video frame included in the video window and the video frame in which each video object in the video window is located.

8. The method as described in claim 7, characterized in that, The step of dividing the target video into the multiple video windows includes: The first video frame in the target video is used as the starting frame of the first video window in the plurality of video windows; Determine the similarity between the second video frame in the target video and the starting frame of the first video window; If the similarity is less than the similarity threshold, the second video frame is used as the starting frame of the second video window in the plurality of video windows, and the similarity between the third video frame in the target video and the starting frame of the second video window is determined. If the similarity is greater than or equal to the similarity threshold, the second video frame is assigned to the first video window, and the similarity between the third video frame and the starting frame of the first video window is determined.

9. The method as described in claim 7, characterized in that, The spatiotemporal relationship storage diagram includes multiple spatiotemporal relationship tables; The step of generating a spatiotemporal relationship storage graph corresponding to the video window based on the spatial relationships between multiple video objects in each video frame included in the video window and the video frame in which each video object is located in the video window includes: Generate a complete graph corresponding to each of the multiple video frames included in the video window. The complete graph includes multiple object vertices that correspond one-to-one with multiple video objects in the corresponding video frames, and each pair of object vertices has a connection relationship. Based on the video objects in the multiple video frames included in the video window, multiple frequent subgraphs are determined, and each frequent subgraph is a subset of the complete graph corresponding to the multiple video frames included in the video window. Based on the spatial relationships between multiple video objects in each video frame included in the video window and the video frame in which each video object is located, the spatiotemporal relationship tables corresponding to the multiple frequent subgraphs are determined, and the multiple spatiotemporal relationship tables are obtained.

10. The method as described in claim 9, characterized in that, The step of determining multiple frequent subgraphs based on video objects in multiple video frames included in the video window includes: The occurrence count of the complete graph corresponding to each of the plurality of video frames is determined, wherein the occurrence count of the complete graph refers to the number of times the complete graph appears in the complete graph corresponding to the plurality of video frames; The object vertices corresponding to any i video objects in the plurality of video frames are combined to generate a plurality of derived subgraphs. The plurality of derived subgraphs are complete graphs composed of the object vertices corresponding to any i video objects, where i takes the value from 2 to N, and N is the number of video objects included in the plurality of video frames. The occurrence count of the plurality of derived subgraphs is determined, wherein the occurrence count of the derived subgraph refers to the number of times the derived subgraph appears in the complete graph corresponding to the plurality of video frames; Based on the occurrence counts of the complete graphs corresponding to the multiple video frames and the occurrence counts of the multiple derived subgraphs, the multiple derived subgraphs are filtered to obtain the multiple frequent subgraphs.

11. The method as described in claim 9, characterized in that, The spatiotemporal relationship storage graph also includes a directed frequent structure graph, which includes multiple subgraph vertices that correspond one-to-one with the multiple frequent subgraphs; The step of determining the spatiotemporal relationship tables corresponding to the multiple frequent subgraphs based on the spatial relationships between multiple video objects in each video frame included in the multiple video windows and the video frame in which each video object is located in the video window includes: Based on the inclusion relationship between the multiple frequent subgraphs, the directed frequent structure graph is created, and the vertices of the frequent subgraphs with fewer video objects are the starting points of the directed edges, while the vertices of the frequent subgraphs with more video objects are the ending points of the directed edges. For each frequent subgraph in the plurality of frequent subgraphs, if there is no subset of the frequent subgraph in the plurality of frequent subgraphs, then a spatiotemporal relationship table corresponding to the frequent subgraph is generated based on the spatial relationship between every two video objects in the video objects corresponding to all object vertices included in the frequent subgraph and the video frames to which every two video objects belong. If a subset of the frequent subgraph exists in the plurality of frequent subgraphs, a spatiotemporal relationship table corresponding to the frequent subgraph is generated based on the spatial relationship between every two video objects in the video objects corresponding to all object vertices included in the frequent subgraph, the video frames in which every two video objects are located, and the subset of the frequent subgraph. The spatiotemporal relationship table corresponding to the frequent subgraph is used to store the spatial relationship between at least one pair of video objects newly added to the frequent subgraph compared to the subset of the frequent subgraph, and the video frames in which the at least one pair of video objects are located.

12. The method according to any one of claims 7-11, characterized in that, The structured index further includes an attribute index structure, which includes multiple layers of attribute information, each layer corresponding to an attribute type; the method further includes: The target video is composed of multiple video objects. Each pair of video objects is combined to obtain multiple pairs of video objects. According to the order of the attribute types corresponding to the multi-layer attribute information, the attribute information of the multiple pairs of video objects is stored in the corresponding layer attribute information, and the association relationship of the attribute information of the same pair of video objects is established in the multi-layer attribute information, as well as the association relationship between the last layer attribute information and at least one pair of video objects.

13. A video query device, characterized in that, The apparatus includes at least one module for implementing the method according to any one of claims 1-12.

14. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program and the processor being used to execute the computer program to implement the method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on the computer, cause the computer to perform the method described in any one of claims 1-12.

16. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method according to any one of claims 1-12.

Images