Video summarization method, system, device and medium based on object occlusion graph structure
By segmenting the object pipeline in the monitoring video and using the MCMC algorithm to generate summary video, the problem of timing, integrity and collision avoidance in the prior art is solved, and the video summary is improved Flexibility and efficiency.
Patent Information
- Application Number
- CN202210311826.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-28
AI Technical Summary
When processing monitoring videos, the existing object-based video digest method is difficult to meet multiple goals such as timing, integrity and collision avoidance, and the minimum processing unit is the entire object pipeline, which is insufficient flexibility.
Using the method based on the object occlusion diagram structure, the object pipelines in the monitoring video are segmented, and the summary video is sampled and generated through the MCMC algorithm to solve the problem of position conflict between the pipeline segments and establish the graph structure to ensure timing and integrity.
The position conflict problem caused by object circulating occlusion was successfully solved. The generated video digest can meet the goals of timing, integrity and collision avoidance, and improve the flexibility and efficiency of video digest.
Smart Images

Figure CN114693632B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer image and video processing, and in particular to a video summarization method, system, device and medium based on an object occlusion graph structure. Background Art
[0002] With the development of science and technology, surveillance cameras are now used in many places for 24 hours a day. For people, it becomes difficult and inefficient to browse a large amount of surveillance videos just to obtain key information. Therefore, it is particularly important to develop methods to extract key information from videos.
[0003] Object-based video summarization often takes the object pipeline of the foreground object in the surveillance video as the overall processing unit, separates it from the basically fixed background, and offsets it on the time axis through certain methods while keeping its position area in the picture unchanged, thereby achieving the purpose of reducing the redundancy of video information in time and space at the same time. This technology can generate shorter video summaries from longer videos, which is convenient for people to quickly browse and extract key information. Compared with frame-based video summarization, this method has the significant advantage of reducing spatial information redundancy, but its minimum processing unit is the entire object pipeline, which is not flexible enough and difficult to meet multiple goals such as timing, integrity, and collision avoidance at the same time. Summary of the invention
[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a video summarization method, system, device and medium based on an object occlusion graph structure.
[0005] The technical solution adopted by the present invention is:
[0006] A video summarization method based on object occlusion graph structure comprises the following steps:
[0007] Obtain surveillance video, extract pipeline information based on the surveillance video, and obtain the object pipeline;
[0008] The object pipeline is divided into pipeline segments according to the occlusion relationship;
[0009] Create edges between pipeline segments to build a graph structure;
[0010] Solve the location conflict problem between pipeline segments and improve the graph structure;
[0011] Based on the graph structure, the MCMC algorithm is used to sample pipeline segments and generate summary videos.
[0012] Furthermore, extracting pipeline information according to the monitoring video to obtain the object pipeline includes:
[0013] The KCF tracking algorithm is used to extract the pipeline information in the surveillance video and obtain the object pipeline O i , i∈[1,N], N is the number of foreground objects; the pipeline information includes the frame number and the position and size of the bounding box of the foreground object.
[0014] Furthermore, the establishing of edges between pipeline segments and the preliminary construction of a graph structure include:
[0015] Use pipe segments as nodes;
[0016] For the jth pipeline segment on the same pipeline i At the node and node Add bidirectional edges between adjacent nodes;
[0017] For pipeline segments with occlusion relationships between different pipelines, bidirectional edges are established between the nodes.
[0018] Furthermore, when a loop formed by cyclic occlusion of multiple objects appears in the graph structure, position conflict problems will occur;
[0019] The method of solving the position conflict problem between pipeline segments and improving the graph structure includes:
[0020] Find the ring formed by the cyclic occlusion of multiple objects in the graph structure, so that the pipeline segments in the same time period in the ring cut each other; for some pipeline segments in the same time period that were originally in different pipelines and not connected, additional bidirectional edges are established. Even if they are not occluded in the original video, occlusion connections are established for them, so that the pipeline segments in the ring can be fully synchronized during traversal updates.
[0021] Furthermore, when a composite ring formed by multiple nested and interlaced rings appears in the graph structure, the problem of position conflict between pipeline segments is solved and the graph structure is improved, including:
[0022] If no nodes in the loop are detected in the traversal graph, for the current node, determine whether other nodes are in the same loop as the current node;
[0023] Traverse the nodes currently in the ring and obtain the maximum composite ring by recursively searching for nodes;
[0024] For all pipeline segments within the largest composite ring, the pipeline segments with overlapping time periods are divided from each other;
[0025] After the pipe segments in the ring are divided from each other, additional shielding connections are selectively established for the divided pipe segments in the ring, so that no position conflict occurs in the ring.
[0026] Furthermore, based on the graph structure, the pipeline segments are sampled using the MCMC algorithm to generate summary videos, including:
[0027] After perfecting the graph structure, the pipeline segments are sampled twice by MCMC to determine the optimal location of the pipeline segments;
[0028] The first MCMC sampling makes the overall pipeline segment meet the requirement that the timing is basically consistent with the original video: first, all graphs are offset to the starting point of the video time, and the MCMC algorithm is used to randomly sample the reverse pipeline segments, iterating multiple times until the graph structure is consistent with the timing of the original video graph structure within the allowable error range;
[0029] The second MCMC sampling minimizes the integrity loss and collision loss of the video summary: the pipeline segments are randomly sampled, the integrity loss and collision loss are calculated in multiple iterations, and finally the result with the smallest comprehensive loss and consistent timing is selected to generate the summary video.
[0030] Furthermore, the video summarization method further comprises the step of visualizing the summary video:
[0031] For the shortened or elongated pipeline segments, a linear sampling method is used to select or supplement video frames, so that the object motion becomes faster or slower in the summary video;
[0032] Opencv is used to visualize the processed pipeline data of the foreground object on a fixed background to generate a summary video.
[0033] Further,
[0034] Another technical solution adopted by the present invention is:
[0035] A video summarization system based on an object occlusion graph structure, comprising:
[0036] The pipeline acquisition module is used to acquire the monitoring video, extract the pipeline information according to the monitoring video, and obtain the object pipeline;
[0037] A pipeline segmentation module is used to segment the object pipeline into pipeline segments according to the occlusion relationship;
[0038] A graph building module, used to establish edges between pipeline segments and build graph structures;
[0039] The graph improvement module is used to solve the position conflict problem between pipeline segments and improve the graph structure;
[0040] The pipeline sampling module is used to sample pipeline segments based on the graph structure and use the MCMC algorithm to generate summary videos.
[0041] Another technical solution adopted by the present invention is:
[0042] A video summarization device based on an object occlusion graph structure, comprising:
[0043] at least one processor;
[0044] at least one memory for storing at least one program;
[0045] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0046] Another technical solution adopted by the present invention is:
[0047] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.
[0048] The beneficial effects of the present invention are as follows: the present invention segments object pipelines according to the occlusion relationship between them, establishes a graph structure, applies the MCMC random sampling method to the object pipelines segmented according to the intrinsic occlusion relationship of the video, retains its inherent logic, and successfully solves the position conflict problem caused by the ring formed by the cyclic occlusion of the object. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0050] Figure 1 is a diagram illustrating a method of dividing object pipes into each other according to the mutual occlusion relationship in an embodiment of the present invention;
[0051] Figure 2 is an illustration of a bidirectional connection method between object pipeline segments in an embodiment of the present invention;
[0052] Figure 3 is a schematic diagram of an example of a position conflict caused by a ring formed by cyclic occlusion of multiple objects in an embodiment of the present invention;
[0053] Figure 4 is a schematic diagram of a method for resolving position conflicts in an embodiment of the present invention;
[0054] Figure 5 is a diagram illustrating a method for finding a maximum composite ring in an embodiment of the present invention;
[0055] Figure 6 is an illustration of a method for selectively establishing an occlusion connection after segmentation within a ring in an embodiment of the present invention;
[0056] Figure 7 is an illustration of a greedy algorithm for adding missing occluded connections in an embodiment of the present invention;
[0057] Figure 8 is a flowchart illustrating a video summarization method based on an object occlusion graph structure and pipeline segmentation optimization in an embodiment of the present invention;
[0058] Fig. 9 is a comparison diagram of the video summary and the original video on the data set 1 in an embodiment of the present invention;
[0059] Fig.10 is a comparison diagram of the video summary and the original video on data set 2 in an embodiment of the present invention;
[0060] Fig.11 is a comparison diagram of the video summary and the original video on data set 3 in an embodiment of the present invention;
[0061] Fig.12 is a flowchart of Algorithm 1 in an embodiment of the present invention;
[0062] Fig.13 is a flowchart of Algorithm 2 in an embodiment of the present invention;
[0063] Fig.14 is a flowchart of Algorithm 3 in an embodiment of the present invention;
[0064] Fig.15 is a flowchart of Algorithm 4 in an embodiment of the present invention;
[0065] Fig.16 is a flowchart of Algorithm 5 in an embodiment of the present invention;
[0066] Fig.17 is a flowchart of Algorithm 6 in an embodiment of the present invention;
[0067] Fig.18 It is a flowchart of the steps of a video summarization method based on an object occlusion graph structure in an embodiment of the present invention. DETAILED DESCRIPTION
[0068] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0069] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0070] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0071] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0072] like Fig.18 As shown, this embodiment provides a video summarization method based on an object occlusion graph structure, comprising the following steps:
[0073] S1. Obtain monitoring video, extract pipeline information according to the monitoring video, and obtain the object pipeline;
[0074] S2, dividing the object pipeline into pipeline segments according to the occlusion relationship;
[0075] S3, establish edges between pipeline segments and construct a graph structure;
[0076] S4, solve the location conflict problem between pipeline segments and improve the graph structure;
[0077] S5. Based on the graph structure, the MCMC algorithm is used to sample the pipeline segments and generate summary videos.
[0078] The method proposed in this embodiment mainly segments the object pipelines according to the occlusion relationship between them, establishes a graph structure, and refines the minimum processing unit into pipeline segments, making the graph structure more flexible and making the generated video summary meet the three goals of timing, integrity, and collision avoidance as much as possible. We invented this graph structure for the first time, applied the MCMC random sampling method to the object pipeline segmented according to the intrinsic occlusion relationship of the video, retained its inherent logic, and successfully solved the position conflict problem caused by the ring formed by the cyclic occlusion of the object.
[0079] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.
[0080] like Figure 8 As shown, this embodiment provides a video summarization method based on object occlusion graph structure and pipeline segmentation optimization, which specifically includes:
[0081] (1) Extract pipeline data
[0082] For the original surveillance video, the KCF tracking algorithm is used to extract the pipeline information such as the frame number, bounding box position and size of the foreground objects (such as people and cars), and obtain the object pipeline Oi, i∈[1,N], N is the number of foreground objects.
[0083] (2) Divide the pipeline into segments based on the occlusion relationship
[0084] The object pipeline is divided according to whether there is occlusion between objects. Figure 1 As shown in (a), the horizontal axis is time, that is, the number of video frames, and the vertical axis is the object number. i and O i In from the s ij Frame to e ij Occlusion occurs during the time period of the frame, so we divide it into several segments according to the occlusion relationship. It is worth noting that if two objects occlude each other multiple times in a time period, we will also merge them into one segment. For some pipeline segments that are occluded by multiple objects at the same time, we will consider the occlusion relationship one by one and further divide the original pipeline segment, such as Figure 1 As shown in (c)(d). For the non-occluded segment, we also use a fixed-length time segment τ to divide it into smaller segments, such as Figure 1 (b) as shown. Figure 1 middle: Figure 1 (a) Establish occlusion connections between occlusion segments; Figure 1 (b) Fixed-length segmentation of non-blocked segments; Figure 1 (c) Multiple object occlusion segments have overlapping parts; Figure 1 (d) The occluded segments with overlapping parts are iteratively segmented.
[0085] (3) Constructing a graph structure for pipeline segments
[0086] We take each pipeline segment as a node. For the jth pipeline segment on the same pipeline i, We add bidirectional edges between the node and its adjacent nodes. For pipeline segments with occlusion relationships between different pipelines, we also establish bidirectional edges between them. In this way, object pipelines that are not occluded by any other objects can form a graph by themselves, and a group of objects connected by occlusion relationships can also form a graph. Figure 2 Shown is a simple example of a graph consisting of a group of objects.
[0087] (4) Solve the location conflict problem and improve the graph structure
[0088] 1) Position conflict caused by cyclic occlusion
[0089] Generally, object-based video summarization often uses offset pipeline position and change of object speed (i.e. pipeline length) to reduce information redundancy. We refine the minimum processing unit to the pipeline segment and take measures such as offsetting the pipeline segment position and changing the pipeline length.
[0090] We define a global offset variable T for the entire graph k , for a graph G k , select the earliest object pipeline segment as the root node segment and offset it by T k .like Figure 2 As shown, O i The first segment is the root segment. After the offset, the root segment position (frame number) becomes:
[0091]
[0092] In addition, we also analyze each pipeline segment Define a scale variable This changes the length of the pipe segment After scaling, the root node segment length becomes:
[0093]
[0094] The root node segment position becomes:
[0095]
[0096] After changing the position or length of a pipeline segment, we use the DFS algorithm to traverse and update the entire graph. The DFS algorithm is a depth-first search algorithm (DFS), which traverses the nodes of the tree along the depth of the tree or graph, searching the branches of the tree or graph as deeply as possible. When all the edges of the current node have been explored, the search will trace back to the starting node of the edge that found the current node. If there are still nodes connected to the node that have not been traversed, one of them is selected as the source node and the above process is repeated. The whole process is repeated until all the nodes of the tree or graph are visited.
[0097] Assuming the pipeline segment The updated location is Then the update rule is:
[0098] ① If the next node to be traversed is the node behind the current node on the same pipeline So The location is updated to
[0099] ② If the next node to be traversed is the previous node on the same pipeline as the current node So The location is updated to
[0100] ③ If the next node to be traversed is a node S that has an occluded connection with the current node x y , then S x y Directly inherit the current node The location is updated to
[0101] In addition, traversing nodes with occlusion relationships takes precedence over traversing adjacent nodes in the same pipeline.
[0102] The above traversal update algorithm has no problem for most simple examples. However, for a graph structure with a ring formed by multiple objects circling and occluding each other, position conflicts may occur during the operation of the algorithm. Figure 3 As shown, let's assume that the updated position of pipeline segment A has been determined. The above traversal update algorithm will update the pipeline segments along the gray route to pipeline segment B in sequence. However, the last frame of A may not be seamlessly connected with the first frame of B, forming an unnecessary gap or causing them to overlap, resulting in a position conflict.
[0103] We have adopted a solution to this problem. We first find the ring formed by multiple objects cyclically occluding each other, so that the pipeline segments in the same time period in the ring cut each other, and then selectively establish additional bidirectional edges for some pipeline segments in the same time period that were originally not connected in different pipelines. Even if they are not occluded in the original video, occlusion connections are established for them, so that the pipeline segments in the ring can be completely synchronized during traversal updates. A simple example is Figure 4 As shown in the figure, the same color fills a pair of occluded segments in the ring, and the segments filled with diagonal lines are non-occluded segments in the original video in the ring. The original traversal update order from pipeline segment A to B is as follows Figure 4 (a). Figure 4 In (b), we cut the pipeline segments in the same period in the ring and add additional occlusion connections, so the traversal order of the pipeline segments in the ring becomes Figure 4 The gray path in (c) will not cause position conflicts. Figure 4 middle: Figure 4 (a) Graph structure and update path before processing the ring; Figure 4 (b) The graph structure after processing the ring; Figure 4 (c) Updated path after processing the ring.
[0104] 2) Find the largest composite ring
[0105] For surveillance videos with busy scenes and more occlusions, the object pipeline is more complicated, and it is very likely that there will be a composite ring formed by several nested and interlaced rings, such as Figure 5 (a) is not like Figure 4 There are many ways to find a ring. Here we use a more efficient method for this situation (Algorithm 1). Algorithm 1 is as follows Fig.12 As shown. Figure 5 In the example, Figure 5 The pipe segments filled with diagonal lines in (b) form a maximum composite ring. First, we traverse the nodes in the graph that are not detected in the ring. For the current node, we use Algorithm 3 to determine whether other nodes are in the same ring as the current node. Algorithm 3 is as follows: Fig.14 However, since the function of Algorithm 3 is a necessary but not sufficient condition to determine whether two nodes in a graph belong to the same ring, there may be nodes in this ring that have not been detected, such as Figure 5 (c), so we need to use Algorithm 2 recursively to find these nodes. Algorithm 2 is as follows: Fig.13 As shown. In this way, the maximum composite ring can be obtained.
[0106] Figure 5 The simple example in (d) illustrates why Algorithm 3 is a necessary but insufficient condition. Starting from the red node in the upper left corner to the star node, the black path is path 1 found by Algorithm 3, but after interrupting the edges along the way, the DFS algorithm cannot find a second path to the star node anyway, as shown by the blue path.
[0107] Figure 5 In (e), Algorithm 2 traverses the node currently in the ring and uses Algorithm 3 to detect whether its neighboring nodes are in the same ring. When Algorithm 3 detects neighboring nodes, as long as they are in the same ring, they can be detected. Figure 5 As shown in (e), for a pair of adjacent nodes in the same ring, the black direct path and the blue path do not interfere with each other. No matter which path Algorithm 3 finds and interrupts first, the second path can be found. Figure 5 middle: Figure 5 (a) The original graph structure; Figure 5 (b) The segment filled with oblique lines is the pipeline segment inside the ring; Figure 5 (c) The pipeline segment originally in the ring cannot be found using only Algorithm 3; Figure 5 (d) The reason why Algorithm 3 misses pipeline segments; Figure 5 (e) The reason why Algorithm 3 is definitely effective for directly connected pipeline segments; Figure 5 (f) The pipe sections within the ring are cut into each other; Figure 5 (g) The pipe segments within the ring are connected after being cut.
[0108] 3) Further segmentation
[0109] For all pipeline segments within the largest composite ring, the pipeline segments with overlapping time periods are divided into segments, such as Figure 5 (f) as shown.
[0110] 4) Selectively establish occlusion connections
[0111] for Figure 5 In the example, after the pipe segments in the ring are divided into each other, additional blocking connections are established, such as Figure 5 As shown in (g). For some special graph structures, in order to maximize their flexibility, we selectively add occlusion connections according to the sufficient conditions in Algorithm 4. Algorithm 4 is as follows: Fig.15 However, this sufficient condition alone may miss some connections that should be added. Therefore, we use Algorithm 5 to calculate the number of potential conflicting positions in the ring, and use the greedy algorithm of Algorithm 6 to add connections to them so that there will be no position conflicts in the ring. Algorithm 5 is as follows Fig.16 shown.
[0112] Figure 6 (b) is a solution that retains structural flexibility to the maximum extent. The pipeline segments in the green box are partially paired and can be freely expanded and contracted. In Algorithm 4, the sufficient condition for adding an additional occlusion connection between a pair of pipeline segments is: for the pipeline segments in the ring that are completely in the same time period as the pair of pipeline segments, interrupt their connections with the left and right neighbor pipeline segments, and check whether their left neighbor pipeline segments and right neighbor pipeline segments are reachable. If both are reachable, then establish an occlusion connection. Figure 6 (c) is a positive example where a connection can be established. Figure 6 (d) is a counterexample to this condition and no connection can be established. Figure 6 middle: Figure 6 (a) Establishing shielded connections between all the divided pipeline segments will result in a loss of flexibility; Figure 6 (b) A part of the graph structure is locally connected and can be freely expanded and contracted; Figure 6 (c) Positive examples that can establish connections according to Algorithm 4; Figure 6 (d) Counterexample of Algorithm 4.
[0113] However, establishing connections based only on the sufficient conditions in Algorithm 4 may result in some connections that should have been established being missed in some rare cases, thus causing position conflicts. Figure 7 As shown, Figure 7 (a) is the graph structure of the original video, and the pipe segments filled with diagonal lines are all inside the ring. Figure 7In (b), the only additional connection established according to Algorithm 4 is the purple connection, and the dotted connection is the connection that should have been established but was omitted. Therefore, we use Algorithm 5 to calculate the number of potential conflicting positions, and use this as a basis to use the greedy algorithm in Algorithm 6 to calculate which connection among the unestablished connections can reduce the number of potential conflicting positions the most, establish this connection, and then use Algorithm 4 to continue to add connections that can be established based on the established connection. Algorithm 6 is as follows Fig.17 As shown. Figure 7 As shown in (c), after the bold purple bidirectional connection is established, thin purple connections can be added one after another. Figure 7 In (d), the greedy algorithm continues to add connections until there are no potential conflicting locations. Figure 7 middle: Figure 7 (a) The pipe segments within the ring are cut into each other; Figure 7 (b) the connections that should be established within the ring; Figure 7 (c) The bold purple bidirectional edges are the edges selected by the greedy algorithm, and the thin purple bidirectional edges are the new connections that meet the conditions after they are established; Figure 7 (d) Gradually establish connections using a greedy algorithm until there are no potential conflicting locations.
[0114] (5) Combined with the energy equation, the Markov chain Monte Carlo (MCMC) algorithm is used to sample the pipeline segment
[0115] After perfecting the graph structure, we perform two MCMC samplings on the pipeline segments to determine the optimal locations of the pipeline segments to meet the three goals of timing, completeness, and collision avoidance.
[0116] The first MCMC sampling makes the overall pipeline segment meet the requirement that the timing is basically consistent with the original video. We first offset all graphs to the starting point of the video time, and use the MCMC algorithm to randomly sample the reverse pipeline segments (offset the pipeline segments or change the length of the pipeline segments), iterating millions of times until the graph structure is consistent with the original video graph structure in timing within the allowable error range.
[0117] The second MCMC sampling minimizes the integrity loss and collision loss of the video summary. Similarly, the pipeline segments are randomly sampled, and the integrity loss and collision loss are calculated in millions of iterations. Finally, the result with the smallest comprehensive loss and consistent timing is selected to generate the summary video.
[0118] (6) Visual summary video
[0119] For the shortened or elongated pipeline segments, we use linear sampling to select or supplement video frames, and the movement of these objects will become faster or slower in the summary video. With these pipeline segments and the original video data, we can use OpenCV to visualize them on a fixed background and generate summary videos.
[0120] We filmed in areas with busy scene activities and used self-filmed videos as input datasets. Videos with more occlusions can better reflect the effect of this summary method. For a specific surveillance video, we can specify the length of the video summary we want to generate, the scalable ratio range of the pipeline segment, and the error range allowed by the timing. Compared with the general uniform fast forward, our method can also specify the scaling ratio range for the occluded segment and the non-occluded segment respectively, so that it is appropriate to emphasize the relationship between pipelines, such as slowing down the occluded segment appropriately.
[0121] Fig. 9 , Fig.10 , Fig.11 The following are comparisons of video summaries and original videos on three datasets. Fig. 9 In the corresponding dataset 1, the video was compressed from 2 minutes and 47 seconds to 20 seconds; Fig.10 In the corresponding dataset 2, the video was compressed from 6 minutes and 44 seconds to 1 minute and 20 seconds; Fig.11 In the corresponding dataset 3, the video was compressed from 1 minute 58 seconds to 15 seconds. The left side of the comparison image shows the video summary, and the right side shows the original video. The objects marked with blue boxes are the corresponding objects that are basically unchanged in the comparison image, and the red boxes indicate that the speed of the corresponding objects has changed significantly or they are different objects. It can be seen that some objects avoid collisions by becoming faster or slower, and some scenes in the video summary are also more compact than the original video, reducing the redundancy of the scenes, and the relationship between objects is also preserved.
[0122] In summary, the method of this embodiment has the following advantages and beneficial effects compared with the prior art:
[0123] (1) The method of this embodiment segments the original video object pipelines according to the occlusion relationship between them, establishes a graph structure, and refines the minimum processing unit into pipeline segments, making the graph structure more flexible and improving the possibility of generating a better video summary that meets the target conditions.
[0124] (2) The method of this embodiment can effectively reduce video information redundancy for surveillance videos with busy scenes and many occlusions between foreground objects, and generate summary videos that can simultaneously meet the three goals of timing, integrity, and collision avoidance.
[0125] This embodiment also provides a video summarization system based on an object occlusion graph structure, including:
[0126] The pipeline acquisition module is used to acquire the monitoring video, extract the pipeline information according to the monitoring video, and obtain the object pipeline;
[0127] A pipeline segmentation module is used to segment the object pipeline into pipeline segments according to the occlusion relationship;
[0128] A graph building module, used to establish edges between pipeline segments and build graph structures;
[0129] The graph improvement module is used to solve the position conflict problem between pipeline segments and improve the graph structure;
[0130] The pipeline sampling module is used to sample pipeline segments based on the graph structure and use the MCMC algorithm to generate summary videos.
[0131] A video summarization system based on an object occlusion graph structure of this embodiment can execute a video summarization method based on an object occlusion graph structure provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0132] This embodiment also provides a video summarization device based on an object occlusion graph structure, including:
[0133] at least one processor;
[0134] at least one memory for storing at least one program;
[0135] When the at least one program is executed by the at least one processor, the at least one processor implements Fig.18 The method shown.
[0136] A video summarization device based on an object occlusion graph structure of this embodiment can execute a video summarization method based on an object occlusion graph structure provided by a method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0137] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Fig.18 The method shown.
[0138] This embodiment also provides a storage medium storing instructions or programs capable of executing a video summarization method based on an object occlusion graph structure provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0139] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0140] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0143] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0144] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0145] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0146] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0147] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A video summarization method based on object occlusion graph structure, characterized in that: The following steps are involved: Obtain surveillance video, extract pipeline information based on the surveillance video, and obtain the object pipeline; The object pipeline is divided into pipeline segments according to the occlusion relationship; Create edges between pipeline segments to build a graph structure; Solve the location conflict problem between pipeline segments and improve the graph structure; Based on the graph structure, the MCMC algorithm is used to sample pipeline segments and generate summary videos; When there are loops formed by multiple objects cyclically occluding each other in the graph structure, position conflicts will occur; The method of solving the position conflict problem between pipeline segments and improving the graph structure includes: Find the loop formed by multiple objects cyclically occluding in the graph structure, so that the pipeline segments in the same time period in the loop cut each other; additional bidirectional edges are established for some pipeline segments in the same time period that were originally in different pipelines and not connected, and occlusion connections are established for them even if they are not occluded in the original video, so that the pipeline segments in the loop can be completely synchronized during traversal updates; When a composite ring formed by multiple nested and interlaced rings appears in the graph structure, the problem of position conflict between pipeline segments is solved and the graph structure is improved, including: If no nodes in the loop are detected in the traversal graph, for the current node, determine whether other nodes are in the same loop as the current node; Traverse the nodes currently in the ring and obtain the maximum composite ring by recursively searching for nodes; For all pipeline segments within the largest composite ring, the pipeline segments with overlapping time periods are divided from each other; After the pipe segments in the ring are divided from each other, additional shielding connections are selectively established for the divided pipe segments in the ring, so that no position conflict occurs in the ring.
2. The video summarization method based on object occlusion graph structure according to claim 1, characterized in that: The extracting pipeline information according to the monitoring video to obtain the object pipeline includes: The KCF tracking algorithm is used to extract the pipeline information in the surveillance video and obtain the object pipeline O i , i∈[1,N], N is the number of foreground objects; the pipeline information includes the frame number and the position and size of the bounding box of the foreground object.
3. The video summarization method based on object occlusion graph structure according to claim 1, characterized in that: The process of establishing edges between pipeline segments and preliminarily constructing a graph structure includes: Use pipe segments as nodes; For the jth pipeline segment N on the same pipeline i i j , at node N i j and node N i j Add bidirectional edges between adjacent nodes; For pipeline segments with occlusion relationships between different pipelines, bidirectional edges are established between the nodes.
4. The video summarization method based on object occlusion graph structure according to claim 1, characterized in that: Based on the graph structure, the MCMC algorithm is used to sample the pipeline segments to generate summary videos, including: After perfecting the graph structure, the pipeline segments are sampled twice by MCMC to determine the optimal location of the pipeline segments; The first MCMC sampling makes the overall pipeline segment meet the requirement that the timing is basically consistent with the original video: first, all graphs are offset to the starting point of the video time, and the MCMC algorithm is used to randomly sample the reverse pipeline segments, iterating multiple times until the graph structure is consistent with the timing of the original video graph structure within the allowable error range; The second MCMC sampling minimizes the integrity loss and collision loss of the video summary: the pipeline segments are randomly sampled, the integrity loss and collision loss are calculated in multiple iterations, and finally the result with the smallest comprehensive loss and consistent timing is selected to generate the summary video.
5. The video summarization method based on object occlusion graph structure according to claim 1, characterized in that: The video summarization method further comprises the step of visualizing the summary video: For the shortened or elongated pipeline segments, a linear sampling method is used to select or supplement video frames, so that the object motion becomes faster or slower in the summary video; Opencv is used to visualize the processed pipeline data of the foreground object on a fixed background to generate a summary video.
6. A video summarization system based on object occlusion graph structure, characterized in that: include: The pipeline acquisition module is used to acquire the monitoring video, extract the pipeline information according to the monitoring video, and obtain the object pipeline; A pipeline segmentation module is used to segment the object pipeline into pipeline segments according to the occlusion relationship; A graph building module, used to establish edges between pipeline segments and build graph structures; The graph improvement module is used to solve the position conflict problem between pipeline segments and improve the graph structure; The pipeline sampling module is used to sample pipeline segments based on the graph structure and use the MCMC algorithm to generate summary videos; When there are loops formed by multiple objects cyclically occluding each other in the graph structure, position conflicts will occur; The method of solving the position conflict problem between pipeline segments and improving the graph structure includes: Find the loop formed by multiple objects cyclically occluding in the graph structure, so that the pipeline segments in the same time period in the loop cut each other; additional bidirectional edges are established for some pipeline segments in the same time period that were originally in different pipelines and not connected, and occlusion connections are established for them even if they are not occluded in the original video, so that the pipeline segments in the loop can be completely synchronized during traversal updates; When a composite ring formed by multiple nested and interlaced rings appears in the graph structure, the problem of position conflict between pipeline segments is solved and the graph structure is improved, including: If no nodes in the loop are detected in the traversal graph, for the current node, determine whether other nodes are in the same loop as the current node; Traverse the nodes currently in the ring and obtain the maximum composite ring by recursively searching for nodes; For all pipeline segments within the largest composite ring, the pipeline segments with overlapping time periods are divided from each other; After the pipe segments in the ring are divided from each other, additional shielding connections are selectively established for the divided pipe segments in the ring, so that no position conflict occurs in the ring.
7. A video summarization device based on an object occlusion graph structure, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 5.
8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 5 when executed by the processor.
