Rapid video concentration method based on 3D mapping table

Abstract

The invention belongs to the technical field of image processing and video monitoring, and relates to a 3D mapping table-based rapid video concentration method, which comprises the following steps of: firstly, storing a mapping relationship between a target and a concentrated video by constructing a 3D mapping table; when a new target is detected, the position and the scaling of the new target in the 3D mapping table are determined by querying the 3D mapping table according to the matching condition of the new target and the existing target tube, and then the position and the scaling of the new target in the concentrated video are determined; and extracting a corresponding original target from the monitoring video, and carrying out image fusion processing frame by frame by adopting an image fusion algorithm in combination with the extracted background picture of the monitoring video to generate a concentrated video. According to the method, the 3D mapping table is constructed, the mapping relation between the target and the target tube and the concentrated video is established, and for the detected new target, the position and the scaling of the new target in the concentrated video are quickly determined by querying the 3D mapping table, so that the execution efficiency of video concentration is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and video monitoring, and relates to a fast video concentration method based on a 3D mapping table. Background Art

[0002] With the rapid advancement of computer networks and video technology, a large number of surveillance cameras have been installed in cities to meet public safety needs. These cameras run 24 hours a day and generate massive amounts of video data every day. However, manually browsing this data is extremely time-consuming and laborious, and it is difficult to process such a large amount of information efficiently and accurately. Therefore, how to fully and efficiently utilize this video data has become a key problem. In recent years, video concentration technology has gradually become a research hotspot in the field of surveillance video as an effective solution.

[0003] Video condensation technology is a target-based video summarization method that can compress long videos into short videos, making it easy to quickly retrieve and browse surveillance data. This technology uses the target as the basic processing unit. First, it extracts the background image in the surveillance video through background modeling, then uses target detection and instance segmentation technology to identify the foreground target, and uses target tracking technology to match and associate the continuously appearing foreground targets to generate target trajectories (i.e., target tubes). Then, the target tubes are moved on the timeline, rearranged, and fused with the background image to finally generate a condensed video. This technology not only eliminates the temporal and spatial redundancy in the original video, but also retains the dynamic characteristics of the moving target.

[0004] Video condensation concentrates moving targets in time, which can help users quickly browse moving targets in the video and save video storage space. However, the application of video condensation technology also has the following problems: First, pseudo-collisions will occur between different targets in the condensed video, which will cause multiple targets to block each other, affecting the visual browsing effect of the condensed video; among them, pseudo-collision refers to moving the target tube on the time axis during the condensation and rearrangement process, so that two target tubes that did not collide in the source video collide after rearrangement. Second, the condensation and rearrangement process is very time-consuming, which is specifically reflected in: the rearrangement of the target tube is the most important link in video condensation. Generally, by setting the loss function, the rearrangement of the target tube is converted into a multi-objective optimization problem, and the optimization algorithm such as simulated annealing is used to perform a global search to find the optimal solution, but the optimization problem has the disadvantages of slow convergence speed and long time consumption. Therefore, how to minimize pseudo-collisions, improve the visual browsing effect, and improve the efficiency of the target tube rearrangement of video condensation while improving the compression rate of the condensed video is an important problem that needs to be solved in the field of video condensation technology. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a fast video condensation method based on a 3D mapping table to improve the execution efficiency of video condensation and reduce pseudo collisions between objects.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] This fast video condensation method based on 3D mapping table first stores the mapping relationship between the target and the condensed video by constructing a 3D mapping table MT; new When, according to the new target O new The matching status with the existing target tube is determined by querying the 3D mapping table MT to determine the new target O new The position and scale ratio in the 3D mapping table MT are used to determine the new target O. new The position and scaling ratio in the condensed video; then the corresponding original target is extracted from the surveillance video, and combined with the background picture of the extracted surveillance video, the image fusion algorithm is used to perform image fusion processing frame by frame to generate a condensed video.

[0008] Specifically, the fast video condensation method comprises the following steps:

[0009] Step 1: Input a surveillance video, analyze the image frame size of the surveillance video, construct a 3D mapping table MT for storing the mapping relationship between the target and the condensed video; initialize the 3D mapping table MT and the target tube list; wherein the target tube is the target trajectory;

[0010] Step 2: Perform target detection and instance segmentation on the surveillance video, and use target tracking technology to track the detected new target O new Match with existing target tubes; adopt different strategies according to the matching results (according to different situations) to determine the new target O new and new target tube T new Position in the condensed video, and the new target O new zoom ratio in condensed video;

[0011] Step 3: Extract the original target from the surveillance video and new The position and scaling information in the condensed video is used to perform target scaling and position rearrangement. Combined with the extracted surveillance video background image, an image fusion algorithm is used to perform image fusion processing frame by frame to generate a condensed video.

[0012] Specifically, the above step 1 includes the following steps:

[0013] Step 11: Analyze the image frame of the surveillance video, obtain the image width W and height H, and construct a 3D mapping table MT with a three-dimensional array structure. The three dimensions are: frame number, pixel horizontal coordinate, and pixel vertical coordinate;

[0014] The elements in the 3D mapping table MT are represented as MT(f,x,y), where f represents the frame number, (x,y) corresponds to the pixel position coordinates in the image frame, and MT∈R MAX×W×H , MAX represents the maximum frame length of the 3D mapping table MT, and R is a real number set; when the value of MT(f,x,y) is 0, it means that the (x,y) pixel point of the condensed video frame corresponding to the f frame is not occupied by any target; when the value of MT(f,x,y) is 1, it means that the (x,y) pixel point of the condensed video frame corresponding to the f frame has been occupied by at least one target;

[0015] Step 12, initialize the information of the 3D mapping table MT, including the search start frame SF, the loop counter CC and the table element data MT (f, x, y), set the search start frame SF initial value to 0, the loop counter CC initial value to 0, and all element data items of the 3D mapping table to 0;

[0016] Step 13, a continuous frame sequence of the same target detected in the monitoring video constitutes a target tube, multiple target tubes constitute a target tube list, target tube attribute description items are defined, and the target tube list is initialized to be empty; wherein, the target tube attribute description items include: ID number tube_id, starting frame number tube_fs in the condensed video, ending frame number tube_fe in the condensed video, end flag end_flag and target scaling ratio list list_scale_ratio, represented as a five-tuple (tube_id, tube_fs, tube_fe, end_flag, list_scale_ratio); wherein, when end flag is FALSE, it indicates that the target tube is not ended and new targets can continue to be added; when end flag is TRUE, it indicates that the target tube is complete and no new targets will be added.

[0017] Specifically, the above step 2 includes the following steps:

[0018] Step 21: Perform multi-target detection on the current image frame of the surveillance video to obtain multiple new targets O new , using target tracking technology, the new target O new Match with the existing target tube; wherein each of the new targets O new Stored in the form of an image mask, define the target pixel as 1 and the non-target pixel as 0;

[0019] Step 22: According to the new target O newThe matching results with the existing target tube can be divided into two cases: if the new target O new If the existing target tube cannot be matched, go to step 23; if the new target O new If an existing target tube is matched, the process goes to step 24;

[0020] Step 23: If the new target O new If the existing target tube cannot be matched, the 3D mapping table MT and the target tube list are queried, and the search start frame SF and table element data MT (f, x, y) in the 3D mapping table MT are updated as needed to obtain the new target O. new Generate a new target tube T new ; Then, based on the principle of combining high enrichment rate and low pseudo-collision rate, the new target O is determined new and new target tube T new The position in the 3D mapping table MT, and then determine the new target O new and new target tube T new The position in the condensed video is used to determine the new target O new The scaling ratio is 1; where a pseudo collision refers to a collision between two targets that does not exist in the original video but is generated in the condensed video due to condensation processing;

[0021] Step 24: If the new target O new Match the existing target tube and add the new target O new Place it at the end of the corresponding target tube; Based on the principle of combining low pseudo collision rate and high image fidelity, an adaptive scaling strategy is used to scale the new target O new Perform dynamic scaling to determine the new target O new Position and scale within the condensed video.

[0022] Furthermore, the above step 23 includes the following steps:

[0023] Step 231: determine whether the search start frame SF in the 3D mapping table MT needs to be updated. If it needs to be updated, recalculate a new search start frame SF, and reset the local element data values ​​of the 3D mapping table MT according to the new search start frame SF.

[0024] Step 232: new target O new Generate a new target tube T new , define a new target tube T new The first frame is the new target O new ; T for the new target new Assign an ID number to the new target new The ID number tube_id is set to the number of existing target tubes tubeCount, so that all target tube numbers in the new target tube list are continuous, and then tubeCount is incremented by 1, which can be expressed as follows using formula (1):

[0025]

[0026] Step 233: query the 3D mapping table MT, and determine the new target O according to the principle of combining high video concentration rate and low false collision rate. new The position of the new target O is determined in the 3D mapping table MT. new and new target tube T new The position in the condensed video is used to determine the new target O new The scaling factor is 1.

[0027] Furthermore, the above step 231 includes the following steps:

[0028] Step 2311, read the five-tuple information (tube_id, tube_fs, tube_fe, end_flag, list_scale_ratio) of the target tube in the target tube list, the number of target tubes tubeCount, the search start frame SF of the 3D mapping table MT, the loop counter CC and the table maximum frame length MAX, calculate the current minimum filling frame number min_frame by formula (2), calculate the frame number SF' in the condensed video corresponding to the search start frame SF by formula (3),

[0029]

[0030] SF'=(SF+CC*MAX) Formula (3)

[0031] In formula (2), min() is a minimum value function; min_fs takes the minimum value of the start frame tube_fs of the m target tubes with the largest ID number in the condensed video, min_fe takes the minimum value of the last frame tube_fe of the target tubes with end flag end_flag FALSE in the condensed video, and min_frame takes the smaller value of min_fs and min_fe as the current minimum filling frame number;

[0032] Step 2312: If min_frame-SF'>α, it is determined that the search start frame SF of the 3D mapping table MT needs to be updated, and the process goes to step 2313; otherwise, it does not need to be updated; wherein α is the minimum frame number threshold manually set;

[0033] Step 2313: If an update is required, first save the search start frame SF value as old_SF, then increase SF by (min_frame-SF'-α), that is, SF=SF+(min_frame-SF'-α); since the 3D mapping table MT is used cyclically, compare the search start frame SF value with the table maximum frame length MAX value, if SF<MAX, execute step 2314; otherwise, execute step 2315;

[0034] Step 2314: If SF<MAX, then only the corresponding element values ​​in the 3D mapping table MT need to be updated. Specifically, all the table element values ​​in the 3D mapping table MT from the old_SF frame to the SF frame, that is, the interval [old_SF, SF), are set to 0.

[0035] Step 2315: If SF>=MAX, the SF value needs to be corrected (such as remainder) first, that is, SF=SF%MAX, and the loop counter CC is incremented by 1, that is, CC=CC+1; update the local element values ​​in the 3D mapping table MT in two stages: the first stage frame number is in the interval [old_SF, MAX-1], and the second stage frame number is in the interval [0, SF), and all the table element values ​​in the two intervals are set to 0.

[0036] In addition, the above step 233 includes the following steps:

[0037] Step 2331, starting from the search start frame SF, read the 3D mapping table MT element data MT (f, x, y) (f>=SF), f is the frame number, according to the pixel point (x, y) with a non-zero mask value of the new target image, search for the corresponding 3D mapping table frame pixel point value MT (f, x, y), sum the values ​​of all pixels, and record the earliest frame number with a cumulative sum of 0 as f s , define a new target O new In the 3D mapping table MT, the frame number FN(0 new ), then FN(O new ) = f s , the calculation formula (4) is:

[0038]

[0039] Step 2332: Based on the new target O new The frame number in the 3D mapping table MT, the table cycle counter CC and the table maximum frame length MAX, calculate the new target O new The frame number in the condensed video is denoted as FN'(O new ), determine the new target O new At the location of the condensed video, the calculation formula (5) is:

[0040] FN′(O new)=FN(O new )+CC*MAX Formula (5)

[0041] Step 2333: Update the information of the 3D mapping table MT according to the new target image mask information. The updating method is:

[0042] According to the new target image mask information, find the corresponding frame FN(0) in the 3D mapping table MT new ) and set all pixel values ​​to 1. The specific update formula (6) is expressed as:

[0043]

[0044] At the same time, the new target O new The scaling factor is set to 1, and the new target O is updated. new The corresponding target tube quintuple information includes the starting frame number tube_fs in the condensed video, the ending frame number tube_fe in the condensed video, and the target scaling ratio list list_scale_ratio.

[0045] Furthermore, the above step 24 includes the following steps:

[0046] Step 241: Set the new target O new In the condensed video, the frame position is set to the frame number of the last frame of the matched target tube plus 1 to ensure the continuity of the target tube; correspondingly, the new target O new The frame position in the 3D mapping table MT is set to the frame number of the last frame of the matched target tube plus 1; at the same time, the quintuple information of the target tube is updated;

[0047] Step 242: Define new target O new The frame position in the 3D mapping table MT is recorded as FN(0 new ), read the 3D mapping table MT FN (0 new ) frame element data value, and the new target O new The image mask value is compared to determine whether there is a false collision; Col(0 new ) records whether there is a collision relationship. If Col(O new ) value is 1, it means there is a false collision. new ) value is 0, it means there is no false collision, and the judgment method is as follows formula (7):

[0048]

[0049] Step 243: If there is a false collision, the new target O is adjusted in an adaptive manner. new Size, for the new target O new Scaling is performed to reduce the new target O newThe collision area with the existing target in the current frame; the new target scaling ratio resizeRatio is related to the ratio of the collision area to the total area overlapRatio, and the calculation method of the collision area to the total area overlapRatio is as follows:

[0050]

[0051] In formula (8), #(·) is a function to count the number of non-zero points; the new target O new The calculation method of the scaling ratio resizeRatio is as follows:

[0052]

[0053] In formula (9), α1, α2 and β are adjustable parameters, and 0≤α1≤α2≤1, 0<β<1; when the overlapRatio (i.e., the area overlap ratio) of the collision area to the total area is less than the first threshold α1, resizeRatio is set to 1, that is, the new target O new No scaling is performed; when the overlapRatio of the collision area to the total area (i.e., the area overlap ratio) is greater than the second threshold α2, resizeRatio is set to a fixed value (1-α2*β); when the overlapRatio of the collision area to the total area (i.e., the area overlap ratio) is between the two, the value of resizeRatio is calculated by the formula (1-overlapRatio*β);

[0054] Step 244: After determining the value of resizeRatio, the new target O is resized according to the value. new Perform scaling processing to obtain the processed new target O n ' ew , O n ' ew = resize(O new , resizeRatio), the resize() function is an image scaling function, which resizes the new target O according to the resizeRatio ratio new Perform scaling processing; then according to the new target O after scaling processing n ' ew For the FNth (O) of the 3D mapping table MT new ) frame is updated, and the update formula (10) is as follows:

[0055]

[0056] Step 245: Update the target scaling ratio list list_scale_ratio of the current target tube with the resizeRatio value to record the scaling ratio of the current target.

[0057] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0058] 1) Construct a 3D mapping table MT to establish the mapping relationship between the target and the target tube and the condensed video. new When the new target O is determined by querying the 3D mapping table MT new The position in the 3D mapping table MT is used to quickly determine the new target O new In the location of condensed video, the execution efficiency of video condensation target tube rearrangement is improved.

[0059] 2) When the new target O new When matching to an existing target tube, in order to ensure the continuity of the target tube, the new target O new Placing it at the tail of the target tube may cause pseudo collision; based on the principle of combining low pseudo collision rate and high image fidelity, an adaptive scaling strategy is used to perform dynamic target scaling processing, which effectively reduces or even avoids pseudo collisions and improves the visual browsing effect of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are incorporated in and constitute a part of this specification and, together with the description, serve to explain the principles of the present invention.

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0062] Figure 1 A flowchart of a fast video condensation method based on a 3D mapping table provided by the present invention;

[0063] Figure 2 A schematic diagram of a 3D mapping table storage structure provided by the present invention;

[0064] Figure 3 A schematic diagram of a method for determining the position of a new target that is not matched to an existing target tube in the present invention;

[0065] Figure 4 It is a schematic diagram of a scaling processing method for matching a new target to an existing target tube in the present invention;

[0066] FIG. 5( a ) is a schematic diagram of a method for updating a 3D mapping table search for a start frame new_SF smaller than MAX in the present invention;

[0067] FIG. 5( b ) is a schematic diagram of a method for updating a 3D mapping table search for a start frame new_SF greater than MAX-1 in the present invention. DETAILED DESCRIPTION

[0068] Here, exemplary embodiments will be described in detail, and the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples consistent with some aspects of the present invention described in detail in the appended claims.

[0069] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0070] Example

[0071] See also Figure 2 As shown, the storage structure of the 3D mapping table MT is a three-dimensional array structure, t is the time axis, with image frames as units, and x and y represent the coordinate axes of the image frames; if the value of the 3D mapping table MT element is 0, it means that the element is not occupied by the target, and if the value of the 3D mapping table MT element is 1, it means that the element has been occupied by the target; Figure 2 The frame states of two cases, occupied by the target and not occupied by the target, are shown in FIG. One frame is occupied by the target, and the table element values ​​of the part occupied by the target are all 1. The other frame is not occupied by the target, and all elements of the frame are 0. The initial state of the 3D mapping table MT is that all element values ​​are 0.

[0072] See also Figure 3 As shown, this embodiment provides a method for determining the position of a new target that is not matched to an existing target tube. Specifically, if the new target O new If the target tube is not matched, query the 3D mapping table MT, start from the search start frame SF, read the element data of the 3D mapping table MT by frame (f represents the frame number, f>=SF), and find the corresponding 3D mapping table MT frame pixel value according to the pixel point (x, y) with non-0 mask value of the new target image. Sum the values ​​of all pixels. The frame number with the earliest cumulative sum of 0 is set as the new target O. new The position of MT in the 3D mapping table.

[0073] See also Figure 4 As shown, this embodiment provides a scaling processing method for a new target matched to an existing target tube. Specifically, if the new target O new Match to target tube, new target O newThe frame position is the frame number of the last frame of the matching target tube plus 1; when the new target O new After the position is determined, the new target O new There may be a false collision with other existing targets. To reduce or avoid false collisions, an adaptive scaling strategy is used to dynamically scale the target. The scaling factor is calculated according to the ratio of the collision area to the new target area, and the new target O is scaled according to the scaling factor. new The scaling process is performed and the scaled target is finally fused into the condensed video.

[0074] 5, this embodiment provides a method for updating a 3D mapping table MT search start frame SF. Specifically, when a new target O new When the existing target tube cannot be matched, a new target tube needs to be generated. At this time, it is necessary to determine whether the search start frame SF of the 3D mapping table MT needs to be updated. If it needs to be updated, the search start frame old_SF is saved first, and then the new search start frame new_SF is recalculated. The 3D mapping table MT adopts a circular use method. According to the relationship between new_SF and the maximum frame length MAX of the 3D mapping table MT, the table element data between old_SF and new_SF in the 3D mapping table MT is set to 0. Among them, Figure 5 (a) is a schematic diagram of the update processing method when the new search start frame new_SF is less than MAX, and Figure 5 (b) is a schematic diagram of the update processing method when the new search start frame new_SF is greater than MAX-1.

[0075] See also Figure 1 As shown, the present invention provides a fast video concentration method based on a 3D mapping table MT, which specifically includes the following steps:

[0076] Step 1: Input surveillance video, analyze the image frame size of surveillance video, construct 3D mapping table MT, which is used to record the time and space information of the target in the condensed video, so as to establish the mapping relationship between the target and the condensed video; initialize 3D mapping table MT information and target tube list; wherein the target tube is the target trajectory.

[0077] The specific process of step 1 above is as follows:

[0078] Step 11: Analyze the image frame of the surveillance video, obtain the image width W and height H, and construct a 3D mapping table MT with a three-dimensional array structure, such as Figure 2 As shown in the figure, the three dimensions are: frame number, pixel horizontal coordinate, pixel vertical coordinate; the elements in the 3D mapping table are represented as MT(f,x,y), where f represents the frame number, (x,y) corresponds to the pixel position coordinates in the image frame, and MT∈R MAX×W×H; Wherein, MAX represents the maximum frame length of the 3D mapping table, and R is a real number set; when the MT(f,x,y) value is 0, it means that the (x,y) pixel point of the condensed video frame corresponding to the f frame is not occupied by any target; when the MT(f,x,y) value is 1, it means that the (x,y) pixel point of the condensed video frame corresponding to the f frame has been occupied by at least one target;

[0079] Step 12, initialize the information of the 3D mapping table MT, including the search start frame SF, the loop counter CC and the table element data MT (f, x, y), set the search start frame SF initial value to 0, the loop counter CC initial value to 0, and all element data items of the 3D mapping table MT to 0;

[0080] Step 13, a continuous frame sequence of the same target detected in the monitoring video constitutes a target tube, multiple target tubes constitute a target tube list, target tube attribute description items are defined, and the target tube list is initialized to be empty; wherein, the target tube attribute description items include: ID number tube_id, starting frame number tube_fs in the condensed video, ending frame number tube_fe in the condensed video, end flag end_flag and target scaling ratio list list_scale_ratio, represented as a five-tuple (tube_id, tube_fs, tube_fe, end_flag, list_scale_ratio); wherein, when end flag is FALSE, it indicates that the target tube is not ended and new targets can continue to be added; when end flag is TRUE, it indicates that the target tube is complete and no new targets will be added.

[0081] Step 2: Perform target detection and instance segmentation on the surveillance video, and use target tracking technology to locate the new target O new Match with the existing target tube; according to the matching results, different strategies are adopted according to different situations to determine the new target O by querying the 3D mapping table MT new The position and scale ratio in the 3D mapping table MT are used to determine the new target O. new The position and scaling ratio in the condensed video are updated simultaneously with the 3D mapping table MT information and target tube information.

[0082] The specific process of step 2 above is as follows:

[0083] Step 21: Use Yolo to perform multi-target detection on the current image frame of the surveillance video to obtain multiple new targets O new , using target tracking technology such as DeepSort method, the new target O new Match with the existing target tube; among them, the new target O new Stored in the form of an image mask, that is, the target pixel is 1 and the non-target pixel is 0;

[0084] Step 22: According to the new target O new The matching results with the existing target tube can be divided into two cases. If the new target O new If the existing target tube cannot be matched, go to step 23; if the new target O new If an existing target tube is matched, the process goes to step 24;

[0085] Step 23: For the new target O new If the existing target tube cannot be matched, query the 3D mapping table MT and the target tube list, update the 3D mapping table MT as needed to search the start frame SF and the table element data MT (f, x, y), and find the new target O. new Generate a new target tube T new ; Then, based on the principle of combining high enrichment rate and low pseudo-collision rate, the new target O is determined new and new target tube T new The position in the 3D mapping table MT, and then determine the new target O new and new target tube T new The position in the condensed video is used to determine the new target O new The scaling ratio is 1; where a pseudo collision refers to a collision between two targets that does not exist in the original video but is generated in the condensed video due to condensation processing;

[0086] Step 24: For the new target O new If the target tube is matched, the new target O new Put it at the end of the corresponding target tube; according to the principle of combining low false collision rate and high image fidelity, an adaptive scaling strategy is adopted to calculate the new target O new Perform dynamic target scaling processing to determine the new target O new Position and scale in the condensed video, such as Figure 4 shown.

[0087] The specific process of the above step 23 is as follows:

[0088] Step 231: determine whether the search start frame SF in the 3D mapping table MT needs to be updated. If it needs to be updated, recalculate a new search start frame SF, and reset the local element data values ​​of the 3D mapping table MT according to the new search start frame SF.

[0089] Step 232: new target O new Generate a new target tube T new , new target O new That is, the first frame of the new target tube, an ID number is assigned to the new target tube, and the new target tube T newThe ID number tube_id is set to the number of existing target tubes tubeCount, so that all target tube numbers in the new target tube list are continuous, and then tubeCount is incremented by 1, which can be expressed as follows using formula (1):

[0090]

[0091] Step 233: query the 3D mapping table MT, and determine the new target O based on the principle of combining high video concentration rate and low false collision rate. new The position of the new target O is determined in the 3D mapping table MT. new and new target tube T new Position in the condensed video, set a new goal O new The scaling factor is 1, such as Figure 3 shown.

[0092] The specific process of the above step 231 is as follows:

[0093] Step 2311, read the five-tuple information (tube_id, tube_fs, tube_fe, end_flag, list_scale_ratio) of the target tube in the target tube list, as well as the number of target tubes in the list tubeCount, the search start frame SF of the 3D mapping table MT, the loop counter CC and the table maximum frame length MAX, calculate the current minimum filling frame number min_frame by formula (2), calculate the frame number SF' in the condensed video corresponding to the search start frame SF by formula (3),

[0094]

[0095] SF'=(SF+CC*MAX) Formula (3)

[0096] In formula (2), min() is a minimum value function; min_fs takes the minimum value of the start frame tube_fs of the m target tubes with the largest ID number in the condensed video, min_fe takes the minimum value of the last frame tube_fe of the target tubes with end flag end_flag FALSE in the condensed video, and min_frame takes the smaller value of min_fs and min_fe as the current minimum filling frame number;

[0097] Step 2312: if min_frame-SF'>α, it is determined that the search start frame SF of the 3D mapping table MT needs to be updated, and the process goes to step 2313; otherwise, it does not need to be updated; wherein α is a manually set minimum frame number threshold, which is 100 in this embodiment;

[0098] Step 2313: If it needs to be updated, first save the search start frame SF value as old_SF, then increase SF by (min_frame-SF'-α), that is, SF=SF+(min_frame-SF'-α); since the 3D mapping table MT is used cyclically, compare the search start frame SF value with the maximum frame length MAX value of the table, if SF<MAX, execute step 2314; otherwise, execute step 2315;

[0099] Step 2314, as shown in FIG. 5(a), if SF<MAX, then only the corresponding element values ​​in the 3D mapping table MT need to be updated. The specific method is: all the table element values ​​in the 3D mapping table MT from the old_SF frame to the SF frame, that is, the interval [old_SF, SF), are set to 0;

[0100] Step 2315, as shown in Figure 5(b), if SF>=MAX (equivalent to SF>MAX-1), it is necessary to first perform modulus processing on the SF value, that is, SF=SF%MAX, and the loop counter CC is incremented by 1, that is, CC=CC+1; update the local element values ​​in the 3D mapping table in two stages, the first stage frame number is in the interval [old_SF, MAX-1], the second stage frame number is in the interval [0, SF), and all the table element values ​​in the two intervals are set to 0.

[0101] The specific process of the above step 233 is as follows:

[0102] Step 2331, starting from the search start frame SF, read the 3D mapping table MT element data MT (f, x, y) (f> = SF), f is the frame number, according to the new target O new The pixel (x, y) with non-zero mask value in the image is used to find the corresponding 3D mapping table frame pixel value MT (f, x, y), and the values ​​of all pixels are summed up. The frame number with the earliest cumulative sum of 0 is recorded as f. s , define a new target O new In the 3D mapping table MT, the frame number FN(0 new ), then FN(O new ) = f s , the calculation formula (4) is:

[0103]

[0104] Step 2332: Based on the new target O new The frame number in the 3D mapping table MT, the table cycle counter CC and the table maximum frame length MAX, calculate the new target O new The frame number in the condensed video is denoted as FN'(O new ), determine the new target O new At the location of the condensed video, the calculation formula (5) is:

[0105] FN′(O new )=FN(O new )+CC*MAX Formula (5)

[0106] Step 2333: According to the new goal O new The image mask information updates the information of the 3D mapping table MT, and the updating method is:

[0107] According to the new goal new Image mask information, find the corresponding frame FN (0 in the 3D mapping table MT new ) and set all pixel values ​​to 1. The specific update formula (6) is expressed as:

[0108]

[0109] At the same time, the new target O new The scaling factor is set to 1, and the new target O is updated. new Corresponding target pipe five-tuple information.

[0110] The specific process of the above step 24 is as follows:

[0111] Step 241: Set the new target O new In the condensed video, the frame position is set to the frame number of the last frame of the matched target tube plus 1 to ensure the continuity of the target tube; correspondingly, the new target O new The frame position in the 3D mapping table MT is set to the frame number of the last frame of the matched target tube plus 1; at the same time, the quintuple information of the target tube is updated;

[0112] Step 242: New Target O new The frame position in the 3D mapping table MT is recorded as FN(0 new ), read the 3D mapping table MT FN (0 new ) frame element data value, and the new target O new The image mask value is compared to determine whether there is a false collision; Col(0 new ) records whether there is a collision relationship. If Col(O new ) value is 1, it means there is a false collision. new ) value is 0, it means there is no false collision, and the judgment method is as follows formula (7):

[0113]

[0114] Step 243: In the case of a false collision, the target size is adjusted in an adaptive manner. new Perform scaling processing to reduce the new target O newThe collision area with the existing target in the current frame; the new target scaling ratio resizeRatio is related to the ratio of the collision area to the total area overlapRatio, and the calculation method of the collision area to the total area overlapRatio is as follows:

[0115]

[0116] In formula (8), #(·) is a function to count the number of non-zero points; the new target O new The calculation method of the scaling ratio resize_ratio is as follows:

[0117]

[0118] In formula (9), α1, α2 and β are adjustable parameters, and 0≤α1≤α2≤1, 0<β<1; here, α1=0.05, α2=0.5, β=0.8; when the overlapRatio (i.e., the area overlap ratio) of the collision area to the total area is less than 0.05, resizeRatio is set to 1, that is, for the new target O new No scaling is performed; when the overlapRatio of the collision area to the total area (i.e., the area overlap ratio) is greater than 0.5, resizeRatio is set to a fixed value of 1-α2×β=0.6; when the overlapRatio of the collision area to the total area (i.e., the area overlap ratio) is between the two, the value of resizeRatio is calculated by the formula (1-overlapRatio*β);

[0119] Step 244: After determining the value of resizeRatio, the new target O is resized according to the value. new Perform scaling processing to obtain the processed new target O n ' ew , O n ' ew = resize(O new , resizeRatio), the resize() function is an image scaling function, which resizes the image in proportion to the resizeRatio. new Perform scaling processing; then according to O n ' ew For the 3D mapping table, new ) frame is updated, and the update formula (10) is as follows:

[0120]

[0121] Step 245: Update the target scaling ratio list list_scale_ratio of the current target tube with the resizeRatio value to record the scaling ratio of the current target.

[0122] Step 3: Extract the original target from the surveillance video and new The position and scaling information in the condensed video is used to perform target scaling and position rearrangement. Combined with the extracted surveillance video background image, the Poisson fusion algorithm is used to perform image fusion processing frame by frame to generate a condensed video, which effectively improves the efficiency of video condensation and reduces pseudo-collisions between targets.

[0123] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention.

[0124] It should be understood that the present invention is not limited to what has been described above and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A fast video condensation method based on a 3D mapping table, characterized in that: First, a 3D mapping table MT is constructed to store the mapping relationship between the target and the condensed video; when a new target O is detected new When, according to the new target O new The matching status with the existing target tube is determined by querying the 3D mapping table MT to determine the new target O new The position and scale ratio in the 3D mapping table MT are used to determine the new target O. new Position and scale in the condensed video; Then the corresponding original target is extracted from the surveillance video, and combined with the background picture of the extracted surveillance video, the image fusion algorithm is used to perform image fusion processing frame by frame to generate a condensed video.

2. The fast video condensation method based on 3D mapping table according to claim 1, characterized in that: The steps include: Step 1: construct a 3D mapping table MT according to the image frames of the surveillance video to store the mapping relationship between the target and the condensed video, and initialize the 3D mapping table MT and the target tube list; Step 2: By performing target detection and instance segmentation on the surveillance video, the detected new target O is segmented based on the target tracking technology. new Match with existing target tubes and use different strategies according to the matching results to determine the new target O new and new target tube T new Position in the condensed video, and the new target O new zoom ratio in condensed video; Step 3: Extract the original target from the surveillance video and new The position and scaling information in the condensed video is used to perform target scaling and position rearrangement. Combined with the extracted surveillance video background image, an image fusion algorithm is used to perform image fusion processing frame by frame to generate a condensed video.

3. The fast video condensation method based on 3D mapping table according to claim 2, characterized in that: The step 1 specifically includes: Step 11: Analyze the image frame of the surveillance video to obtain the image width W and height H, and construct a 3D mapping table MT with a three-dimensional array structure. The elements in the 3D mapping table MT are represented as MT(f,x,y); where f represents the frame number, (x,y) corresponds to the pixel position coordinates in the image frame, and MT∈R MAX×W×H ; MAX represents the maximum frame length of the 3D mapping table MT, and R is a real number set; when the value of MT(f,x,y) is 0, it means that the (x,y) pixel point of the condensed video frame corresponding to the f frame is not occupied by any target; when the value of MT(f,x,y) is 1, it means that the (x,y) pixel point of the condensed video frame corresponding to the f frame has been occupied by at least one target; Step 12, initializing information of the 3D mapping table MT, the information of the 3D mapping table MT including the search start frame SF, the loop counter CC and the table element data MT (f, x, y); setting the initial value of the search start frame SF to 0, the initial value of the loop counter CC to 0, and all element data items of the 3D mapping table MT to 0; Step 13: A continuous frame sequence of the same target detected in the monitoring video constitutes a target tube, multiple target tubes constitute a target tube list, target tube attribute description items are defined, and the target tube list is initialized to be empty.

4. The fast video condensation method based on 3D mapping table according to claim 3, characterized in that: The target tube attribute description items include: ID number tube_id, starting frame number tube_fs in the condensed video, ending frame number tube_fe in the condensed video, end flag end_flag and target scaling ratio list list_scale_ratio, expressed as a quintuple (tube_id, tube_fs, tube_fe, end_flag, list_scale_ratio).

5. The fast video condensation method based on 3D mapping table according to claim 2, characterized in that: The step 2 specifically includes: Step 21: Perform multi-target detection on the current image frame of the surveillance video to obtain multiple new targets O new , using target tracking technology, the new target O new Match with the existing target tube; wherein each of the new targets O new Stored in the form of an image mask, define the target pixel as 1 and the non-target pixel as 0; Step 22: According to the new target O new The matching result with the existing target management adopts the corresponding execution strategy: if the new target O new If the target tube cannot be matched, the process goes to step 23, otherwise, the process goes to step 24; Step 23: If the new target O new If the existing target tube cannot be matched, the 3D mapping table MT and the target tube list are queried, and the search start frame SF and table element data MT (f, x, y) in the 3D mapping table MT are updated as needed to obtain the new target O. new Generate a new target tube T new ; Then, based on the principle of combining high enrichment rate and low pseudo-collision rate, the new target O is determined new and new target tube T new The position in the 3D mapping table MT, and then determine the new target O new and new target tube T new The position in the condensed video is used to determine the new target O new The scaling factor is 1; Step 24: If the new target O new If the existing target tube is matched, the new target O new Place it at the end of the corresponding target tube; Based on the principle of combining low pseudo collision rate and high image fidelity, an adaptive scaling strategy is used to scale the new target O new Perform dynamic scaling to determine the new target O new Position and scale within the condensed video.

6. The fast video condensation method based on 3D mapping table according to claim 5, characterized in that: The step 23 specifically includes: Step 231: determine whether the search start frame SF in the 3D mapping table MT needs to be updated. If it needs to be updated, recalculate a new search start frame SF, and reset the local element data values ​​of the 3D mapping table MT according to the new search start frame SF. Step 232: new target O new Generate a new target tube T new , define a new target tube T new The first frame is the new target O new ; T for the new target new Assign an ID number to the new target new The ID number tube_id is set to the number of existing target tubes tubeCount, so that all target tube numbers in the new target tube list are continuous, and then tubeCount is incremented by 1, which can be expressed as follows using formula (1): Step 233: query the 3D mapping table MT, and determine the new target O according to the principle of combining high video concentration rate and low false collision rate. new The position of the new target O is determined in the 3D mapping table MT. new and new target tube T new The position in the condensed video is used to determine the new target O new The scaling factor is 1.

7. The fast video condensation method based on 3D mapping table according to claim 6, characterized in that: The step 231 specifically includes: Step 2311, read the five-tuple information of the target tube in the target tube list, the number of target tubes in the target tube list tubeCount, the search start frame SF of the 3D mapping table MT, the loop counter CC and the table maximum frame length MAX, calculate the current minimum filling frame number min_frame by formula (2), calculate the frame number SF' in the condensed video corresponding to the search start frame SF by formula (3), SF'=(SF+CC*MAX) Formula (3) In formula (2), min() is a minimum function; min_fs is the minimum value of the starting frame tube_fs of the m target tubes with the largest ID number in the condensed video, min_fe is the minimum value of the last frame tube_fe of the target tubes with end flag end_flag FALSE in the condensed video, and min_frame is the smaller value of min_fs and min_fe; Step 2312: If min_frame-SF'>α, it is determined that the search start frame SF of the 3D mapping table MT needs to be updated, and the process goes to step 2313; otherwise, it does not need to be updated; wherein α is the minimum frame number threshold manually set; Step 2313: If an update is required, first save the search start frame SF value as old_SF, and then increase SF by (min_frame-SF'-α); since the 3D mapping table MT is used cyclically, compare the search start frame SF value with the table maximum frame length MAX value, if SF < MAX, execute step 2314; otherwise, execute step 2315; Step 2314: if SF<MAX, then the corresponding element values ​​in the 3D mapping table MT need to be updated. The specific method is: all table element values ​​in the 3D mapping table MT from the old_SF frame to the SF frame, that is, the interval [old_SF, SF), are set to 0; Step 2315: If SF>=MAX, the SF value needs to be corrected first, and the loop counter CC is incremented by 1; the local element values ​​in the 3D mapping table are updated in two stages: the first stage frame number is in the interval [old_SF, MAX-1], and the second stage frame number is in the interval [0, SF), and all the table element values ​​in the two intervals are set to 0.

8. The fast video condensation method based on 3D mapping table according to claim 6, characterized in that: The step 233 specifically includes: Step 2331, starting from the search start frame SF, read the 3D mapping table MT element data MT (f, x, y) (f>=SF), f is the frame number, according to the pixel point (x, y) with a non-zero mask value of the new target image, search for the corresponding 3D mapping table frame pixel point value MT (f, x, y), sum the values ​​of all pixels, and record the earliest frame number with a cumulative sum of 0 as f s , define a new target O new In the 3D mapping table MT, the frame number FN(0 new ), then FN(O new ) = f s , the calculation formula (4) is: Step 2332: Based on the new target O new The frame number in the 3D mapping table MT, the table cycle counter CC and the table maximum frame length MAX, calculate the new target O new The frame number in the condensed video is denoted as FN'(O new ), and its calculation formula (5) is: FN′(O new )=FN(O new )+CC*MAX Formula (5) Step 2333: Update the information of the 3D mapping table MT according to the new target image mask information. The updating method is: according to the new target image mask information, find the corresponding frame FN(0) in the 3D mapping table MT. new ) corresponding to the pixel point, and set all pixel point values ​​to 1, which is expressed by the following formula (6); at the same time, update the target tube quintuple information corresponding to the new target; 9. The fast video condensation method based on 3D mapping table according to claim 5, characterized in that: The step 24 specifically includes: Step 241: Set the new target O new In the condensed video, the frame position is set to the frame number of the last frame of the matched target tube plus 1 to ensure the continuity of the target tube; correspondingly, the new target O new The frame position in the 3D mapping table MT is set to the frame number of the last frame of the matched target tube plus 1; at the same time, the quintuple information of the target tube is updated; Step 242: Define new target O new The frame position in the 3D mapping table MT is recorded as FN(0 new ), read the 3D mapping table MT FN (0 new ) frame element data value, and the new target O new The image mask value is compared to determine whether there is a false collision; Col(O new ) records whether there is a collision relationship: If Col(O new ) value is 1, it means there is a false collision. new ) value is 0, it means there is no false collision, and the judgment method is as follows formula (7): Step 243: If there is a false collision, the new target O is adjusted using an adaptive scaling strategy. new Size, for the new target O new Perform scaling processing to reduce the new target O new The collision area with the existing target in the current frame, the new target O new The scaling ratio resizeRatio is related to the ratio of the collision area to the total area overlapRatio; wherein the calculation method of the ratio of the collision area to the total area overlapRatio is as follows: In formula (8), #(·) is a function to count the number of non-zero points, and the new target O new The calculation method of the scaling ratio resizeRatio is as follows: In formula (9), α1, α2 and β are adjustable parameters, and 0≤α1≤α2≤1, 0<β<1; when the overlapRatio of the collision area to the total area is less than the first threshold α1, the new target O new The scaling ratio resizeRatio is set to 1, that is, for the new target O new No scaling is performed; when the overlapRatio of the collision area to the total area is greater than the second threshold α2, resizeRatio is set to a fixed value (1-α2*β); when the overlapRatio of the collision area to the total area is between the two, the value of resizeRatio is calculated by the formula (1-overlapRatio*β); Step 244: After determining the value of resizeRatio, the new target O is resized according to the value. new Perform scaling processing to obtain the scaled new target O n ' ew , O n ' ew = resize(O new , resizeRatio), the resize() function is an image scaling function, and then according to the scaled new target O n ' ew For the 3D mapping table MT, new ) frame is updated, and the update formula (10) is as follows: Step 245: Update the target scaling ratio list list_scale_ratio of the current target tube with the resizeRati o value to record the scaling ratio of the current new target.

Images