A method for sparse representation of video motion based on line flow

Abstract

The present invention discloses a method for sparse characterization of video motion based on line flow, including: obtaining a sketch map of each frame of image; obtaining an initial sketch motion field of the t-th frame of image; obtaining an updated initial sketch motion field of the t-th frame of image according to whether the sketch line segment is a repeated sketch line segment; removing the noisy sketch line segments, and taking the remaining sketch line segments as motion sketch line segments to obtain the sketch motion field of the t-th frame of image; dividing adjacent motion sketch line segments in the sketch motion field of the t-th frame of image into the same region to obtain a set of division results; for each division region in the set of division results, calculating the set of motion sketch line segments of each frame of image; obtaining a set of trajectories based on the trajectories of each division region obtained from the set of motion sketch line segments; calculating the motion vector of each motion sketch line segment in the set of motion sketch line segments of the t-th frame of image to obtain a set of motion vectors. The method of the present invention has less noise and better extraction effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for sparse representation of video motion based on line flow. Background Art

[0002] British neurologist and psychologist Marr pointed out that human vision is essentially an information processing process and proposed a theoretical framework for visual computing. This theoretical framework for visual computing has played a huge role in the subsequent research and development of computer vision. In this theoretical framework for visual computing, Marr divided the process of obtaining object information from images into the following three stages: 1) initial sketch, 2) 2.5D sketch, and 3) 3D model. Essentially, the initial sketch is a symbolic representation of the combination of image gray-scale changes, geometric feature distributions, and structural information. Based on Marr's proposed visual computing theory, many scholars have studied methods for extracting the initial sketch in images. Inspired by Marr's theoretical framework for visual computing, Guo et al. proposed an initial sketch model that combines a sparse coding model for representing geometric structures and a Markov random field for representing texture structures. Guo et al. pointed out that models based on sparse coding theory have better representation capabilities for geometric structure regions with lower information entropy, while models based on Markov random field theory have better representation capabilities for texture structure regions with higher information entropy. Therefore, by dividing an image into sketchable and non-sketchable parts and using different models for each part, better expression of image content can be achieved. The successful application of the visual computing theory shows that it can not only represent the structural information of images, but also provides new means and platforms for visual analysis and understanding.

[0003] As a pre-task of video analysis technology, optical flow can improve the performance of video-related tasks, such as object tracking, object segmentation, object detection, saliency detection, action recognition, anomaly detection, quality enhancement, scene classification, and crowd event detection, by calculating the motion information of objects between adjacent frames. However, optical flow captures the motion information of videos in units of pixels and does not consider the consistency within the motion region, resulting in complex calculations and limited practical applications. Usually, the frame sequence of a video contains rich motion information. However, the motion of an object is related, rather than the independent motion of pixels or small regions. Current motion information extraction technologies do not reflect this correlation. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a method for sparse representation of video motion based on line flow.

[0005] The technical problems to be solved by the present invention are realized through the following technical solutions:

[0006] A method for sparse representation of video motion based on line flow, the representation method comprising:

[0007] S1. Sketch the image frame sequence in video V to obtain a sketch map of each frame of image;

[0008] S2. For each moment t, by superimposing the sketch map of the t-th frame onto the sketch map of the (t + K)-th frame, obtain the initial sketch motion field of the t-th frame of image;

[0009] S3. Determine whether a sketch line segment is a repeated sketch line segment according to the matching degree between any two sketch line segments in the initial sketch motion field of the t-th frame of image. If so, delete the latter sketch line segment from the initial sketch motion field of the t-th frame of image to obtain the updated initial sketch motion field of the t-th frame of image;

[0010] S4. Remove the noisy sketch line segments in the updated initial sketch motion field of the t-th frame of image, and use the remaining sketch line segments as motion sketch line segments to obtain the sketch motion field of the t-th frame of image;

[0011] S5. Based on the clustering method, divide the adjacent motion sketch line segments in the sketch motion field of the t-th frame of image into the same region to obtain a set of division results;

[0012] S6. For each division region in the set of division results, calculate the set of motion sketch line segments of each frame of image;

[0013] S7. Obtain a set of trajectories based on the trajectories of each division region obtained from the set of motion sketch line segments;

[0014] S8. Based on the set of trajectories, calculate the motion vectors of each motion sketch line segment in the set of motion sketch line segments of the t-th frame of image to obtain a set of motion vectors.

[0015] Optionally, the set of sketch line segments of the sketch map of the t-th frame of image is represented as N is the number of frames of video V, is the i-th sketch line segment in the set of sketch line segments of the sketch map of the t-th frame of image, and are the x coordinate of the midpoint, the y coordinate of the midpoint, the angle and the length of the i-th sketch line segment respectively, is the set of sketch line segments S of the sketch map of the t-th frame of image t in the number of sketch line segments.

[0016] Optionally, step S2 includes:

[0017] S2.1. Obtain the set of sketch line segments of K + 1 sketch maps from the t-th frame to the (t + K)-th frame in the set of sketch line segments of the sketch maps of all frame images, where K is the number of sketch maps superimposed on the sketch map of the t-th frame image;

[0018] S2.2. Add the set of sketch line segments of the K + 1 sketch maps obtained in step S2.1 to the initial sketch motion field corresponding to the t-th frame image, and the initial sketch motion field is expressed as:

[0019]

[0020] where, is the initial sketch motion field of the t-th frame image, S t+k the set of sketch line segments of the sketch map of the (t + K)-th frame image.

[0021] Optionally, step S3 includes:

[0022] S3.1. Traverse any two sketch line segments and

[0023] in the initial sketch motion field of the t-th frame image and the sketch line segment S3.2. Use the matching function to calculate the matching degree

[0024] between the sketch line segment and the matching degree threshold τ. If the matching degree is less than the matching degree threshold τ, it means that the sketch line segment and the sketch line segment are not duplicate sketch line segments. If the matching degree is greater than or equal to the matching degree threshold τ, it means that the sketch line segment and the sketch line segment are duplicate sketch line segments, and then delete the sketch line segment from the initial sketch motion field of the t-th frame image to obtain the updated initial sketch motion field of the t-th frame image.

[0025] Optionally, the matching function is expressed as:

[0026]

[0027] where, is the matching degree between the sketch line segment and the sketch line segment λ1, λ2, λ3 and λ4 are trade-off parameters, and λ1 = λ2. The sketch line segment the sketch line segment and The x coordinate of the midpoint, the y coordinate of the midpoint, the angle, and the length of the i-th sketch line segment, respectively, and The x coordinate of the midpoint, the y coordinate of the midpoint, the angle, and the length of the j-th sketch line segment, respectively, g x (·, ·), g y (·, ·), g θ (·, ·) and g l (·, ·) are the components of the metric function on the x coordinate of the corresponding midpoint, the y coordinate of the midpoint, the angle θ, and the length l, respectively.

[0028] Optionally, step S4 includes:

[0029] S4.1. Set a set of parameters where Z is the total number of steps, (r z , min z ) indicates that the minimum number of sketch line segments within the radius r z is min z ;

[0030] S4.2. For the initial sketch motion field of the updated t-th frame image, the number of midpoints of other sketch line segments within the circle centered at the midpoint of each sketch line segment with a radius of r z , when the number of midpoints of other sketch line segments within the circle is less than min z , the sketch line segment is a noise sketch line segment, and the noise sketch line segment is removed from the initial sketch motion field of the updated t-th frame image, and the remaining sketch line segments are used as motion sketch line segments to obtain the final sketch motion field of the t-th frame image composed of motion sketch line segments.

[0031] Optionally, step S5 includes:

[0032] S5.1. Obtain the number of clusters according to the number |Ψ t | of motion sketch line segments in the sketch motion field of the t-th frame image, where the number of clusters is c num is the maximum number of motion sketch line segments in each cluster;

[0033] S5.2. Use the K-means algorithm to divide the sketch motion field of the t-th frame image into clusters;

[0034] S5.3. Based on clusters, obtain the final set of partitioning results where C t is the set of all partitions of the sketch motion field of the t-th frame image, is the c-th partition region of the sketch motion field of the t-th frame image.

[0035] Optionally, step S6 includes:

[0036] S6.1. Obtain the set of sketch line segments of K + 1 sketch maps from the t-th frame to the (t + K)-th frame from the set of sketch line segments of the sketch maps of all frame images;

[0037] S6.2. Sequentially determine whether the sketch line segments in the set of K + 1 sketch line segments from the t-th frame to the (t + K)-th frame are in the c-th divided region. If so, store the sketch line segment in the moving sketch line set to obtain the set of moving sketch line segments belonging to each frame image in the c-th divided region.

[0038] Optionally, step S7 includes:

[0039] S7.1. Calculate the means of the four components of the x coordinate of the midpoint, the y coordinate of the midpoint, the angle θ, and the length l of all the sketch line segments in the set of moving sketch line segments of each frame image to obtain the corresponding trajectory.

[0040] S7.2. Repeatedly execute step S7.1 until the trajectories corresponding to all frame images are obtained, and use the trajectories corresponding to all frame images as the final trajectory set.

[0041] Optionally, step S8 includes:

[0042] S8.1. For each set of trajectories in the trajectory set, calculate the mean of the first half and the mean of the second half of the trajectory in the four components of the x coordinate of the midpoint, the y coordinate of the midpoint, the angle θ, and the length l;

[0043] S8.2. Obtain the motion vector of the corresponding moving sketch line segment according to the mean of the first half and the mean of the second half of the trajectory, and use the motion vector as the motion vector of the sketch line segment belonging to the t-th frame image in the corresponding divided region;

[0044] S8.3. Combine the motion vectors of the sketch line segments belonging to the t-th frame image in all divided regions to obtain the set of motion vectors of all the moving sketch line segments in the t-th frame image.

[0045] Compared with the prior art, the beneficial effects of the present invention:

[0046] The method for sparse representation of video motion information based on line flow provided by the present invention removes the repeated sketch line segments generated by the background and the noise sketch line segments generated due to reasons such as camera noise, illumination change, and slight jitter, so as to represent the running information in the video in a sparse form. Therefore, the video motion information representation method proposed by the present invention has a sparser representation. Compared with the video motion information representation method based on optical flow, this algorithm uses the motion vectors of the motion sketch line segments to represent the motion information in the video, providing a new method for representing video motion information in a sparser form, thereby reducing the storage space.

[0047] The following will further elaborate on the present invention in conjunction with the accompanying drawings. Description of the Drawings

[0048] Figure 1 is a schematic flowchart of a method for sparse representation of video motion based on line flow provided by an embodiment of the present invention;

[0049] Figures 2a - 2f is a comparison diagram before and after sketching of video frame images in the present invention, where, Figure 2a 、 Figure 2c and Figure 2e are the original images of three video frame images, Figure 2b 、 Figure 2d and Figure 2f are the sketch images after sketching the original images of three video frame images;

[0050] Figure 3 is a schematic diagram of the initial sketch motion field in the present invention;

[0051] Figure 4 is a schematic diagram of the result after removing repeated sketch line segments in the present invention;

[0052] Figure 5 is a schematic diagram of the sketch motion field after removing noise sketch line segments in the present invention;

[0053] Figure 6 is a schematic diagram of the divided sketch motion field in the present invention;

[0054] Figure 7a 、 Figure 7b and Figure 7c are schematic diagrams of the division results belonging to three video frames in the present invention;

[0055] Figure 8 is a schematic diagram of the visualization method of line flow in the present invention;

[0056] Figures 9a - 9b is a comparison diagram of the present invention and the optical flow method, where, Figure 9a is the effect diagram of the line flow extracted in the present invention, Figure 9bIt is the effect diagram of the extracted optical flow. Detailed implementation manners

[0057] The following further describes the present invention in detail with reference to specific embodiments, but the implementation manners of the present invention are not limited thereto.

[0058] Embodiment 1

[0059] In the 1980s, Marr summarized the research results on human vision in aspects such as psychophysics, neurophysiology, and anatomy, pointed out that human vision is essentially a process of information processing, and proposed a framework prototype of the visual computing theory. Later, Guo, Zhu Songchun and others, based on the sketch theory in Marr's visual computing theory, proposed an initial sketch model and method applicable to natural images, and realized image compression and reconstruction using the sketch information of natural images.

[0060] Drawing on the initial sketch model proposed by Zhu Songchun and others, the present invention provides a method for sparse representation of video motion based on line flow for the motion information in a video on the basis of the adjacent time consistency and local spatial consistency of the video. Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for sparse representation of video motion based on line flow provided by an embodiment of the present invention. The present invention provides a method for sparse representation of video motion based on line flow, including:

[0061] S1. Sketch the image frame sequence in video V to obtain a sketch map of each frame of the image in video V.

[0062] Here, the sketch map of each frame of the image corresponds to a set of sketch line segments. The set of sketch line segments of the sketch map of the t-th frame of the image is denoted as N is the number of frames of video V, is the i-th sketch line segment in the set of sketch line segments of the sketch map of the t-th frame of the image, and are respectively the x coordinate of the midpoint of the i-th sketch line segment, the y coordinate of the midpoint, the angle and the length, is the number of sketch line segments in the set of sketch line segments S t of the sketch map of the t-th frame of the image. The set of sketch line segments of the sketch maps of all frames of the image is

[0063] In a specific embodiment, step S1 includes:

[0064] S1.1. For the input image frame sequence For each frame image, design an edge-line template and use a detection operator to calculate a response map with CFAR characteristics and a gradient-based response map.

[0065] S1.2. Fuse the two response maps obtained in S1.1 to obtain an intensity map.

[0066] S1.3. Use non-maximum suppression operation and double-threshold edge connection operation to extract the edge-line map from the intensity map obtained in S1.2.

[0067] S1.4. Sketch each curve in the edge-line map in a straight-line approximation manner, calculate the coding length gain of each sketched line based on the edge-line model of the graph, and obtain the sketch map of the image through sketch tracing.

[0068] S1.5. Further evaluate and prune the obtained sketch map using the coding length gain of the sketched lines to obtain the sketch map of the image; the set of sketch line segments of the sketch maps of all frame images is

[0069] Please refer to Figure 2a 、 Figure 2b and Figure 2c , Figure 2a 、 Figure 2c and Figure 2e in which are the original images of three video frame images, Figure 2b 、 Figure 2d and Figure 2f in which are the sketch maps after sketching the original images of three video frame images.

[0070] S2. For each moment t, by superimposing the sketch map of the t-th frame to the sketch map of the (t + K)-th frame, obtain the initial sketch motion field of the t-th frame image.

[0071] Specifically, for each moment t, by superimposing the sketch map of the t-th frame to the sketch map of the (t + K)-th frame, construct the initial sketch motion field of the t-th frame image and then obtain the set Ψ of the initial sketch motion fields of all frame images init .

[0072] In a specific embodiment, step S2 includes:

[0073] S2.1. Obtain the set of sketch line segments of K + 1 sketch maps from the t-th frame to the (t + K)-th frame in the set of sketch line segments of the sketch maps of all frame images, where K is the number of sketch maps superimposed on the sketch map of the t-th frame image.

[0074] Specifically, first set a hyperparameter K, which is the number of sketch maps superimposed on the sketch map of the t-th frame image. Then obtain the set S of K + 1 sketch line segments from the t-th frame to the (t + K)-th frame from the set S of sketch line segments of the sketch maps of all frame imagest , S t+1 ,...S t+k ,...,, S t+K .

[0075] S2.2. Add the set of sketch line segments (S t , S t+1 ,...S t+k ,...,, S t+K ) of the K + 1 sketch maps obtained in step S2.1 to the initial sketch motion field corresponding to the t-th frame image, and this initial sketch motion field is represented as:

[0076]

[0077] wherein, is the initial sketch motion field of the t-th frame image, and S t+k is the set of sketch line segments of the sketch map of the (t + K)-th frame image.

[0078] Please refer to Figure 3 , Figure 3 which shows the initial sketch motion field obtained in this step, where K = 9.

[0079] S3. Determine whether a sketch line segment is a duplicate sketch line segment according to the matching degree of any two sketch line segments in the initial sketch motion field of the t-th frame image. If so, delete the latter sketch line segment from the initial sketch motion field of the t-th frame image to obtain the updated initial sketch motion field of the t-th frame image.

[0080] Specifically, according to the set of initial sketch motion fields Ψ init obtained in step S2, for the initial sketch motion field of the t-th frame image, use the matching function φ m (·, ·) to calculate the matching degree and of any two sketch line segments Then, judge whether any two sketch line segments in the initial sketch motion field are duplicate sketch line segments through the set matching degree threshold τ. If the sketch line segment and the sketch line segment are duplicate sketch line segments, then remove the sketch line segment .

[0081] In a specific embodiment, step S3 includes:

[0082] S3.1. Traverse any two sketch line segments and

[0083] S3.2. Calculate the sketch line segments using the matching function and the sketch line segments to calculate the matching degree

[0084] Here, the matching function is expressed as:

[0085]

[0086] Among them, is the matching degree between the sketch line segment and the sketch line segment λ1, λ2, λ3, and λ4 are trade-off parameters, and λ1 = λ2. The sketch line segment The sketch line segment and are respectively the x coordinate of the midpoint, the y coordinate of the midpoint, the angle, and the length of the midpoint of the i-th sketch line segment, and are respectively the x coordinate of the midpoint, the y coordinate of the midpoint, the angle, and the length of the midpoint of the j-th sketch line segment, g x (·, ·), g y (·, ·), g θ (·, ·), and g l (·, ·) are respectively the components of the metric function on the x coordinate of the corresponding midpoint, the y coordinate of the midpoint, the angle θ, and the length l.

[0087] S3.3. Judge the matching degree to determine the relationship with the matching degree threshold τ. If the matching degree is less than the matching degree threshold τ, it indicates that the sketch line segment and the sketch line segment are not duplicate sketch line segments. If the matching degree is greater than or equal to the matching degree threshold τ, it indicates that the sketch line segment and the sketch line segment are duplicate sketch line segments. Then, the sketch line segment is deleted from the initial sketch motion field of the t-th frame image to obtain the updated initial sketch motion field of the t-th frame image.

[0088] Here:

[0089]

[0090] Among them, τ is the matching degree threshold, indicates whether the sketch line segment and the sketch line segment are duplicate sketch line segments. That is to say, when the matching degree of two sketch line segments is greater than or equal to τ, takes the value of 1, and the sketch line segment and the sketch line segments are regarded as repeated sketch line segments, otherwise the value is taken as 0, and the sketch line segments and the sketch line segments are regarded as non-repeated sketch line segments.

[0091] If the sketch line segment and the sketch line segment are repeated sketch line segments, then the latter sketch line segment is removed from the initial sketch sports field in the middle.

[0092] As Figure 4 shown, it is the result after removing the repeated sketch line segments in this step, and the removed sketch line segments are as Figure 4 shown in the blue box.

[0093] S4. Remove the noisy sketch line segments in the initial sketch sports field of the updated t-th frame image, and use the remaining sketch line segments as the motion sketch line segments to obtain the sketch sports field of the t-th frame image.

[0094] Specifically, after removing the repeated sketch line segments, a set of parameters is set to gradually remove the noisy sketch line segments to obtain the final sketch sports field Ψ t of the t-th frame image, and the sketch line segments in the sketch sports field are called motion sketch line segments.

[0095] In a specific embodiment, step S4 includes:

[0096] S4.1. Set a set of parameters where Z is the total number of steps, (r z , min z ) means that the minimum number of sketch line segments within the radius r z is min z ;

[0097] S4.2. For the initial sketch sports field of the updated t-th frame image, the number of midpoints of other sketch line segments within the circle with the midpoint of each sketch line segment as the center and r z as the radius. When the number of midpoints of other sketch line segments within this circle is less than min z , this sketch line segment is a noisy sketch line segment. Remove this noisy sketch line segment from the initial sketch sports field of the updated t-th frame image, and use the remaining sketch line segments as the motion sketch line segments to obtain the final sketch sports field Ψ t composed of motion sketch line segments.

[0098] As Figure 5 shown, it is the sketch sports field after removing the noisy sketch line segments in this step, and the removed sketch line segments are asFigure 5 as shown in the red box

[0099] S5. Based on the clustering method, divide the adjacent motion sketch line segments in the sketch motion field of the t-th frame image into the same region to obtain a set of division results.

[0100] Specifically, use the sketch motion field Ψ of the t-th frame image obtained in step S4 t , and divide the adjacent motion sketch line segments into the same region through a clustering algorithm to obtain the division where c num is the maximum number of sketch line segments in each cluster, is the c-th division region and also the c-th cluster after clustering, is the number of clusters.

[0101] In a specific embodiment, step S5 includes:

[0102] S5.1. First, define the maximum number of sketch line segments in each cluster as c num , and then, according to the number of motion sketch line segments |Ψ t | in the sketch motion field of the t-th frame image, obtain the number of clusters, where the number of clusters is c num is the maximum number of motion sketch line segments in each cluster.

[0103] S5.2. Use the K-means algorithm to divide the sketch motion field of the t-th frame image into clusters;

[0104] S5.3. Based on the clusters, obtain the final set of division results where C t is the set of all divisions of the sketch motion field of the t-th frame image, is the c-th division region of the sketch motion field of the t-th frame image.

[0105] As Figure 6 shown, it is the sketch motion field divided by the clustering algorithm in this step.

[0106] S6. For each division region in the set of division results, calculate the set of motion sketch line segments of each frame image.

[0107] Specifically, use the division result C obtained in step S5 t , and in each division region , calculate the set of motion sketch line segments belonging to each frame where is the set of motion sketch lines belonging to the (t + K)-th frame within the c-th division region.

[0108] In a specific embodiment, step S6 includes:

[0109] S6.1. Obtain the set of sketch line segments S t , S t+1 ,... S t+k ,...,, S t+K .

[0110] S6.2. Sequentially determine whether the sketch line segments in the set of K + 1 sketch line segments from the t-th frame to the (t + K)-th frame are in the c-th divided region. If so, store the sketch line segment in the moving sketch line set to obtain the set of moving sketch line segments belonging to each frame image in the c-th divided region.

[0111] Specifically, calculate whether the sketch line segments t in the set of sketch line segments S are in the c-th divided region . If so, store the sketch line segment in the moving sketch line set ; and calculate the set of sketch line segments S t , S t+1 ,... S t+k ,...,, S t+K sequentially in the above manner; finally obtain the set of moving sketch line segments belonging to each frame in the c-th divided region and the set of sets of moving sketch line segments belonging to each frame in all divisions C For example, t as shown in

[0112] such as Figure 7a , Figure 7b and Figure 7c , which are the moving sketch line segments belonging to three video frames obtained in this step.

[0113] S7. Obtain a trajectory set based on the trajectories of each divided region obtained from the set of moving sketch line segments.

[0114] Specifically, use the set of moving sketch lines in the c-th divided region obtained in step S6 to calculate the trajectory from the t-th frame to the (t + K)-th frame in the c-th divided region and then obtain the trajectory sets of all divisions

[0115] In a specific embodiment, step S7 includes:

[0116] S7.1. Calculate the mean of the four components of the midpoint x-coordinate, the midpoint y-coordinate, the angle θ and the length l of all sketch segments in the motion sketch segment set of each frame image to obtain the corresponding trajectory.

[0117] Here, the method for calculating the mean is: calculate the mean of the x-coordinates of the midpoints of all sketch line segments in the motion sketch line segment set of each frame image, calculate the mean of the y-coordinates of the midpoints of all sketch line segments in the motion sketch line segment set of each frame image, calculate the mean of the angles θ of all sketch line segments in the motion sketch line segment set of each frame image, and calculate the mean of the lengths l of all sketch line segments in the motion sketch line segment set of each frame image.

[0118] S7.2, repeat step S7.1 until the trajectories corresponding to all frame images are obtained, and the trajectories corresponding to all frame images are taken as the final trajectory set.

[0119] Specifically, first, get all the sketch segments in Ψct in, For the cth partition area The number of motion sketch lines belonging to the t+kth frame in the t+kth frame; secondly, calculate the sketch line segment set The mean of all sketch line segments in the four components of x, y, θ and l is obtained Likewise, The average value of the sketch line segment set is calculated separately Then, the trajectory from the tth frame to the t+Kth frame in the cth partition area is obtained. Finally, for all Partition to get the trajectory set

[0120] S8. Based on the trajectory set, calculate the motion vector of each motion sketch line segment in the motion sketch line segment set of the t-th frame image to obtain a motion vector set.

[0121] Specifically, using the trajectory set Trt obtained in step S7, for the trajectory from the tth frame to the t+Kth frame in the cth divided area Calculate the set of motion sketch segments belonging to the tth frame The motion vector of each motion sketch line segment in Then we get the motion vector set of all motion sketch line segments in the tth frame in is the number of all motion sketch segments in the tth frame.

[0122] In a specific embodiment, step S8 includes:

[0123] S8.1. For each set of trajectories in the trajectory set, calculate the mean of the first half and the mean of the second half of the trajectory in four components: the x - coordinate of the mid - point, the y - coordinate of the mid - point, the angle θ, and the length l.

[0124] Specifically, for the trajectory from the t - th frame to the (t + K)-th frame in the c - th partition region Calculate the mean of the first half and the mean of the second half of the trajectory in four components: x, y, θ, and l of the first half of the trajectory and the mean of the second half of the trajectory

[0125] S8.2. Obtain the motion vector of the corresponding motion sketch segment according to the mean of the first half and the mean of the second half of the trajectory, and use this motion vector as the motion vector of the sketch segment belonging to the t - th frame image in the corresponding partition region.

[0126] Here, the motion vector is and use the motion vector as the motion vector of the sketch segment belonging to the t - th frame in the c - th partition region.

[0127] S8.3. Combine the motion vectors of the sketch segments belonging to the t - th frame image in all partition regions to obtain the set of motion vectors of all motion sketch segments in the t - th frame image.

[0128] Here, combine all the motion vectors of the sketch segments belonging to the t - th frame in the partition regions, and then obtain the set of motion vectors of all motion sketch segments in the t - th frame This set of motion vectors is the sparse representation of the motion information in the t - th frame of the video, and is called the line flow of the t - th frame.

[0129] As Figure 8 shown, it is a visualization method of the line flow, using colors to represent the motion direction of the motion sketch segments, and using the depth of the colors to represent the motion distance of the motion sketch segments; as Figure 9a shown, it is the visualization of the line flow extracted in this step.

[0130] The method for sparsely representing video motion information based on line flow provided by the present invention preserves the abrupt parts such as boundaries and contours in the video images by using the initial sketch model in step S1, removes parts such as textures in the video images, and obtains a set of sketch line segments that can sparsely represent the images. Therefore, based on this, the information in the video is represented in a sparse form. In steps S3 and S4, the present invention further removes the repeated sketch line segments generated by the background and the noise sketch line segments generated due to reasons such as camera noise, illumination change, and slight jitter, so as to represent the running information in the video in a sparse form. Therefore, the video motion information representation method proposed by the present invention has a more sparse representation. Compared with the video motion information representation method based on optical flow, this algorithm uses the motion vectors of the motion sketch line segments to represent the motion information in the video, provides a new method for representing video motion information in a more sparse form, and thus reduces the storage space.

[0131] The effect of the present invention can be further illustrated by the following simulation results.

[0132] 1. Simulation conditions

[0133] The hardware conditions for the simulation of the present invention are: the graphics workstation HP Z840 in the Intelligent Sensing and Image Understanding Laboratory. The software conditions for the simulation of the present invention are: Windows 10 system, PyCharm 2020, Visual Studio 2017, Python3.6, OpenCV 2.1.0.

[0134] The parameters for the simulation of the present invention are shown in Table 1:

[0135] Table 1

[0136]

[0137] 2. Simulation content

[0138] Using a set of video frame images, the present invention extracts the line flow that represents the motion information in the video in a sparse form. The experimental results are shown in Figure 9 of the accompanying drawings, where Figure 9a is the effect diagram of the line flow extracted by the present invention, Figure 9b is the effect diagram of the extracted optical flow.

[0139] The comparison of the storage space of the motion information between the present invention and the optical flow method is shown in Table 2.

[0140] Table 2

[0141] Method Video frame image size Size of storage space Optical flow method 200×200 22.2KB The present invention 200×200 4.48KB

[0142] 3. Analysis of simulation results:

[0143] As can be seen from Figure 9, the motion information extracted by optical flow has obvious noise and is relatively dense. The line flow extracted by the present invention has a sparser representation form and less noise, achieving a better representation of the motion information. As can be seen from Table 2, the storage space of the motion information extracted by the present invention is significantly reduced compared with that of optical flow, indicating that the present invention can extract sparser motion information.

[0144] In summary, the present invention realizes the extraction of motion information in a video, which not only has less noise but also has a better extraction effect. It can represent the motion information in the video in a sparser form and occupies less storage space at the same time.

[0145] It should be noted that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.

[0146] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0147] Although the present invention has been described in connection with various embodiments herein, however, in the process of implementing the claimed invention, those skilled in the art can understand and realize other variations of the disclosed embodiments by viewing the accompanying drawings and the disclosure. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality of cases. Certain measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0148] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for sparse representation of video motion based on line flow, characterized in that, The described characterization method includes: S1. Sketchify the image frame sequence in video V to obtain a sketch map for each frame of the image; S2. For each moment t, by superimposing the sketch map of the t-th frame onto the sketch map of the (t + K)-th frame, obtain the initial sketch motion field of the t-th frame image; S3. Determine whether a sketch line segment is a repeated sketch line segment according to the matching degree between any two sketch line segments in the initial sketch motion field of the t-th frame image. If so, delete the latter sketch line segment from the initial sketch motion field of the t-th frame image to obtain the updated initial sketch motion field of the t-th frame image; S4. Remove the noise sketch line segments in the updated initial sketch motion field of the t-th frame image, and use the remaining sketch line segments as motion sketch line segments to obtain the sketch motion field of the t-th frame image; S5. Based on the clustering method, divide the adjacent motion sketch line segments in the sketch motion field of the t-th frame image into the same region to obtain a set of division results; S6. For each division region in the set of division results, calculate the set of motion sketch line segments for each frame of the image; S7. Obtain a set of trajectories based on the trajectories of each division region obtained from the set of motion sketch line segments; S8. Based on the set of trajectories, calculate the motion vectors of each motion sketch line segment in the set of motion sketch line segments of the t-th frame image to obtain a set of motion vectors.

2. The method for sparse representation of video motion based on line flow according to claim 1, wherein The set of sketch line segments of the sketch of the t-th frame image is denoted as N is the number of frames of video V, is the i-th sketch line segment in the set of sketch line segments of the sketch of the t-th frame image, and are the x coordinate of the midpoint, the y coordinate of the midpoint, the angle and the length of the i-th sketch line segment respectively, is the set S of sketch line segments of the sketch of the t-th frame image t The number of sketch line segments in it.

3. The method for sparsely representing video motion based on line flow according to claim 1, wherein Step S2 includes: S2.

1. Obtain the set of sketch line segments of K + 1 sketch maps from the t-th frame to the (t + K)-th frame in the set of sketch line segments of the sketch maps of all frame images, where K is the number of sketch maps superimposed on the sketch map of the t-th frame image; S2.

2. Add the set of sketch line segments of the K + 1 sketch maps obtained in step S2.1 to the initial sketch motion field corresponding to the t-th frame image, and this initial sketch motion field is expressed as: Among them, is the initial sketch motion field of the t-th frame image, S t+k is the set of sketch line segments of the sketch of the (t + K)-th frame image.

4. The method for sparse representation of video motion based on line flow according to claim 1, wherein Step S3 includes: S3.

1. Traverse any two sketch line segments in the initial sketch motion field of the t-th frame image and S3.

2. Calculate the sketch line segments using the matching function and the sketch line segments to calculate the matching degree S3.

3. Determine the matching degree The relationship with the matching degree threshold τ. If the matching degree is less than the matching degree threshold τ, it indicates that the sketch line segment and the sketch line segment are not repeated sketch line segments. If the matching degree is greater than or equal to the matching degree threshold τ, it indicates that the sketch line segment and the sketch line segment are repeated sketch line segments. Then, delete the sketch line segment from the initial sketch motion field of the t-th frame image to obtain the updated initial sketch motion field of the t-th frame image.

5. The method for sparse characterization of video motion based on line flow according to claim 4, characterized in that The described matching function is expressed as: Among them, is the matching degree between and the sketch line segment , λ1, λ2, λ3, and λ4 are weighing parameters, and λ1 = λ2. The sketch line segment The sketch line segment and are respectively the x coordinate of the midpoint, the y coordinate of the midpoint, the angle, and the length of the i-th sketch line segment. and are respectively the x coordinate of the midpoint, the y coordinate of the midpoint, the angle, and the length of the j-th sketch line segment. g x (·, ·), g y (·, ·), g θ (·, ·), and g l (·, ·) are respectively the components of the metric function on the x coordinate of the corresponding midpoint, the y coordinate of the midpoint, the angle θ, and the length l.

6. The method for sparse representation of video motion based on line flow according to claim 1, characterized in that Step S4 includes: S4.

1. Set a set of parameters Among them, Z is the total number of steps, (r z , min z ) indicates that the minimum number of sketch segments within the radius r z is min z ; S4.

2. For the initial sketch motion field of the updated t-th frame image, the number of midpoints of other sketch line segments within a circle centered at the midpoint of each sketch line segment with a radius of r z . When the number of midpoints of other sketch line segments within this circle is less than min z , this sketch line segment is a noisy sketch line segment. Remove this noisy sketch line segment from the initial sketch motion field of the updated t-th frame image, and use the remaining sketch line segments as motion sketch line segments to obtain the final sketch motion field of the t-th frame image composed of motion sketch line segments.

7. The method for sparsely representing video motion based on line flow according to claim 1, wherein Step S5 includes: S5.

1. Obtain the number of clusters based on the number |Ψ| of moving sketch line segments in the sketch sports field of the t-th frame image, where the number of clusters is t |, and c num is the maximum number of moving sketch line segments in each cluster; S5.

2. Divide the sketch motion field of the t-th frame image into clusters using the K-means algorithm; S5.

3. Based on clusters, obtain the final set of partitioning results where C t is the set of all partitions of the sketch sports field of the t-th frame image, is the c-th partition region of the sketch sports field of the t-th frame image.

8. The method for sparsely representing video motion based on line flow according to claim 1, wherein Step S6 includes: S6.

1. Obtain the set of sketch line segments of K + 1 sketch maps from the t-th frame to the (t + K)-th frame in the set of sketch line segments of the sketch maps of all frame images; S6.

2. Determine in sequence whether the sketch segments in the K+1 sketch segment sets from the t-th frame to the (t+K)-th frame are in the c-th division region. If so, store the sketch segment in the moving sketch line set to obtain the moving sketch line sets belonging to each frame image in the c-th division region.

9. The method for sparse representation of video motion based on line flow according to claim 1, wherein Step S7 includes: S7.

1. Calculate the mean values of the four components of the x coordinate of the midpoint, the y coordinate of the midpoint, the angle θ, and the length l of all sketch line segments in the set of motion sketch line segments of each frame of the image to obtain the corresponding trajectory; S7.

2. Repeat step S7.1 until the trajectories corresponding to all frame images are obtained, and use the trajectories corresponding to all frame images as the final set of trajectories.

10. The method for sparse representation of video motion based on line flow according to claim 1, characterized in that, Step S8 includes: S8.

1. For each set of trajectories in the set of trajectories, calculate the mean values of the first half and the second half of the trajectory in the four components of the x coordinate of the midpoint, the y coordinate of the midpoint, the angle θ, and the length l; S8.

2. Obtain the motion vector of the corresponding motion sketch line segment according to the mean values of the first half and the second half of the trajectory, and use this motion vector as the motion vector of the sketch line segment belonging to the t-th frame image in the corresponding division region; S8.

3. Combine the motion vectors of the sketch line segments belonging to the t-th frame image in all division regions to obtain the set of motion vectors of all motion sketch line segments in the t-th frame image.

Images