Detection frame completion method and device based on frame skipping detection and related components

Abstract

The invention discloses a detection frame complementing method and device based on frame skip detection and a related component, and the method comprises the steps: cutting a video stream every other preset frame, and obtaining a group of detection frames; performing target detection on the detection frame to obtain feature information of the detection frame of each target in the detection frame; integrating the feature information of the detection frames of each target in all the detection frames to obtain a temporary track of the corresponding target, and calculating the feature information of a candidate frame of the corresponding target in a to-be-interpolated frame between two adjacent detection frames according to the temporary track by using a target dynamic interpolation algorithm; and integrating the feature information of the detection frames of each target in all the detection frames and the feature information of all the candidate frames to obtain a complete motion compensation track of the corresponding target. According to the invention, motion compensation and full-frame continuous rendering are carried out on the candidate frame through the dynamic interpolation algorithm, so that the stability and continuity of multi-target detection can be maintained while the calculation amount is reduced and the detection speed is improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method, device and related components for completing a detection frame based on frame skipping detection. Background Art

[0002] With the rapid development of video processing technology, frame skipping detection technology, as a key link in video analysis and processing, is becoming increasingly important. In recent years, this field has made significant progress. Traditional filtering technology and interpolation methods have been continuously optimized, and frame skipping detection technology has made significant progress, but the challenges it brings cannot be ignored. Especially in target tracking tasks, frame skipping detection often leads to discontinuity of the target trajectory. Due to the existence of frame skipping, it is difficult for the tracking algorithm to accurately capture the complete motion trajectory of the target, resulting in breakpoints or drift in the tracking results. In addition, frame skipping may also aggravate the jitter problem of the target frame during target detection. During the detection process, due to the sudden change of information between adjacent frames, the position and size of the target frame may fluctuate violently, seriously affecting the accuracy and reliability of the detection results. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device and related components for completing a detection frame based on frame skipping detection, aiming to solve the problems of incoherent target trajectory and severe jitter of the target frame during target detection caused by frame skipping detection.

[0004] In a first aspect, an embodiment of the present invention provides a method for completing a detection frame based on frame skipping detection, comprising:

[0005] Obtain an input video stream, and crop the video stream every predetermined frame to obtain a set of detection frames;

[0006] Performing target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of a detection frame of each target in the detection frame;

[0007] Integrate the feature information of the detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and aggregate the temporary trajectories of each target to form a trajectory cluster;

[0008] For each temporary track in the track cluster, a target dynamic interpolation algorithm is used to calculate feature information of a candidate box of a corresponding target in a to-be-interpolated frame between two adjacent detection frames;

[0009] The feature information of the detection frame of each target in all detection frames and the feature information of the candidate frame in all frames to be interpolated are integrated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target.

[0010] In a second aspect, an embodiment of the present invention provides a detection frame completion device based on frame skipping detection, comprising:

[0011] An acquisition unit, used for acquiring an input video stream, and cropping the video stream every predetermined frame to obtain a set of detection frames;

[0012] A detection unit, configured to perform target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of a detection frame of each target in the detection frame;

[0013] an aggregation unit, configured to integrate feature information of detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and to aggregate the temporary trajectories of the targets into a trajectory cluster;

[0014] A calculation unit, configured to calculate, for each temporary track in the track cluster, feature information of a candidate frame of a corresponding target in a to-be-interpolated frame between two adjacent detection frames using a target dynamic interpolation algorithm;

[0015] The integration unit is used to integrate the feature information of the detection frame of each target in all the detection frames and the feature information of the candidate frames in all the frames to be interpolated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target.

[0016] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for completing the detection frame based on frame skipping detection described in the first aspect is implemented.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for completing a detection frame based on frame skipping detection described in the first aspect above is implemented.

[0018] The present invention discloses a method, device and related components for completing a detection frame based on frame skipping detection. The method comprises: obtaining an input video stream, cropping the video stream at every predetermined frame to obtain a group of detection frames; performing target detection on the detection frames in real time by using a target detection algorithm to obtain feature information of the detection frames of each target in the detection frames; integrating the feature information of the detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and aggregating the temporary trajectories of each target to form a trajectory cluster; for each temporary trajectory in the trajectory cluster, using a target dynamic interpolation algorithm to calculate the feature information of a candidate frame of the corresponding target in a frame to be interpolated between two adjacent detection frames; integrating the feature information of the detection frames of each target in all detection frames and the feature information of the candidate frames in all frames to be interpolated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target. The present invention uses a dynamic interpolation algorithm to perform motion compensation and full-frame continuous rendering on multiple target candidate frames, and infers the continuous motion trajectory of the detection frame in real time, which can reduce the amount of calculation and improve the detection speed while obtaining a continuous target trajectory and a stable detection frame, thereby maintaining the stability and continuity of road traffic multi-target detection. The embodiment of the present invention also provides a detection frame completion device based on frame skipping detection, a computer-readable storage medium, and a computer device, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0020] Figure 1 A flowchart of a detection frame completion method based on frame skipping detection is shown;

[0021] Figure 2 It is a schematic block diagram of a detection frame completion device based on frame skipping detection. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0024] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0025] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] See also Figure 1 This embodiment provides a method for completing a detection frame based on frame skipping detection, including:

[0027] S101: Obtain an input video stream, and crop the video stream every predetermined frame to obtain a group of detection frames;

[0028] Specifically, use the cv2.VideoCapture() function to create a video capture object, then use the cap.read() function in a loop to read a frame of image data from the video capture object, and use all the read image data as a set of detection frames.

[0029] cv2.VideoCapture() supports capturing videos from multiple sources such as files, cameras, or network URLs. Its acquisition method is flexible and can be applied to a wide range of application scenarios. The cap.read() function is an efficient way to read video frames. It is optimized internally to ensure that the video stream can be read smoothly while minimizing the use of system resources.

[0030] The predetermined frame may be realized by setting the number of frames of the cycle. For example, the predetermined frame is set to 10 frames, so a cycle may be set every 10 frames to achieve cropping of the video stream every 10 frames.

[0031] S102: performing target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of a detection frame of each target in the detection frame;

[0032] In some embodiments, target detection is performed on a detection frame in real time through a target detection algorithm, including: adjusting the size of the detection frame according to the size required by the target detection algorithm; then inputting the adjusted detection frame into a backbone network of the target detection algorithm for feature extraction to obtain a feature map; then inputting the feature map into a prediction branch (bounding box) of the target detection algorithm to obtain the position information and confidence of the bounding box; then using a predefined anchor box and a model setting decoder to parse the position information and confidence of all bounding boxes; then removing all duplicate boxes in the bounding box according to the confidence and through non-maximum suppression (NMS) to obtain the optimal bounding box of the target, and using the optimal bounding box of the target as the detection box of the target.

[0033] Adjust the size of the detection frame according to the size required by the target detection algorithm to ensure that the input data meets the algorithm requirements and improve the accuracy of detection. Remove all duplicate boxes in the bounding box according to the confidence and through non-maximum suppression to obtain the optimal bounding box of the target, which helps to eliminate redundant detection results and improve the accuracy of target detection.

[0034] Specifically, the feature map is input into the prediction branch (bounding box) of the target detection algorithm to obtain the position information and confidence of the bounding box, including: dividing the feature map into multiple grid units, each grid unit is responsible for predicting the object contained in the grid. For example, the feature map is divided into S×S grids (S is a preset value, such as 7, 19, etc.), and each grid unit predicts B bounding boxes and the confidence and category probability of these bounding boxes. The bounding box is used to locate the position of the target in the image, the confidence reflects the possibility of the bounding box containing the target and the accuracy of the bounding box, and the category probability is used to identify the type of the target.

[0035] By dividing the feature map into multiple grid cells, each grid cell independently predicts the object inside it, thus achieving fine-grained detection of objects in the image. This method can better capture local details in the image, thereby improving the accuracy of detection.

[0036] Among them, the target detection algorithm can be an R-CNN algorithm, a Fast R-CNN algorithm, a FasterR-CNN algorithm or a YOLOv5 algorithm, etc.

[0037] In some embodiments, target detection is performed on a detection frame, including: performing denoising preprocessing on an input detection frame; then using a recurrent neural network to classify the preprocessed detection frame into foreground and background point by point to roughly segment the preprocessed detection frame to obtain foreground data; then using a closed operation and a median filter to eliminate depressions and burrs in the foreground data; then selecting a seed point in the foreground data after the elimination process and generating a temporary detection frame; then using a fully convolutional neural network to determine the temporary detection frame, and performing precise regression on the boundary of the detection frame determined to be the foreground to obtain a final detection frame.

[0038] By using a recurrent neural network to perform coarse segmentation on the detection frame, and then using a fully convolutional neural network to determine the detection frame for fine segmentation, the detection frame acquisition method can obtain a more complete and pure target in different situations. This method can effectively detect long and short targets in the detection frame as well as close and difficult-to-segment targets.

[0039] Among them, the recurrent neural network can be a basic recurrent neural network (RNN), a long short-term memory network (Long Short-Term Memory, LSTM), a gated recurrent unit (GRU), a bidirectional recurrent neural network (Bi-directional RNN), a clock frequency driven recurrent neural network (Clockwork RNNs, CW-RNNs) or a deep recurrent neural network (Deep RNNs). The fully convolutional neural network can be a FCN fully convolutional neural network, a U-Net fully convolutional neural network or a SegNet fully convolutional neural network.

[0040] Furthermore, a recurrent neural network is used to classify the preprocessed detection frame into foreground and background point by point to perform rough segmentation on the preprocessed detection frame, and the obtained foreground data includes:

[0041] The preprocessed detection frame is input into the convolutional neural network for feature extraction to obtain a feature vector. The feature vector is divided into small blocks to convert the feature vector into a one-dimensional feature vector sequence. The one-dimensional feature vector sequence is then input into the RNN layer to obtain the confidence of each feature vector in the one-dimensional feature vector sequence. The fully connected layer (Dense layer) is then used to classify the confidence of each feature vector in the one-dimensional feature vector sequence, and the probability of each feature vector belonging to the foreground or background is output. Then, the feature vector is determined to belong to the foreground according to the preset threshold. If the probability that the feature vector belongs to the foreground is greater than the preset threshold, the feature vector is determined to be the foreground, and all the feature vectors belonging to the foreground are aggregated to obtain the foreground data.

[0042] Convolutional neural networks (CNNs) are used for feature extraction. CNNs excel in image and video processing due to their powerful local feature capture capabilities and spatial hierarchical structure understanding capabilities. Through multi-layer convolution and pooling operations, rich, high-level feature representations can be automatically extracted from preprocessed detection frames, providing strong support for subsequent classification tasks. After the RNN layer, a fully connected layer (Dense layer) is used to classify the confidence of the feature vector, further improving the refinement and accuracy of the classification. The Dense layer is fully connected to learn and extract the most discriminative features, thereby more accurately determining whether each feature vector belongs to the foreground or background.

[0043] Furthermore, using closing operation and median filtering to eliminate the depressions and burrs in the foreground data includes: if there are many depressions and small breaks in the foreground data, you can first use closing operation to fill these depressions and connect the broken parts, and then use median filtering to remove possible burrs or other noise. If the depressions in the foreground data are not the main problem, but mainly burrs or noise at the edge, then you can directly use median filtering to process it.

[0044] Furthermore, selecting seed points in the foreground data after the elimination process and generating a temporary detection frame includes: randomly selecting seed points on the foreground data after the elimination process, using the selected seed points as the center points, and flipping the seed points with a distance from the current interval endpoint being half the window length to obtain a temporary detection frame.

[0045] By using the longer distance between the seed point and the current interval endpoint as the half-window length to generate a temporary detection frame, this design is adaptive. Foreground objects of different positions and sizes may require temporary detection frames of different sizes to accurately frame them. By dynamically adjusting the half-window length, this method can more flexibly adapt to different situations and improve the accuracy and applicability of the temporary detection frame.

[0046] In some embodiments, for the case where detection frames of different targets overlap in the same detection frame, the following method can be used to identify the overlapping detection frames. Specifically, the detection frames of each target in the detection frame are detected using a target detector, and then the features in the detection frames of each detected target are extracted, including appearance features and motion features, and then the similarity between each target in the previous and next frames is calculated by the cosine distance between the features, and then the detection frames of the two targets with the greatest similarity in the previous and next frames are used as the detection frames of the same target.

[0047] This method ensures the continuity of target tracking by calculating the similarity between the targets in the previous and next frames and treating the two detection frames with the highest similarity as the same target. When the target detection frames overlap in a frame, the target can be accurately tracked by matching the similarity across frames, thereby identifying the detection frames of the overlapping targets.

[0048] In this embodiment, the feature information includes: the width, height and center point position information of the detection frame. The target detection algorithm is used to perform target detection on the detection frame in real time, and the feature information of the detection frame of each target in the detection frame includes:

[0049] The target detection algorithm is used to detect the target in the detection frame of the i-th frame, and the width and height (w i1 ,h i1 )、(w i2 ,h i2 ),…(w im ,h im )…,(w ic ,h ic ) and the center point location information (x i1 ,y i1 )、(x i2 ,y i2 ),…(x im ,y i2 )…,(x ic ,y ic );

[0050] Among them, x im Indicates the x-coordinate and y-coordinate of the center point of the detection box of the m-th target in the i-th frame im represents the y coordinate of the center point of the detection box of the mth target in the i-th frame, w im Indicates the width of the detection box of the mth target in the i-th frame, h im Represents the height of the detection box of the mth target in the i-th frame, where m is an integer between 1 and c, and c is the number of targets.

[0051] The target detection algorithm is used to detect the target in the detection frame in real time, and the feature information of the detection frame of each target in the detection frame is obtained, which also includes:

[0052] The target detection algorithm is used to detect the target in the detection frame of the i+nth frame, and the width and height (w (i+n)1 ,h (i+n)1 )、(w (i+n)2 ,h (i+n)2 ),…(w (i+n)m ,h (i+n)m )…,(w (i+n)c ,h (i+n)c) and the center point location information (x (i+n)1 ,y (i+n)1 )、(x (i+n)2 ,y (i+n)2 ),……,(x (i+n)c ,y (i+n)c );

[0053] Among them, x (i+n)m Indicates the x-coordinate and y-coordinate of the center point of the detection box of the m-th target in the i+n-th frame. (i+n)m represents the y coordinate of the center point of the detection box of the mth target in the i+nth frame, w (i+n)m Indicates the width of the detection box of the mth target in the i+nth frame, h (i+n)m Indicates the height of the detection box of the mth target in the i+nth frame.

[0054] The above method is used to obtain the width, height and center point position information of the detection frame of each target in two adjacent detection frames (i.e., the detection frame of the i-th frame and the detection frame of the i+n-th frame). Then, according to the width, height and center point position information of the detection frame of each target and using the target dynamic interpolation algorithm, the width, height and center point position information of the candidate frame of the corresponding target in the frame to be interpolated between the two adjacent detection frames can be calculated, wherein n frames are predetermined frames, i.e., the video stream is cropped every n frames.

[0055] S103: Integrate feature information of detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and aggregate the temporary trajectories of the targets into a trajectory cluster;

[0056] S104: for each temporary track in the track cluster, using a target dynamic interpolation algorithm to calculate feature information of a candidate box of a corresponding target in a to-be-interpolated frame between two adjacent detection frames;

[0057] Specifically, the feature information of the candidate frame of the corresponding target in the to-be-interpolated frame between two adjacent detection frames is calculated using the target dynamic interpolation algorithm, including:

[0058] The center point position information (x km ,y km ):

[0059]

[0060]

[0061] The width and height (w km ,h km ):

[0062]

[0063]

[0064] Among them, i <k<i+n。

[0065] In the above formula, by fixing i and k and changing the value of m, the feature information of the candidate boxes of all c targets in the kth frame between the i-th frame and the i+n-th frame can be obtained.

[0066] The above formula can be used to calculate the feature information of all candidate frames of the interpolated frames between two adjacent detection frames. By inserting the obtained feature information of the candidate frames of the interpolated frames between the detection frames, a smoother and more continuous motion trajectory of each target can be obtained.

[0067] In this embodiment, using the target dynamic interpolation algorithm to calculate the feature information of the candidate frame of the corresponding target in the frame to be interpolated between two adjacent detection frames also includes: obtaining the predetermined number of frames n when cropping the video stream, and setting the number of frames to be interpolated between two adjacent detection frames to n-1; for each frame to be interpolated, using the target dynamic interpolation algorithm according to the temporary trajectory to calculate the feature information of the candidate frame in the corresponding frame to be interpolated between two adjacent detection frames.

[0068] Specifically, by changing the values ​​of i, k, and m at the same time, feature information of multi-target candidate boxes missing in all intermediate frames in the video stream can be obtained.

[0069] The candidate box feature information generated by the dynamic interpolation algorithm can fill the gaps between detection frames, enhance the continuity and integrity of the data, and provide a richer information basis for subsequent processing.

[0070] S105: Integrate the feature information of the detection frames of each target in all the detection frames and the feature information of the candidate frames in all the frames to be interpolated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target.

[0071] The predetermined order may be to arrange all the detection frames and the to-be-interpolated values ​​in the order of frame numbers from small to large according to the frame numbers of all the detection frames and the to-be-interpolated values, so as to obtain a complete motion compensation trajectory of the corresponding target. Alternatively, the predetermined order may be to arrange all the detection frames and the to-be-interpolated values ​​in the order of frame numbers from large to small according to the frame numbers of all the detection frames and the to-be-interpolated values, so as to obtain a complete motion compensation trajectory of the corresponding target.

[0072] This embodiment utilizes the prior target frame information output by the multi-target detection algorithm. And by introducing a dynamic interpolation algorithm, it is possible to implement a sophisticated motion compensation strategy for the multi-target candidate frame, and to realize full-frame continuous rendering technology. In turn, the continuous and smooth motion trajectory of the target can be derived in real time. The use of the dynamic interpolation algorithm significantly reduces unnecessary computational burden, thereby greatly improving the processing speed without sacrificing detection accuracy. At the same time, the algorithm can ensure that the generated target trajectory is continuous and the target frame is presented smoothly without jumping, which is crucial to maintaining the stability and consistency of multi-target detection in road traffic scenarios.

[0073] See also Figure 2 This embodiment provides a detection frame completion device 200 based on frame skipping detection, including:

[0074] An acquisition unit 201 is used to acquire an input video stream, and to crop the video stream every predetermined frame to obtain a set of detection frames;

[0075] The detection unit 202 is used to perform target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of a detection frame of each target in the detection frame;

[0076] The collection unit 203 is used to integrate the feature information of the detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and to collect the temporary trajectories of each target to form a trajectory cluster;

[0077] A calculation unit 204 is used to calculate feature information of a candidate box of a corresponding target in a to-be-interpolated frame between two adjacent detection frames using a target dynamic interpolation algorithm for each temporary track in the track cluster;

[0078] The integration unit 205 is used to integrate the feature information of the detection frame of each target in all the detection frames and the feature information of the candidate frames in all the frames to be interpolated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target.

[0079] Furthermore, the detection unit 202 includes:

[0080] A size adjustment subunit, used to adjust the size of the detection frame according to the size required by the target detection algorithm;

[0081] A feature extraction subunit, used for inputting the adjusted detection frame into the backbone network of the target detection algorithm to extract features and obtain a feature map;

[0082] Inputting the feature map into the prediction branch of the object detection algorithm to obtain the position information and confidence of the bounding box;

[0083] Use the predefined anchor boxes and model settings to set up the decoder to parse the location information and confidence of all bounding boxes;

[0084] According to the confidence level and through non-maximum suppression, all duplicate boxes in the bounding box are removed to obtain the optimal bounding box of the target, and the optimal bounding box of the target is used as the detection box of the target.

[0085] Furthermore, the feature information includes: the width, height and center point position information of the detection frame, and the detection unit 202 also includes:

[0086] The first target detection subunit is used to perform target detection on the detection frame of the i-th frame through the target detection algorithm, and obtain the width and height (w i1 ,h i1 )、(w i2 ,h i2 ),…(w im ,h im )…,(w ic ,h ic ) and the center point location information (x i1 ,y i1 )、(x i2 ,y i2 ),…(x im ,y i2 )…,(x ic ,y ic );

[0087] Among them, x im Indicates the x-coordinate and y-coordinate of the center point of the detection box of the m-th target in the i-th frame im represents the y coordinate of the center point of the detection box of the mth target in the i-th frame, w im Indicates the width of the detection box of the mth target in the i-th frame, h im Represents the height of the detection box of the mth target in the i-th frame, where m is an integer between 1 and c, and c is the number of targets.

[0088] Furthermore, the detection unit 202 further includes:

[0089] The second target detection subunit is used to perform target detection on the detection frame of the i+nth frame by using the target detection algorithm to obtain the width and height (w (i+n)1 ,h (i+n)1 )、(w (i+n)2 ,h (i+n)2 ),…(w (i+n)m ,h (i+n)m )…,(w (i+n)c ,h (i+n)c) and the center point location information (x (i+n)1 ,y (i+n)1 )、(x (i+n)2 ,y (i+n)2 ),……,(x (i+n)c ,y (i+n)c );

[0090] Among them, x (i+n)m Indicates the x-coordinate and y-coordinate of the center point of the detection box of the m-th target in the i+n-th frame. (i+n)m represents the y coordinate of the center point of the detection box of the mth target in the i+nth frame, w (i+n)m Indicates the width of the detection box of the mth target in the i+nth frame, h (i+n)m Indicates the height of the detection box of the mth target in the i+nth frame.

[0091] Furthermore, the calculation unit 204 includes:

[0092] The first calculation subunit is used to calculate the center point position information (x km ,y km ):

[0093]

[0094]

[0095] The second calculation subunit is used to calculate the width and height (w km ,h km ):

[0096]

[0097]

[0098] Among them, i <k<i+n。

[0099] Furthermore, the calculation unit 204 further includes:

[0100] A setting subunit is used to obtain a predetermined number of frames n when cropping the video stream, and set the number of frames to be interpolated between two adjacent detection frames to n-1;

[0101] The feature information calculation subunit is used to calculate the feature information of the candidate frame in the corresponding frame to be interpolated between two adjacent detection frames using the target dynamic interpolation algorithm according to the temporary trajectory for each frame to be interpolated.

[0102] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0103] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the method provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0104] The present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method provided in the above embodiment can be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0105] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0106] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive.

[0107] Inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for completing a detection frame based on frame skipping detection, characterized in that: include: Obtain an input video stream, and crop the video stream every predetermined frame to obtain a set of detection frames; Performing target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of a detection frame of each target in the detection frame; Integrate the feature information of the detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and aggregate the temporary trajectories of each target to form a trajectory cluster; For each temporary track in the track cluster, a target dynamic interpolation algorithm is used to calculate feature information of a candidate box of a corresponding target in a to-be-interpolated frame between two adjacent detection frames; The feature information of the detection frame of each target in all detection frames and the feature information of the candidate frame in all frames to be interpolated are integrated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target.

2. The method for completing a detection frame based on frame skipping detection according to claim 1, characterized in that: The performing target detection on the detection frame in real time by using a target detection algorithm includes: Adjust the size of the detection frame according to the size required by the target detection algorithm; Inputting the adjusted detection frame into the backbone network of the target detection algorithm for feature extraction to obtain a feature map; Inputting the feature map into the prediction branch of the object detection algorithm to obtain the position information and confidence of the bounding box; Use the predefined anchor boxes and model settings to set up the decoder to parse the location information and confidence of all bounding boxes; According to the confidence level and through non-maximum suppression, all duplicate boxes in the bounding box are removed to obtain the optimal bounding box of the target, and the optimal bounding box of the target is used as the detection box of the target.

3. The method for completing a detection frame based on frame skipping detection according to claim 1, characterized in that: The feature information includes: the width, height and center point position information of the detection frame.

4. The method for completing a detection frame based on frame skipping detection according to claim 3, characterized in that: The target detection is performed on the detection frame in real time by using a target detection algorithm to obtain feature information of the detection frame of each target in the detection frame, including: The target detection algorithm is used to detect the target in the detection frame of the i-th frame, and the width and height (w i1 ,h i1 )、(w i2 ,h i2 ),…(w im ,h im )…,(w ic ,h ic ) and the center point location information (x i1 ,y i1 )、(x i2 ,y i2 ),…(x im ,y i2 )…,(x ic ,y ic ); Among them, x im Indicates the x-coordinate and y-coordinate of the center point of the detection box of the m-th target in the i-th frame im represents the y coordinate of the center point of the detection box of the mth target in the i-th frame, w im Indicates the width of the detection box of the mth target in the i-th frame, h im Represents the height of the detection box of the mth target in the i-th frame, where m is an integer between 1 and c, and c is the number of targets.

5. The method for completing a detection frame based on frame skipping detection according to claim 4, characterized in that: The step of performing target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of the detection frame of each target in the detection frame also includes: The target detection algorithm is used to detect the target in the detection frame of the i+nth frame, and the width and height (w (i+n)1 ,h (i+n)1 )、(w (i+n)2 ,h (i+n)2 ),…(w (i+n)m ,h (i+n)m )…,(w (i+n)c ,h (i+n)c ) and the center point location information (x (i+n)1 ,y (i+n)1 )、(x (i+n)2 ,y (i+n)2 ),……,(x (i+n)c ,y (i+n)c ); Among them, x (i+n)m Indicates the x-coordinate and y-coordinate of the center point of the detection box of the m-th target in the i+n-th frame. (i+n)m represents the y coordinate of the center point of the detection box of the mth target in the i+nth frame, w (i+n)m Indicates the width of the detection box of the mth target in the i+nth frame, h (i+n)m Indicates the height of the detection box of the mth target in the i+nth frame.

6. The method for completing a detection frame based on frame skipping detection according to claim 5, characterized in that: The method of using the target dynamic interpolation algorithm to calculate the feature information of the candidate frame of the corresponding target in the to-be-interpolated frame between two adjacent detection frames includes: The center point position information (x km ,y km ): The width and height (w km ,h km ): Among them, i <k<i+n。 7. The method for completing a detection frame based on frame skipping detection according to claim 6, characterized in that: The method of using the target dynamic interpolation algorithm to calculate the feature information of the candidate frame of the corresponding target in the to-be-interpolated frame between two adjacent detection frames also includes: Obtaining a predetermined number of frames n when cropping the video stream, and setting the number of frames to be interpolated between two adjacent detection frames to n-1; For each frame to be interpolated, feature information of a candidate frame in a corresponding frame to be interpolated between two adjacent detection frames is calculated using a target dynamic interpolation algorithm according to the temporary trajectory.

8. A detection frame completion device based on frame skipping detection, characterized in that: include: An acquisition unit, used for acquiring an input video stream, and cropping the video stream every predetermined frame to obtain a set of detection frames; A detection unit, configured to perform target detection on the detection frame in real time by using a target detection algorithm to obtain feature information of a detection frame of each target in the detection frame; an aggregation unit, configured to integrate feature information of detection frames of each target in all detection frames in a predetermined order to obtain a temporary trajectory corresponding to the target, and to aggregate the temporary trajectories of the targets into a trajectory cluster; A calculation unit, configured to calculate, for each temporary track in the track cluster, feature information of a candidate box of a corresponding target in a to-be-interpolated frame between two adjacent detection frames using a target dynamic interpolation algorithm; The integration unit is used to integrate the feature information of the detection frame of each target in all the detection frames and the feature information of the candidate frames in all the frames to be interpolated in a predetermined order to obtain a complete motion compensation trajectory of the corresponding target.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for completing the detection frame based on frame skipping detection according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method for completing a detection frame based on frame skipping detection according to any one of claims 1 to 7.

Images