A dynamic sparse key frame video target detection method and device and a storage medium

Abstract

The application discloses a dynamic sparse key frame video target detection method and device and a storage medium. The application utilizes high correlation between video frames, proposes a real-time video target detection post-processing method based on key frame target matching and interpolation prediction of non-key frames by using a matching result, adaptively adjusts a key frame interval through displacement speed of inter-frame targets, and calculates a target frame matching score and histogram similarity between video frames to constrain a time when a post-processing algorithm takes effect, so that the video target detection speed can be greatly improved under the condition of sacrificing a small accuracy. The method can be widely applied to all static target detectors.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a method, device and storage medium for detecting targets in dynamic sparse keyframe videos. Background Art

[0002] With the continuous development of artificial intelligence, deep learning models, represented by convolutional neural networks, have been widely studied in computer vision. Object detection, a hot research area in computer vision, has broad applications in areas such as facial recognition, autonomous driving, intelligent security, medical assistance, and traditional industries. Existing object detection algorithms can be further categorized into two types: two-stage object detection and one-stage object detection. The former first generates candidate bounding boxes and then performs classification, with representative algorithms such as Faster-RCNN and Mask-RCNN. The latter directly regresses the object's category and location, such as the YOLO series of algorithms. To meet the frame rate requirements of real-time object detection, neural network algorithms such as YOLO and MobileNet are often used due to their high speed and low parameter count.

[0003] With the help of graphics processing units (GPUs), the inference speed of the above-mentioned target detection algorithms can already meet real-time requirements. However, with the development of Internet of Things technology, the demand for deploying target detection algorithms on various edge computing devices is increasing. Due to many limitations such as cost and computing power, the deployment of existing target detection algorithms still exceeds the capabilities of most edge device hardware, especially for video target detection, which has higher real-time requirements. Compared with static images, videos have rich context and temporal information that can be utilized. Within a certain frame interval, the position and appearance of the target will not change drastically. Therefore, the characteristics of videos can be fully utilized to improve the performance of existing target detectors. Currently, there are several main methods to improve the speed of video target detection: (1) Only perform complex neural network inference on sparse key frames, and use optical flow networks to calculate the optical flow of targets in non-key frames, guide the features of key frames to propagate to non-key frames, or redesign new small networks for non-key frames. However, the amount of calculation is also large. In the case of changing backgrounds, the optical flow network is also prone to information loss, which reduces the accuracy. (2) Post-process the static target detection results. For example, T-CNN combines detection and tracking, uses the target tracking algorithm to track the generated candidate boxes to generate link sequences, and re-scores the detection results. Seq Bbox Matching uses the union intersection between target boxes to match keyframe targets, generates tubelets by linking across video frames, and uses these links to optimize detection results.

[0004] In summary, most existing methods are based on offline videos and require inference information from current and future frames. They are less capable of handling target detection in real-time video streams, and the implementation process is relatively complex. In complex and changeable scenarios such as actual monitoring, the performance of the algorithm will also be greatly affected. Summary of the Invention

[0005] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, device and storage medium for dynamic sparse key-frame video object detection.

[0006] The technical solution adopted in the present invention is:

[0007] A method for detecting objects in dynamic sparse key-frame videos, comprising the following steps:

[0008] S1. Initialize detection parameters, including key frame interval value d, target frame matching threshold and image similarity threshold;

[0009] S2, acquiring the t-th frame image in the video data in real time, and determining whether the t-th frame image is a key frame;

[0010] S3. If the t-th frame image is a key frame, perform target detection on the key frame using a preset network model to obtain a first target detection result; match the obtained first target detection result with the pre-saved second target detection result to obtain a target frame matching similarity, and determine whether the target frame matching similarity is higher than the target frame matching threshold;

[0011] S5. If the target frame matching similarity is greater than or equal to the target frame matching threshold, predict and save target frame detection results for a plurality of frames between the tth frame and the t+dth frame according to the first target detection result and the second target detection result; and update the second target detection result according to the first target detection result.

[0012] S6. If the target frame matching similarity is lower than the target frame matching threshold, save the first target detection result, reinitialize the detection parameters, obtain the next frame image, and return to step S2;

[0013] S7. If the t-th frame image is not a key frame, calculate the histogram similarity between the t-th frame image and the t-1-th frame image, and determine whether the histogram similarity is higher than the image similarity threshold;

[0014] S8. If the histogram similarity is higher than or equal to the image similarity threshold, obtain the detection result of the t-th frame according to the saved target frame detection result;

[0015] S9. If the histogram similarity is lower than the image similarity threshold, use the preset network model to perform target detection on the t-th frame, obtain and save the target detection result; reinitialize the detection parameters, obtain the next frame image and return to execute step S2.

[0016] Furthermore, the target frame matching similarity is calculated in the following way:

[0017] match score = ∑IoU ij *(conf i +conf j ) / l

[0018] Among them, IoU ij Indicates the IoU value of the target box of the two frames, conf i Indicates the confidence score of the target box detected in the first frame image, conf j represents the confidence score of the target box detected in the second frame image, and l is the number of matched targets between the two images.

[0019] Furthermore, before the step of predicting and saving target frame detection results of a plurality of frames of images between the tth frame and the t+dth frame according to the first target detection result and the second target detection result, the following steps are also included:

[0020] Calculate the histogram similarity between the t-th frame image and the saved last key frame image;

[0021] If the calculated histogram similarity is greater than or equal to the image similarity threshold, then executing the step of predicting and saving target frame detection results of several frames of images between the tth frame and the t+dth frame according to the first target detection result and the second target detection result;

[0022] If the calculated histogram similarity is lower than the image similarity threshold, the first target detection result is saved, the detection parameters are reinitialized, the next frame image is acquired, and the process returns to step S2.

[0023] Furthermore, the calculation formula of the histogram similarity is:

[0024]

[0025] Among them, H1 and H2 represent the histograms of two frames of images. Represents the mean of the histogram data, H1(I), H2(I) represent the histogram data matrix, and N represents the number of histogram sub-segment bins.

[0026] Furthermore, predicting and saving target frame detection results of a plurality of frames of images between the tth frame and the t+dth frame according to the first target detection result and the second target detection result includes:

[0027] Assume that the second target detection result is: The first target detection result is m, n are the number of target boxes respectively, where is the center point coordinate of the target frame, is the width of the target box, is the height of the target frame;

[0028] Calculate the coordinates of the center point of the target box in the next k frame:

[0029]

[0030]

[0031] Where N is the number of matching target boxes, and d2 is the last key frame interval;

[0032] The target frame detection result of the future k-th frame image is obtained based on the calculated coordinates of the center point of the target frame and the width and height of the target frame in the first target detection result or the second target detection result.

[0033] Furthermore, the step S5 further includes the step of adjusting the key frame interval value:

[0034] Calculate the average single-frame displacement Δw of the center position of the matching target box:

[0035]

[0036] The average single-frame displacement Δw is compared with a preset motion speed value, and the key frame interval value is increased or decreased according to the comparison result.

[0037] Furthermore, the initialization detection parameters include:

[0038] The preset network model is used to perform target detection on the first d frames of images, and the target detection results are saved.

[0039] Furthermore, after the preset network model is used to perform target detection on the image, the following steps are also included:

[0040] The NMS algorithm is used to process the detected target frame to obtain the category, confidence, target frame center point coordinates, width and height of each target as the target detection result.

[0041] When predicting the target detection results of non-key frames, the width and height of the target box remain unchanged, and the confidence and category adopt the maximization strategy. The predicted target detection results are

[0042] Another technical solution adopted by the present application is:

[0043] A dynamic sparse key frame video target detection device comprises:

[0044] At least one processor;

[0045] At least one memory for storing at least one program;

[0046] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0047] Another technical solution adopted by the present application is:

[0048] A computer readable storage medium, wherein a processor executable program is stored, the processor executable program is used for executing the above method when executed by a processor.

[0049] The present application has the beneficial effects that: the present application utilizes the high correlation between video frames, proposes a real-time video target detection post-processing method based on key frame target matching and non-key frame interpolation prediction using the matching result, and calculates the target frame matching score and the histogram similarity between video frames to constrain the timing of the post-processing algorithm to take effect; the present application method can be applied to all static target detectors, and can greatly improve the video target detection speed with a small sacrifice in accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0051] Figure 1 is a flow chart of a dynamic sparse key frame video target detection method in the embodiments of the present application;

[0052] Figure 2 is a schematic diagram of a target frame matching algorithm in the embodiments of the present application;

[0053] Figure 3 is a framework diagram of a dynamic sparse key frame video target detection method in the embodiments of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0056] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0057] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0058] To address existing technical issues, designing a low-complexity video target detection method that meets real-time requirements is key to solving the problem of insufficient computing power in edge computing devices. This method combines existing advanced static target detection algorithms. By calculating the displacement velocity of targets between matching video frames, it dynamically and adaptively determines the interval between key frames. Non-key frame detection results are derived using a bilinear interpolation algorithm. Inter-frame similarity evaluation is also incorporated to constrain the prediction results, reducing the impact of sudden changes in video scenes. This method can significantly improve the speed of video target detection with minimal computational cost and accuracy loss, and is applicable to all static target detectors.

[0059] like Figure 1 and Figure 3 As shown, this embodiment provides a method for detecting objects in dynamic sparse keyframe videos, comprising the following steps:

[0060] S1. Initialize detection parameters, including key frame interval value d, target frame matching threshold and image similarity threshold;

[0061] S2, acquiring the t-th frame image in the video data in real time, and determining whether the t-th frame image is a key frame;

[0062] S3. If the t-th frame image is a key frame, perform target detection on the key frame using a preset network model to obtain a first target detection result; match the obtained first target detection result with the pre-saved second target detection result to obtain a target frame matching similarity, and determine whether the target frame matching similarity is higher than the target frame matching threshold;

[0063] S5. If the target frame matching similarity is greater than or equal to the target frame matching threshold, predict and save target frame detection results for a plurality of frames between the tth frame and the t+dth frame according to the first target detection result and the second target detection result; and update the second target detection result according to the first target detection result.

[0064] S6. If the target frame matching similarity is lower than the target frame matching threshold, save the first target detection result, reinitialize the detection parameters, obtain the next frame image, and return to step S2;

[0065] S7. If the t-th frame image is not a key frame, calculate the histogram similarity between the t-th frame image and the t-1-th frame image, and determine whether the histogram similarity is higher than the image similarity threshold;

[0066] S8. If the histogram similarity is higher than or equal to the image similarity threshold, obtain the detection result of the t-th frame according to the saved target frame detection result;

[0067] S9. If the histogram similarity is lower than the image similarity threshold, use the preset network model to perform target detection on the t-th frame, obtain and save the target detection result; reinitialize the detection parameters, obtain the next frame image and return to execute step S2.

[0068] The above method is explained in detail below with reference to specific embodiments.

[0069] This embodiment is based on a static target detector. After weighing the speed and accuracy of existing target detection algorithms, the single-stage target detection algorithm yolov4 can be used as the benchmark model. Similar to existing work, the embodiment of the present invention uses two standard ImageNet VID and ImageNet DET data sets for training and testing. The ImageNet VID data set contains 3862 training sets and 555 validation sets. There are 30 categories in the data set. ImageNet DET is a large target detection data set with 200 categories. We only use data with the same categories as ImageNet VID. Taking into account the size of the data set and the redundancy of the video, we only randomly extracted data from 12 categories, and randomly extracted 1 / 10 frames from each video for training. Through training, a basic target detection model can be obtained. Specifically, the opencv tool can be used to obtain the video frame I at the tth moment. t Information serves as input to the model.

[0070] See also Figure 1 First, we set the initial value of the dynamic keyframe interval and the thresholds for target frame matching and image similarity scores. In this embodiment, the initial value of the keyframe interval is set to d = 3, and the thresholds for target frame matching and image similarity scores are 0.1 and 0.5, respectively. To perform real-time post-processing, this embodiment only uses the detection results of the current frame and historical frames. In the initial target frame matching, the first d frames are inferred using the basic target detector. When the frame counter reaches the keyframe interval, the current t-th frame is determined to be a keyframe and inferred using the basic target detector. After the NMS algorithm, the detection results of each target are obtained, including the category, confidence, target frame center coordinates, width and height, recorded as (class, conf, x, y, w, h).

[0071] Each time a key frame arrives, we save the key frame detection results as data for the next key frame matching. Subsequently, we match the target frame of the current t-th frame with the td-th frame in the history. The target frame matching algorithm is as follows:

[0072] The target frame information of the td frame is recorded as:

[0073]

[0074] The target frame information of the tth frame is recorded as:

[0075]

[0076] Among them, m, n are the number of target boxes respectively,

[0077] Calculate the intersection-over-union matrix of the target box:

[0078]

[0079] The intersection over union (IoU) is used to describe the degree of overlap between two frames. The larger the IoU, the higher the degree of overlap. The above steps match the target frames between the two frames and calculate their IoU. The calculated IoU matrix is ​​used as the distance matrix for target frame matching. Since the target frame matching can be approximated to the maximum weight matching problem of the bipartite graph, the KM algorithm can be used to calculate the final matching pair result.

[0080] like Figure 2 As shown in the figure, due to the complexity and variability of actual scenes, after obtaining the matching results, the detection results between key frames may also vary greatly. Judging whether the targets are completely matched based solely on the IoU value is sometimes unreliable. Therefore, the method of this embodiment weights the IoU value of the matching target with each confidence score to calculate the matching score for this time, which is used to further constrain the matching results, as shown below:

[0081] match score=∑IoU ij *(conf i +conf j ) / l (2)

[0082] Among them, l is the number of targets matched between key frames, conf i ,conf j are the confidences of the detection boxes on the two images respectively.

[0083] After obtaining the matching information of the key frames, the image similarity evaluation module of this embodiment is introduced next. In real-world scenarios, there may be sudden changes in scenes or objects between consecutive video frames. At this time, the results of key frame matching or non-key frame prediction will be wrong, thereby affecting the detection accuracy. To prevent such situations, this embodiment uses image histogram similarity with low computational complexity as an evaluation constraint. When the current frame is a key frame, the similarity between the histogram of the tn-th frame image and the current t-th frame image is calculated. The specific steps are as follows:

[0084] 1) Calculate the histogram matrices H1 and H2 of the two frames.

[0085] 2) Calculate the bus distance of the histogram:

[0086]

[0087] Distance is the histogram similarity score between the two frames. If either the matching confidence or the similarity score falls below the set threshold, the subsequent dynamic keyframe interpolation module is abandoned and the initialization operation is re-entered. Once the above two constraints are met, the dynamic keyframe interval d is selected and the prediction results for the next d frames are calculated and saved.

[0088] Based on the continuity of the context of the video frames, it can be considered that in a sequence of continuous frames with very short intervals, the object is approximately moving at a constant speed, and the position information of the object changes very little. If all frames use the basic object detector to infer, a large amount of useless calculation will be caused, and the speed of video object detection will be reduced. Similar to the previous video object detection method, the embodiment discards the inference of the basic object detector on the non-key frame, and further designs a low-complexity dynamic key frame interpolation method. According to the detection results (class, conf, x, y, w, h) of the matching target frame and the t-th frame key frame, the average single-frame displacement Aw of the matching target frame center position can be calculated:

[0089]

[0090] wherein N is the number of matching target frames, d2 is the interval of the previous key frame, and the interval of the key frame is dynamically selected according to the size of Aw.

[0091] As an optional implementation, the object motion speed is divided into three levels: slow, middle and fast according to the displacement size. The target under slow motion has a relatively fixed position, so the interval frame size can be set larger. For the target under fast motion, the position changes greatly. If the fixed key frame interval method is used, when the key frame interval is set larger, it is easy to cause target drift and other error prediction results. According to the object motion speed, the interval of the key frame is limited in the corresponding three speed regions. The interval frame value is updated each time the key frame arrives. In the case of improving the video object detection speed, the precision loss will be less. The initial key frame threshold of the method is set to 3, and the interval values corresponding to the three speed levels are 1, 2 and 3 respectively. The threshold value of the key frame interval is not limited to 3, and the threshold value can be changed according to the actual scene needs. When the initial threshold value is set larger, the detection speed will be faster, but the detection accuracy will also be reduced to a certain extent. The speed and accuracy need to be balanced according to the use scene.

[0092] Further, the key frame interpolation strategy is as follows:

[0093] The position of the target frame center point of the future d cur -1 frame is calculated:

[0094]

[0095] The width and height of the target frame remain unchanged, and the confidence and class adopt the maximum strategy, and the matching frame target frame confidence score is selected as the score and class of the prediction frame. Finally, the predicted result is:

[0096]

[0097] Save and output the detection results of the current frame.

[0098] When a non-keyframe arrives, the image similarity evaluation module is first entered, which calculates the histogram similarity between the current frame and the previous frame using the same calculation method as above. When the similarity score exceeds the set value, the frame is considered to be in the same scene as the keyframe, and the prediction results stored in the keyframe interpolation module are directly queried without the need for complex neural network inference. When the similarity score falls below the set value, the prediction results of the current non-keyframe are discarded, and the basic object detector is used to re-detect it, re-initialize the parameters, and enter the next keyframe matching. At this point, the image detection process for one frame in the real-time video stream is complete, and the current step can be repeated for subsequent video frames.

[0099] The method in this embodiment was trained and tested on a subset of the ImageNet-VID dataset. The initial threshold for the keyframe interval was 3, the target box matching score was set to 0.1, and the image similarity threshold was 0.5. The test results are shown in Table 1. The detection speed was increased by about 70%, while the accuracy was only reduced by about 1%.

[0100] Table 1

[0101] methods mAP (%) Run times (ms) yolov3 74.33 17.5 yolov3+ dynamic keyframe strategy 73.16 9.4 yolov4 74.45 34.5 yolov4+ dynamic keyframe strategy 73.21 24.6

[0102] In summary, based on the existing basic target detectors, the method of the present invention designs a lightweight real-time video target detection post-processing algorithm. The basic target detectors used in the method of the present invention include but are not limited to the Yolo series algorithms, which can theoretically be applied to any target detection algorithm. The designed dynamic key frame matching module and image similarity constraint module can greatly improve the video target detection speed with less loss of accuracy. The deployment method is simple and can be well applied to target detection in edge computing devices. In addition, the key frame interval and the similarity threshold can be selectively adjusted according to the actual usage scenario, so as to achieve a better balance between accuracy and speed.

[0103] This embodiment also provides a dynamic sparse key frame video object detection device, including:

[0104] at least one processor;

[0105] at least one memory for storing at least one program;

[0106] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0107] The dynamic sparse key frame video target detection device of the embodiment can execute the dynamic sparse key frame video target detection method provided by the method embodiment of the application, can execute the implementation steps of any combination of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0108] The application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method shown in the method embodiment of the application. Figure 1

[0109] The embodiment further provides a storage medium storing instructions or programs that can execute the dynamic sparse key frame video target detection method provided by the method embodiment of the application. When the instructions or programs are executed, the implementation steps of any combination of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method are possessed.

[0110] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0111] In addition, although the application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be within the routine skill of the engineer, given the teachings of the present disclosure. Thus, the application illustrated in the claims is enabled without undue experimentation by those skilled in the art using ordinary skill, given the teachings of the present disclosure. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the application, which is determined by the full scope of the appended claims and their equivalents. ​

[0112] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0114] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0115] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their equivalents, can be employed for implementation: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0116] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0117] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0118] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A dynamic sparse key frame video target detection method, characterized in that: The following steps are involved: S1. Initialize detection parameters, including key frame interval value d, target frame matching threshold and image similarity threshold; S2, acquiring the t-th frame image in the video data in real time, and determining whether the t-th frame image is a key frame; S3. If the t-th frame image is a key frame, use a preset network model to perform target detection on the key frame to obtain a first target detection result; Matching the obtained first target detection result with the pre-saved second target detection result to obtain a target frame matching similarity, and determining whether the target frame matching similarity is higher than the target frame matching threshold; S5. If the target frame matching similarity is greater than or equal to the target frame matching threshold, predict and save target frame detection results for a plurality of frames between the tth frame and the t+dth frame according to the first target detection result and the second target detection result; and update the second target detection result according to the first target detection result. S6. If the target frame matching similarity is lower than the target frame matching threshold, save the first target detection result, reinitialize the detection parameters, obtain the next frame image, and return to step S2; S7. If the t-th frame image is not a key frame, calculate the histogram similarity between the t-th frame image and the t-1-th frame image, and determine whether the histogram similarity is higher than the image similarity threshold; S8. If the histogram similarity is higher than or equal to the image similarity threshold, obtain the detection result of the t-th frame according to the saved target frame detection result; S9. If the histogram similarity is lower than the image similarity threshold, use the preset network model to perform target detection on the t-th frame, obtain and save the target detection result; reinitialize the detection parameters, obtain the next frame image and return to step S2; The method of predicting and saving target frame detection results of a plurality of frames of images between the tth frame and the t+dth frame according to the first target detection result and the second target detection result includes: Assume that the second target detection result is: The first target detection result is m, n are the number of target boxes respectively, where is the center point coordinate of the target frame, is the width of the target box, is the height of the target frame; Calculate the coordinates of the center point of the target box in the next k frame: Where N is the number of matching target boxes, and d2 is the last key frame interval; Obtain the target frame detection result of the kth frame image in the future according to the calculated coordinates of the center point of the target frame and the width and height of the target frame in the first target detection result or the second target detection result; The step S5 further comprises the step of adjusting the key frame interval value: Calculate the average single-frame displacement Δw of the center position of the matching target box: The average single-frame displacement Δw is compared with a preset motion speed value, and the key frame interval value is increased or decreased according to the comparison result.

2. The method for detecting target in dynamic sparse keyframe video according to claim 1, wherein: The target frame matching similarity is calculated in the following way: match score=∑IoU ij *(conf i +conf j ) / l Among them, IoU ij Indicates the IoU value of the target box of the two frames, conf i Indicates the confidence score of the target box detected in the first frame image, conf j represents the confidence score of the target box detected in the second frame image, and l is the number of matched targets between the two images.

3. The method for detecting target in dynamic sparse keyframe video according to claim 1, wherein: Before the step of predicting and saving target frame detection results of a plurality of frames of images between the tth frame and the t+dth frame according to the first target detection result and the second target detection result, the following steps are also included: Calculate the histogram similarity between the t-th frame image and the saved last key frame image; If the calculated histogram similarity is higher than or equal to the image similarity threshold, the step of predicting and saving the target frame detection results of several frames of images between the tth frame and the t+dth frame based on the first target detection result and the second target detection result is executed; if the calculated histogram similarity is lower than the image similarity threshold, the first target detection result is saved, the detection parameters are reinitialized, the next frame of image is obtained, and the process returns to step S2.

4. A method for detecting target in dynamic sparse keyframe video according to claim 1 or 3, characterized in that: The calculation formula of the histogram similarity is: Among them, H1 and H2 represent the histograms of two frames of images. Represents the mean of the histogram data, H1(I), H2(I) represent the histogram data matrix, and N represents the number of histogram sub-segment bins.

5. The method for detecting target in dynamic sparse key-frame video according to claim 1, wherein: The initialization detection parameters include: The preset network model is used to perform target detection on the first d frames of images, and the target detection results are saved.

6. The method for detecting target in dynamic sparse key-frame video according to claim 1, wherein: After using the preset network model to detect the target in the image, the following steps are also included: The NMS algorithm is used to process the detected target frame to obtain the category, confidence, target frame center coordinates, width and height of each target as the target detection result; When predicting the target detection results of non-key frames, the width and height of the target box remain unchanged, and the confidence and category adopt the maximization strategy. The predicted target detection results are 7. A dynamic sparse key frame video target detection device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images