0-1 multi-class object classification dynamic skip frame method for surveillance video data

Abstract

The application relates to the technical field of video analysis, and particularly discloses a dynamic frame skipping method for 0-1 multi-class target classification of monitoring video data. The method fully excavates the corresponding relationship between the change of the maximum and minimum values of the number of multi-class targets and the change trend of the video state, and dynamically adjusts the frame skipping step length according to the corresponding relationship. Compared with the monitoring video data processing mode of frame-by-frame calculation or fixed frame skipping step length, the method can greatly improve the video processing speed while minimizing the loss of the recall rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video analysis, and particularly relates to a 0-1 multi-class target classification dynamic frame skipping method for monitoring video data. BACKGROUND

[0002] Nowadays, cameras are increasingly widely used in people's daily life and have brought important effects to people's work, and have become an indispensable part in daily life. Cameras are widely deployed in various places of life, such as stores, campuses, restaurants and airports, and the like, public places, the main purpose is to realize real-time recording of events in public places, and then real-time and post-event analysis of the recorded information, so as to more quickly and efficiently carry out social management.

[0003] In the related application of monitoring video analysis processing, 0-1 multi-class target classification is an important application direction. The 0-1 multi-target classification in the monitoring video mainly refers to finding the video frames containing multiple types of targets at the same time from the monitoring video, and storing these frames as key frames for subsequent use.

[0004] In the aspects of related departments' crackdown on and prevention of crimes, timely and efficient acquisition of 0-1 multi-class target classification information in the monitoring video can provide relevant clues for law enforcement personnel. For example, finding video frames containing both people and bicycles from the monitoring video helps the police to lock down the car thieves. In addition, finding video frames containing both people and motorcycles from the monitoring video at the non-motor vehicle lane of the traffic intersection is conducive to the relevant departments to find the illegal drivers afterwards.

[0005] In the daily management of shopping malls, supermarkets and the like, timely acquisition of 0-1 multi-class target classification information in real-time monitoring video can help the relevant management personnel to better perform the management functions. For example, real-time feedback of video frames containing both people and lit cigarettes from the monitoring video helps the shopping mall management personnel to find the smokers and timely dispatch the staff to persuade them, so as to maintain a good public environment.

[0006] In the aspect of community management, timely acquisition of 0-1 multi-class target classification information in the monitoring video is conducive to helping the property personnel to carry out relevant management work. For example, finding video frames containing both people and electric vehicles from the monitoring video of the elevator helps the management personnel to find the relevant owners and negotiate with them to perform the relevant regulations of electric vehicles not allowed to enter the elevator.

[0007] Efficient video analysis for 0-1 multi-class target classification plays an important role in various aspects of intelligent security and social governance, and has received extensive attention from researchers in recent years. At the same time, the data generated by monitoring equipment is also massive. In 2020, about half of the 770 million cameras in use worldwide were in China. Therefore, how to obtain effective 0-1 multi-class target classification information from the monitoring videos generated by hundreds of millions of cameras is a great challenge. The main technical difficulties of analyzing massive monitoring video data are:

[0008] Similar video frames often exist in video data, and these similar video frames are easy to cause redundant calculation, thereby increasing the calculation cost and further increasing the calculation time of the video. How to quickly filter out irrelevant video frames and quickly locate the key frames that the user wants is also a challenge in target detection in massive monitoring videos.

[0009] In view of the urgent need for 0-1 multi-class target classification information in real-world applications and the related challenges of massive video processing, the present application proposes a dynamic frame skipping method for 0-1 multi-class target classification of monitoring video data. SUMMARY

[0010] In order to solve the problem of large calculation overhead caused by calculating a large number of similar video frames in the video analysis process, the present application provides a dynamic frame skipping method for 0-1 multi-class target classification of monitoring video data, which can greatly improve the processing speed of the video while minimizing the loss of key frame recall rate.

[0011] To achieve the above purpose, the present application adopts the following technical solutions:

[0012] A dynamic frame skipping method for 0-1 multi-class target classification of monitoring video data, which dynamically adjusts the frame skipping step of the video analysis process based on the different changes of the minimum value minNum of the number of multi-class targets in the video frame and the maximum value maxNum of the number of multi-class targets, including the following steps:

[0013] Step 1: Let M represent the video frame sequence that needs to be detected and initialize it, CheckedObject is the target class set to be detected and is initialized, stepLength is a set of optional frame skipping steps arranged in ascending order, and is initialized.

[0014] Step 2: Select the first frame in the video frame sequence M, judge the state of the video frame, if none of the n types of targets that the user is interested in appears, jump to step 3; if some of the n types of targets that the user is interested in appear, jump to step 4; if all of the n types of targets that the user is interested in appear, jump to step 5.

[0015] Step 3: If the state of the video frame is that none of the n types of objects of interest to the user appears, then the frame skipping step is gradually increased according to the step sequence in stepLength, but not more than the maximum frame skipping step maxStep, until the state of the video changes. If the state of the video frame is that some of the n types of objects of interest to the user appears, then step 4 is executed; if the state of the video frame is that all of the n types of objects of interest to the user appears, then step 5 is executed.

[0016] Step 4: If the state of the video frame is that some of the n types of objects of interest to the user appears, then a minimum frame skipping step minStep is selected from stepLength and used to skip to the next frame, and the change of maxNum compared with the previous frame is calculated. If maxNum continuously increases, then the frame skipping step is gradually increased, but not more than the maximum frame skipping step maxStep. When the frame skipping step reaches maxStep, the subsequent frame skipping step is repeatedly changed between minStep and maxStep. If maxNum continuously decreases, then the frame skipping step is maintained as a fixed frame skipping step conStep until the state of the video changes. If the state of the video frame is that none of the n types of objects of interest to the user appears, then step 3 is executed; if the state of the video frame is that all of the n types of objects of interest to the user appears, then step 5 is executed.

[0017] Step 5: If the state of the video frame is that all of the n types of objects of interest to the user appears, then a minimum frame skipping step minStep is selected from stepLength and used to skip to the next frame, and the change of minNum compared with the previous frame is calculated. If minNum continuously increases, then the frame skipping step is gradually increased, but not more than the maximum frame skipping step maxStep. If minNum decreases, then the frame skipping step is repeatedly changed between minStep and maxStep until the state of the video changes. If the state of the video frame is that none of the n types of objects of interest to the user appears, then step 3 is executed; if the state of the video frame is that some of the n types of objects of interest to the user appears, then step 4 is executed.

[0018] Step 6: Steps 3, 4 and 5 are repeatedly executed in the whole process of the video frame until all the frames in set M are processed.

[0019] Preferably, in step 3, when the state of the video frame is that none of the n types of objects of interest to the user appears, minNum is 0 and maxNum is 0. At this time, the possibility that the next state of the video frame is that all of the n types of objects of interest to the user appears is smaller compared with the possibility that the video frame still remains in the state that none of the n types of objects of interest to the user appears, i.e. the possibility that the frame skipping operation skips some key frames is smaller, so the frame skipping step is gradually increased to speed up the processing of the video, but the step cannot exceed the maximum frame skipping step maxStep.

[0020] Preferably, in step 4, when the video frame is in the state of partial appearance of the n types of targets of interest to the user, minNum=0 but maxNum is not 0, and at this time, maxNum is divided into two cases of increasing and decreasing. When maxNum increases, the next state of the video frame is more likely to be the state of full appearance of the n types of targets of interest to the user or the state of partial appearance of the n types of targets of interest to the user than the state of full non-appearance of the n types of targets of interest to the user. If the step continues to increase, it is possible to skip some key frames, and at this time, the frame skipping step should be gradually reduced or kept unchanged. According to actual needs, in order to speed up the video processing speed and at the same time avoid missing the key frames as much as possible due to too large frame skipping amplitude, the frame skipping step can be repeatedly changed between the minimum value minStep and the maximum value maxStep. When maxNum decreases, the next state of the video frame is more likely to be the state of full non-appearance of the n types of targets of interest to the user than the state of full appearance of the n types of targets of interest to the user or the state of partial appearance of the n types of targets of interest to the user, and at this time, it can be assumed that the likelihood of the three states is equal. Under this assumption, the likelihood of the next state of the video frame being the state of full non-appearance of the n types of targets of interest to the user is increased compared with the case of increasing maxNum, but it is considered that the next state of the video frame is still likely to be the state of full appearance of the n types of targets of interest to the user. In addition, in order to speed up the video processing speed and at the same time reduce the missing of key frames as much as possible, the frame skipping step can be directly set to conStep, and maxStep>=conStep>=minStep, and the step is kept unchanged for frame skipping operation.

[0021] Preferably, in step 5, when the video frame is in the state of full appearance of the n types of targets of interest to the user, minNum is not 0 and maxNum is not 0. When minNum increases, the next state of the video frame is more likely to be the state of full appearance of the n types of targets of interest to the user than the state of full non-appearance of the n types of targets of interest to the user or the state of partial appearance of the n types of targets of interest to the user. At this time, the frame skipping step can be gradually increased, and the purpose is to speed up the video processing speed, at the same time, the frames between the starting id and the ending id of each frame skipping are saved as key frames, which can also maximize the reduction of missing of key frames and improve the recall rate. When minNum decreases, the next state of the video frame is likely to be not the state of full appearance of the n types of targets of interest to the user, and at this time, if the frame skipping step is too large, a large number of non-key frames may be saved, and at this time, the frame skipping step should be gradually reduced or kept unchanged.

[0022] The dynamic frame skipping method for 0-1 multi-type target classification of monitoring video data provided by the present application solves the problem of large computational overhead caused by calculating a large number of similar video frames in the video analysis process. Compared with the frame-by-frame calculation or the fixed frame skipping step video processing method, the method can greatly improve the video processing speed while minimizing the loss of recall rate. Attached Figure Description

[0023] Figure 1 A schematic diagram illustrating the implementation process of a dynamic frame skipping method for 0-1 multi-class target classification of surveillance video data;

[0024] Figure 2 This is a diagram illustrating the 0-1 multi-class target classification process in a dynamic frame-skipping method for 0-1 multi-class target classification of surveillance video data. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. The purpose of the present invention is to solve the problem of high computational overhead caused by calculating a large number of video frames with similar content during video analysis. The present invention first determines the video frame set M, initializes the target set CheckedObject to be queried, and initializes the optional skip step length set stepLength, which is arranged in ascending order.

[0026] refer to Figure 1 and Figure 2 When analyzing specific videos, this application proposes a dynamic frame-skipping method for 0-1 multi-class target classification of surveillance video data. This method uses the different changes in the minimum value minNum and the maximum value maxNum of the number of multi-class targets in the video frame as a reference to dynamically adjust the frame-skipping step size in the video analysis process. Specifically, it performs the following steps:

[0027] Step 1: Let M represent the video frame sequence for which object detection is required and initialize it, CheckedObject be the set of target types to be detected and initialize it, and stepLength be the set of optional frame skipping step lengths, arranged in ascending order, and initialize it.

[0028] Step 2: Select the first frame in the video frame sequence M and determine which state it belongs to. If none of the n types of targets that the user is interested in appear, proceed to step 3; if some of the n types of targets that the user is interested in appear, proceed to step 4; if all of the n types of targets that the user is interested in appear, proceed to step 5.

[0029] Step 3: If the video frame state shows that none of the n types of targets of interest to the user have appeared, then gradually increase the frame skipping step size according to the step size sequence in stepLength, but do not exceed the maximum frame skipping step size maxStep, until the video state changes. If the next state shows that some of the n types of targets of interest to the user have appeared, then proceed to step 4; if the next state shows that all of the n types of targets of interest to the user have appeared, then proceed to step 5.

[0030] In step 3, the video frame state is that none of the n types of objects of interest to the user appears, minNum is 0 and maxNum is 0. At this time, the next state of the video frame is that the possibility of all the n types of objects of interest to the user appearing is compared with the possibility of the video frame still being in the state of none of the n types of objects of interest to the user appearing, i.e. the possibility of the frame skipping operation skipping part of the key frames is small, at this time, the frame skipping step is gradually increased to speed up the processing speed of the video, but the step cannot exceed the maximum frame skipping step maxStep.

[0031] In step 4, the video frame state is that part of the n types of objects of interest to the user appears, a minimum frame skipping step minStep is first selected from stepLength and the next frame is jumped to with this step, and the change of maxNum of the next frame compared with the previous frame is calculated. If maxNum continuously increases, the frame skipping step is gradually increased, but cannot exceed the maximum frame skipping step maxStep. When the frame skipping step reaches maxStep, the subsequent frame skipping step is made to vary between minStep and maxStep. If maxNum continuously decreases, the frame skipping step is maintained at a fixed frame skipping step conStep, until the video state changes. If the next state is that none of the n types of objects of interest to the user appears, step 3 is executed, and if the next state is that all the n types of objects of interest to the user appear, step 5 is executed.

[0032] In step 4, when the frame is partially appeared, minNum = 0 but maxNum is not 0, and maxNum is divided into two cases: increasing and decreasing. When maxNum is increasing, the next state of the video frame is more likely to be all appeared or partially appeared than not appeared. If the step is continuously increased, some key frames will be skipped. At this time, the step should be gradually decreased or kept unchanged. According to the actual needs, in order to speed up the video processing speed, while trying to avoid the missed detection of key frames caused by too large step, the step can be changed between minStep and maxStep. When maxNum is decreasing, the next state of the video frame is not appeared, partially appeared or all appeared, and the possibility cannot be intuitively judged. Here, it is assumed that the possibility of the three states is equal. Under this assumption, the possibility of the next state of the video frame is not appeared is increased than when maxNum is increased, but the next state of the video frame is still likely to be all appeared. In order to speed up the video processing speed, while trying to reduce the missed detection of key frames as much as possible, the step can be directly set to conStep, and maxStep >= conStep >= minStep, and the step is kept for the jump operation.

[0033] Step 5: When the state of the video frame is all appeared, a minimum jump step minStep is selected from stepLength and the next frame is jumped to, and the change of minNum is calculated. If minNum is continuously increased, the jump step is gradually increased, but not more than maxStep. If minNum is decreased, the jump step is changed between minStep and maxStep until the state of the video is changed. If the next state of the video frame is not appeared, step 3 is executed. If the next state of the video frame is partially appeared, step 4 is executed.

[0034] In step 5, when the video frame is in a state that all the n types of targets of interest of the user appear, minNum is not 0 and maxNum is not 0. When minNum increases, the next state of the video frame is more likely to be that all the n types of targets of interest of the user appear than that all the n types of targets of interest of the user do not appear or that part of the n types of targets of interest of the user appear. At this time, the frame skipping step can be gradually increased, and the purpose is to speed up the processing speed of the video. At the same time, the frames between the start id and the end id of each frame skipping are saved as key frames, which can also maximize the reduction of key frame missing and improve the recall rate. When minNum decreases, the next state of the video frame is not likely to be that all the n types of targets of interest of the user appear. At this time, if the frame skipping step is too large, a large number of non-key frames may be saved. At this time, the frame skipping step should be gradually reduced or maintained unchanged.

[0035] Step 6: Steps 3, 4 and 5 are repeatedly executed in the whole process of the video frame until all the frames in the set M are processed.

[0036] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A 0-1 multi-class object classification dynamic skip frame method for monitoring video data, the method taking different changes of the minimum value minNum of the number of multi-class objects in the video frame and the maximum value maxNum of the number of multi-class objects as a reference basis to dynamically adjust the skip frame step of the video analysis process, characterized in that, The method comprises the following steps: Step 1: Let M represent a video frame sequence in which target detection needs to be performed and initialize it, let cheObj represent a target category set to be detected and initialize it, and let stepLength represent an optional frame skipping step length set arranged in ascending order of step length and initialize it; Step 2: Select the first frame in the video frame sequence M, determine which state it belongs to in the video stream, and jump to step 3, step 4, or step 5; Step 3: Determine the state of the video frame, if none of the n types of targets of interest to the user appear, gradually increase the frame skipping step length according to the step length sequence in stepLength, but not more than the maximum frame skipping step length maxStep, until the video frame state changes; if the next state of the video frame is that some of the n types of targets of interest to the user appear, execute step 4; if the next state of the video frame is that all of the n types of targets of interest to the user appear, execute step 5; Step 4: The state of the video frame is that some of the n types of targets of interest to the user appear, first select a minimum frame skipping step length minStep from stepLength and jump to the next frame with this step length, and calculate the change of maxNum compared with the previous frame; if maxNum continuously increases, gradually increase the frame skipping step length, but not more than the maximum frame skipping step length maxStep; When the frame skipping step length reaches maxStep, make the subsequent frame skipping step length vary between minStep and maxStep; If maxNum continuously decreases, keep the frame skipping step length unchanged at a fixed frame skipping step length conStep until the video frame state changes; if the next state of the video frame is that none of the n types of targets of interest to the user appear, execute step 3; if the next state of the video frame is that all of the n types of targets of interest to the user appear, execute step 5; Step 5: The state of the video frame is that all of the n types of targets of interest to the user appear, select a minimum frame skipping step length minStep from stepLength and jump to the next frame with this step length, and calculate the change of minNum compared with the previous frame; if minNum continuously increases, gradually increase the frame skipping step length, but not more than the maximum frame skipping step length maxStep; if minNum decreases, make the frame skipping step length vary between minStep and maxStep until the video state changes; if the next state of the video frame is that none of the n types of targets of interest to the user appear, execute step 3; if the next state of the video frame is that some of the n types of targets of interest to the user appear, execute step 4; Step 6: Repeat steps 3, 4, and 5 throughout the process of the video frame until all frames in the set M are processed.

2. The 0-1 multi-class object classification dynamic frame skipping method for surveillance video data according to claim 1, wherein, When the state of the video frame in step 3 is that none of the n types of targets of interest to the user appear, minNum is 0 and maxNum is 0, at this time, gradually increase the frame skipping step length to speed up the processing of the video, but the step length is not more than the maximum frame skipping step length maxStep.

3. The 0-1 multi-class object classification dynamic frame skipping method for surveillance video data according to claim 1, wherein, When the state of the video frame in step 4 is that some of the n types of targets of interest to the user appear, minNum = 0 but maxNum is not 0.

4. The 0-1 multi-class object classification dynamic frame skipping method for surveillance video data according to claim 1, wherein, In step 5, minNum is non-zero and maxNum is non-zero when the video frame is at the point where all n-class objects of interest are present.

Images