Methods for capturing and identifying animal behavior segments of interest in long-term video surveillance

Abstract

This invention discloses a method for capturing and identifying animal behavior segments from long-term video surveillance. The method includes: acquiring live video stream data; using a motion-sensitive detector to discover meaningful video segments and footage; outputting a processed list of motion regions and writing it to a log; constructing a SlowFast semantic analysis model; segmenting the video data and recording timestamp information; obtaining video frame groups corresponding to individual video segments; obtaining the semantics of each video segment; classifying the captured motion footage into background and various material categories, outputting different timestamp information of the motion occurrences, writing it to a log, and obtaining animal motion clips, submitting the information to a backend server for post-processing and scheduling; and driving the camera to rotate based on the motion category results. This method can help media professionals acquire animal motion footage and reduce the workload of secondary editing of large amounts of video.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and relates to a video content analysis method, in particular to a method for capturing and identifying interesting animal behavior segments for long-time video monitoring. BACKGROUND

[0002] With the continuous development of video processing technology and the emergence of various live media streams, video content analysis methods have attracted the interest of the academic and industrial communities. The relatively mature products have been successfully applied to various monitoring systems to realize automatic detection and automatic alarm functions. However, for animal monitoring videos, due to the scarcity and variety of data, the current mainstream animal media programs or live broadcasts still need manual analysis and editing, and the use of artificial intelligence algorithms for event detection is still a challenging problem in the field, which has the following technical difficulties:

[0003] 1. For 24-hour live signals, there are long redundant segments in the monitoring video, and the collected pictures have influences such as light, shadow, and occlusion, and the quality of the video pictures is uneven. This limits the deployment and application of algorithms in complex daily videos.

[0004] 2. The current research goal of motion analysis and understanding at home and abroad is mainly based on humans, and there is still a lack of mature technology and data support for animal motion analysis.

[0005] 3. The same action has different forms in different animal species, such as "tiger running" and "rabbit running", which leads to the fact that animal actions are individually divided into a category, greatly increasing the total number of action categories to be detected, and greatly increasing the difficulty of action classification. SUMMARY

[0006] In order to overcome the above-mentioned deficiencies in the prior art, the application provides a method for capturing and identifying interesting animal behavior segments for long-time video monitoring. This method can help media practitioners obtain animal motion materials and reduce the workload of media practitioners for secondary editing of a large number of videos.

[0007] The purpose of the application is achieved by the following technical solutions:

[0008] A method for capturing and identifying interesting animal behavior segments for long-time video monitoring, comprising the following steps:

[0009] Step S1: acquiring video live stream video data;

[0010] Step S2: proposing an action-sensitive detector based on motion region attention, using the action-sensitive detector to discover meaningful video segments and materials, and positioning the positions of the targets with significant semantics in the shots;

[0011] Step S3: Output the processed list of motion regions and write it to the log;

[0012] Step S4: Construct the semantic analysis model of SlowFast, which includes a slow channel and a fast channel, wherein:

[0013] The slow channel is used to extract spatial semantic information, and its input is low-frame-rate sampled video frames;

[0014] The fast channel is used to extract temporal semantic information, and its input is high-frame-rate sampled video frames;

[0015] The output of the fast channel is sent to the slow channel through a lateral connection to achieve information interaction between the two channels and generate more effective video semantic analysis results.

[0016] Step S5: Segment the video data from step S1 to obtain one or more video segments, and record the timestamp information of the one or more video segments;

[0017] Step S6: Perform frame extraction on the one or more video segments to obtain a group of video frames corresponding to a single video segment;

[0018] Step S7: Input the video frame group corresponding to the one or more video segments into the SlowFast model constructed in step S4 to obtain the semantics of each video segment;

[0019] Step S8: Using semantic information, classify the motion images captured in step S3 into background and various material categories, output different timestamp information of the motion occurrence, write it into the log, obtain the animal motion clip, and submit the information to the backend server for post-processing and scheduling.

[0020] Step S9: Drive the camera to rotate based on the motion category result from step S8.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] 1. The motion-sensitive detector proposed in this invention can efficiently identify video semantics, filter out other redundant information, and has a fast processing speed and real-time performance. For captured motion segments, the motion-sensitive detector can achieve a recall rate of over 95%.

[0023] 2. The Top-5 accuracy of the deep video semantic analysis model based on SlowFast proposed in this invention can reach over 90%.

[0024] 3. This invention can help media professionals obtain animal movement materials, reduce the workload of secondary editing of a large number of videos, and provide a new approach for subsequent animal movement capture and behavior recognition. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method for capturing and identifying animal behavior segments of interest in long-term video surveillance according to the present invention;

[0026] Figure 2 This is a structural diagram of a motion-sensitive detector based on motion region focus;

[0027] Figure 3 This is a schematic diagram of a video semantic analysis network structure based on SlowFast.

[0028] Figure 4 This is a schematic diagram showing the relationship between noise suppression and the motion region;

[0029] Figure 5 This is a schematic diagram of the capture and identification results of behavioral fragments of animals of interest. Detailed Implementation

[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0031] This invention provides a method for capturing and identifying animal behavior segments of interest in long-term video surveillance. It integrates key artificial intelligence technologies for the acquisition, transmission, distribution, and production of video information, and constructs an intelligent video semantic analysis system using image processing and deep learning algorithms. This system intelligently acquires animal movement elements from massive amounts of video data and accurately locates the animal targets within them. The resulting target location information and motion semantics can be applied to other downstream tasks, such as driving surveillance camera movement and intelligent editing and pushing, saving media professionals the workload of secondary editing. Figure 1 As shown, the method includes the following steps:

[0032] Step S1: Obtain video data from the live video stream.

[0033] In this step, the live video stream is obtained by calling the provided interface.

[0034] Step S2: Propose a motion-sensitive detector based on motion region focus to discover meaningful video clips and footage, and simultaneously locate the semantically significant targets within the shot. For example... Figure 2 As shown, the specific steps are as follows:

[0035] Step S21: Perform grayscale image processing.

[0036] In this step, the OpenCV-python library is used for image grayscale conversion.

[0037] Step S22: Use foreground detection based on the existing KNN model to remove noise with small variations. The specific steps are as follows:

[0038] Step S221: For a new pixel value at a certain location in the image, compare it with the historical information of that pixel value (including the pixel values ​​of the previous few frames and the judgment of whether the pixel is foreground or background). If the difference between the pixel values ​​is within a set threshold (which needs to be adjusted according to the scene and resolution), then the new pixel value is considered to match the historical information and is in the "potential" category; otherwise, it is not in the "potential" category.

[0039] Step S222: After all the historical information in step S221 has been compared, if the number of times it matches the historical information exceeds a set threshold, then the new pixel is classified as a "potential background point". If the number of background points in the matched historical information exceeds a set threshold, then the new pixel is classified as a background point.

[0040] Step S223: Update the historical information with the new pixel.

[0041] Step S23: Use dilation and erosion operators, region drawing, etc. to connect adjacent moving regions together, and save the list of regions that have detected motion.

[0042] In this step, after finding the foreground using KNN, an image boundary extraction algorithm is used to find the boundaries of the foreground image. Due to the influence of light and other perturbations, the extracted boundaries include not only the boundaries of the target object but also noise boundaries. Therefore, suppressing noise boundaries and associating the true moving regions is important. This step mainly uses dilation and erosion operators for noise reduction and uses region rendering to associate the moving regions. The visualization effect is as follows: Figure 4 As shown. Figure 4 In the image, the top left shows the initial motion region drawing, the top right shows the first denoising step, the bottom left shows the motion region association result, and the bottom right shows the result of the detection box on the image.

[0043] Step S24: To prevent abnormal frames caused by drastic changes in illumination, the sensitive motion detector detects abnormal frames by judging the motion gradient between related frames, the slope of the area change of related frames, the ratio of the frame motion area to the average motion area of ​​the related frames, and the magnitude of the change in the number of frame regions.

[0044] Step S25: If an abnormally changing frame appears in S24, the frame is considered a non-action frame; otherwise, it is considered a normal frame. Whether the frame is a non-action frame depends on the detection result obtained in S23.

[0045] Step S3: Output the processed list of motion regions and write it to the log.

[0046] In this step, for frames judged to be normal motion: the KNN detection results are used directly; for frames judged to have huge changes in motion area: the existing intersection operation is used in the motion area of ​​the preceding and following frames, and the frame is judged to be affected by other factors such as occlusion and lighting based on the correlation ratio threshold.

[0047] Step S4: Construct the semantic analysis model for SlowFast.

[0048] like Figure 3 As shown, the sparse frame sampling branch on the semantic analysis model structure of SlowFast is the "Slow Pathway" in the SlowFast model. The "Slow Pathway" mainly focuses on extracting spatial semantic information, and its input is low-frame-rate sampled video frames (in the experiment, it was set to skip 16 frames per sampling, which means only 2 frames are sampled per second if the video is at 30fps). Low frame-rate sampling means that this "Slow Pathway" uses a video frame sequence with a large temporal span. Therefore, the "Slow Pathway" is not sensitive to temporal changes in the video, but is better at capturing the spatial information of video frames. This is because the "Slow Pathway" has more 3D convolutional kernels.

[0049] Figure 3 The branch below is the "Fast Pathway" in the Slow-Fast model. The Fast Pathway primarily focuses on extracting temporal semantic information. Its input is sampled video frames at a high frame rate (experimentally set to α times that of the Slow Pathway, typically 8). Even when using higher temporal resolution as input, the same high temporal resolution is maintained as output, without temporal downsampling. Furthermore, to ensure this path focuses on temporal information extraction, the Fast Pathway model reduces computation in the video space during design; its convolutional channels (determined by the number of kernels) are typically set to β times that of the Slow Pathway, usually 1 / 8.

[0050] To achieve information exchange between the "fast channel" and the "slow channel," this invention uses lateral connectivity, a common algorithm in computer vision, such as fusing features from different layers in feature pyramids for 2D image object detection; a similar fusion strategy is also employed in Two Stream, a mainstream method for video detection. Here, as in the structure... Figure 3As shown, the output of the fast channel is sent to the slow channel through a lateral connection, realizing information interaction between the two channels and producing more effective video semantic analysis results.

[0051] Step S5: Segment the video data from step S1 to obtain one or more video segments, and record the timestamp information of the one or more video segments.

[0052] In this step, the video data is divided into several 70-second video segments, and timestamp information is recorded for subsequent assembly of the video segments.

[0053] Step S6: Perform frame extraction on the one or more video segments to obtain a group of video frames corresponding to a single video segment.

[0054] In this step, for animal videos, which consist mostly of background and less foreground, there is little variation between adjacent frames. Frame extraction can reduce the computational load of the model and highlight the information contained in the video frames.

[0055] Step S7: Input the video frame group corresponding to the one or more video segments into the trained SlowFast model to obtain the semantics of each video segment.

[0056] Step S8: Using semantic information, classify the motion images captured in Step S3 into background and various material categories, output different timestamp information of the motion occurrences, write it into the log, and obtain the animal motion clip. The recognition results are as follows: Figure 5 As shown. Furthermore, the obtained logs and animal motion clips are submitted to the backend server for later processing and scheduling.

[0057] Step S9: Drive the camera to rotate based on the motion category result in step S8.

[0058] In this step, based on action priority and target position, a cyclic correction algorithm is used to drive the camera rotation, ensuring the target animal is centered in the frame to improve footage quality. The specific implementation process is as follows:

[0059] Step S91: Obtain the JSON file containing the coordinates of the captured moving target from step S3.

[0060] Step S92: Based on the coordinates, set the x and y axis deflection directions of the spherical camera and perform a small-amplitude deflection to correct the position.

[0061] Step S93: Activate the motion-sensitive detector, detect the position of the moving target again, and repeat steps S91 and S912 until the target is in the center area of ​​the lens.

Claims

1. A method for capturing and recognizing interesting animal behavior clips for long-time video surveillance, characterized in that The method comprises the following steps: Step S1: acquiring video live streaming video data; Step S2: proposing a motion-sensitive detector based on motion region attention, using the motion-sensitive detector to find meaningful video clips and materials, and positioning the positions of the semantic expression significant targets in the shots; Step S3: outputting the processed motion region list and writing it into a log; Step S4: constructing a SlowFast semantic analysis model; Step S5: segmenting the video data in step S1 to obtain one or more video clips and recording the timestamp information of the one or more video clips; Step S6: performing frame extraction processing on the one or more video clips to obtain a video frame group corresponding to a single video clip; Step S7: inputting the video frame group corresponding to the one or more video clips into the SlowFast model constructed in step S4 to obtain the semantics of each video clip; Step S8: classifying the motion pictures captured in step S3 into backgrounds and various material categories through semantic information, outputting different timestamp information of the motion, writing it into a log, and obtaining animal action clips and submitting information to a backend server for post-processing and scheduling; Step S9: driving the camera to rotate according to the motion category result of step S8; The specific steps of step S2 are as follows: Step S21: performing grayscale image processing; Step S22: using KNN model-based foreground detection to remove small change noise; The specific steps of step S22 are as follows: Step S221: comparing a new pixel value at a certain position of an image with historical information of the pixel value, and if the difference between the pixel values is within a set threshold, the new pixel value is considered to match the historical information and is a potential class; Otherwise, it is not a potential class; Step S222: after all historical information comparison in step S221 is completed, if the number of matches with historical information exceeds a set threshold, the new pixel point is classified as a potential background point, and if the number of background points in the matched historical information exceeds a set threshold, the new pixel point is classified as a background point; Step S223: updating the new pixel point to the historical information; Step S23: using an expansion and erosion operator and a region drawing method to connect adjacent motion regions together and save the detected motion region list; Step S24: to prevent abnormal frames caused by severe changes in illumination, the motion-sensitive detector detects abnormal change frames by judging the motion gradient between associated frames, the area change slope of associated frames, the ratio of the frame motion area to the average motion region area of associated frames, and the frame region number change amplitude; Step S25: if an abnormal change frame appears in S24, the frame is considered to be a non-motion frame; otherwise, it is considered to be a normal frame, and whether the frame is a non-motion frame depends on the detection result obtained in S23.

2. The method for capturing and recognizing animal behavior segments of interest for long-time video surveillance according to claim 1, characterized in that In step S1, the provided interface is called to acquire live video streaming.

3. The method for capturing and recognizing animal behavior segments of interest for long-time video surveillance according to claim 1, characterized in that In step S21, OpenCV-python library is used for image grayscale processing.

4. The method for capturing and recognizing animal behavior segments of interest for long-time video surveillance according to claim 1, characterized in that In the step S3, for the frame judged as normal motion, the KNN detection result is directly used; for the frame judged as great change of motion region, intersection operation is used on the front and rear frames of motion region, and whether the frame is affected by occlusion and light is judged according to the correlation proportion threshold.

5. The method for animal behavior segment capturing and recognizing of interest for long-time video surveillance according to claim 1, characterized in that In the step S4, the semantic analysis model of SlowFast includes a slow channel and a fast channel, wherein: The slow channel is used for extracting spatial semantic information, and the input is a low frame rate sampling video frame; The fast channel is used for extracting timing semantic information, and the input is a high frame rate sampling video frame; The output result of the fast channel is sent into the slow channel through lateral connection, so as to realize information interaction of the two channels and generate more effective video semantic analysis result.

6. The method for capturing and recognizing animal behavior segments of interest for long-time video surveillance according to claim 5, characterized in that The convolution channel of the fast channel is set as β times of the slow channel, and β is set as 1 / 8.

7. The method for animal behavior segment capturing and recognizing of interest for long-time video surveillance according to claim 1, characterized in that In the step S5, the video data is divided into several video segments of 70s, and timestamp information is recorded so as to splice the video segments in the future.

8. The method for animal behavior segment capturing and recognizing of interest for long-time video surveillance according to claim 1, characterized in that In the step S9, according to the action priority and the target position, a loop correction algorithm is used to drive the camera to rotate, so that the target animal is in the middle area of the lens, and the material quality is improved, and the specific implementation process is as follows: Step S91: obtain the json file containing the motion target coordinates in the step S3; Step S92: according to the coordinates, set the deflection direction of the spherical camera in x and y axes, and execute small deflection to correct the position; Step S93: enable the action sensitive detector, detect the motion target position again, repeat the step S91 and the step S92, and until the target is in the middle area of the lens.

Images