Lightweight extraction method for key frames of moving target of interest in video stream

Abstract

The invention relates to the technical field of video processing, is mainly applied to an edge computing environment, and particularly relates to a video stream interested moving target key frame lightweight extraction method. The method comprises the steps of obtaining an original video stream, performing picture clipping, and establishing an image space coordinate system based on the clipped video stream; determining and marking a high-frequency appearing area of the interested moving target as an interested area in the coordinate system; performing frame image de-duplication processing on the clipped video stream; under the coordinate system, inter-frame motion detection is carried out on the de-duplicated video stream, frame images moving in the region of interest are screened out, and a target motion video stream is formed; and applying a lightweight target recognition algorithm to the target motion video stream, and extracting a key frame of an interested motion target. According to the method, the unified coordinate reference is established, region focusing, time redundancy removal and motion screening are performed in sequence, the number of redundant frames needing to be processed is reduced before target recognition, and the real-time performance and processing efficiency of key frame extraction on edge computing equipment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a lightweight method for extracting keyframes of moving targets of interest in a video stream. Background Technology

[0002] Video processing technology, especially the detection and tracking of moving targets in video streams, has wide applications in fields such as intelligent transportation, security monitoring, and behavior analysis. These applications typically rely on fixed edge computing devices deployed on-site (such as processors built into traffic cameras and edge servers) for real-time analysis. Due to the inherent limitations of computing resources, storage capacity, and power consumption, edge computing devices place higher demands on the processing efficiency and real-time performance of algorithms.

[0003] Traditional target recognition and keyframe extraction methods typically apply complex target recognition algorithms, such as YOLO and SSD, directly to each frame of the video stream. YOLO stands for You Only LookOnce, and SSD stands for Single Shot MultiBox Detector, which represents a single-shot target detection algorithm.

[0004] However, the above methods face severe challenges in edge computing scenarios: First, the original video stream contains a large number of redundant frames with highly similar or static backgrounds. Full-frame processing would severely consume the limited computing resources of edge devices, leading to increased processing latency and making it difficult to meet real-time requirements. Second, edge devices typically need to continuously process massive video streams, and directly applying heavyweight recognition algorithms would quickly reach their performance bottleneck, affecting the overall throughput and stability of the system. In addition, existing solutions lack a pre-frame screening mechanism for edge environments and fail to effectively utilize the spatiotemporal locality of target motion to pre-condense video information, resulting in a large amount of computation being wasted on regions and frames unrelated to the target of interest. Some existing optimization schemes attempt to alleviate the pressure by reducing the complexity of the recognition model or randomly selecting frames, but these methods sacrifice the accuracy and completeness of keyframe extraction and cannot fundamentally solve the computational redundancy problem caused by the 'full-frame processing' mode.

[0005] Therefore, in resource-constrained environments of edge computing devices, improving the real-time extraction of keyframes of moving targets of interest and the overall energy efficiency of the system has become a pressing technical challenge in the field of edge-side video intelligent analysis. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a lightweight extraction method for video stream moving target keyframes that features multi-layer filtering and progressive steps, addressing the shortcomings of existing technologies in the extraction of video moving target keyframes on edge computing devices, which suffers from insufficient real-time performance and high computational resource consumption.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a lightweight method for extracting keyframes of moving targets of interest in a video stream, comprising: The original video stream is acquired, and the original video stream is cropped to obtain a cropped video stream. An image space coordinate system is then established based on the cropped video stream. In the image space coordinate system, the region corresponding to the frequency of the occurrence of the moving target of interest being greater than a preset threshold is determined, and the region is marked as the region of interest; The cropped video stream is subjected to frame image deduplication processing to obtain a deduplicated video stream; In the image space coordinate system, inter-frame motion detection is performed on the deduplicated video stream to filter out frame images in which motion occurs in the region of interest, thus forming a target motion video stream; A target recognition algorithm is applied to the target motion video stream to identify frame images containing the target of interest, thereby obtaining keyframes of the target of interest.

[0008] In one implementation, the labeling of the region of interest includes: Based on the historical frames or real-time analysis data of the cropped video stream, the spatial location of the target of interest in motion is statistically analyzed. The regions corresponding to spatial locations that appear more frequently than a preset threshold are marked as the regions of interest.

[0009] In one implementation, the cropped video stream undergoes frame deduplication processing, including: The cropped video stream is subjected to lightweight processing on a frame-by-frame basis, which includes image size reduction and grayscale conversion. Extract adjacent frame images from the cropped video stream after lightweight processing, wherein the adjacent frame images include the current frame image and the next frame image of the current frame image; Calculate the hash values ​​of the current frame image and the next frame image respectively; Calculate the Hamming distance between two frames based on the hash value of the current frame image and the hash value of the next frame image; Based on the comparison result between the Hamming distance and the preset threshold, the cropped video stream is subjected to frame image deduplication processing.

[0010] In one implementation, filtering out frame images where motion occurs within the region of interest includes: Inter-frame difference processing is performed on adjacent frame images in the deduplicated video stream to obtain the difference result, and the difference result is binarized to obtain a binarized image. Contour detection and extraction are performed on the binarized image; The detected contours are then area-calculated and filtered. Based on the filtered contour area and the position of the region of interest in the image space coordinate system, frame images containing valid motion information are selected.

[0011] In one implementation, based on the filtered contour area and the position of the region of interest in the image space coordinate system, frame images containing valid motion information are filtered out, including: When the area of ​​the outline is greater than a preset area threshold, the minimum bounding rectangle of the outline is determined. Based on the minimum bounding rectangle, the target motion region is determined on the corresponding adjacent frame images; Determine whether the center point of the target motion region is located inside the region of interest; If yes, then the current frame image and the next frame image in the adjacent frame images are retained; if not, then the current frame image is discarded to obtain a frame image containing valid motion information.

[0012] In one implementation, a target recognition algorithm is applied to the target motion video stream to identify frame images containing the target of interest, thereby obtaining keyframes of the target of interest, including: The target recognition algorithm is applied to each frame of the target motion video stream to obtain the probability value of each frame containing the target of interest. Frames with probability values ​​greater than a preset probability threshold are identified as keyframes for the moving target of interest.

[0013] In one implementation, the target recognition algorithm is an existence recognition network or a target detection model; wherein, the existence recognition network sequentially includes an input layer, a depthwise separable convolutional layer, a linear activation layer, an inverse residual module group, a channel attention layer, a global average pooling layer, a fully connected layer, and an output layer.

[0014] Secondly, embodiments of the present invention also provide a lightweight extraction device for keyframes of moving targets of interest in a video stream. The device is implemented based on the lightweight extraction method for keyframes of moving targets of interest in a video stream, and includes the following components: The video preprocessing module is used to acquire and trim the raw video stream and establish an image space coordinate system; A region marking module is used to mark high-frequency occurrence regions of moving targets of interest as regions of interest in the image spatial coordinate system; The deduplication module is used to perform inter-frame similarity analysis and deduplication on the cropped video stream to generate a deduplicated video stream. The motion filtering module is used to extract frame images containing effective motion information from the deduplicated video stream based on the region of interest and inter-frame motion detection, forming a target motion video stream. The target recognition module is used to apply a lightweight target recognition algorithm to the target motion video stream and extract key frames of the target of interest.

[0015] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a lightweight extraction program for keyframes of moving targets of interest in a video stream stored in the memory and executable on the processor. When the processor executes the lightweight extraction program for keyframes of moving targets of interest in a video stream, it implements the steps of the lightweight extraction method for keyframes of moving targets of interest in a video stream described above.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a lightweight extraction program for keyframes of moving targets of interest in a video stream. When the lightweight extraction program for keyframes of moving targets of interest in a video stream is executed by a processor, it implements the steps of the lightweight extraction method for keyframes of moving targets of interest in a video stream described above.

[0017] Beneficial Effects: The lightweight extraction method for keyframes of moving targets of interest in video streams according to this invention includes: acquiring the original video stream, cropping the image, and establishing an image space coordinate system based on the cropped video stream; marking the high-frequency occurrence areas of moving targets of interest as regions of interest in the coordinate system; performing frame image deduplication on the cropped video stream; performing inter-frame motion detection on the deduplicated video stream in the coordinate system, and selecting frame images where motion occurs within the regions of interest to constitute the target motion video stream; applying a lightweight target recognition algorithm to the target motion video stream to extract keyframes of moving targets of interest. By establishing a unified coordinate reference and sequentially performing region focusing, temporal redundancy removal, and motion filtering, the number of redundant frames to be processed before target recognition is significantly reduced, significantly improving the real-time performance and processing efficiency of keyframe extraction on edge computing devices, and solving the problem of insufficient real-time performance of video analysis in resource-constrained environments. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of the present invention; Figure 2 A structural diagram of the lightweight extraction device for keyframes of moving targets of interest in a video stream provided by the present invention; Figure 3This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] As mentioned earlier, existing technologies suffer from insufficient real-time performance and high computational redundancy in edge computing environments. To address these issues, this invention provides a lightweight extraction method for keyframes of moving targets of interest in video streams, employing multi-layered filtering and progressive steps. This method overcomes the shortcomings of existing technologies, such as insufficient real-time performance and high computational resource consumption, in extracting keyframes of moving targets of interest on edge computing devices.

[0021] This embodiment provides a lightweight method for extracting keyframes of moving targets of interest in video streams, primarily targeting edge computing devices with limited computing resources, storage, and power consumption, such as smart cameras deployed at traffic intersections, edge servers, or embedded industrial control computers. This method employs a four-level progressive processing framework of "region focusing, temporal redundancy removal, motion initial screening, and lightweight recognition," significantly reducing the computational load while ensuring extraction accuracy, thus meeting the real-time analysis needs of the edge. In this embodiment, as... Figure 1 As shown, the lightweight extraction method for keyframes of moving targets of interest in a video stream specifically includes the following steps: S100, acquire the original video stream, crop the original video stream to obtain the cropped video stream, and establish an image space coordinate system based on the cropped video stream; S200, in the image space coordinate system, determine the region corresponding to the frequency of the occurrence of the moving target of interest being greater than a preset threshold, and mark the region as the region of interest; S300, perform frame image deduplication processing on the cropped video stream to obtain a deduplicated video stream; S400, In the image space coordinate system, perform inter-frame motion detection on the deduplicated video stream, and filter out the frame images in which the motion occurs in the region of interest to form the target motion video stream; S500, apply a target recognition algorithm to the target motion video stream to identify frame images containing the target of interest motion, and obtain key frames of the target of interest motion.

[0022] The method for extracting frames of moving targets of interest based on steps S100, S200, S300, S400, and S500 can be applied to traffic monitoring. When deploying intelligent traffic cameras (with built-in edge computing modules) directly above urban intersections, it is necessary to extract key frames of freight vehicles (moving targets of interest) passing through in real time for subsequent license plate recognition or vehicle type statistics. The specific application process is as follows: 1) Cropping + Coordinate System Establishment: The camera continuously captures a 1080p raw video stream. The edge processing module automatically detects and crops out the road area in the image, removes irrelevant parts such as the sky and green belts, and generates a 720p cropped video stream.

[0023] 2) Region of Interest (ROI) Marking: After initial deployment, the system automatically learns for 24 hours. It uses a built-in lightweight detector (or manual preset) to define the locations of all vehicles. Statistical analysis reveals that vehicles are concentrated in three lane areas (straight lane, left-turn lane, and right-turn lane). The system marks the corresponding rectangular areas of these three lanes (e.g., [x1:y1, x2:y2], [x3:y3, x4:y4], [x5:y5, x6:y6] in the image coordinate system) as "ROIs," with x1, x2, x3, x4, x5, and x6 representing the six horizontal coordinates, and y1, y2, y3, y4, y5, and y6 representing the six vertical coordinates.

[0024] 3) Frame Image Deduplication: For the cropped video stream, the hash values ​​of adjacent frames are calculated consecutively. In situations with smooth traffic and vehicles queuing (e.g., waiting at a red light), multiple consecutive frames are highly similar. The system retains only the first frame of each similar sequence, dynamically reducing the video stream frame rate from 30 FPS to approximately 5-15 FPS (depending on traffic density), generating a deduplicated video stream. This significantly reduces the number of frames to process. FPS stands for Frames Per Second.

[0025] 4) Motion Filtering: Calculate inter-frame differences for the deduplicated video stream. When a vehicle moves, the difference image shows significant changes at the vehicle's outline. After outline detection, small outlines caused by swaying leaves or changing shadows are filtered out. Key Judgment: Calculate the center point of the remaining outlines. For example, a car driving in a straight lane always has its motion outline center within the ROI of the straight lane, so all frames of that video segment are retained. However, a bird flying across the sky (a non-ROI) has its motion outline center outside any ROI, and all related frames are discarded. Ultimately, only those frames that exhibit significant motion within the "lane ROI" are retained, forming the "target motion video stream." This step can further reduce the frame count by more than 60%.

[0026] 5) Target Recognition and Keyframe Extraction: The "target motion video stream" is input into a pre-trained "truck presence recognition network." This network is a lightweight CNN (Convolutional Neural Network), specifically designed to analyze and determine frame-by-frame whether a truck is present in an image. For example, when a car passes by, the network outputs a probability of 0.2. Since 0.2 is less than the threshold of 0.7, this frame is discarded. When a truck passes by, the network outputs a probability of 0.9. Since 0.9 is greater than the threshold of 0.7, this frame is marked as a keyframe of interest. The system outputs these keyframes (which may only account for 1% to 5% of the total number of frames in the original video stream) and their timestamps for uploading to the cloud or triggering local alarms.

[0027] In this embodiment, each frame of the original video stream goes through steps S300, S400, and S500. The average time required to perform step S300 for each frame is less than the average time required to perform step S400, and the average time required to perform step S400 is less than the average time required to perform step S500. That is, step S500 is the most time-consuming. Therefore, this embodiment first uses S300 and S400 to remove some frames that do not reflect motion information, reducing the computational burden on step S500 and thus improving the real-time performance of extracting frames of interest. In other words, steps S300 (deduplication) and S400 (motion filtering) employ image processing operations with extremely low computational complexity, and their average processing time per frame is much lower than that of step S500 (neural network inference). Through layer filtering in S300 and S400, the number of frames that need to enter S500 for expensive neural network computation can be reduced to 90%-95%. This enables the entire system to perform real-time keyframe extraction for specific targets (such as trucks) on resource-constrained edge devices, solving the high latency and high load problems caused by full-frame analysis in traditional solutions. At the same time, by focusing on ROI and motion information, the targeting and accuracy of keyframe extraction are also improved.

[0028] In this embodiment, step S100 acquires the original video stream, crops the original video stream to obtain a cropped video stream, and establishes an image space coordinate system based on the cropped video stream. The aim is to eliminate fixed, unchanging background interference (such as the sky or green belts) from the original image, focusing on the effective area that may contain moving targets, and establishing a unified coordinate reference for all subsequent spatial location determinations; that is, installing cameras in scenarios requiring traffic monitoring, acquiring the original video stream of the monitored scene through the cameras, cropping the image to obtain a cropped video stream, and establishing an image space coordinate system based on the cropped video stream.

[0029] In this embodiment, step S200 is based on the historical frames or real-time analysis data of the cropped video stream to count the regions where the target of interest appears frequently; the high-frequency regions are taken as regions of interest, and the coordinates of the regions of interest are marked in the image space coordinate system.

[0030] The historical frames of the cropped video stream are the other frames in the cropped video stream besides the current frame. In this embodiment, a manual statistical method is used to identify the regions where the moving targets of interest appear frequently based on the historical frames, and the coordinates of the regions of interest are manually marked in the image space coordinate system.

[0031] Before extracting the final keyframes, this step first determines the region of interest (ROI). This has two advantages: first, it can significantly reduce the computational area required for subsequent steps (especially motion detection and target recognition), thereby reducing the computational load; second, it can guide the system to focus more on the change information within the ROI, filtering out a large number of invalid movements that occur in non-interested areas, thereby more accurately selecting keyframes containing valid information and achieving efficient and accurate keyframe extraction.

[0032] Step S300 in this embodiment includes the following specific steps S301, S302, S303 and S304: S301, Image Lightweighting and Frequency Domain Conversion: Lightweight preprocessing is performed on each frame of the cropped video stream, including resizing and then grayscale conversion to significantly reduce the data volume. Subsequently, the processed time-domain image signal is converted into a frequency-domain signal through Discrete Cosine Transform (DCT).

[0033] S302, Extract adjacent frame images: From the lightweight image sequence, extract two adjacent frames as a group, namely the current frame image and its next frame image.

[0034] S303, Calculate hash value: Calculate the hash value of the current frame image and the next frame image respectively. Specifically, calculate the hash value based on the frequency domain signal of the image.

[0035] S304, Calculate Hamming distance: Based on the two hash values ​​obtained in S303, calculate the Hamming distance between them to quantify the degree of difference between the content of two image frames. The smaller the distance, the more similar the images are; conversely, the larger the distance, the greater the difference. Compare the calculated Hamming distance with a preset distance threshold to determine whether to retain the current frame image, and iteratively perform deduplication processing.

[0036] When the Hamming distance is less than or equal to a preset distance threshold, the two frames are determined to be highly similar and redundant. In this case, the current frame image is discarded, the next frame image is retained, and the next frame image is updated as the new current frame image. A new round of comparison is then performed with its next frame to remove duplicates until a deduplicated video stream is obtained.

[0037] Alternatively, when the Hamming distance is greater than a preset distance threshold, it is determined that the content of the two frames is significantly different. Then, the current frame image and the next frame image are retained, and the next frame image is updated to the current frame image. The updated current frame image is then used to continue to compare with its next frame for a new round of deduplication processing until a deduplicated video stream is obtained.

[0038] The above processing is performed iteratively. Specifically: Starting from the beginning of the video stream, the system takes the first frame as the current frame and compares it with the second frame (the next frame). If the Hamming distance is less than or equal to a preset distance threshold, the first frame is discarded, the second frame is retained, and the second frame is used as the new current frame; if it is greater than the threshold, both the first and second frames are retained, and the second frame is used as the new current frame. Then, the new current frame is compared and decided again with its next frame (the third frame in the original sequence), and this process is repeated until the comparison between the penultimate frame and the last frame of the video stream is completed. At this point, the deduplication process of the entire original video stream is complete, ultimately resulting in a deduplicated video stream without highly similar redundant frames.

[0039] For example, suppose the video stream contains five frames F1, F2, F3, F4, and F5 captured in chronological order. The deduplication process is as follows: First, compare the Hamming distance between F1 and F2. If it is less than or equal to a preset distance threshold, discard F1 and keep F2. Next, using F2 as the current frame, compare its Hamming distance with F3. If it is greater than the preset distance threshold, keep both F2 and F3. Then, using F3 as the current frame, compare its Hamming distance with F4. If it is less than or equal to the preset distance threshold, discard F3 and keep F4. Finally, using F4 as the current frame, compare its Hamming distance with F5. If it is greater than the preset distance threshold, keep both F4 and F5. After the above processing, F1 and F3 in the original video stream are discarded because they are similar to the content of subsequent frames. The final deduplicated video stream contains only F2, F4, and F5.

[0040] Since high similarity between adjacent images usually means content redundancy, this embodiment effectively removes redundant frames through this method, thereby significantly reducing the number of frames to be processed before entering the target recognition step with greater computational overhead, and alleviating the overall computational pressure of the system.

[0041] In this embodiment, step S400 is used to determine and filter the motion regions in each frame of the deduplicated video stream, including the following specific steps S401, S402, S403 and S404: S401, Inter-frame Differentiation and Binarization Processing: Inter-frame difference processing is performed sequentially on adjacent frames (current frame and next frame) in the deduplicated video stream. This involves calculating the absolute difference between corresponding pixel values ​​between two frames to obtain a difference image. The difference image is then converted to grayscale and binarized using a fixed threshold (15 in this embodiment): pixels with grayscale values ​​greater than 15 are set to 1, otherwise to 0, resulting in a binarized image containing only 0 and 1 pixels. If no target motion occurs between adjacent frames, the pixel values ​​in the difference image are theoretically all 0; conversely, non-zero regions in the difference image reflect the motion information of the moving target.

[0042] S402, Contour Extraction: Contour extraction is performed on the binarized image to obtain contour regions that may correspond to moving targets. Contour extraction can be implemented using image processing algorithms, including the findContours method in OpenCV. OpenCV stands for Open Source Computer Vision Library, and findContours represents image contour detection algorithms.

[0043] S403, Contour Area Calculation and Screening: The detected contours are then area-calculated. This step aims to distinguish between effective motion and noise based on contour size, as motion contours generated by real targets typically have larger areas, while noise contours generated by swaying leaves, changes in lighting, etc., have smaller areas. By comparing the calculated contour area with a preset area threshold, contours with significant areas can be initially screened out, while small-area noise interference can be eliminated.

[0044] S404, motion frame filtering based on contour area and position: Based on the calculated contour area and its positional relationship with the region of interest (ROI), the deduplicated video stream is further filtered to ultimately form the target motion video stream. Specifically, this involves the following two cases: Case 1: When the contour area is greater than the set area threshold, it indicates that there is a contour generated by a significant moving target between adjacent frames. Then, the following steps S404a, S404b, S404c, and S404d are executed: S404a, defines the minimum bounding rectangle of the contour.

[0045] S404b, Mark the target movement area. The smallest bounding rectangle is projected onto the current frame and the next frame image respectively. The target motion region is marked in the image coordinate system of the two frames, and the coordinates of the center point of the rectangle are recorded. This region represents the range of pixel changes caused by the target motion.

[0046] S404c determines the positional relationship between the target's moving area and the ROI. It is determined whether the center point of the target motion region is located inside a pre-marked region of interest (ROI). In this embodiment, the relative positional relationships include the center of the target motion region being inside the ROI and the center of the target motion region being outside the ROI.

[0047] S404d performs frame filtering based on positional relationships.

[0048] Based on the judgment result of S404c, the deduplicated video stream is processed as follows: if the center point of the target motion region is located inside the region of interest, the current frame image and the next frame image are retained. If the center point of the target motion region is located outside the region of interest, the current frame image is discarded, and the next frame image is retained. Subsequently, the next frame image is updated to the new current frame, and steps S401-S404 are repeated to continue a new round of filtering with subsequent frames.

[0049] Scenario 2: When the contour area is less than or equal to the set area threshold, it indicates that there is no significant motion between adjacent frames or that it is only noise. In this case, the current frame image is directly discarded, the next frame image is updated to the current frame, and a new round of S401-S404 processing continues with subsequent frames. In this case, noise contours with too small an area (such as noise generated by swaying leaves or changes in shadows) can also be filtered out.

[0050] Through the above iterative process, combined with the dual constraints of contour area and spatial position, small-area noise and invalid motion in non-interest areas are filtered out, and the final retained frame images constitute the target motion video stream. The target motion video stream only contains frames in which effective motion occurs within the region of interest, thereby further reducing the number of frames to be processed before entering the lightweight target recognition step.

[0051] In this embodiment, step S500 includes the following specific steps: applying a target recognition algorithm to each frame of the target motion video stream, predicting the probability of a moving target of interest appearing in each frame; pre-setting a probability threshold, determining that a moving target of interest exists in a frame if the predicted probability is greater than the threshold, and filtering it out. Conversely, if the probability is less than or equal to the threshold, the frame is considered not to contain a moving target of interest and is discarded. Finally, all frames determined to contain moving targets of interest are extracted as the final keyframes for the moving targets of interest.

[0052] In this embodiment, the target recognition algorithms that can be used include existence recognition networks (such as FastExist-Net designed in this embodiment), YOLO, SSD, and other models. FastExist-Net represents an existence recognition network. To adapt to the stringent resource constraints of edge devices, this embodiment preferably uses a specially designed lightweight existence recognition network. The existence recognition network can achieve efficient target existence classification, thereby completely relieving the computational burden on edge devices while ensuring the overall accuracy of the system.

[0053] The Existence Recognition Network abandons the redundant bounding box regression and complex multi-class classification modules in traditional detection models, focusing its core task on an efficient binary classification problem: quickly determining whether a moving target of interest exists in an input image. By employing a minimal input size, a lightweight backbone network based on depthwise separable convolutions, and a channel attention mechanism, this network achieves a significant reduction in the number of model parameters and computational complexity while ensuring high recall (i.e., minimizing missed targets).

[0054] Forward reasoning process of existence recognition network: The frame images from the target motion video stream output in step S400 are input into the presence recognition network. First, they are scaled to a fixed low resolution or the frame image size is scaled to 64×64 pixels before being input into the network. After forward inference, the network directly outputs a scalar value representing the probability of the presence of a moving target of interest in the input image. The system compares this probability value with a preset threshold to decide whether to extract the frame image as a keyframe.

[0055] Network architecture of the existence identification network: The presence detection network in this embodiment includes a cascaded input layer, a depthwise separable convolutional layer, a corrected linear layer, an inverted residual module, an attention layer, a global average pooling layer, a fully connected layer, and an output layer. The input layer is a normalization layer, and the kernel size of the convolutional layers is [missing information]. The convolutional layer has a stride of 2 and 16 output channels. The corrected linear layer uses ReLU6 (Rectified Linear Unit). There are three inverse residual modules, cascaded sequentially. Each module has a dilation factor of 6, and their output channels are 32, 64, and 128 respectively, with strides of 2, 2, and 1 respectively. Each inverse residual module includes a first pointwise convolutional layer to increase channel dimension, a deep convolutional layer for spatial feature filtering, and a second pointwise convolutional layer to decrease channel dimension. Both the deep convolutional layer and the second pointwise convolutional layer are followed by a batch normalization layer and a linear activation function. An attention layer is connected to the last cascaded inverse residual module and adaptively calibrates the weights of each feature channel. The attention layer obtains global information through a global average pooling layer, and then generates weight coefficients for each channel through a bottleneck structure consisting of two fully connected layers and a sigmoid function (Sigmoid represents an activation function). These weight coefficients are then multiplied channel by channel with the feature map of the frame image output by the convolutional layer.

[0056] Training strategy for existence recognition networks: This embodiment uses a combination of focus loss function and knowledge distillation loss function to train the existence recognition network. That is, a large, high-precision teacher model is used to preprocess the training dataset to generate soft labels. The existence recognition network acts as a student model, fitting both the real hard labels and the soft labels provided by the teacher model during training. This achieves higher discrimination accuracy and robustness in a lightweight architecture.

[0057] The training of the Presence Detection Network consists of a first stage of basic training and a second stage of distillation training. Basic training involves using real, manually labeled data and employing a focus loss function to initially train the Presence Detection Network, giving it basic discrimination capabilities. Distillation training involves selecting a high-precision, general-purpose object detection model pre-trained on a dataset. For example, YOLOv5 or RetinaNet pre-trained on the COCO dataset can be used as the teacher model. YOLOv5 represents the fifth generation of YOLOv5, COCO stands for Common Objects in Context, which is an image dataset, and RetinaNet represents a single-stage object detection algorithm.

[0058] The training set images are input into the teacher model to obtain its predictions (such as confidence scores for vehicle categories), and then converted into soft labels. The teacher model is then frozen, and the total loss function is applied. The existence recognition network was fine-tuned to improve its generalization and classification abilities by learning from the teacher model. It is calculated using the following formula: ; ; In the formula, Representing the focus loss function, this embodiment uses the loss function of the YOLO model described above. As the focus loss function, Represents the knowledge distillation loss function. represent The weight, represent The weight, The labels represent the output of the teacher model. The label represents the output of the existence identification network. Represents divergence, This represents an adjustment parameter. Symbols representing differential calculations.

[0059] This embodiment also provides a lightweight extraction device for keyframes of moving targets of interest in a video stream, such as... Figure 2 As shown, the device comprises the following components: The video preprocessing module 01 is used to acquire the original video stream, perform image cropping on the original video stream to obtain the cropped video stream, and establish an image space coordinate system based on the cropped video stream. The region marking module 02 is used to determine, in the image space coordinate system, the region corresponding to the frequency of the occurrence of the moving target of interest being greater than a preset threshold, and mark the region as the region of interest; The deduplication module 03 is used to perform frame image deduplication processing on the cropped video stream to obtain a deduplicated video stream; The motion filtering module 04 is used to perform inter-frame motion detection on the deduplicated video stream in the image space coordinate system, and filter out frame images in which the motion occurs in the region of interest to form a target motion video stream; The target recognition module 05 is used to apply a target recognition algorithm to the target motion video stream to identify frame images containing the target of interest, and obtain key frames of the target of interest.

[0060] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which is as follows: Figure 3As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a lightweight method for extracting keyframes of moving targets of interest in a video stream. The display screen of the terminal device can be a liquid crystal display (LCD) or an e-ink display.

[0061] Those skilled in the art will understand that Figure 3 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0062] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a lightweight extraction program for keyframes of moving targets of interest in a video stream stored in the memory and executable on the processor. When the processor executes the lightweight extraction program for keyframes of moving targets of interest in a video stream, it implements the following operation instructions: The original video stream is acquired, and the original video stream is cropped to obtain a cropped video stream. An image space coordinate system is then established based on the cropped video stream. In the image space coordinate system, the region corresponding to the frequency of the occurrence of the moving target of interest being greater than a preset threshold is determined, and the region is marked as the region of interest; The cropped video stream is subjected to frame image deduplication processing to obtain a deduplicated video stream; In the image space coordinate system, inter-frame motion detection is performed on the deduplicated video stream to filter out frame images in which motion occurs in the region of interest, thus forming a target motion video stream; A target recognition algorithm is applied to the target motion video stream, and the frame images containing the target of interest motion are identified as keyframes of the target of interest motion.

[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting key frames of moving targets of interest in video streams, characterized in that, The method comprises the following steps: acquiring an original video stream, performing picture cropping on the original video stream to obtain a cropped video stream, and establishing an image space coordinate system based on the cropped video stream; in the image space coordinate system, determining a region corresponding to a frequency of appearance of a target moving object greater than a preset threshold, and marking the region as a region of interest; performing frame image deduplication processing on the cropped video stream to obtain a deduplicated video stream; in the image space coordinate system, performing inter-frame motion detection on the deduplicated video stream to screen out frame images in which motion occurs in the region of interest, and constructing a target moving video stream; applying a target recognition algorithm to the target moving video stream to identify frame images containing the target moving object, and obtaining key frames of the target moving object.

2. The method of claim 1, wherein, The marking of the region of interest comprises the following steps: based on historical frames or real-time analysis data of the cropped video stream, counting spatial positions of appearance of the target moving object; marking a region corresponding to a spatial position with a frequency of appearance higher than a preset threshold as the region of interest.

3. The method of claim 1, wherein, The frame image deduplication processing on the cropped video stream comprises the following steps: performing lightweight processing on the cropped video stream in units of frame images, wherein the lightweight processing comprises image size reduction and grayscale processing; extracting adjacent frame images from the cropped video stream after lightweight processing, wherein the adjacent frame images comprise a current frame image and a next frame image of the current frame image; respectively calculating hash values of the current frame image and the next frame image; calculating a Hamming distance between the two frame images according to the hash values of the current frame image and the next frame image; performing frame image deduplication processing on the cropped video stream according to a comparison result of the Hamming distance and a preset distance threshold.

4. The method of claim 1, wherein, The screening of frame images in which motion occurs in the region of interest comprises the following steps: performing inter-frame difference processing on adjacent frame images in the deduplicated video stream to obtain a difference result, and performing binaryzation processing on the difference result to obtain a binary image; performing contour detection and extraction on the binary image; performing area calculation and screening on the detected contour; based on the screened contour area and a position of the region of interest in the image space coordinate system, screening out frame images containing effective motion information.

5. The method of claim 4, wherein, The screening of frame images containing effective motion information based on the screened contour area and the position of the region of interest in the image space coordinate system comprises the following steps: when the contour area is greater than a preset area threshold, determining a minimum circumscribed rectangle of the contour; based on the minimum circumscribed rectangle, determining a target motion region on the corresponding adjacent frame image; judging whether a center point of the target motion region is located inside the region of interest; if yes, retaining the current frame image and the next frame image in the adjacent frame image; if no, discarding the current frame image to obtain frame images containing effective motion information.

6. The method of claim 1, wherein, The application of the target recognition algorithm to the target moving video stream to identify frame images containing the target moving object to obtain key frames of the target moving object comprises the following steps: The target recognition algorithm is applied to each frame image of the target motion video stream respectively to obtain a probability value of each frame image containing the motion target of interest; A frame image with a probability value greater than a preset probability threshold is determined as a key frame of the motion target of interest.

7. The method of claim 1-6, wherein, The target recognition algorithm is an existence recognition network or a target detection model; the existence recognition network sequentially includes an input layer, a depth separable convolution layer, a linear activation layer, a reverse residual module group, a channel attention layer, a global average pooling layer, a full connection layer, and an output layer.

Images