Video Frame Resilience Extraction Spatiotemporal Parameter Adaptive Optimization Control Method

Abstract

This invention relates to the field of video processing technology, specifically to an adaptive optimization control method for spatiotemporal parameters in video frame resilience extraction. The invention performs deduplication processing on historical video streams based on a first threshold to obtain a deduplicated video stream. Then, it performs frame removal processing on the deduplicated video stream based on a second threshold to obtain a target motion video stream. Keyframes containing the target of interest are extracted, and the region of interest (ROI) is extracted from the RIO stream composed of these keyframes. The stability of the RIO stream is assessed, determining the frame drop rate from deduplication, the false alarm rate from removal, and the degree of jump between keyframes for the target of interest. Finally, the two thresholds are optimized based on stability, frame drop rate, false alarm rate, and jump degree. The optimized first threshold accurately identifies duplicate frames, and the optimized second threshold accurately identifies frames to be removed, thereby improving the accuracy of keyframe extraction from real-time video streams.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to an adaptive optimization control method for extracting spatiotemporal parameters of video frame resilience. Background Technology

[0002] Extracting keyframes containing the target of interest from a video stream requires deduplication and removal of frames that do not record target motion changes before keyframe extraction. Deduplication and removal remove redundant images to reduce computational burden and improve performance. Both deduplication and removal involve threshold design to identify duplicate images and frames that do not record target motion changes.

[0003] In existing technologies, fixed thresholds cannot adapt to dynamic scene changes, leading to a decrease in keyframe extraction accuracy. Once a threshold is designed, existing technologies consistently use that threshold for keyframe extraction, meaning they use static thresholds. However, changes in the external environment cause variations in the captured video stream, rendering static thresholds unsuitable for extracting keyframes from dynamic video streams.

[0004] In summary, existing technologies cannot adequately guarantee the resilience and accuracy of the keyframe extraction process.

[0005] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an adaptive optimization control method for spatiotemporal parameters in video frame resilience extraction, solving the problem that existing technologies cannot adequately guarantee resilience and accuracy during keyframe extraction. The "resilience extraction" referred to in this invention aims to ensure that the keyframe extraction method maintains high accuracy and robustness even when video scenes, lighting, and target motion patterns change.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an adaptive optimization control method for spatiotemporal parameters of video frame resilience extraction, comprising:

[0009] The historical video stream is sequentially deduplicated, frame images are removed, and targets of interest are identified to obtain a video stream of interest consisting of keyframes containing the targets of interest. Regions of interest are extracted from the video stream of interest, and the stability of the regions of interest is determined. The regions of interest are spatial parameters.

[0010] Obtain the first threshold used for deduplication and calculate the frame drop rate caused by deduplication. The first threshold is used to determine the repetition of frame images.

[0011] The second threshold used for frame image removal processing is obtained, and the false alarm rate caused by frame image removal processing is calculated. The second threshold is used to determine the degree of motion of the target object between frame images. The target object includes moving objects, and the moving objects include the target of interest. The first threshold and the second threshold are time-series parameters.

[0012] Determine the degree of jump of the target of interest between keyframes;

[0013] Based on the stability, the frame drop rate, the false alarm rate, and the degree of jump, the first threshold and the second threshold are optimized. The optimized first threshold and the second threshold are used to process the real-time video stream to extract key frames.

[0014] In one implementation, extracting a region of interest from the video stream of interest and determining the stability of the region of interest includes:

[0015] Determine the heat map corresponding to the frame image of the video stream of interest, the heat map being used to characterize the location of the target object recorded by the video stream of interest;

[0016] The connected regions of the heatmap are determined, and the connected regions of the heatmaps corresponding to each frame image are taken as the local regions of interest corresponding to each frame image. The local regions of interest corresponding to a set number of consecutive frame images are accumulated to obtain the regions of interest corresponding to the historical video stream.

[0017] The stability of the region of interest (ROI) corresponding to the historical video stream is determined based on the difference between the ROI corresponding to the historical video stream and the pre-defined ROI.

[0018] In one implementation, the frame drop rate resulting from deduplication is statistically analyzed, including:

[0019] Count the number of discarded frames resulting from deduplication;

[0020] The total number of original frame images contained in the historical video stream is counted;

[0021] The frame drop rate is calculated based on the number of dropped frame images and the total number of original frame images.

[0022] In one implementation, the false alarm rate resulting from frame image removal processing is calculated, including:

[0023] The number of frames in the remaining frames after frame image removal processing that do not contain the target of interest is obtained as the number of false alarm frames.

[0024] The total number of frame images retained after frame image removal is counted, and the false alarm rate is obtained based on the number of false alarm frames and the total number of frame images retained after frame image removal.

[0025] In one implementation, identifying the target of interest includes:

[0026] Based on the target detection model, the category and location of the target object are identified;

[0027] Based on the category of the target objects, the target of interest and its location are identified from the target objects.

[0028] In one implementation, determining the degree of jump of the target of interest between keyframes includes:

[0029] Determine the intersection-union ratio (CIG) of the target pixels of interest on adjacent keyframes;

[0030] Based on the intersection-union ratio, determine the jump index of the target of interest between adjacent keyframes;

[0031] Determine the mean of the jump index, and use the mean of the jump index as the jump degree.

[0032] In one implementation, optimizing the first threshold and the second threshold based on the stability, the frame drop rate, the false alarm rate, and the degree of hopping includes:

[0033] The stability, the frame drop rate, the false alarm rate, and the degree of jump are compared with their respective preset thresholds;

[0034] Based on the comparison results, the first threshold and / or the second threshold are increased, decreased, or maintained according to a preset adjustment strategy.

[0035] Beneficial Effects: This invention deduplicates historical video streams to obtain deduplicated video streams. It then removes frames containing non-moving objects from these deduplicated video streams to obtain target motion video streams. Keyframes containing the target of interest are extracted from these target motion video streams. Regions of interest (ROIs) are extracted from the RIO streams composed of these keyframes, and the stability of these RIOs is assessed. The frame drop rate from deduplication, the false alarm rate from removal, and the degree of jump between keyframes are determined. Finally, a first threshold and a second threshold are optimized based on stability, frame drop rate, false alarm rate, and jump degree. These optimized thresholds are then used to extract keyframes from the real-time video stream. As the above analysis shows, this invention continuously optimizes the first and second thresholds, enabling accurate identification of duplicate frames based on the optimized first threshold and accurate identification of removed frames based on the optimized second threshold. This improves the accuracy of keyframe extraction from subsequent video streams (i.e., real-time video streams). Attached Figure Description

[0036] Figure 1 This is an overall flowchart of the present invention;

[0037] Figure 2 This is a schematic diagram illustrating the interaction between modules of different levels in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram illustrating the relationship between the core processing flow and the intelligent control center in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram illustrating adaptive control based on an analyzer, a decision engine, and an actuator in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of spatiotemporal two-dimensional analysis decision execution adaptive control in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0042] Research has shown that extracting keyframes containing the target of interest from a video stream requires deduplication and removal of frames that do not record target motion changes before keyframe extraction. Deduplication and removal remove redundant images to reduce the computational burden on keyframe extraction and improve performance. These processes involve designing temporal and spatial thresholds to identify duplicate images and frames that do not record target motion changes.

[0043] Once a threshold is designed, existing technologies continue to use that threshold for keyframe extraction. In other words, existing technologies use static thresholds to extract keyframes. However, due to changes in the external environment, the captured video stream changes, making static thresholds unsuitable for extracting keyframes from dynamic video streams.

[0044] To address the aforementioned technical problems, this invention provides an adaptive optimization control method for spatiotemporal parameters in video frame resilience extraction, solving the problem of reduced keyframe extraction resilience and accuracy in existing technologies. In specific implementation, the historical video stream is first subjected to deduplication, frame image removal, and target of interest (ROI) identification, resulting in a video stream of interest composed of keyframes containing the ROI. The region of interest (ROI) is extracted from this video stream, and its stability is determined. The frame drop rate generated by deduplication of the historical video stream based on a first threshold, and the false alarm rate generated by removing frames from the deduplicated video stream that do not record the ROI based on a second threshold, are then calculated. The degree of transition of the ROI between adjacent keyframes is further calculated. Finally, the first and second thresholds are optimized based on stability, frame drop rate, false alarm rate, and degree of transition. Subsequently, the optimized first threshold is used to deduplicate new video streams, and the optimized second threshold is used to remove frames from these new video streams. This invention continuously optimizes the first and second thresholds to extract high-quality keyframes from real-time video streams.

[0045] For example, historical video streams of traffic intersections are acquired, and regions of interest (ROIs) are marked on the video window of these streams. ROIs represent the locations of moving objects such as bicycles, electric scooters, and cars around the intersection. The stability of ROIs is determined by comparing the new ROI with the old ROI to see if they have changed. The old ROI is the pre-labeled ROI, while the new ROI is extracted from the historical video stream. For instance, if the target of interest is a car driving at the intersection, mapping the car's position on the image yields the pre-labeled ROI; the new ROI is the region containing the target of interest extracted from the historical video stream based on a heatmap.

[0046] Based on a first threshold, duplicate frames are selected from the historical video stream and removed to obtain a deduplicated video stream. The frame drop rate resulting from the deduplication process is also calculated. For example, if the original video stream contains 100 frames and the deduplicated video stream obtained after deduplication contains only 80 frames, then the frame drop rate is 20%.

[0047] Based on a second threshold, frames from several adjacent frames in the deduplicated video stream where the target object has not changed motion are identified and removed to obtain the target motion video stream. The false alarm rate caused by the removal process is then calculated. For example, if the target motion video stream contains 80 frames, and target of interest identification is performed on these 80 frames, and the target of interest does not appear in 10 of them, then the false alarm rate is... .

[0048] Finally, the target of interest is identified from the target objects in the target motion video stream. For example, if the target of interest is a car, the key frame containing the car is extracted from the target motion video stream, and then the degree of transition of the car between key frames is analyzed.

[0049] Based on the stability, frame drop rate, false alarm rate and jump degree mentioned above, the first threshold and the second threshold are optimized, and the optimized first threshold and the second threshold are used to process the new video stream in order to extract high-quality keyframes from the new video stream.

[0050] Example 1: The video frame resilience extraction spatiotemporal parameter adaptive optimization control method of this embodiment can be applied to a terminal device, which can be a terminal product with video acquisition function, such as a computer. In this embodiment, as... Figure 1 As shown, the video frame resilience extraction spatiotemporal parameter adaptive optimization control method specifically includes the following steps:

[0051] S100, the historical video stream is sequentially deduplicated, frame image removed, and target of interest identified to obtain a video stream of interest composed of key frames where the target of interest is located. The region of interest is extracted from the video stream of interest and the stability of the region of interest is determined. The region of interest is a spatial parameter.

[0052] S200, obtain the first threshold used for deduplication processing, and calculate the frame drop rate caused by deduplication processing. The first threshold is used to determine the repetition of frame images.

[0053] S300, obtain the second threshold used for frame image removal processing, and calculate the false alarm rate caused by frame image removal processing. The second threshold is used to determine the degree of motion of the target object between frame images. The target object includes moving objects, and the moving objects include the target of interest. The first threshold and the second threshold are time-series parameters.

[0054] S400, determine the degree of jump of the target of interest between keyframes;

[0055] S500, based on the stability, the frame drop rate, the false alarm rate and the degree of jump, optimize the first threshold and the second threshold, and use the optimized first threshold and the second threshold to process the real-time video stream to extract key frames.

[0056] In this embodiment, the real-time video stream is the new video stream.

[0057] Steps S100, S200, S300, S400, and S500 are embedded into a computer that transmits data with the camera. The computer is equipped with a preprocessing module, a first-order module, a second-order module, a third-order module, and an adaptive module. The interaction between the first-order module, the second-order module, the third-order module, and the adaptive module is as follows: Figure 2 As shown, the camera sends the raw video stream to a preprocessing module and a first-order module on the computer. The preprocessing module marks the region of interest based on step S100, and the first-order module completes step S200. The first-order module processes the historical video stream based on a first threshold to obtain a deduplicated video stream. The first-order module then sends the deduplicated video stream to a second-order module, which sends the current hash value, the removed frames, and the retained frames to an adaptive module. The second-order module processes the deduplicated video stream based on a second threshold to obtain the target motion video stream, which then sends the target motion video stream to a third-order module. The second-order module sends the contour area threshold to the adaptive module. The third-order module extracts keyframes from the target motion video stream and sends the keyframes to the adaptive module.

[0058] The adaptive module consists of, for example Figure 4 The diagram shows a system composed of a repetition analyzer, a false alarm rate analyzer, a content change analyzer, and a region of interest (ROI) dynamic analyzer (i.e., a ROI planner). The ROI planner calculates the stability of the ROI; the repetition analyzer calculates the frame drop rate based on the deduplicated video stream; the false alarm rate analyzer calculates the false alarm rate based on the number of frames in the target motion video stream and the number of keyframes identified from it; the content change analyzer calculates the jump level based on keyframes; and the adaptive module's decision engine layer optimizes the first and second thresholds, as well as the ROI, based on the stability frame drop rate, false alarm rate, and jump level. Figure 2 As shown, the adaptive module feeds back the optimized first threshold to the first-order module, enabling the first-order module to perform deduplication on the new video stream based on the optimized first threshold. The adaptive module feeds back the optimized second threshold and region of interest to the second-order module, enabling the second-order module to extract the target motion video stream from the new video stream based on the optimized second threshold and region of interest. The third-order module then extracts the keyframe containing the target of interest from the motion video stream. The data interaction between the first-order, second-order, and third-order modules is as follows: Figure 3 As shown.

[0059] In this embodiment, the adaptive module can also optimize only the first threshold and the second threshold without optimizing the region of interest. Therefore, the adaptive module in this embodiment can also feed back the first threshold and the second threshold to the first-order module and the second-order module respectively without feeding back the region of interest to the second-order module.

[0060] Step S100, which involves extracting the region of interest from the video stream of interest and determining the stability of the region of interest, includes the following specific steps S101, S102, and S103:

[0061] S101, determine the heat map corresponding to the frame image of the video stream of interest, the heat map being used to characterize the location of the target object recorded by the video stream of interest.

[0062] Detection of the first For a target object in a frame image, a heat value is injected at the bottom center point of the target object's bounding box. The heat value then diffuses from this bottom center point, creating a heat map. In other words, the distribution area of ​​heat values ​​on the heat map represents the location of the target object. The bounding box represents the outline of the target object, and the bottom center point represents the contact point between the target object and the ground.

[0063] Due to the presence of noise, the heatmap needs to be denoised. The denoising process is as follows:

[0064] ;

[0065] In the formula, Representing the Pixels on a frame heatmap Popularity value This represents the heatmap after denoising. Represents the x-coordinate of a pixel. The x-axis represents the center of the heatmap. The vertical coordinate of a pixel. The vertical axis represents the center of the heatmap. Representing the Confidence level of a target object Represents gain. This represents the extent of the spread.

[0066] S102, determine the connected regions of the heatmap, and take the connected regions of the heatmap corresponding to each frame image as the local regions of interest corresponding to each frame image. Accumulate the local regions of interest corresponding to a set number of consecutive frame images to obtain the region of interest corresponding to the historical video stream.

[0067] The heatmap of each frame image is binarized to obtain a binarized heatmap. The binarized heatmap is then dilated and eroded to fill the internal holes and smooth the boundaries. Finally, the circumscribed polygons of the connected regions are extracted from the heatmap as the local region of interest (ROI) corresponding to that frame image. The ROIs corresponding to multiple consecutive frames are accumulated to obtain the final ROI.

[0068] The threshold used for binarization described above can be 40% of the maximum heat value on the heatmap. The bounding polygon described above is the minimum bounding rectangle. The connected regions in this embodiment do not include the connected regions with the minimum area, because the connected regions with the minimum area may be connected regions generated by noise.

[0069] S103, based on the difference between the region of interest corresponding to the historical video stream and the pre-defined region of interest, determine the stability of the region of interest corresponding to the historical video stream.

[0070] If the region of interest (ROI) corresponding to a historical video stream differs significantly from a pre-defined ROI, then the stability of the ROI corresponding to the historical video stream is low; otherwise, the stability of the ROI corresponding to the historical video stream is high. The intersection-union ratio (IU / U) of two ROIs represents their difference. The IU / U is calculated by dividing the area of ​​the union of two ROIs by the area of ​​their intersection.

[0071] In this embodiment, the region of interest can be manually set first, then adaptively optimized, and the region of interest can be periodically reset to zero.

[0072] The deduplication process in step S100 includes: performing lightweight processing on the historical video stream in units of frame images; extracting adjacent frame images from the lightweight processed historical video stream, wherein the adjacent frame images include the current frame image and the next frame image of the current frame image; calculating the hash values ​​of the current frame image and the next frame image respectively; determining the Hamming distance between the current frame image and the next frame image based on the hash values ​​of the current frame image and the next frame image; and performing frame image deduplication processing on the historical video stream according to the Hamming distance to obtain a deduplicated video stream.

[0073] This involves first reducing the size of each frame in the original video stream and then converting it to grayscale to achieve lightweight processing of each frame. After lightweight processing, the time-domain image signal is converted into a frequency-domain signal using discrete cosine transform. The hash value of the image is calculated using the frequency-domain signal, and the Hamming distance between two adjacent frames is calculated based on the hash values ​​of two adjacent frames.

[0074] When the Hamming distance is less than or equal to the first threshold (the first threshold is the threshold of the Hamming distance), the current frame image contained in the historical video stream is removed and the next frame image is retained. The next frame image is then updated to the current frame image. The historical video stream is repeatedly deduplicated using the updated current frame image until a deduplicated video stream is obtained.

[0075] Alternatively, when the Hamming distance is greater than the first threshold, the current frame image and the next frame image are retained, and the next frame image is updated to the current frame image. The updated current frame image is then used to perform deduplication processing on the historical video stream in a loop until a deduplicated video stream is obtained.

[0076] In other words, if the Hamming distance between two adjacent frames is less than or equal to a first threshold, the current frame of the two adjacent frames is removed from the historical video stream, and the next frame is retained. The next frame is then updated to the current frame. The Hamming distance between the updated current frame and its next frame is then calculated again. If the distance is less than or equal to the first threshold, the updated current frame is removed, and its next frame is retained. This process is repeated on the historical video stream until the second-to-last frame is updated to the current frame, completing the deduplication process.

[0077] The step S200 of calculating the frame drop rate caused by deduplication includes: calculating the number of dropped frame images caused by deduplication; calculating the total number of original frame images contained in the historical video stream; and calculating the frame drop rate based on the number of dropped frame images and the total number of original frame images.

[0078] use Represents the frame drop rate. , This represents the number of frames that were discarded. This represents the total number of original frame images. The number of discarded frame images is the total number of original frame images minus the number of frame images in the deduplicated video stream.

[0079] The frame image removal process in step S100 includes: sequentially performing inter-frame difference processing, grayscale processing, and binarization processing on adjacent frame images in the deduplicated video stream to obtain a binarized image; extracting contours from the binarized image; filling the contours with area; and processing the deduplicated video stream based on the area filled on the binarized image of the adjacent frame image, the region of interest on the adjacent frame image, and the second threshold to obtain a target motion video stream composed of frame images recording the motion information of the target object.

[0080] Inter-frame difference processing involves calculating the absolute difference between pixel values ​​at the same location in two adjacent frames to obtain a difference image. This difference image is then converted to a grayscale image, and finally binarized to obtain the preprocessed adjacent frame images. The threshold used for binarization is 15, meaning that pixel values ​​greater than 15 in the grayscale image are set to 1, while pixel values ​​less than or equal to 15 are set to 0. In other words, the binarized image contains only 0 and 1 pixel values.

[0081] If the position of the target object does not change when acquiring adjacent frames, meaning the target object does not generate motion information, then theoretically all pixel values ​​in the difference image of adjacent frames will be 0. Therefore, the difference image of adjacent frames can be used to characterize whether the target object has moved.

[0082] The above-mentioned area filling of contours has the following beneficial effects: The purpose of area filling of contours is to identify which contours are valid and which are invalid. For example, unclosed contours cannot be filled, and unclosed contours will not be the contours formed by the target object, because the contours of the target object must be closed. Therefore, the contours of the target object can be identified through area filling.

[0083] When the area filled on the binarized image is greater than the second threshold, the target motion region is marked not only on the current frame image and the next frame image in adjacent images, but also on the coordinates of the center point of the smallest bounding rectangle. The target motion region is used to characterize the region where the pixel values ​​change due to the motion of the target object in the frame image. When the center of the target motion region is outside the region of interest, the current frame image in the adjacent frame image is deleted, while the next frame image in the adjacent frame image is retained. The next frame image is then used as the current frame image in the new adjacent frame image. The frame image in the deduplicated video stream that is after and adjacent to the next frame image is used as the next frame image in the new adjacent frame image. This process is repeated until the target motion video stream is obtained. Alternatively, when the center of the target motion region is inside the region of interest, the current frame image and the next frame image in the adjacent frame image are retained. The next frame image is then used as the current frame image in the new adjacent frame image. The frame image in the deduplicated video stream that is after and adjacent to the next frame image is used as the next frame image in the new adjacent frame image. This process is repeated until the target motion video stream is obtained.

[0084] When the area filled on the binarized image of the adjacent frame is less than or equal to the second threshold, it indicates that there is no target object in the adjacent frame image, or that the position of the target object has not changed during the time the surveillance camera collects the adjacent frame images. In both cases, the adjacent frame images collected contain redundant images. Therefore, it is necessary to delete the current frame image in the adjacent frame images to remove the redundant images.

[0085] In addition to using a second threshold to remove frames that do not record the motion information of the target object from the deduplicated video stream to obtain the target motion video stream, this embodiment can also use a frame difference threshold to remove frames that do not record the motion information of the target object from the deduplicated video stream to obtain the target motion video stream. That is, if the difference between adjacent frames in the deduplicated video stream is greater than the frame difference threshold, one of the adjacent frames is removed; otherwise, the adjacent frames are retained to obtain the target motion video stream. The frame difference threshold is then optimized in the same way as the second threshold.

[0086] The step S300 of calculating the false alarm rate caused by frame image removal processing includes: calculating the number of frames in the remaining frame images (which are frame images in the target motion video stream) after frame image removal processing where the target of interest does not appear, thus obtaining the number of false alarm frames; calculating the total number of frame images after frame image removal processing, and obtaining the false alarm rate based on the number of false alarm frames and the total number of frame images after frame image removal processing.

[0087] use Represents the false alarm rate. , This represents the number of frames in the moving video stream where the target of interest (ROI) does not appear. All moving objects are targets, which can be further divided into targets of interest and non-targets of interest. For example, both electric bikes and cars moving at an intersection are targets, but if the target is a car, then the car is the target of interest. This represents the total number of frames in the target motion video stream. For example, if the target objects are a moving electric scooter and a car, and target interest identification is performed on the target motion video stream, finding that the car does not appear in 10 frames, then... The value is 10.

[0088] The identification of the target of interest in step S100 includes: applying a target recognition algorithm to each frame of the target motion video stream to predict the probability of the target of interest appearing in each frame; and extracting the frame image where the target of interest is located from each frame image based on the probability of the target of interest appearing in each frame image.

[0089] In other words, a target recognition algorithm is applied to each frame of the target motion video stream to predict the probability that the target object in the frame image is the target of interest. When the probability is greater than a preset value, the target object is determined to be the target of interest; otherwise, the target object is not the target of interest.

[0090] In this embodiment, step S400, determining the degree of jump of the target of interest between keyframes, includes: determining the cross-union ratio (CUP) of the target of interest pixels in adjacent keyframes; determining the jump index of the target of interest between adjacent keyframes based on the CUP; determining the mean of the jump index, and using the mean of the jump index as the degree of jump.

[0091] Preferably, the jump index is calculated based on the intersection-union ratio (IU), that is, based on the following formula, the jump index generated by the target of interest between adjacent keyframes is calculated. :

[0092] ;

[0093] In the formula, Representing the A set consisting of the pixels of the target of interest on each keyframe. Representing the The set of pixels representing the target of interest on keyframes, the first keyframe being... The first keyframe and the first Each keyframe constitutes an adjacent keyframe. The symbol representing the intersection of sets. The union symbol represents sets. A symbol representing the number of pixels within a statistical set.

[0094] Each adjacent keyframe corresponds to one Calculate all The mean value yields the degree of hopping of the target of interest between adjacent keyframes. ,in It represents the mean.

[0095] In another embodiment, the jump index can also be calculated using the following formula. :

[0096] ;

[0097] In the formula, represent and The Jaccard similarity coefficient is used to macroscopically measure the stability of an object of interest. This represents the average trajectory of all targets of interest. The trajectory is formed by the positions of the detection boxes where the same target of interest appears in adjacent keyframes, and the detection box positions represent the locations of the targets of interest. and These represent two weighting coefficients.

[0098] In step S500, the first threshold and the second threshold are optimized based on the stability, the frame drop rate, the false alarm rate, and the hopping degree. This involves comparing the stability, frame drop rate, false alarm rate, and hopping degree with their respective preset thresholds. Based on the comparison results, the first threshold and / or the second threshold are increased, decreased, or maintained according to a preset adjustment strategy. Specifically, this includes the following four scenarios:

[0099] First case: If the stability is higher than the stability threshold, the frame drop rate is lower than the drop threshold, the false alarm rate is lower than the false alarm threshold, and the jump degree is lower than the jump threshold, then increase the first threshold and the second threshold.

[0100] use This represents the first threshold used for processing historical video streams. The Hamming distance mentioned above is represented by... This represents the first threshold after the increase. , This represents the increase factor of the first threshold. It is an integer.

[0101] use This represents the second threshold used in processing deduplicated video streams. Representing the area mentioned above, using This represents the second threshold after the increase. , This represents the increase factor for the second threshold. The value can be 0.1.

[0102] When the stability of the region of interest is high and the degree of jump is low, it indicates that the position of the moving target does not change frequently during the period of acquiring the historical video stream. Therefore, the historical video stream should contain many duplicate frames. However, using the current first threshold to process the original video stream results in a low frame drop rate, which is inconsistent with reality. This indicates that the current first threshold is not suitable for deduplication, and the current first threshold needs to be increased to remove more duplicate frames. Based on the same principle, when the position of the moving target does not change frequently during the period of acquiring the historical video stream, the target should appear continuously in the frame images, leading to a higher false alarm rate. However, the calculated false alarm rate is lower than expected, which is inconsistent with reality. Therefore, this indicates that the current second threshold is not suitable for filtering moving targets, and the current second threshold needs to be increased.

[0103] The first case in this embodiment is applicable to processing video streams acquired in stable scenes, where a stable scene is one in which the position of the target object is relatively stable.

[0104] The second scenario: If the stability is higher than the stability threshold, the frame drop rate is higher than the drop threshold, and the jump degree is higher than the jump threshold, then the first threshold and the second threshold are reduced.

[0105] use This represents the first threshold after reduction. , The reduction factor represents the first threshold. It is an integer.

[0106] use This represents the second threshold after reduction. , The reduction factor representing the second threshold, in this example equal .

[0107] The second scenario in this example mainly corresponds to situations where the target motion is significant but over-filtered (which can be called a missed detection scenario). Its optimization can be found in the adaptive control process described in this article, such as... Figure 5 As shown.

[0108] The third scenario: If the stability is higher than the stability threshold and the false alarm rate is higher than the false alarm threshold, then the second threshold is increased while the first threshold remains unchanged.

[0109] In this case, , Represents greater than The increase factor, The value can be 0.2.

[0110] The third scenario in this embodiment is applicable to processing video streams acquired in false alarm scenarios. False alarm scenarios refer to video frames that do not contain the target of interest in a relatively high proportion after being filtered by the frame image removal process.

[0111] The fourth scenario: If the stability is lower than the stability threshold, then the first threshold is reduced while the second threshold remains unchanged.

[0112] A stability threshold below the stability threshold indicates that the center of the region of interest has shifted, which means that the probability of repeated frames in the video stream has decreased. Therefore, the first threshold can be reduced while the region of interest is updated, with the goal of tracking the region of interest.

[0113] In this embodiment, when A sharp increase (e.g.) When the threshold is greater than ten times the jump threshold, and the target of interest on the keyframe is located inside the edge of the region of interest, the region of interest needs to be re-marked and the first threshold needs to be re-initialized in the new video stream.

[0114] This embodiment uses This represents the region of interest extracted from the historical video stream. Represents the original video stream, using This represents a new video stream, captured after the historical video streams. and Intersection and Union Ratio Judgment Does it match? For example, when the crossover ratio is greater than 0.4, then Matched with So based on right Perform smoothing processing, using Represents the result after smoothing. :

[0115] ;

[0116] In the formula, represent The weights, using Replacement for extraction from live video stream Based on Extract the keyframe containing the target of interest from the real-time video stream.

[0117] if Not matched Then directly based on Extract the keyframe containing the target of interest from the real-time video stream.

[0118] This embodiment also includes a counter, which is used to count matches. of .

[0119] In this embodiment, the target object is determined to be stationary if the moving distance of the target object in the continuous frame image is less than a set distance. Once the target object is determined to be stationary, the region of interest is initialized. At this time, the target object used to mark the region of interest is accurate. It is impossible for the moving target object to move less than the set distance in the continuous frame image. Therefore, it means that the target object selected for marking the region of interest is wrong, which leads to the marked region of interest being wrong. Therefore, it is necessary to initialize the region of interest to adjust it.

[0120] Example 2, based on Example 1, calculates the region of interest in the planner, calculates the frame drop rate and false alarm rate in the analyzer, and determines in the decision engine whether to increase the first threshold and the second threshold or decrease the first threshold and the second threshold.

[0121] Both the planner and the analyzer operate in parallel with the decision engine, and the planner and the analyzer exchange data with the decision engine through shared memory.

[0122] The planner operates at a low frequency, the analyzer at a medium frequency, and the decision engine at a medium frequency. The planner, analyzer, and decision engine operate at different frequencies, which can reduce the instantaneous computational pressure.

[0123] The size of the data output by the planner, analyzer, and decision engine must be strictly limited to a range consisting of the maximum and minimum values. The data output by the decision engine must also be subjected to a first-order low-pass filter to avoid system jitter caused by parameter abrupt changes.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video frame resilience extraction spatiotemporal parameter adaptive optimization control method, characterized in that, include: The historical video stream is sequentially deduplicated, frame images are removed, and targets of interest are identified to obtain a video stream of interest consisting of key frames containing the targets of interest. Extract the region of interest from the video stream of interest and determine the stability of the region of interest, wherein the region of interest is a spatial parameter; Obtain the first threshold used for deduplication and calculate the frame drop rate caused by deduplication. The first threshold is used to determine the repetition of frame images. The second threshold used for frame image removal processing is obtained, and the false alarm rate caused by frame image removal processing is calculated. The second threshold is used to determine the degree of motion of the target object between frame images. The target object includes moving objects, and the moving objects include the target of interest. The first threshold and the second threshold are time-series parameters. Determine the degree of jump of the target of interest between keyframes; Based on the stability, the frame drop rate, the false alarm rate, and the degree of jump, the first threshold and the second threshold are optimized. The optimized first threshold and the second threshold are used to process the real-time video stream to extract key frames. Based on the stability, the frame drop rate, the false alarm rate, and the degree of hopping, the first threshold and the second threshold are optimized, including: The stability, the frame drop rate, the false alarm rate, and the degree of jump are compared with their respective preset thresholds; If the comparison results show that the stability is higher than the stability threshold, the frame drop rate is lower than the drop threshold, the false alarm rate is lower than the false alarm threshold, and the jump degree is lower than the jump threshold, then the first threshold and the second threshold are increased. If the comparison results show that the stability is higher than the stability threshold, the frame drop rate is higher than the drop threshold, and the jump degree is higher than the jump threshold, then the first threshold and the second threshold are reduced. If the comparison result shows that the stability is higher than the stability threshold and the false alarm rate is higher than the false alarm threshold, then the second threshold is increased while the first threshold remains unchanged. If the comparison result indicates that the stability is lower than the stability threshold, then the first threshold is reduced while the second threshold remains unchanged.

2. The video frame resilience extraction spatiotemporal parameter adaptive optimization control method as described in claim 1, characterized in that, Extracting the region of interest from the video stream of interest and determining the stability of the region of interest includes: Determine the heat map corresponding to the frame image of the video stream of interest, the heat map being used to characterize the location of the target object recorded by the video stream of interest; The connected regions of the heatmap are determined, and the connected regions of the heatmaps corresponding to each frame image are taken as the local regions of interest corresponding to each frame image. The local regions of interest corresponding to a set number of consecutive frame images are accumulated to obtain the regions of interest corresponding to the historical video stream. The stability of the region of interest (ROI) corresponding to the historical video stream is determined based on the difference between the ROI corresponding to the historical video stream and the pre-defined ROI.

3. The video frame resilience extraction spatiotemporal parameter adaptive optimization control method as described in claim 1, characterized in that, The frame drop rate resulting from deduplication is statistically analyzed, including: Count the number of discarded frames resulting from deduplication; The total number of original frame images contained in the historical video stream is counted; The frame drop rate is calculated based on the number of dropped frame images and the total number of original frame images.

4. The video frame resilience extraction spatiotemporal parameter adaptive optimization control method as described in claim 1, characterized in that, The false alarm rate caused by frame image removal processing is statistically analyzed, including: The number of frames in the remaining frames after frame image removal processing that do not contain the target of interest is obtained as the number of false alarm frames. The total number of frame images retained after frame image removal is counted, and the false alarm rate is obtained based on the number of false alarm frames and the total number of frame images retained after frame image removal.

5. The video frame resilience extraction spatiotemporal parameter adaptive optimization control method as described in claim 1, characterized in that, Identify targets of interest, including: Based on the target detection model, the category and location of the target object are identified; Based on the category of the target objects, the target of interest and its location are identified from the target objects.

6. The video frame resilience extraction spatiotemporal parameter adaptive optimization control method as described in claim 1, characterized in that, Determining the degree of jump of the target of interest between keyframes includes: Determine the intersection-union ratio (CIG) of the target pixels of interest on adjacent keyframes; Based on the intersection-union ratio, determine the jump index of the target of interest between adjacent keyframes; Determine the mean of the jump index, and use the mean of the jump index as the jump degree.

Images