Heterogeneous ai chip event-driven spatiotemporal roi monitoring closed-loop video compression method and system

Abstract

The application provides a heterogeneous AI chip event-driven space-time ROI monitoring closed-loop video compression method and system, belongs to the technical field of intelligent video monitoring, computer vision and video coding, and is used for solving the problems of AI inference frame and coding frame time asynchronization, ROI picture flicker caused by detection frame jitter and fixed ROI coding parameter unable to adapt to bandwidth change in related technologies. The application writes video frames into a shared buffer and forms a monitoring branch and a coding branch based on the same video frame, the monitoring branch performs target detection and generates a monitoring event state, the coding branch determines a coding priority according to the event state and generates a space-time importance map, determines the quantization parameter offset of each coding unit after asymmetric time domain smoothing, and then adjusts the coding parameter in combination with closed-loop code rate control and quality feedback, so that event-driven adaptive video compression is realized, the key target recognition accuracy is ensured, and the background video code rate is greatly reduced.

Description

Technical Field

[0001] This application relates to the technical fields of intelligent video surveillance, computer vision and video coding, and in particular to a closed-loop video compression method and system for event-driven spatiotemporal ROI monitoring of heterogeneous AI chips. Background Technology

[0002] Video surveillance systems typically require continuous acquisition, transmission, and storage of high-definition video. In edge computing scenarios, heterogeneous AI chips with AI inference units and video encoding units are widely deployed in smart cameras, smart analysis boxes, and industrial monitoring equipment to simultaneously perform video content analysis and video compression encoding tasks. However, the target detection frame rate of the AI ​​inference unit is usually much lower than the encoding frame rate of the video encoding unit, resulting in most encoded frames failing to obtain real-time target detection results, making it difficult to achieve content-aware adaptive encoding.

[0003] In existing technologies, ROI video encoding typically sets different quantization parameter offsets based on manually specified fixed regions or target detection boxes. Some schemes use target detection algorithms to extract key targets in video frames and set the detection box regions as high-priority ROI regions. Other schemes utilize target tracking algorithms to track detected targets across frames to reduce the computational overhead of target detection. Furthermore, closed-loop bitrate control technology is widely used in video encoders, dynamically adjusting encoding parameters by feeding back the error between the actual number of encoded bits and the target bitrate.

[0004] However, the aforementioned existing technologies still have the following drawbacks: First, the time asynchrony between the AI ​​inference frequency and the encoding frame rate results in a large number of encoded frames lacking accurate ROI information, and directly reusing the detection results of the previous frame will cause target position deviation; Second, the inter-frame jitter and brief missed detections of the target detection box will cause the ROI region to frequently jump between encoded frames, resulting in obvious flickering of the compressed image quality; Third, the ROI quantization parameter offset is usually a fixed value, which cannot be dynamically adjusted according to the actual bitrate consumption and scene complexity, which may lead to a serious deterioration in the quality of key targets when the bitrate is limited; Fourth, existing encoders lack closed-loop verification of the completion quality of the compressed ROI region recognition task, and over-compression may lead to the dilemma that the human eye can see it but the machine recognition task fails. Summary of the Invention

[0005] This application provides a closed-loop video compression method and system for event-driven spatiotemporal ROI monitoring of heterogeneous AI chips. It can achieve adaptive video compression for machine vision tasks through event state-driven dynamic mapping of encoding priorities, asymmetric temporal smoothing, and closed-loop quality feedback for recognition tasks.

[0006] In a first aspect, this application provides a video compression method. The method is applied to a processor having an AI inference unit and a video encoding unit, and the method includes:

[0007] Acquire video frames, write the video frames into a shared buffer, and form a monitoring branch and an encoding branch based on the same video frame;

[0008] The monitoring branch performs image preprocessing on the video frame and then sends it to the AI ​​inference unit for target detection to obtain the detected target.

[0009] The detected target is tracked, a target trajectory is generated, and a unique target identifier is assigned to each target;

[0010] Based on the target trajectory and the preset static monitoring area, determine the overlap relationship and movement trend between the target and the static monitoring area, and generate the monitoring event status;

[0011] The encoding priority of the target is determined based on the state of the monitored event, and the target position in the current video frame to be encoded is predicted based on the target trajectory.

[0012] A spatiotemporal importance map is generated based on the target location, and the spatiotemporal importance map contains the spatiotemporal importance of each coding unit in the video frame;

[0013] The spatiotemporal importance map is smoothed in the temporal domain. The quantization parameter offset of each coding unit is generated based on the smoothed spatiotemporal importance map. The final coding quantization parameter of each coding unit is determined based on the quantization parameter offset and the basic quantization parameter.

[0014] The video encoding unit compresses and encodes the video frames according to the final encoding quantization parameters of each encoding unit to generate a compressed video stream;

[0015] Video frames are extracted from the compressed video stream for quality assessment, and the encoding parameters are adjusted based on the quality assessment results.

[0016] By adopting the above technical solution, video frames are written to a shared buffer and form a parallel dual-branch structure based on the same frame, eliminating data duplication between monitoring and encoding, reducing memory overhead and transmission latency. The monitoring branch uses AI inference units for target detection, and the encoding branch uses video encoding units for compression encoding. The two branches work in parallel based on the same video frame, enabling AI inference and video encoding to be executed in a pipelined parallel manner. By generating monitoring event states and dynamically mapping encoding priorities, the spatiotemporal behavior state of the target is directly converted into encoding control signals, realizing event-driven encoding decisions. Through cross-frame target position prediction, low-frequency AI inference results can cover every encoding frame, solving the problem of mismatch between AI inference frame rate and encoding frame rate. Through per-encoding-unit calculation of spatiotemporal importance maps and quantization parameter offsets, fine-grained ROI encoding control is realized. Through post-compression quality evaluation and parameter adjustment, a complete closed-loop feedback link is formed, enabling the encoder to adaptively adjust the encoding strategy according to the actual task quality after compression.

[0017] Furthermore, the monitoring branch performs image preprocessing on the video frames and then sends them to the AI ​​inference unit for target detection, specifically including:

[0018] The monitoring branch scales the video frame to generate a low-resolution monitoring image, and performs color space conversion and pixel value normalization on the low-resolution monitoring image.

[0019] The preprocessed low-resolution monitoring image is sent to the AI ​​inference unit, which performs target detection, instance segmentation, key point recognition, or abnormal state recognition operations, and outputs the detected target information.

[0020] The detection target information includes target category, detection confidence level, target bounding box, target service priority, and the acquisition timestamp of the video frame;

[0021] The target bounding box is a rectangular bounding box or a segmented region outline, and the target service priority is pre-configured according to the target category.

[0022] By adopting the above technical solutions, video frames are scaled to generate low-resolution monitoring images, reducing the amount of input data for the AI ​​inference unit and decreasing inference latency and computational resource consumption. Color space conversion and pixel value normalization ensure that the input data meets the data format requirements of the AI ​​inference unit, guaranteeing detection accuracy. The AI ​​inference unit supports multiple visual tasks such as target detection, instance segmentation, key point recognition, and abnormal state recognition, expanding the range of detectable target types. The detected target information includes target category, confidence level, bounding box, business priority, and acquisition timestamp, providing complete structured input data for subsequent event state determination, encoding priority mapping, and cross-frame position prediction. The target bounding box supports both rectangular boxes and segmented region outlines, balancing detection efficiency and accuracy.

[0023] Furthermore, predicting the target position in the current video frame to be encoded based on the target trajectory specifically includes:

[0024] Obtain the inference timestamp of the AI ​​inference unit, the acquisition timestamp of the video frame, and the encoding timestamp of the video frame to be encoded;

[0025] When the detected target output by the AI ​​inference unit does not cover the video frame to be encoded, the position coordinates of the detected target in the video frame are interpolated or extrapolated in the temporal domain according to the first time difference between the inference timestamp and the acquisition timestamp and the second time difference between the encoding timestamp and the acquisition timestamp, so as to generate a predicted target region corresponding to the video frame to be encoded.

[0026] The predicted target region is generated based on at least one of the following information: the target motion speed and direction calculated based on the position difference of the detected target in two adjacent frames, the predicted position obtained by performing Kalman filtering or state estimation on the position coordinates of the detected target, the optical flow field obtained by optical flow calculation on the video frame, the motion vector generated by the video coding unit, and the matching result obtained by feature point matching on adjacent video frames.

[0027] By adopting the above technical solution, the inference timestamp, acquisition timestamp, and encoding timestamp are obtained. The time difference relationship among the three is used for coordinate interpolation or extrapolation, so that the predicted target area is accurately aligned with the video frame to be encoded in the time domain. This solves the problem of ROI position deviation caused by the time asynchrony between the AI ​​inference frame and the encoding frame. When the AI ​​inference does not cover the current encoding frame, the cross-frame prediction mechanism can be automatically triggered, realizing full frame rate ROI coverage under low-frequency AI inference conditions. The predicted target area is generated based on at least one of the following information: target velocity direction, Kalman filtering, optical flow field, encoder motion vector, and feature matching. The fusion of multi-source information improves the accuracy and robustness of target position prediction. In particular, the motion vector generated by the video encoding unit is reused as the prediction basis without additional computational overhead, making full use of the hardware resources of heterogeneous chips.

[0028] Furthermore, based on the target trajectory and the preset static monitoring area, the overlap relationship and movement trend between the target and the static monitoring area are determined, and a monitoring event status is generated, specifically including:

[0029] Calculate the overlap ratio between the target bounding box and the static monitoring area. The overlap ratio is the ratio of the intersection area of ​​the target bounding box and the static monitoring area to the area of ​​the target bounding box, or the ratio of the intersection area to the area of ​​the static monitoring area, or the ratio of the intersection area to the union area of ​​the target bounding box and the static monitoring area.

[0030] The monitoring event status is determined based on the comparison result between the overlap ratio and the preset overlap threshold, the positional relationship between the target center point and the boundary of the static monitoring area, the movement direction of the target crossing the boundary of the static monitoring area, and the number of consecutive frames the target stays in the static monitoring area.

[0031] The monitored event status includes the status outside the area, the status of approaching the area, the status of entering the area, the status of staying within the area, the status of leaving and maintaining the status, and the status of recovery.

[0032] When the monitoring event status is "entering", the target encoding priority is immediately increased to the first priority level;

[0033] When the monitored event is in the leave hold state, the target encoding priority gradually decreases from the first priority level to the second priority level within a preset hold duration according to a preset decay rate.

[0034] By adopting the above technical solution, the accuracy of event determination and anti-interference ability are improved by calculating the overlap ratio between the target bounding box and the static monitoring area, and comprehensively considering multiple dimensions such as overlap ratio threshold comparison, center point position relationship, crossing direction, and continuous dwell time. Six monitoring event states are defined: outside the area, approaching, entering, staying within the area, leaving and maintaining, and recovering. The spatiotemporal interaction relationship between the target and the monitoring area is refined, providing rich event semantics for fine-tuning of encoding priority. When the target enters the monitoring area, the encoding priority is immediately increased to the first priority level, ensuring real-time capture and high-quality encoding of critical events. After the target leaves, the encoding priority gradually decreases within a preset holding time rather than dropping immediately, avoiding visual flicker caused by sudden changes in image quality when the target just leaves the monitoring area, while reserving encoding quality margin for possible subsequent target return.

[0035] Furthermore, a spatiotemporal importance map is generated based on the target location, specifically including:

[0036] The video frame is divided into multiple coding units, which are macroblocks, coding tree units, or ROI grid units supported by a hardware encoder.

[0037] For the k-th coding unit in the t-th frame, spatial importance is calculated based on the spatial coverage relationship between the coding unit and the target location, the target category priority, and the context protection region. The spatial importance is the product of the overlap weight between the target bounding box and the coding unit and the target category priority, wherein the overlap weight is calculated based on the distance decay between the coding unit and the center point of the target bounding box.

[0038] The context protection region is the area formed by expanding the target bounding box outward according to a preset expansion ratio, and the context protection region covers the motion edge region around the target.

[0039] The spatiotemporal importance is obtained by performing time-domain filtering on the spatial importance;

[0040] The quantization parameter offset corresponding to the context protection region is between the quantization parameter offset corresponding to the target core region and the quantization parameter offset corresponding to the background region.

[0041] By adopting the above technical solution, video frames are divided into coding units and spatiotemporal importance is calculated for each unit, providing a fine-grained control basis for subsequent coding unit-level quantization parameter allocation. Spatial importance is the product of the overlap weight between the target bounding box and the coding unit and the target category priority. The overlap weight is calculated based on the distance decay between the coding unit and the target center point, so that the importance distribution smoothly transitions from the target center outward, avoiding image quality gaps at the ROI boundary. The context protection region is formed by extending the target bounding box outward, covering the motion edge region around the target, protecting the temporal continuity of the target motion and behavioral context information, which is beneficial to subjective image quality and subsequent video analysis tasks. The quantization parameter offset of the context protection region is between the core region and the background region, realizing a three-level coding quality gradient of the target core region, context protection region, and background region, reducing background bitrate overhead while ensuring target quality.

[0042] Furthermore, temporal smoothing is performed on the spatiotemporal importance map, specifically including:

[0043] The spatiotemporal importance map is recursively filtered in the time domain to obtain the current frame importance and the previous frame importance of the k-th coding unit in the t-th frame.

[0044] Compare the importance of the current frame with the importance of the previous frame;

[0045] When the importance of the current frame is greater than or equal to the importance of the previous frame, the importance of the current frame is smoothed using the first filtering inertia coefficient to obtain the smoothed importance.

[0046] When the importance of the current frame is less than the importance of the previous frame, the importance of the current frame is smoothed using the second filtering inertia coefficient to obtain the smoothed importance.

[0047] The first filter inertia coefficient is smaller than the second filter inertia coefficient.

[0048] By adopting the above technical solution, asymmetric temporal filtering is implemented on the spatiotemporal importance map. When the importance increases, a smaller filtering inertia coefficient is used, enabling the ROI region to respond quickly to the sudden appearance or rapid entry of the target, ensuring the timely capture of critical events. When the importance decreases, a larger filtering inertia coefficient is used, causing the ROI region to decay slowly, effectively suppressing frequent jumps in the ROI region and image quality flicker caused by target detection frame jitter and brief missed detections. The asymmetric filtering parameter design is specifically designed for the asymmetric timeliness requirements of target appearance and disappearance in monitoring scenarios. The fast rise and slow fall response characteristics ensure the timeliness of event capture while improving the temporal stability of the encoded image quality.

[0049] Further, the final coding quantization parameters for each coding unit are determined based on the quantization parameter offset and the basic quantization parameters, specifically including:

[0050] Obtain the preset target bitrate and count the actual number of encoded bits in the encoded video frames;

[0051] The difference between the target bitrate and the actual number of encoded bits is calculated as the bitrate error, and the basic quantization parameters of the next frame or the next image group are updated based on the bitrate error.

[0052] The final encoding quantization parameter is the algebraic sum of the basic quantization parameter and the quantization parameter offset, wherein the quantization parameter offset is determined according to the smoothed importance and importance quantization mapping relationship.

[0053] When the actual number of encoded bits exceeds the target bit rate, the quantization parameter offset of the corresponding region is increased in order of priority: background region, context region, non-alarm target region, and alarm core target region.

[0054] By adopting the above technical solution, the target bitrate is obtained and the actual number of encoded bits is counted. The bitrate error is calculated and the basic quantization parameters of the next frame or the next image group are updated accordingly, forming a closed-loop bitrate control. This ensures that the actual output bitrate converges to the target bitrate, avoiding bitrate runaway caused by ROI encoding. Finally, the encoding quantization parameter is the algebraic sum of the basic quantization parameter and the quantization parameter offset, realizing the collaborative control of superimposed ROI differentiated encoding under the global bitrate control framework. When the bitrate exceeds the limit, the quantization parameter offset is increased step by step according to the priority order of background area, context area, non-alarm target area, and alarm core target area. That is, the encoding quality is sacrificed step by step according to the order of business importance from low to high, giving priority to ensuring that the encoding quality of alarm core targets is not affected, thus achieving rigid guarantee of the quality of critical tasks under bitrate-limited conditions.

[0055] Furthermore, video frames are extracted from the compressed video stream for quality assessment, and encoding parameters are adjusted based on the quality assessment results. Specifically, this includes:

[0056] Keyframes or ROI images are extracted from the compressed video stream according to a preset period, and quality assessment or lightweight target recognition operations are performed on the extracted keyframes or ROI images.

[0057] When any of the following triggering conditions are met, the quantization parameter offset corresponding to the ROI region is reduced or the inference frequency of the AI ​​inference unit is increased: the confidence of the compressed target detection decreases by more than a preset confidence threshold, the clarity index of the ROI region decreases by more than a preset clarity threshold, the number of target feature points is less than a preset feature point number threshold, text recognition, license plate recognition, or instrument recognition fails, or the reliability of continuous frame target tracking decreases by more than a preset reliability threshold.

[0058] When the device load exceeds a preset load threshold, the device temperature exceeds a preset temperature threshold, or the network bandwidth is lower than a preset bandwidth threshold, the inference frequency of the AI ​​inference unit in non-alarm scenarios is reduced.

[0059] When a target approaches the static monitoring area or the monitoring event is in an alarm state, the inference frequency of the AI ​​inference unit is increased.

[0060] By adopting the above technical solution, key frames or ROI images are extracted from the compressed bitstream according to a preset period to perform quality assessment or lightweight target recognition. This achieves periodic detection of the task quality of the compressed ROI region, allowing for the evaluation of key region quality without decoding the entire frame, thus reducing the computational overhead of quality feedback. When detection confidence decreases, clarity decreases, feature points are insufficient, recognition fails, or tracking reliability decreases, the quantization parameter offset is proactively reduced or the AI ​​inference frequency is increased. The quality of the recognition task completion is taken as the direct target of encoding optimization, overcoming the limitation of traditional encoders that only optimize subjective human quality, and realizing a closed loop of encoding quality for machine vision tasks. When the device load, temperature, or bandwidth exceeds the threshold, the inference frequency in non-alarm scenarios is reduced, and computing power is proactively released to ensure system stability when resources are limited. When the target approaches the monitoring area or enters the alarm state, the inference frequency is proactively increased, and the investment of perception resources is dynamically increased when critical events occur, improving the perception accuracy and response speed in high-risk scenarios.

[0061] Secondly, this application provides a video compression system. The system is used to perform the method as described in the first aspect, and the system includes:

[0062] The video acquisition module is used to acquire video frames, write the video frames into a shared buffer, and form a monitoring branch and an encoding branch based on the same video frame;

[0063] The AI ​​inference module is set in the monitoring branch and is used to perform target detection after image preprocessing of the video frame to obtain the detected target.

[0064] The target tracking module is used to track the detected target, generate the target trajectory, and assign a unique target identifier to each target;

[0065] The event status determination module is used to determine the overlap relationship and movement trend between the target and the static monitoring area based on the target trajectory and the preset static monitoring area, and to generate the monitoring event status.

[0066] The ROI generation module is used to determine the encoding priority of the target based on the state of the monitored event, predict the target position in the current video frame to be encoded based on the target trajectory, and generate a spatiotemporal importance map based on the target position. The spatiotemporal importance map contains the spatiotemporal importance of each encoding unit in the video frame.

[0067] A time-domain smoothing module is used to smooth the spatiotemporal importance map in the time domain;

[0068] The quantization parameter calculation module is used to generate the quantization parameter offset of each coding unit based on the smoothed spatiotemporal importance map, and to determine the final coding quantization parameter of each coding unit based on the quantization parameter offset and the basic quantization parameter.

[0069] A video encoding module, located in the encoding branch, is used to compress and encode the video frame according to the final encoding quantization parameters of each encoding unit to generate a compressed video stream;

[0070] The quality feedback module is used to extract video frames from the compressed video stream for quality assessment and adjust the encoding parameters based on the quality assessment results.

[0071] The output module is used to output compressed video streams and metadata.

[0072] By adopting the above technical solutions, the video acquisition module achieves zero-copy sharing of video frames between the monitoring branch and the encoding branch through a shared buffer, reducing data transport overhead; the AI ​​inference module and the video encoding module are deployed in the monitoring branch and the encoding branch respectively, realizing hardware-level pipeline parallelism of AI inference and video encoding, giving full play to the multi-core parallel processing capabilities of heterogeneous AI chips; the target tracking module and the event state determination module convert the semantics of video content into structured event state signals, providing task-oriented control input for encoding decisions; the ROI generation module integrates event priority and cross-frame target position prediction to generate a spatiotemporal importance map, realizing a complete conversion link from event semantics to encoding parameters; the temporal smoothing module suppresses inter-frame jitter in the ROI region, improving the temporal stability of encoded image quality; the quantization parameter calculation module combines bitrate closed-loop feedback and hierarchical priority step-by-step degradation strategy to achieve key target quality assurance under the premise of controllable bitrate; the quality feedback module uses the quality of the recognition task in the compressed ROI region as a feedback signal, enabling the system to have task-oriented self-optimization capabilities, forming a complete perception, decision-making, encoding, and feedback closed loop.

[0073] In summary, this application has at least the following beneficial effects:

[0074] A closed-loop video compression method and system for event-driven spatiotemporal ROI monitoring of heterogeneous AI chips is provided. By dynamically mapping the encoding priority driven by event state and asymmetric temporal smoothing, the background video bitrate is reduced while ensuring the recognition accuracy of key targets.

[0075] By predicting target location across frames, low-frequency AI inference results can be covered in every encoded frame, thus solving the problem of missing ROI caused by the mismatch between inference frame rate and encoding frame rate in edge heterogeneous AI chips.

[0076] By using closed-loop feedback on the quality of the compressed recognition task, the shortcomings of traditional encoders that only optimize subjective image quality for the human eye while ignoring machine recognition tasks are overcome, thus achieving adaptive optimization of encoding quality for machine vision tasks.

[0077] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0078] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0079] Figure 1 A schematic diagram of a video compression system according to an embodiment of this application is shown.

[0080] Figure 2 A flowchart of a video compression method according to an embodiment of this application is shown. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0082] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0083] This application provides a closed-loop video compression method and system for event-driven spatiotemporal ROI monitoring of heterogeneous AI chips. Through dynamic mapping of encoding priorities driven by event state and closed-loop quality feedback oriented towards recognition tasks, it significantly reduces the background video bitrate while ensuring the clarity of key targets, and realizes dynamic adaptation of encoding strategies to target behavior.

[0084] Firstly, embodiments of this application disclose a video compression system. The video compression system provided in this application can be deployed in edge intelligent cameras, intelligent video analysis boxes, industrial vision inspection equipment, or video surveillance servers, and is used for real-time intelligent analysis and adaptive compression encoding of high-definition videos collected in monitoring scenarios.

[0085] Figure 1 A schematic diagram of a video compression system according to an embodiment of this application is shown.

[0086] Reference Figure 1 The system is built on a heterogeneous processor with an AI inference unit and a video encoding unit. This heterogeneous processor includes a central processing unit (CPU), an AI inference unit, a video encoding unit, a memory controller, and video input / output interfaces. The CPU is communicatively connected to the AI ​​inference unit, the video encoding unit, and the memory controller. The CPU performs logical operations corresponding to target tracking, event state determination, spatiotemporal importance map generation, temporal smoothing, quantization parameter calculation, and quality feedback. The AI ​​inference unit performs neural network inference operations corresponding to target detection. The video encoding unit performs video frame compression encoding operations. The system also includes shared physical memory coupled to the processor. This shared physical memory is divided into a shared buffer and a bitstream output buffer. The shared buffer caches raw video frame data, and the bitstream output buffer caches the compressed video bitstream. The system also includes video input and video output interfaces coupled to the processor. The video input interface connects to a video acquisition device to acquire raw video frames, and the video output interface outputs the compressed video bitstream and metadata.

[0087] The video capture device is a high-definition network camera, USB camera, or HDMI camera. The captured video frames are sent to the processor via the video input interface. The processor writes the video frames to a shared buffer in shared physical memory, and associates each video frame with a frame number, capture timestamp, image width and height information, and pixel format information in the shared buffer. Based on the same video frame, the processor internally forms a monitoring branch and an encoding branch. Both branches access the video frame data through the same physical address in the shared buffer, eliminating the need for data copying between the two branches. The processing frame rate of the monitoring branch is lower than that of the encoding branch; for example, the monitoring branch performs AI inference at a rate of eight to twelve frames per second, while the encoding branch performs video compression encoding at a rate of twenty-five or thirty frames per second.

[0088] In the monitoring branch, the central processing unit first performs image preprocessing on the video frames. Image preprocessing includes scaling the video frames to generate low-resolution monitoring images, performing color space conversion on the low-resolution monitoring images to convert the input pixel format to a data format supported by the AI ​​inference unit, and performing normalization on the pixel values ​​of the low-resolution monitoring images to map the pixel values ​​to the numerical range required for neural network inference. The preprocessed low-resolution monitoring images are then sent to the AI ​​inference unit, which loads a pre-trained target detection neural network model and performs inference operations, outputting target detection information. Target detection information includes target category, detection confidence, target bounding box, target service priority, and video frame acquisition timestamp. Target categories include at least personnel, motor vehicles, non-motor vehicles, fireworks, safety helmets, instrument pointers, and equipment components. The target bounding box is a rectangular bounding box or a segmented region outline. Target service priority is pre-set during the system configuration phase based on the target category; for example, personnel and fireworks are configured as the highest priority, while motor vehicles and non-motor vehicles are configured as the second highest priority.

[0089] The central processing unit (CPU) performs target tracking processing on the detected targets output by the AI ​​inference unit. Target tracking is implemented using a combination of Kalman filtering and Hungarian matching algorithms. The Kalman filtering algorithm predicts the target's state vector in the next frame, while the Hungarian matching algorithm associates the detected target in the current frame with existing target trajectories. The CPU assigns a unique target identifier to each tracked target and retains this identifier for a preset tracking survival time after the target disappears from the detection field of view. The preset tracking survival time ranges from two to five seconds, preferably three seconds. The CPU generates a target trajectory based on the target detection results and historical trajectory data. The target trajectory includes the target's position coordinate sequence, velocity information, and direction of motion information for each frame in the processed video frame sequence.

[0090] The central processing unit (CPU) determines the monitoring event status based on the target trajectory and a preset static monitoring area. The static monitoring area is pre-defined by the system user through a configuration interface and stored in a configuration area within shared physical memory. The static monitoring area can be any combination of one or more of the following: rectangular, polygonal, polyline, electronic fence, or masked area. The CPU calculates the overlap ratio between the target bounding box and the static monitoring area. This overlap ratio is the ratio of the intersection area to the target bounding box area, or the intersection area to the static monitoring area area, or the intersection area to the union area of ​​the target bounding box and the static monitoring area. The specific ratio used is determined by the system configuration, with the ratio of the intersection area to the target bounding box area being preferred. The CPU comprehensively determines the monitoring event status based on a comparison of the overlap ratio with a preset overlap threshold, the positional relationship of the target center point relative to the boundary of the static monitoring area, the direction of the target's movement across the boundary of the static monitoring area, and the number of consecutive frames the target spends within the static monitoring area. Monitoring event statuses include: outside the area, approaching, entering, staying within the area, leaving and holding, and recovering. The preset overlap threshold ranges from 30% to 70%, preferably 50%. The threshold for the number of consecutive frames is from five to fifteen, preferably ten.

[0091] The central processing unit determines the target's coding priority based on the monitored event status. The coding priority is divided into multiple levels, including at least a first priority level, a second priority level, and a third priority level. When the monitored event status is "entering," the target's coding priority immediately increases to the first priority level. When the monitored event status is "staying within the area," the target's coding priority remains at the first priority level. When the monitored event status is "approaching," the target's coding priority increases to the second priority level. When the monitored event status is "outside the area" or "recovering," the target's coding priority decreases to the third priority level. When the monitored event status is "leaving and holding," the target's coding priority gradually decreases from the first priority level to the second priority level within a preset holding time according to a preset decay rate. The preset holding time ranges from three to ten seconds, preferably five seconds. The preset decay rate uses either a linear decay method or an exponential decay method, preferably a linear decay method.

[0092] The central processing unit (CPU) predicts the target position in the current video frame to be encoded based on the target trajectory. Since the inference frame rate of the AI ​​inference unit is lower than the encoding frame rate of the video encoding unit, not every video frame to be encoded has a corresponding real-time detected target. The CPU obtains the inference timestamp of the AI ​​inference unit, the acquisition timestamp of the video frame, and the encoding timestamp of the video frame to be encoded. When the detected target output by the AI ​​inference unit does not cover the current video frame to be encoded, the CPU performs temporal interpolation or extrapolation on the position coordinates of the detected target in the video frame based on the first time difference between the inference timestamp and the acquisition timestamp, and the second time difference between the encoding timestamp and the acquisition timestamp, to generate a predicted target region corresponding to the current video frame to be encoded. The temporal interpolation or extrapolation uses a linear interpolation algorithm, a quadratic interpolation algorithm, or a cubic spline interpolation algorithm, with linear interpolation being preferred. The predicted target region is generated based on at least one of the following information: the target's motion velocity and direction calculated from the position difference between two adjacent detected targets; the predicted position obtained by Kalman filtering the detected target's position coordinates; the optical flow field obtained by optical flow calculation of video frames; the motion vector generated by the video coding unit during motion estimation; and the matching result obtained by performing scale-invariant feature transformation or accelerated robust feature matching on adjacent video frames. When the motion vector generated by the video coding unit is used as the prediction basis, the central processing unit directly reads the motion vector data from the motion estimation module of the video coding unit, without needing to perform additional optical flow calculations.

[0093] The central processing unit (CPU) generates a spatiotemporal importance map based on the target location. The CPU first divides the current video frame to be encoded into multiple coding units. These coding units are macroblocks defined by the H.264 coding standard, coding tree units defined by the H.265 coding standard, or ROI grid units supported by the hardware encoder. The size of the coding unit is 16x16 pixels, 32x32 pixels, or 64x64 pixels, with the specific size determined according to the coding standard and the configuration of the hardware coding units; 32x32 pixels is preferred. The CPU then... The first frame Each coding unit calculates its spatial importance based on the spatial coverage relationship between the coding unit and the target location, the target category priority, and the context protection region. Spatial importance is the product of the overlap weight between the target bounding box and the coding unit, and the target category priority. The overlap weight is calculated using a Gaussian decay function or a linear decay function based on the distance between the coding unit and the center point of the target bounding box, preferably using a Gaussian decay function. The context protection region is the area formed by expanding the target bounding box outward according to a preset expansion ratio, which ranges from 100% to 300%, preferably 200%, meaning the width and height of the context protection region are three times the width and height of the target bounding box, respectively. The context protection region covers the motion edge region surrounding the target. The central processing unit obtains spatiotemporal importance by performing temporal filtering on the spatial importance, using a first-order recursive filtering method.

[0094] The central processing unit (CPU) performs temporal smoothing on the spatiotemporal importance map. The CPU recursively filters each coding unit in the spatiotemporal importance map in the temporal domain to obtain the... Frame number The importance of the current frame and the importance of the previous frame are calculated for each coding unit. The importance of the current frame is compared with the importance of the previous frame. When the importance of the current frame is greater than or equal to the importance of the previous frame, the importance of the current frame is smoothed using a first filtering inertia coefficient to obtain a smoothed importance. When the importance of the current frame is less than the importance of the previous frame, the importance of the current frame is smoothed using a second filtering inertia coefficient to obtain a smoothed importance. The first filtering inertia coefficient is less than the second filtering inertia coefficient. The value range of the first filtering inertia coefficient is 10% to 30%, preferably 20%. The value range of the second filtering inertia coefficient is 60% to 90%, preferably 80%. Through the above asymmetric filtering configuration, the spatiotemporal importance map responds quickly when importance increases and decays slowly when importance decreases.

[0095] The central processing unit (CPU) generates a quantization parameter offset for each coding unit based on the smoothed spatiotemporal importance map. The mapping between the quantization parameter offset and the smoothed importance is either linear or piecewise linear, preferably piecewise linear. Coding units with importance higher than a first threshold correspond to a first quantization parameter offset; coding units with importance between the first and second thresholds correspond to a second quantization parameter offset; and coding units with importance lower than the second threshold correspond to a third quantization parameter offset. The absolute value of the first quantization parameter offset is greater than the absolute value of the second quantization parameter offset, and the absolute value of the second quantization parameter offset is greater than the absolute value of the third quantization parameter offset. The CPU further obtains a preset target bitrate and counts the actual number of encoded bits in the encoded video frames within a preset statistical window. The length of the preset statistical window is fifteen to thirty frames, preferably twenty-five frames. The CPU calculates the difference between the target bitrate and the actual number of encoded bits as the bitrate error and updates the basic quantization parameters for the next frame or the next image group based on the bitrate error. The ratio between the bitrate error and the update amount of the basic quantization parameters ranges from 5% to 20%, preferably 10%. The central processing unit (CPU) algebraically sums the basic quantization parameters and the quantization parameter offsets to obtain the final quantization parameters for each coding unit. When the actual number of encoded bits exceeds the target bit rate, the CPU increases the quantization parameter offset of the corresponding region in the following priority order: background region, context region, non-alarm target region, and alarm core target region. The background region is a set of coding units not covered by any target bounding box or context protection region; the context region is a set of coding units covered only by the context protection region and not by the target bounding box; the non-alarm target region is a set of coding units covered by the target bounding box in the outside or near state; and the alarm core target region is a set of coding units covered by the target bounding box in the entering, staying, or leaving holding state.

[0096] In the coding branch, the video coding unit reads video frame data from the shared buffer and receives the final coding quantization parameters for each coding unit from the central processing unit. Based on the received final coding quantization parameters, the video coding unit performs quantization and entropy coding operations on each coding unit of the current video frame to generate compressed video frames. These compressed video frames are then combined into a compressed video stream and written to the stream output buffer. The video coding unit supports either the H.264 or H.265 video coding standard, with H.265 being preferred.

[0097] The central processing unit (CPU) extracts compressed video frames from the bitstream output buffer for quality assessment at a preset period. The preset period is one frame extracted every 30 to 60 frames, preferably one frame every 50 frames. The CPU performs quality assessment or lightweight target recognition operations on the extracted keyframes or ROI images. The quality assessment operation uses a no-reference image quality assessment algorithm to extract sharpness and texture features from the compressed ROI region and calculate a quality score. The lightweight target recognition operation uses a simplified lightweight neural network model to perform target detection on the compressed ROI region. The number of parameters in this lightweight neural network model is less than the number of parameters in the main detection model loaded by the AI ​​inference unit. When any of the following triggering conditions are met, the CPU reduces the quantization parameter offset corresponding to the ROI region or increases the inference frequency of the AI ​​inference unit: the confidence of the compressed target detection decreases by more than a preset confidence threshold; the sharpness index of the ROI region decreases by more than a preset sharpness threshold; the number of target feature points is less than a preset feature point number threshold; text recognition, license plate recognition, or instrument panel recognition fails; or the reliability of continuous frame target tracking decreases by more than a preset reliability threshold. The preset reliability threshold is 10% to 30%, preferably 20%. The preset sharpness threshold is a decrease in sharpness score calculated based on grayscale variance or Laplace variance exceeding 20% ​​to 40%, preferably exceeding 30%. The preset feature point number threshold is fewer than four to eight feature points per ROI region, preferably fewer than six. The preset reliability threshold is a decrease in tracking reliability score exceeding 15% to 35%, preferably exceeding 25%.

[0098] The central processing unit (CPU) also monitors system operating status, including device load, device temperature, and network bandwidth. When the device load exceeds a preset load threshold, the device temperature exceeds a preset temperature threshold, or the network bandwidth falls below a preset bandwidth threshold, the CPU reduces the inference frequency of the AI ​​inference unit in non-alarm scenarios. The preset load threshold is a CPU utilization rate exceeding 80% to 95%, preferably exceeding 90%. The preset temperature threshold is 75°C to 85°C, preferably 80°C. The preset bandwidth threshold is the current network bandwidth being 1.2 to 1.5 times lower than the target bitrate, preferably 1.3 times lower than the target bitrate. When a target approaches a static monitoring area or the monitoring event status is in an entry state, a state of staying within the area, or a state of leaving and holding, the CPU increases the inference frequency of the AI ​​inference unit by one to three times, preferably to twice the current inference frequency.

[0099] The output module reads the compressed video stream from the output buffer via the video output interface and simultaneously acquires metadata generated by the central processing unit. This metadata includes target identifier, target category, target confidence level, ROI grid map or ROI region coordinates generated based on a spatiotemporal importance map, monitoring event type, monitoring event occurrence time, encoding strategy version number, target bitrate, and actual number of encoded bits. The output module embeds the ROI grid map or monitoring event type as supplementary enhancement information into the additional data fields of the compressed video stream, or saves it as a custom user data field in a container format, or outputs it synchronously with the compressed video stream as an independent metadata stream, or writes it to the storage medium as a database record.

[0100] This system achieves data multiplexing of video frames between the monitoring and encoding branches through a shared buffer, eliminating the need to copy image data between the two branches and reducing memory access overhead to a single write plus multiple zero-copy reads. While the AI ​​inference unit performs target detection in the monitoring branch, the video encoding unit performs video compression encoding in the encoding branch. The two branches operate in parallel in a pipelined manner within the processor, with AI inference and video encoding operations overlapping. This reduces end-to-end processing latency to within a single frame acquisition cycle, jointly completing a complete closed-loop processing flow from video acquisition to compressed bitstream output. This system is used to execute the video compression method disclosed in the second aspect of this application.

[0101] Secondly, embodiments of this application disclose a video compression method.

[0102] Figure 2 A flowchart of a video compression method according to an embodiment of this application is shown.

[0103] Reference Figure 2 The method specifically includes the following steps:

[0104] S1: Acquire video frames, write the video frames into a shared buffer, and form a monitoring branch and an encoding branch based on the same video frame.

[0105] The method provided in this application embodiment operates within the aforementioned video compression system. Before executing this step, the system has been powered on and initialized, and the video acquisition device has been started and begun outputting a continuous stream of video frames. The video frames output by the video acquisition device are sent to the processor via the video input interface. Each video frame contains complete image data and associated metadata. The processor, through the memory controller, writes the image data of the video frames into a pre-allocated shared buffer in shared physical memory in a row-by-row pixel arrangement. Simultaneously, it assigns a unique frame sequence number to each frame and reads the time the frame arrives at the processor from the system clock as the acquisition timestamp. The frame sequence number is a monotonically increasing integer sequence starting from zero, used to uniquely identify the acquisition order and temporal relationship of the video frames. The acquisition timestamp is the count value of a high-precision clock counter at the current moment, with a timing accuracy of not less than microseconds, used for calculating time differences in subsequent steps. The shared buffer is a contiguous address space in physical memory, with a storage capacity sufficient to cache at least three complete video frame data. It is managed using a circular buffer structure, maintaining write and read pointers. Each time a new video frame data is written, the write pointer is adjusted according to the formula... Update, in which The variable represents the maximum number of frames that the shared buffer can hold. This represents a pointer to the current write position of the shared buffer; its value is an integer ranging from 0 to... This pointer indicates the starting address offset of the buffer where the next video frame data should be written, and its update operation is used to implement the circular overwrite management of the circular buffer. The value of is not less than three, preferably four, to ensure that there are still historical frames available for the encoding branch to read when a new frame is written, avoiding data overwriting conflicts; at the same time, the count semaphore for a full buffer is incremented by one. Each video frame buffered in the shared buffer is associated with a frame sequence number, acquisition timestamp, image width and height, and pixel format information. Each information item is indexed by the frame sequence number as the primary key, to ensure that subsequent steps can uniquely identify the corresponding image data and its attribute information by using the frame sequence number when accessing video frame data.

[0106] The processor forms a monitoring branch and an encoding branch based on the same video frame. Specifically, when a video frame is written to the shared buffer, its physical memory address is simultaneously notified to both the monitoring and encoding branches. The two branches access the same frame data by referencing this physical memory address, without performing image data copying operations. The monitoring branch retrieves video frames from the shared buffer at a lower processing frame rate, while the encoding branch retrieves them at a higher processing frame rate. For example, the monitoring branch processes eight to twelve inferences per second, and the encoding branch processes twenty-five or thirty encodings per second. The video frames retrieved by the monitoring branch are used for target detection, tracking, and event determination in subsequent steps, while the video frames retrieved by the encoding branch are used for compression encoding in subsequent steps. The two branches share the same frame data source but have different processing cycles; the monitoring branch's processing cycle is longer than the encoding branch's. The output of the monitoring branch is associated with the video frame currently being processed by the encoding branch through its frame sequence number, thus providing the encoding branch with a basis for encoding decisions based on visual content analysis.

[0107] S2: The monitoring branch performs image preprocessing on the video frame and then sends it to the AI ​​inference unit for target detection to obtain the detected target.

[0108] In this step, the monitoring branch scales the video frame to generate a low-resolution monitoring image, performs color space conversion and pixel value normalization on the low-resolution monitoring image, and sends the preprocessed low-resolution monitoring image to the AI ​​inference unit. The AI ​​inference unit performs target detection, instance segmentation, key point recognition, or abnormal state recognition operations, and outputs target detection information. The target detection information includes target category, detection confidence, target bounding box, target service priority, and the acquisition timestamp of the video frame. The target bounding box is a rectangular bounding box or a segmented region outline, and the target service priority is pre-configured according to the target category.

[0109] After the monitoring branch acquires video frames from the shared buffer, it first performs image preprocessing. This preprocessing is controlled by the central processing unit (CPU) and includes scaling, color space conversion, and pixel value normalization. The CPU reads the video frame data from the shared buffer to obtain the original image size and pixel format information. During scaling, the CPU downsamples the original video frame according to a preset target input size and calculates the pixel value of each pixel in the scaled image using bilinear interpolation, generating a low-resolution monitoring image. The AI ​​inference unit supports fixed-size input images, such as 416 pixels wide and 416 pixels high, or 608 pixels wide and 608 pixels high. Let the original image width be... Height is The target input width is Height is Then the horizontal scaling ratio and vertical scaling ratio They are respectively:

[0110]

[0111] in and All are dimensionless positive real numbers. This represents the scaling factor by which the original image width is scaled to the target input width. This represents the scaling factor from the original image height to the target input height. Both are used to map pixel coordinates in the original image to their corresponding positions in the scaled image, and to map the target bounding box coordinates in the detection results from the scaled image space back to the original image space, thus ensuring the consistency of target location information at different resolutions. For the target bounding box coordinates in the scaled image... The pixel at that location corresponds to its mapped position in the original image. for:

[0112]

[0113] in and These are the horizontal and vertical coordinates of a pixel in the scaled image, in pixels, with values ​​ranging from [value range to value]. and ; and These are the horizontal and vertical coordinates of the corresponding mapped position in the original image, in pixels and of floating-point type. Their values ​​are used to locate the nearest pixels needed for interpolation in the original image. The bilinear interpolation algorithm takes... The four nearest pixels are weighted and averaged to suppress high-frequency noise generated during scaling.

[0114] In color space conversion, the central processing unit (CPU) converts the pixel format of the original video frame to the input pixel format required by the AI ​​inference unit. Video frames output by video capture devices are typically encoded in the YCbCr color space, while the object detection neural network model loaded by the AI ​​inference unit is usually trained in the RGB color space; therefore, there is a format difference between the two. Let the components of any pixel in the original color space be... , , The three components in the RGB space after conversion , , Calculated using the linear transformation matrix:

[0115]

[0116] in This is a color space conversion matrix, whose coefficients are determined by the standard conversion specification. This 3×3 real number matrix is ​​used to linearly map the pixel components of the YCbCr color space to the RGB color space. The specific coefficient values ​​are determined by the color space conversion standard used, such as BT.601 or BT.709. The matrix coefficients are pre-loaded into the central processing unit's configuration register during system initialization. After color space conversion, the central processing unit converts the pixel values ​​of each channel of each pixel from integer to floating-point.

[0117] In pixel value normalization, the central processing unit maps the pixel values ​​of each channel of each pixel to a preset numerical range according to the input requirements of the model loaded by the AI ​​inference unit. Let the original pixel values ​​be... The range of values ​​is Normalized pixel values The calculation method is as follows ,in The mean normalization parameter, This is the standard deviation normalization parameter. and The specific values ​​were obtained from statistics based on the training dataset of the object detection neural network model. and All are real scalars, of which This represents the statistical mean of all pixel values ​​in the training dataset. This represents the statistical standard deviation of all pixel values ​​in the training dataset. Both are used to adjust the distribution of input pixel values ​​to the range of data distribution during model training, thereby improving the model's inference accuracy. For models that do not support mean-variance normalization, maximum-minimum normalization can be used. This method converts pixel values ​​from Integer interval linear mapping to The floating-point range, at this time The value range is 0 to 1, used to meet the numerical range requirements of the model input layer. The normalized low-resolution monitoring images are organized into a multi-dimensional tensor according to a batch processing format, with data dimensions of Batch×Channel×Height×Width, representing the batch dimension, channel dimension, height dimension, and width dimension, respectively. The central processing unit writes this tensor data into the input buffer of the AI ​​inference unit and sends an inference start command to the AI ​​inference unit. The AI ​​inference unit loads a pre-trained target detection neural network model and performs forward inference operations on the input tensor. The target detection neural network model is a single-stage target detection network or a two-stage target detection network, preferably a single-stage target detection network. Its output feature map, after decoding, yields the class probability distribution and bounding box regression parameters of multiple detected targets. The AI ​​inference unit selects the class corresponding to the highest probability value as the target class based on the class probability distribution. The maximum probability value is used as the detection confidence level. The target bounding box is obtained by decoding the bounding box regression parameters combined with the input image size. The coordinates of the top left corner and the bottom right corner subscript This represents the index of the target within the detection result set. It is a positive integer, ranging from 1 to the total number of targets detected in the current frame. ; For category labels of enumeration types, A real number between 0 and 1 For A rectangular region defined by four integer coordinates. When the AI ​​inference unit is configured to perform instance segmentation, in addition to detecting the target bounding box, it also outputs a segmentation mask corresponding to the target. The segmentation mask is a binary matrix with the same resolution as the low-resolution monitoring image. A value of 1 for each pixel indicates that it belongs to the target, and a value of 0 indicates that it does not belong to the target. When the AI ​​inference unit is configured to perform keypoint recognition, it outputs the coordinates of multiple keypoints of the target, represented by normalized coordinates relative to the target bounding box. When the AI ​​inference unit is configured to perform abnormal state recognition, it outputs a state classification label for the target, including two categories: normal state and abnormal state. After completing the inference operation, the AI ​​inference unit writes the detected target information to the detection result storage area associated with the frame sequence number in shared physical memory. The target category in the detected target information. Used to indicate the type of target and detect confidence level. The target bounding box is used to characterize the reliability of the detection result. Used to define the spatial extent of a target within a video frame image, and the target's service priority. According to target category The target category-priority mapping table, predefined during the system configuration phase, is used to obtain the target category-priority mapping. This is a positive real number, and its value range is determined according to the system configuration. A typical value is a discrete value between 0.5 and 1.0. This priority is used to assign different weights to different categories of targets in subsequent spatiotemporal importance calculations. For example, the business priority of personnel and fireworks categories is higher than that of general object categories. (Time stamp data is collected.) Used to identify the acquisition time of the original video frame corresponding to the detected target. This is the system clock count value at the exposure time of the video frame, measured in microseconds or milliseconds. It is used in subsequent steps to calculate the time intervals and timing relationships of each event. The central processing unit reads the data from the detection result storage area by frame number and sorts it according to the target category. Detection confidence Target bounding box Target business priority and collection timestamp The structured format is used to extract the target information for subsequent steps such as target tracking and event status determination.

[0118] S3: Track the detected target, generate the target trajectory, and assign a unique target identifier to each target.

[0119] The central processing unit (CPU) reads the detected target information of the current frame from the detection result storage area and performs target tracking processing on these detected targets. Target tracking processing is performed continuously in the time domain, maintaining a list of target trajectories, where each existing target trajectory corresponds to a target being tracked. For newly arrived targets in the current frame... ,in The total number of targets detected in the current frame. It is a non-negative integer, and its value changes dynamically with the actual number of detected targets appearing in the video frame; The set of all detected targets in the current frame, where each element is a subset of the set. Indicates the current frame number Each detection target, including target category Detection confidence Target bounding box and collection timestamp ,in For frame sequence number index, The target number detected in the current frame. It is a positive integer, ranging from 1 to... The central processing unit compares the target set detected in the current frame with the target set detected in the previous frame. Data correlation is performed between them, and the detection results belonging to the same physical target are associated with the same target trajectory.

[0120] The data association operation specifically includes: the central processing unit first calculates the association cost between each detected target in the current frame and each detected target in the previous frame. The association cost uses the intersection-union ratio (IUU) distance metric. Let the detected targets in the previous frame be... The target bounding box is Detect target in the current frame The target bounding box is The intersection and union ratio of the two for:

[0121]

[0122] in The area of ​​the intersection of the two bounding boxes. The area of ​​the union of the two bounding boxes. This is a real number between 0 and 1. The closer the value is to 1, the higher the spatial overlap between the two bounding boxes, and the more likely they are detection results of the same physical target in adjacent frames. (Association Cost Matrix) elements in , It is a real number matrix, and the number of rows is equal to the number of targets detected in the previous frame. The number of columns is equal to the number of targets detected in the current frame. Each element in the matrix Indicates the previous frame number The target and the current frame number The matching cost between two objectives is denoted by a smaller cost, indicating a better match. The central processing unit (CPU) uses the correlation cost matrix as a basis for this process. Execute the Hungarian matching algorithm to find the matching scheme that minimizes the global association cost. During the matching process, when the intersection-union ratio... Greater than or equal to the preset matching threshold At that time, it is allowed to detect the target in the previous frame. Detected target in the current frame Establish a matching relationship. A real-valued threshold between 0 and 1 is used to exclude matching pairs with excessively low intersection-union ratios (IU), preventing the incorrect association of spatially distant targets with the same trajectory. The value ranges from 0.3 to 0.5, preferably 0.4. When the crossover ratio... Less than the preset matching threshold At this time, establishing a matching relationship is prohibited.

[0123] Based on the matching results, the matched detected targets are directly associated with the corresponding existing target trajectories, and the bounding box of the detected targets in the current frame is added. and collection timestamp The target is added to the historical position sequence of the trajectory. An unmatched target detected in the previous frame is considered to have temporarily left the detection field of view or failed to be detected, and the central processing unit starts a target loss timer. An unmatched target detected in the current frame is considered a newly appearing target, and the central processing unit creates a new target trajectory for it. The target loss timer is maintained independently for each unmatched existing target trajectory, with an initial value of zero, and increments by one frame's duration for each unmatched frame. When the target loss timer is less than or equal to the preset tracking survival time... At this time, the target trajectory is preserved, and the central processing unit continues to maintain its trajectory state through prediction. This parameter is a positive real number, expressed in seconds. It defines the maximum duration for which the system maintains the trajectory of a target after it has disappeared from the detection field of view. After this duration, the target will be considered permanently removed. The value ranges from 2 seconds to 5 seconds, preferably 3 seconds. When the target loss timer exceeds the preset tracking survival time... When the target trajectory is determined to have left the monitoring field of view, the central processing unit marks the target status as lost and removes it from the active tracking list.

[0124] During the data association process, the central processing unit uses the Kalman filter algorithm to predict and update the state vector of each target trajectory. Let the target be... State vector at time step ,in and The x and y coordinates of the center point of the target bounding box are given. and The width and height of the target bounding box. , , , These are the rates of change, or velocity components, of the four variables mentioned above. This is an 8-dimensional real-valued vector, with each component in pixels and the velocity component in pixels per second. This state vector fully describes the target's position and size in the image plane and its changing trend, used to predict the target's spatial position at future times. The Kalman filter prediction steps are as follows: , ,in Here is the state transition matrix. It is an 8×8 real matrix, its function is to... The state vector at time t is mapped as The prior state estimate at time step is constructed based on a uniform motion model, describing the natural motion evolution of the target under no external force. for The state vector updated after measurement at time step. for The error covariance matrix at time t. This is an 8×8 real symmetric positive definite matrix, where the diagonal elements represent the uncertainty variance of the estimates of each state component, and the off-diagonal elements represent the covariance between different state components. The process noise covariance matrix is... This is an 8×8 real symmetric positive definite matrix, describing the impact of unmodeled random perturbations in the target motion model on state prediction. State transition matrix. Based on the uniform motion model, a block diagonal matrix is ​​constructed. Each block , The time interval between the current frame and the previous frame. This is a positive real number, expressed in seconds, and its value represents the difference between the timestamps of two adjacent frames. When the detected target successfully matches the target trajectory, the central processing unit detects the target bounding box in the current frame. The center point coordinates, width, and height are used as the observation vector. Perform the Kalman filter update steps:

[0125]

[0126]

[0127]

[0128] in Here is the Kalman gain matrix. This is an 8×4 real matrix, and its function is to balance the reliability of prior estimates and observational data when fusing observational information. A larger gain indicates a higher degree of confidence in the observational data. To map the state vector to the observation space for the observation matrix, It is a 4×8 real matrix, and its function is to extract the observation components of the four dimensions of position and size from the 8-dimensional state vector. To observe the noise covariance matrix, It is a 4×4 real symmetric positive definite matrix, which describes the error statistics of the bounding box measurements during target detection. The larger the value, the lower the reliability of the detection result. This is the updated target state vector. The updated error covariance matrix, It is an 8×8 identity matrix.

[0129] The central processing unit assigns a unique target identifier to each newly created target trajectory. ,in For the target index number, A positive integer is used to uniquely identify a specific target during system operation. Unique target identifiers are assigned using an incrementing sequence of integers starting from 0. Each time a new target trajectory is created, the global counter is incremented and that value is used as the new unique target identifier. When a target trajectory is determined to have left the monitoring field of view and removed from the active tracking list, its unique target identifier is no longer used. However, the assigned identifier is not reassigned to other target trajectories to ensure that the same physical target has a constant and uniquely traceable identifier throughout its entire lifecycle. Target Trajectory Represented as the state sequence of the target in consecutive frames. ,in The frame number of the first frame in which the target appears. The frame number of the latest frame. This is an ordered set of all state vectors of the target in consecutive frames, which completely records the target's full trajectory motion information from its appearance to the current moment. Each state vector... It includes the target's position coordinates and size information within the frame. The central processing unit associates each target trajectory with its corresponding unique target identifier, target category, target service priority, and historical state sequence in the trajectory storage area of ​​shared physical memory for use in subsequent steps for event state determination and cross-frame target position prediction.

[0130] S4: Determine the overlap relationship and movement trend between the target and the static monitoring area based on the target trajectory and the preset static monitoring area, and generate the monitoring event status.

[0131] In this step, the overlap ratio between the target bounding box and the static monitoring area is calculated. The overlap ratio is the ratio of the intersection area of ​​the target bounding box and the static monitoring area to the area of ​​the target bounding box, or the ratio of the intersection area to the area of ​​the static monitoring area, or the ratio of the intersection area to the union area of ​​the target bounding box and the static monitoring area. Based on the comparison result of the overlap ratio with a preset overlap threshold, the positional relationship of the target center point relative to the boundary of the static monitoring area, the movement direction of the target crossing the boundary of the static monitoring area, and the number of consecutive frames the target stays in the static monitoring area, the monitoring event state is determined. The monitoring event state includes the state outside the area, the state of approach, the state of entry, the state of staying in the area, the state of leaving and holding, and the state of recovery.

[0132] The central processing unit reads the trajectory data and unique target identifiers of each target from the trajectory storage area, and simultaneously reads the preset static monitoring area information from the configuration area shared physical memory. The static monitoring area is preset by the system user through the configuration interface and stored in the configuration area as a sequence of polygon vertex coordinates. The static monitoring area supports any one or more combinations of rectangular areas, polygonal areas, polyline areas, electronic fence areas, or mask areas. Each area type is represented by a set of ordered vertex coordinates. To define, where The number of vertices. For a rectangular region, the integer is a positive integer not less than 2. Furthermore, the vertices form a rectangle for a polygonal region. Furthermore, the vertices form closed polygons, which applies to both polyline regions and electronic fence regions. Furthermore, the vertices form an open polyline for a rectangular region. For polygonal regions For broken line areas and electronic fence areas The central processing unit converts the vertex coordinate sequence into a planar geometric region. , For a connected region in a two-dimensional image plane, its geometry is defined by a sequence of vertices. This region is used to delineate the spatial area requiring focused monitoring and to calculate the region's size. area , It is a positive real number, with the unit being the square of pixels. This value is used as one of the denominators when calculating the overlap ratio.

[0133] For each active target trajectory, the central processing unit (CPU) uses its latest state vector... Target bounding box parameters Construct the target bounding box for the current frame , The bounding box represents the rectangular region occupied by the target in the current frame, whose geometric parameters are extracted from the state vector updated by Kalman filtering. Using the center point coordinates ,width and height Defined, its upper left corner coordinates are The coordinates of the lower right corner are The central processing unit calculates the target bounding box. With static monitoring area Intersection area , This represents the spatial overlap between the target bounding box and the static monitoring area. This intersection may be an empty set or a non-empty connected region. The area of ​​the intersection region is calculated. , This is a non-negative real number, expressed in pixels squared, and its value reflects the degree of spatial overlap between the target and the monitored area. Overlap Ratio The calculation method depends on the system configuration and can be selected from the following three options. The first option is the ratio of the intersection area to the area of ​​the target bounding box. ,in The first method measures the area of ​​the target's bounding box, using the target's own area as a baseline to measure the proportion of the monitored area covering the target. The second method measures the ratio of the intersection area to the static monitoring area. This method uses the area of ​​the monitored region as a benchmark to measure the proportion of the monitored region covered by the target. The third method is the ratio of the intersection area to the union area of ​​the target bounding box and the static monitoring area. This method uses the union of the target and the monitoring area as a benchmark to measure the proportion of their overlap to the overall coverage area. The system defaults to the first calculation method, which uses the ratio of the intersection area to the target bounding box area as the overlap ratio. This applies when the target bounding box is completely within the static monitoring area. When the target bounding box does not overlap with the static monitoring area When the target bounding box is partially located inside the static monitoring area .

[0134] The central processing unit further calculates the target center point. Relative to static monitoring area The positional relationship is determined using a point-polygon position determination algorithm. For rectangular or polygonal regions, a ray method is used to determine whether a point is inside the region. Specifically, a ray is emitted horizontally to the right from the target center point, and the number of intersections between the ray and the region boundary is counted. If the number of intersections is odd, the point is determined to be inside the region; if it is even, the point is determined to be outside the region. For polyline regions or electronic fence regions, the positional relationship is determined by calculating the shortest distance from the target center point to each segment of the polyline. To determine, ,in The target center point to the line segment Euclidean distance, This is a non-negative real number, expressed in pixels. This distance is used to determine whether a target has entered an approaching state; the smaller the distance, the closer the target is to the boundary of the monitoring area. Less than the preset distance threshold The system determines that the target is in an approaching state. This is a positive real number, expressed in pixels. This threshold defines the boundary for determining proximity; when the distance between the target's center point and the boundary of the monitoring area is less than this value, the target is considered to be approaching the monitoring area. The distance threshold ranges from 10 to 30 pixels, preferably 20 pixels. The central processing unit also calculates the target's motion direction vector based on the target's center point position in adjacent frames of the target trajectory. , This is a two-dimensional real vector. Its direction represents the target's motion direction from the previous frame to the current frame, and its magnitude represents the pixel distance the target moves during the time interval between frames. It determines whether this motion direction points inside or outside the static monitoring area. The determination method is: along the motion direction... The prediction point is obtained by extending one step forward from the target center point. ,in This is the inter-frame time interval. and The velocity component, measured in pixels per second, is obtained from the Kalman filter state vector. This predicted point is used to estimate the target's position at the next moment. If the predicted point is inside the static monitoring area, the direction of motion is determined to be inward; if it is outside the area, the direction of motion is determined to be outward. The central processing unit also counts the number of consecutive frames the target remains within the static monitoring area. , It is a non-negative integer. This counter records the minimum number of frames in which the target is continuously located inside the monitoring area. It is used to distinguish between instantaneous crossing and continuous stay. The counter is incremented frame by frame when the target's center point is inside the area and is reset to zero when the target's center point moves out of the area.

[0135] The central processing unit integrates the above four judgment criteria and determines the monitoring event state of the current frame according to the state transition logic. Monitoring event states include: outside the region, approach, entry, within the region, leaving and holding, and recovery. The transition conditions between each state are as follows: when the target is in the outside region state and the overlap ratio is... And the shortest distance between the target center point and the boundary of the static monitoring area Greater than the proximity threshold And when the direction of movement does not point towards the interior of the area, maintain the state outside the area. When the target is in the state outside the area and When the direction of movement points towards the interior of the region, the state transitions from outside the region to the approaching state. This occurs when the targets are in the approaching state and the overlap ratio is [not specified]. Greater than or equal to the preset overlap threshold Furthermore, when the direction of movement points towards the interior of the region, the state transitions from the approach state to the entry state. The overlap threshold is a real number between 0 and 1. When the overlap ratio reaches this threshold, it is considered that the target has fully entered the monitoring area, triggering an entry event. The preset overlap threshold is... The value ranges from 0.3 to 0.7, preferably 0.5. When the target is in the entry state and the overlap ratio... Greater than or equal to the preset overlap threshold When the target's center point is located within the static monitoring area, the state transitions from entering the monitoring area to remaining within it. The target remains in the remaining state within the monitoring area for a certain number of consecutive frames. Less than the preset number of frames to stay Maintain a state of staying within the area. This threshold, being a positive integer, specifies the minimum number of consecutive frames required to determine that a target is in a sustained state. The value range is from 5 to 15 frames, preferably 10 frames. When the target is in a stationary state within the area and its center point moves out of the static monitoring area and its movement direction points out of the area, it transitions from the stationary state within the area to the leave-holding state. When the target is in the leave-holding state, the target encoding priority gradually decreases according to a preset decay rate, and the leave timer is started simultaneously. , This is a positive real number, in seconds. This timer records the duration after the target leaves the monitoring area, and the duration after the target leaves the static monitoring area. Less than or equal to the preset duration At that time, maintain the state of being out of control. This is a positive real number, expressed in seconds. This parameter defines the maximum duration for which a target retains its encoding priority after leaving the monitoring area. The value ranges from 3 seconds to 10 seconds, with 5 seconds being preferred. When Greater than When the target is in the leave-hold state, it transitions to the recovery state. This occurs when the target is in the leave-hold state and the overlap ratio is... Greater than or equal to the preset overlap threshold When the target transitions from the leave-hold state to the enter state, it indicates that the target has returned to the static monitoring area during the leave-hold period, and the coding priority should be immediately raised to the first priority level again. When the target is in the recovery state and the overlap ratio is... and And when the direction of movement does not point towards the interior of the area, the target remains in the recovery state. When the target is in the recovery state and When the direction of movement points into the area, the state transitions from the recovery state to the approach state. The above state transition logic realizes a complete closed loop from outside the area through approach, entry, stay within the area, leave and hold, and finally recovery. The approach state and the leave and hold state serve as transition states for entering and exiting the monitoring area, respectively, providing a state basis for increasing the coding priority in advance when the target enters and delaying the reduction of the coding priority when the target leaves.

[0136] The central processing unit associates the monitoring event status corresponding to each target in the current frame with a unique target identifier and writes it to the event status storage area in shared physical memory for subsequent steps to determine encoding priorities. The monitoring event status is stored in the form of enumerated values, with each status corresponding to an integer value. For example, 0 corresponds to the status outside the area, 1 to the status approaching, 2 to the status entering, 3 to the status staying within the area, 4 to the status leaving and holding, and 5 to the status recovering, in order to reduce data storage space and improve reading efficiency.

[0137] S5: Determine the encoding priority of the target based on the monitoring event status, and predict the target position in the current video frame to be encoded based on the target trajectory.

[0138] In this step of the method, when the monitoring event status is in the entry state, the target encoding priority is immediately increased to the first priority level. When the monitoring event status is in the exit hold state, the target encoding priority gradually decreases from the first priority level to the second priority level within a preset hold duration according to a preset decay rate. The inference timestamp of the AI ​​inference unit, the acquisition timestamp of the video frame, and the encoding timestamp of the video frame to be encoded are obtained. When the detected target output by the AI ​​inference unit does not cover the video frame to be encoded, the position coordinates of the detected target in the video frame are interpolated or extrapolated in the temporal domain according to the first time difference between the inference timestamp and the acquisition timestamp and the second time difference between the encoding timestamp and the acquisition timestamp, to generate a predicted target region corresponding to the video frame to be encoded. The predicted target region is generated based on at least one of the following information: the target motion speed and motion direction calculated according to the position difference of the detected target in two adjacent frames, the predicted position obtained by Kalman filtering or state estimation of the position coordinates of the detected target, the optical flow field obtained by optical flow calculation of the video frame, the motion vector generated by the video encoding unit, and the matching result obtained by feature point matching of adjacent video frames.

[0139] The central processing unit reads the monitoring event status corresponding to each target in the current frame from the event status storage area and determines the encoding priority of each target based on the monitoring event status. Encoding Priority This is used to quantify the relative importance of a target during video coding. Its values ​​are divided into multiple discrete levels from high to low based on a combination of business importance and event urgency, where the subscript... For the target index number, The value type is enumeration, and its value is... , or One of them. This embodiment defines at least three levels: the first priority level... The highest priority level is for critical events requiring the highest coding quality assurance; the second priority level... Medium priority, applicable to targets in a transitional state that are approaching or have just left the monitoring area; third priority level. This is the lowest priority and is applicable to ordinary targets that have not interacted with the monitored area. The initial baseline encoding quality for each priority level is preset by the system configuration to quantize parameter offsets. Characterization, This is an integer, ranging from -12 to 0. Negative values ​​indicate a decrease in quantization parameters, thus improving coding quality. 0 indicates no adjustment. This represents the quantization parameter offset corresponding to the first priority level. The quantization parameter offset corresponding to the third priority level with the largest absolute value. Minimum absolute value, i.e. , , , All are negative integers, representing the quantization parameter offsets at the first, second, and third priority levels, respectively. The typical value is -10. The typical value is -6. The typical value is -2. The central processing unit determines the coding priority of each target according to the following mapping rules: when the monitored event status is "entry" or "staying within the area", the coding priority of the target is immediately increased to the first priority level. When the monitored event status is nearing completion, the target encoding priority is increased to the second priority level. When the monitored event status is outside the region or in a recovery state, the target code priority is set to the third priority level. When the monitored event status is in the "away hold" state, the target encoding priority is reduced from the first priority level. The decay rate is gradually reduced according to the preset decay rate until it drops to the second priority level. The target remains at this level until it leaves the hold state. Specifically, the target encoding priority decay in the leave hold state uses a linear decay method. Let the time when the target enters the leave hold state be... , A positive real number in seconds, representing the system time when a target transitions from a stationary state within the area to a leave-and-hold state. At this moment, the target's priority is first priority level. Corresponding quantization parameter offset The preset duration is At present distance The duration is , A non-negative real number in seconds represents the duration the target has been in the leave hold state, and is the quantization parameter offset of the target at the current moment. For: when hour, ,in This is the quantization parameter offset corresponding to the second priority level. and All are negative and Therefore, this difference It is a positive number. along with As the value increases, the coding quality gradually decreases. hour, This means that the priority decays to the second priority level and then stops decreasing. The physical meaning of this linear decay formula is that during the period away from the hold-off state, the target's coding quality linearly transitions from the highest quality corresponding to the first priority level to the medium quality corresponding to the second priority level, with a transition time of... This avoids image flickering caused by a sudden drop in encoding quality when the target leaves the monitoring area. Furthermore, the system also supports exponential decay as an alternative solution; the formula for exponential decay is... ,in The decay rate coefficient, It is a positive real number, with the unit being the reciprocal of seconds, used to control the steepness of the exponential decay curve. A larger value indicates a faster decay. The value is determined through configuration. Users can choose between linear decay and exponential decay modes during the configuration phase; the system defaults to linear decay.

[0140] The central processing unit (CPU) also performs cross-frame target location prediction. Cross-frame target location prediction occurs in the following scenarios: when the AI ​​inference unit does not output a detected target in the current frame, but the encoding branch is currently encoding a video frame, this encoded frame lacks real-time detected target information from the AI ​​inference unit, and it needs to predict the target location in the current encoded frame based on historical detection results. The CPU first obtains three types of timestamp information: the inference timestamp corresponding to the AI ​​inference unit's most recent output of a detected target. This timestamp is the same as the collection timestamp. The same applies to the video frame's capture timestamp, upon which the reasoning result is based; the video frame's capture timestamp The encoding timestamp of the current frame to be encoded in the encoding branch is the moment when the video frame is exposed by the video capture device. This is the start time when the video encoding unit begins encoding the frame. , , All three are system clock count values, with the same unit and an accuracy of at least microseconds. The time difference between them is used to determine the time reference for target position prediction and the scale for interpolation extrapolation. The central processing unit calculates the first time difference. That is, the acquisition time interval between the video frame on which the reasoning is based and the current encoded frame; calculate the second time difference. This refers to the processing delay between the completion of acquisition and the start of encoding of the current encoded frame. and All are real numbers, in seconds. Used to determine the time span of the forecast. Used to determine the delay between the encoding start time and the acquisition time. When the detected target output by the AI ​​inference unit does not cover the current video frame to be encoded, that is, when the frame number of the current encoded frame is inconsistent with the frame number corresponding to the most recent detected target output by the AI ​​inference unit, the central processing unit starts the cross-frame prediction process.

[0141] In the cross-frame prediction process, the central processing unit (CPU) uses the target's position information at the inference time, combined with the target motion model, to perform temporal interpolation or extrapolation of the target's position at the current encoding time. Let's assume the AI ​​inference unit most recently detected the target. The time is At that moment, the center point of the target bounding box is The target bounding box has a width and height of 1000 mm. The current encoded frame's time is The time interval is , For real numbers, the unit is seconds. Time indicates extrapolation prediction, when Time represents interpolation prediction, which is typically used in this scenario. This is extrapolation. The central processing unit acquires the target. Motion velocity vector and the rate of change of size The velocity vector and the rate of size change are derived from the state vector of the Kalman filter. Extract directly from, among which , , , These represent the rates of change of the x-coordinate, y-coordinate, width, and height of the target center point, respectively. The unit is pixels per second. The units are pixels per second. These four rates describe the velocity of the target's movement and the rate of size change in the image plane. Predicted target bounding box parameters. The following was obtained through linear extrapolation calculation using the uniform motion model: , , , , , , , These are the estimated x-coordinates, y-coordinates, width, and height of the predicted target bounding box center point. These predictions are used to generate an approximate ROI region for the current coded frame when AI detection results are lacking. When the absolute value of is positive and small, the above linear extrapolation has high prediction accuracy; when When the absolute value is large or the target motion pattern changes abruptly, the prediction accuracy may decrease, but it can still provide a reasonable estimation range for the target position. The central processing unit determines the applicable range of linear extrapolation based on the frame rate of the actual application scenario. For a coding frame rate of 25fps or 30fps, the time interval between adjacent coding frames... The time interval is approximately 33 or 40 milliseconds, and the AI ​​inference interval is approximately 80 to 125 milliseconds. At the above time scales, the accuracy of linear extrapolation is sufficient to meet the needs of ROI region localization.

[0142] The central processing unit also supports a fusion prediction method based on multiple types of information. When a single source of prediction information is limited by specific conditions, fusing multiple prediction information can improve prediction robustness and accuracy. The first type of prediction information is the target's motion velocity and direction calculated based on the position difference of the detected target in two adjacent frames: assuming the target is in... The position of the moment is ,exist The position of the moment is Then the target's speed is , , here and The unit is pixels per frame, which differs from the unit of the velocity component in Kalman filtering. This simplifies calculations, and the predicted position for the next moment is obtained through linear extrapolation. The second type of prediction information is the predicted position obtained by performing Kalman filtering or state estimation on the detected target's position coordinates: from the Kalman filter prediction steps... The first type of prediction information directly obtains the predicted state vector, which contains the predicted target position and size information. The second type is the optical flow field obtained by optical flow calculation on video frames: the central processing unit calls the optical flow calculation algorithm to calculate the dense optical flow field between adjacent video frames. The median or mean of the optical flow vectors of each pixel within the target bounding box is used as the overall motion displacement vector of the target, and the target position is predicted by translation based on this displacement vector. The third type is the motion vector generated by the video coding unit during motion estimation: when performing inter-frame prediction, the video coding unit calculates motion vectors for each coding block. The central processing unit directly reads these motion vectors from the motion estimation module of the video coding unit, and uses the statistical median or mean of the motion vectors of each coding block within the area covered by the target bounding box as the overall motion displacement of the target. Compared to optical flow calculation, reading motion vectors from the video coding unit requires no additional computational overhead and directly utilizes the intermediate processing results of the hardware coding unit. The fifth type of prediction information is the matching result obtained by matching feature points of adjacent video frames: The central processing unit extracts scale-invariant feature transforms or accelerated robust feature descriptors within the target bounding box region, matches the feature points of the current frame with those of the previous frame, calculates the displacement between the matching point pairs, and takes the median displacement as the target motion displacement for position prediction. After the cross-frame prediction process is completed, the central processing unit associates the generated predicted target region with a unique target identifier and stores it in the prediction result storage area in shared physical memory for use in the generation of the spatiotemporal importance map in subsequent steps.

[0143] S6: Generate a spatiotemporal importance map based on the target location. The spatiotemporal importance map contains the spatiotemporal importance of each coding unit in the video frame.

[0144] In this step, the video frame is divided into multiple coding units, which are macroblocks, coding tree units, or ROI grid units supported by a hardware encoder. For the first... The first frame Each coding unit calculates spatial importance based on its spatial coverage relationship with the target location, target category priority, and context protection region. The spatial importance is the product of the overlap weight between the target bounding box and the coding unit and the target category priority. The overlap weight is calculated based on the distance decay between the coding unit and the center point of the target bounding box. The context protection region is the area formed by the target bounding box expanding outward according to a preset expansion ratio. The context protection region covers the motion edge region around the target. The spatiotemporal importance is obtained by temporal filtering the spatial importance. The quantization parameter offset corresponding to the context protection region is between the quantization parameter offset corresponding to the target core region and the quantization parameter offset corresponding to the background region.

[0145] The central processing unit (CPU) reads the target location information of each target in the current video frame to be encoded from the prediction result storage area in shared physical memory. This includes the target bounding boxes directly detected by the AI ​​inference unit and the predicted target regions generated through the cross-frame prediction process. The CPU divides the current video frame to be encoded into multiple coding units according to the partitioning method supported by the video coding unit. The partitioning method of the coding unit corresponds to the video coding standard. When the video coding unit adopts the H.264 coding standard, the coding unit is a 16×16 pixel macroblock; when the video coding unit adopts the H.265 coding standard, the coding unit is a coding tree unit divided into levels from 8×8 to 64×64 pixels. Its recursive partitioning method is to divide the coding tree unit into four sub-coding units level by level according to the quadtree structure until the preset minimum coding unit size is reached; when the video coding unit supports ROI grid mode, the coding unit is an ROI grid unit configured internally by the hardware encoder. The size of this grid unit is configured to a fixed size according to the encoder capability during system initialization, such as 32×32 pixels or 64×64 pixels. Let the width of the current video frame be... pixels, height is pixels, coding unit size is Pixels and It is a positive integer, determined by the resolution of the video capture device; and If the integer is a positive integer, determined by both the coding standard and system configuration, then the total number of coding units contained in this frame is... ,in This is the floor function, used to round up to cover incomplete coding unit regions at the edges of video frames. It is a positive integer representing the total number of coding units divided in the current frame. This value is used to determine the data dimension of the spatiotemporal importance map.

[0146] Central processing unit for the first The first frame The spatial importance of a coding unit is calculated based on its spatial coverage relationship with the target location, target category priority, and context protection region. When multiple targets exist in a video frame, the central processing unit calculates the spatial importance of each target's contribution to the coding unit, and then takes the maximum value or weighted sum as the final spatial importance of the coding unit. For a single target, let the target be the [number]th [target]. The center point of the target bounding box of the frame is The width of the target bounding box is Height is Encoding unit The center point is subscript Indicates frame number, subscript Indicates the target index, subscript This represents the coding unit index, where the width and height of the coding unit are respectively... and The central processing unit first calculates the encoding unit. With target bounding box Overlap weights The overlap weight characterizes the importance of a coding unit being covered by the target, and its calculation method uses a distance-based Gaussian decay function. Let the coding unit... Normalized distance between the center point and the center point of the target bounding box for ,in and These are the attenuation scale parameters in the width and height directions of the target bounding box, respectively. , , and The unit is pixels. As a natural logarithm constant, this decay scale parameter makes the target bounding box boundary... The overlap weight at that time just drops to its maximum value. That is, the half-width value matches the size of the target bounding box, so that the boundary of the target bounding box is... The overlap weight at that time just drops to its maximum value. Overlapping weights of Gaussian decay functions Calculated as , The weight is a real number between 0 and 1, representing the coding unit. Relative to the target The importance of the coding unit is determined by its proximity to the target center; the closer the coding unit is to the target center, the greater its weight, and vice versa. Gaussian decay is used to calculate the overlapping weights, ensuring a smooth decrease in importance from the target center to the edge. When a coding unit is completely outside the target bounding box but relatively close, its importance decays smoothly rather than dropping directly to zero, thus avoiding abrupt changes in importance in the spatial domain. The central processing unit assigns the overlapping weights... Multiply by target category priority The target pair coding unit is obtained. Spatial importance of contribution Target category priority According to target category The target category-priority mapping table, predefined during the system configuration phase, is used to find targets of different categories with different priority weights. For example, the personnel and fireworks categories are configured with the highest priority weights. Motor vehicle category and non-motor vehicle category are configured with the second highest priority weight. Generally, object categories are configured with lower priority weights. When multiple targets exist in a video frame, the coding unit... Spatial importance Take the maximum value of the spatial importance contributed by all targets to the coding unit:

[0147]

[0148] in For the first The set of all targets in a frame. From 0 to real numbers between The highest target category priority weight configured for the system, this value represents the coding unit Spatial importance in the current frame: the larger the value, the more important the coding unit is for the target detection and recognition task, and the higher the quality should be assigned during encoding.

[0149] The central processing unit also generates a context protection region for the target. The context protection region is the area formed by expanding the target bounding box outwards according to a preset expansion ratio, used to protect the motion edge regions and background context information surrounding the target. Let the width of the target bounding box be... Height is The preset expansion ratio is , A non-negative real number, representing the proportion of the target bounding box's outward expansion relative to the target bounding box size, and the width of the expanded context protection region. and height They are respectively , , and The unit is pixels, and the center point of the context-protected region is the same as the center point of the target bounding box. Preset expansion ratio. The value range is 50.

[0150] The central processing unit (CPU) obtains spatiotemporal importance by performing temporal filtering on the spatial importance. Temporal filtering is used to suppress spatial importance fluctuations caused by inter-frame jitter in target detection, improving the stability of the importance map in the temporal domain. Let the first... Frame number The original spatial importance of each coding unit is The time-domain filtering uses a first-order recursive filtering method to calculate the spatiotemporal importance after smoothing. :

[0151]

[0152] in These are the time-domain filter coefficients. A real number between 0 and 1, used to control the weight of the spatial importance of the current frame in the smoothing result. A larger value indicates a greater weight for the current frame and a faster filtering response. The value range is from 0.2 to 0.5, with 0.3 being the preferred value. For the previous frame The spatiotemporal importance of each coding unit is recursively incorporated, including importance information from historical frames, achieving temporal smoothing. The temporally filtered spatiotemporal importance reflects both the spatial distribution of the target in the current frame and preserves importance information from historical frames, effectively reducing the impact of target bounding box position jitter between frames on the importance map. The central processing unit calculates the spatiotemporal importance of each coding unit individually, generating a spatiotemporal importance map with the same grid size as the coding units of the video frame. ,in This represents the number of rows of coding units in the vertical direction. This refers to the number of columns of coding units in the horizontal direction. for OK A real matrix of columns, where each element corresponds to the spatiotemporal importance of a coding unit. Each element in the spatiotemporal importance graph... The range of values ​​is The closer the value is to 1, the higher the spatiotemporal importance of the coding unit, and the higher the coding quality should be obtained in the subsequent quantization parameter offset allocation.

[0153] Furthermore, the central processing unit (CPU) classifies regions based on the value distribution of each coding unit in the spatiotemporal importance map. For each coding unit, its region type is determined by its positional relationship with the target core region and the context protection region: when the center point of a coding unit is inside the target bounding box, it is classified as the target core region; when the center point of a coding unit is outside the target bounding box but inside the context protection region, it is classified as the context region; and when the center point of a coding unit is outside the context protection region, it is classified as the background region. Based on this classification, the CPU determines that the quantization parameter offset corresponding to the context protection region is between the quantization parameter offset corresponding to the target core region and the quantization parameter offset corresponding to the background region, ensuring a smooth transition in coding quality between regions. The CPU stores the generated spatiotemporal importance map in the importance map storage area in shared physical memory for use in subsequent steps of temporal smoothing and quantization parameter offset generation.

[0154] S7: Perform temporal smoothing on the spatiotemporal importance map, generate the quantization parameter offset for each coding unit based on the smoothed spatiotemporal importance map, and determine the final coding quantization parameter for each coding unit based on the quantization parameter offset and the basic quantization parameter.

[0155] In this step, the encoding units in the spatiotemporal importance map are recursively filtered in the time domain to obtain the first... Frame number The importance of the current frame and the importance of the previous frame of each coding unit are compared. When the importance of the current frame is greater than or equal to the importance of the previous frame, the importance of the current frame is smoothed using a first filtering inertia coefficient to obtain a smoothed importance. When the importance of the current frame is less than the importance of the previous frame, the importance of the current frame is smoothed using a second filtering inertia coefficient to obtain a smoothed importance. The first filtering inertia coefficient is less than the second filtering inertia coefficient. A preset target bitrate is obtained, the actual number of encoded bits in the encoded video frames is counted, and the difference between the target bitrate and the actual number of encoded bits is calculated as the bitrate error. The basic quantization parameters of the next frame or the next image group are updated according to the bitrate error. The final quantization parameter is the algebraic sum of the basic quantization parameter and the quantization parameter offset. The quantization parameter offset is determined according to the smoothed importance and importance quantization mapping relationship. When the actual number of encoded bits exceeds the target bitrate, the quantization parameter offset of the corresponding region is increased step by step according to the priority order of background region, context region, non-alarm target region, and alarm core target region.

[0156] The central processing unit reads the spatiotemporal importance map from the importance map storage area. Furthermore, the spatiotemporal importance map is subjected to asymmetric temporal smoothing to suppress frequent jumps in the spatiotemporal importance map caused by jitter between target detection frames and brief missed detections. Asymmetric temporal smoothing maintains rapid response capability when the target appears while providing a gradual decay transition when the target disappears. The central processing unit recursively filters each coding unit in the spatiotemporal importance map in the temporal domain to obtain the... Frame number Current frame importance of each coding unit and the smoothed importance of the same coding unit in the previous frame. , This represents the importance value of the previous frame after asymmetric temporal smoothing, serving as a historical reference for the temporal smoothing of the current frame. The central processing unit assigns the importance value of the current frame... Importance after smoothing with the previous frame A comparison is made, and different filtering inertia coefficients are selected based on the comparison results to perform asymmetric smoothing. When When this occurs, it indicates that the importance of the coding unit is on the rise, requiring a rapid response to the increase in importance. The first filter inertia coefficient is then used. Perform smoothing to obtain the smoothed importance. .when When this occurs, it indicates that the importance of the encoding unit is decreasing and needs to be gradually reduced to avoid sudden changes in image quality. Therefore, a second filter inertia coefficient is used. Perform smoothing to obtain the smoothed importance. The first filter inertia coefficient Less than the second filter inertia coefficient , and All are real numbers between 0 and 1, controlling the filter strength when importance increases and decreases, respectively. The value ranges from 0.1 to 0.3, preferably 0.2; The value range is from 0.6 to 0.9, preferably 0.8. The relationship between the two coefficients determines the characteristics of asymmetric filtering: during the rise... Smaller, historical frame weights are smaller than current frame weights The larger value allows the upward trend in importance to quickly follow changes in the test results; when it decreases... Larger, with historical frames having larger weights and the current frame having a larger weight. The smaller value allows the decreasing trend of importance to be smoothed out by historical information, resulting in a slow, gradual decay. After asymmetric temporal smoothing, the central processing unit will adjust the smoothed importance... The smooth importance storage area is stored in shared physical memory and used for subsequent generation of quantization parameter offsets.

[0157] The central processing unit (CPU) determines the importance based on the smoothed values. Generate the quantization parameter offset for each coding unit Quantization parameter offset This is an integer value representing the adjustment magnitude of the quantization parameter of this coding unit relative to the basic quantization parameter. The range of the quantization parameter offset is... to Negative values ​​indicate that reducing the quantization parameter improves coding quality. This indicates that no adjustment is needed, meaning the basic quantization parameters are used. The central processing unit then determines the importance based on the smoothed values. Determine the quantization parameter offset by mapping with important quantizations The mapping relationship adopts a piecewise linear mapping method. Let the importance value be... The range is The central processing unit divides the importance range into multiple intervals, each corresponding to a quantization parameter offset. Specifically, when When this coding unit belongs to a high-importance region, the corresponding quantization parameter offset is... , The high importance threshold is a real number between 0 and 1, used to distinguish between high-importance and medium-importance coding units. The range of values ​​is to Preferred This indicates that the region will receive the highest improvement in coding quality; when At that time, the coding unit belongs to the medium importance region, and the corresponding quantization parameter offset is... , The low importance threshold is a real number between 0 and 1. Used to distinguish between medium-importance and low-importance coding units. The range of values ​​is to Preferred This indicates that the region will receive a moderate improvement in coding quality; when At that time, the coding unit belongs to the low importance region, and the corresponding quantization parameter offset is , The range of values ​​is to Preferred This indicates that the region will experience a relatively small improvement in coding quality. The high importance threshold ranges from 0.6 to 0.8, with 0.7 being the preferred value. The threshold value is low importance, ranging from 0.2 to 0.4, preferably 0.3. For pure background coding units with an importance of 0 after smoothing, the quantization parameter offset is set to 0, indicating that the region is encoded using the basic quantization parameter.

[0158] The central processing unit (CPU) also performs closed-loop rate control, dynamically adjusting the basic quantization parameters by feeding back the error between the actual number of encoded bits and the target rate, so that the actual output rate converges to the target rate. The CPU obtains the preset target rate from the system configuration. , This is a positive real number, measured in bits per second (bps), and is preset by the system user during system configuration based on network bandwidth and storage capacity. This value represents the target output bitrate of the encoder. The central processing unit (CPU) counts the actual number of encoded bits in the most recently encoded video frames, with a statistical window size of [value missing]. frame, A positive integer, representing the number of encoded frames included in the bitrate statistics. This window is used to smooth bitrate fluctuations to avoid drastic changes in the underlying quantization parameters due to abnormal bit counts in a single frame. The [number of frames] within the window... The actual number of encoded bits in a frame is , This is a positive integer, expressed in bits. This value is fed back to the central processing unit by the video encoding unit after each frame is encoded. It represents the average number of actual encoded bits within the window. for Actual coding bit rate for ,in For video frame rate, This is a positive real number, measured in frames per second (fps), determined by the frame rate of the video capture device. The central processing unit calculates the difference between the target bit rate and the actual encoded bit rate as the bit rate error. , For real numbers, the unit is bits per second (bps). When the current bitrate is lower than the target value, there is a bitrate margin; when This indicates that the current bitrate exceeds the target value and the bitrate needs to be reduced. When the actual bitrate is lower than the target bitrate, there is a bitrate margin, and the basic quantization parameters can be appropriately reduced to improve the overall encoding quality; when When this occurs, it indicates that the actual bitrate exceeds the target bitrate, and the basic quantization parameters need to be appropriately increased to reduce the encoded bitrate. The central processing unit then determines the bitrate based on the bitrate error. Update the underlying quantization parameters for the next frame or the next group of images. :

[0159]

[0160] in This is the proportional control coefficient. It is a positive real number used to control the amplification factor of the bit rate error on the adjustment of the basic quantization parameter. It is preset by the system configuration and the value range is 2 to 6, preferably 4. It is used to control the magnitude and convergence speed of the bit rate adjustment. The base quantization parameters used for the current frame. It is an integer, and its range of values ​​is determined by the video coding standard; The underlying quantization parameters that will be used in the next frame or the next group of images. The integer value is obtained by rounding the result of the aforementioned formula to the nearest integer. The central processing unit will... The quantization parameter is limited to a preset valid range, determined by the video coding standard. For H.264 or H.265 coding standards, the valid range of the quantization parameter is 0 to 51, where 0 represents the highest coding quality (smallest quantization step size) and 51 represents the lowest coding quality (largest quantization step size). The central processing unit generates the quantization parameter offset based on the smoothed importance. With basic quantization parameters Determine the final coding quantization parameters for each coding unit. , The integer represents the number of digits. Frame number The actual quantization parameter value used by each coding unit directly controls the quantization and coding quality of the coding unit. Final coding quantization parameters. Also limited to the valid range of 0 to 51, for values ​​outside the valid range, the central processing unit truncates them to the nearest valid boundary value.

[0161] The central processing unit also implements a step-by-step degradation strategy when the bit rate exceeds the limit in closed-loop bit rate control. This is based on the actual encoded bit rate... Exceeding the target bitrate And when the amplitude exceeds the preset tolerance range, that is... ,in This is the bitrate tolerance factor. This is a non-negative real number, representing the upper limit of the allowed actual bitrate exceeding the target bitrate. Its value ranges from 0.05 to 0.15, preferably 0.1. The central processing unit increases the quantization parameter offset of the corresponding region in ascending order of priority. Increasing the quantization parameter offset means adjusting towards a positive value. This gradually reduces the coding quality of the region to save bitrate. The progressive degradation is performed in the following priority order: the first level is the background region, i.e., the importance after smoothing. The first level is the encoding unit; the second level is the context region, i.e., the importance after smoothing. The first level is the encoding unit; the third level is the non-alarm target area, i.e., the importance after smoothing. The corresponding encoding unit represents the state where no alarm event has been triggered for the target; the fourth level is the core target area for alarms, i.e., the smoothed importance. Furthermore, it corresponds to the coding units that trigger the entry state, stay state within the region, or leave the holding state of the target. The specific operation of step-by-step degradation is as follows: The central processing unit first increases the quantization parameter offset of all coding units in the background region by one step size based on the current value. , A positive integer represents the increase in the quantization parameter offset during a single downgrading operation. The step size ranges from 1 to 3, preferably 2, which reduces the coding quality of the region by one level. If the bitrate still exceeds the limit, the quantization parameter offset of all coding units in the context region is increased by the step size. If the bit rate still exceeds the limit, continue to increase the quantization parameter offset of all coding units in the non-alarm target area by a second step. , are positive integers and The second step size is smaller than the first step size, with a value range of 1, to avoid causing excessive quality loss to non-alarm targets. The alarm core target area is the last level of quality degradation target, and the minimum step size is only applied to this area after the bitrate has been continuously and severely exceeded and the first three levels of quality degradation measures have been implemented. Deterioration of quality, are positive integers and , Furthermore, after degradation, it is necessary to check whether the quantization parameter offset of the region still meets the preset minimum quality constraint to ensure that the encoding quality of the alarm core target region is not lower than the minimum quality standard configured by the system. Through the above-mentioned step-by-step degradation strategy, when the bit rate is limited, the encoding quality of the background and context regions is sacrificed first, so as to maximize the quality of the alarm core target region recognition task. The central processing unit will finally encode the quantization parameters. The quantization parameter mapping table is organized into units of encoding and written into the encoding parameter storage area in shared physical memory for the video encoding unit to read and execute compression encoding.

[0162] S8: The video encoding unit compresses and encodes the video frame according to the final encoding quantization parameters of each encoding unit to generate a compressed video stream.

[0163] The video encoding unit reads the raw image data of the current video frame to be encoded from the shared buffer in shared physical memory, and simultaneously reads the quantization parameter mapping table corresponding to that frame from the encoding parameter storage area. The quantization parameter mapping table records the final encoding quantization parameters corresponding to each encoding unit of the current video frame. The video coding unit performs compression coding operations row by row and column by column according to the coding order specified by the video coding standard. The compression coding operation includes at least four processing stages: intra-frame prediction or inter-frame prediction, transform coding, quantization, and entropy coding.

[0164] During the predictive coding phase, the video coding unit selects the appropriate prediction mode based on the frame type of the current coding unit. For coding units in I-frames, the video coding unit uses intra-frame prediction mode, constructing the predicted pixel value using the already encoded neighboring pixels in the current frame. Intra-frame prediction supports multiple directional prediction modes, including horizontal, vertical, DC, and various angle prediction modes. The video coding unit selects the optimal intra-frame prediction mode through rate-distortion optimization to minimize the coding cost of the prediction residual. For coding units in P-frames or B-frames, the video coding unit uses inter-frame prediction mode, using the already encoded reference frame as the prediction reference. Motion estimation searches for the image block in the reference frame that best matches the current coding unit as the prediction block. Motion estimation uses a full search algorithm or a fast search algorithm, with the rate-distortion cost function as the matching criterion. Let the pixel value of the current coding unit be... The predicted pixel value is Then predict the residuals for , The value is a real number, representing the coordinates in the encoding unit. The difference between the original pixel value and the predicted value at that location. The smaller the value, the higher the prediction accuracy. and These are the horizontal and vertical coordinates of the pixel within the coding unit, respectively. The smaller the absolute value, the higher the prediction accuracy.

[0165] During the transform coding stage, the video coding unit performs a discrete cosine transform or a discrete sine transform on the prediction residual, converting the spatial domain residual signal into transform coefficients in the frequency domain. Taking the discrete cosine transform as an example, let the coding unit size be... The residual matrix is , The value is the side length (in pixels) of the coding unit, a positive integer, for an H.264 macroblock. For H.265 coding tree units The transform coefficient matrix after the two-dimensional discrete cosine transform can be 8, 16, 32, or 64. elements in The calculation formula is:

[0166]

[0167] in and For the horizontal and vertical frequency indices in the frequency domain, the values ​​range from 0 to... , and The corresponding DC component is the average brightness of the image. and The larger the value, the higher the frequency of detail components; when hour ,when hour ; Definition and same, and These are the normalization coefficients, used to ensure the orthogonality of the transformation matrix. The transformed coefficient matrix. In the matrix, low-frequency components are concentrated in the upper left corner, while high-frequency components are concentrated in the lower right corner.

[0168] During the quantization phase, the video coding unit performs quantization based on the final coding quantization parameters of that coding unit. Perform quantization on the transform coefficients. The quantization operation will transform the coefficients... According to the quantization step size Uniform quantization employs an exponential relationship between the quantization step size and the quantization parameters, defined by the video coding standard employed. For H.264 or H.265 coding standards, the quantization step size doubles for every 6 units increase in the quantization parameters. This exponential relationship means that for every 6-unit change in the quantization parameters, the bitrate approximately halves or doubles. The quantized transform coefficients... Calculated using the following formula:

[0169]

[0170] in For the floor function, For quantization parameters The corresponding quantization step size, It is a positive real number, and its value follows It increases monotonically. The specific values ​​are defined in the standard documentation of the adopted video coding standard. Using smaller quantization parameters results in a smaller quantization step size, retaining more precision in the quantized transform coefficients, leading to higher quality decoded and reconstructed images, but consuming more coding bits. Conversely, using larger quantization parameters results in a larger quantization step size, causing more precision loss in the rounded transform coefficients, leading to lower quality decoded and reconstructed images, but consuming fewer coding bits. Because different coding units are assigned different final coding quantization parameters, coding units with higher importance have lower quantization parameters and smaller quantization losses, achieving differentiated coding quality control based on spatiotemporal importance.

[0171] During the entropy coding stage, the video coding unit processes the quantized transform coefficients. Entropy coding is performed on the predicted pattern information and motion vector information to generate the final compressed bitstream. Entropy coding uses context-adaptive binary arithmetic coding or context-adaptive variable-length coding, which encodes syntax elements according to the probability distribution model of the encoded elements to generate the basic units of the compressed bitstream.

[0172] After completing the four stages of prediction, transform, quantization, and entropy coding for each coding unit, the video coding unit combines the compressed bitstreams of each coding unit into a complete video frame compressed bitstream according to the syntax structure of the coding standard. The video frame compressed bitstream includes frame header information and stripe header information. The frame header information records the coding configuration parameters of the video frame, and the stripe header information records the quantization parameter information of the coding units within each stripe. For video coding units supporting ROI grid mode, the quantization parameter mapping table is configured in the ROI parameter register of the coding unit in the form of an ROI grid diagram. During the encoding process, the video coding unit automatically reads the quantization parameter values ​​of the corresponding coding units from this register, without requiring intervention from the central processing unit on a unit-by-unit basis. The video coding unit writes the generated compressed video bitstream into the bitstream output buffer in shared physical memory as a continuous byte stream, while simultaneously recording the encoding timestamp, frame type, and actual number of encoded bits for the current frame. and the actual number of encoded bits A bitrate statistics area is stored in shared physical memory. The central processing unit (CPU) reads the actual number of encoded bits for each frame from the bitrate statistics area for bitrate error calculation and basic quantization parameter updates in step S7. After completing the compression encoding of the current frame, the video encoding unit sends an encoding completion interrupt signal to the CPU, notifying the CPU that the compression encoding of the current frame is complete and the compressed video stream is available for subsequent steps to read and use.

[0173] S9: Extract video frames from the compressed video stream for quality assessment, and adjust the encoding parameters based on the quality assessment results.

[0174] In this step, keyframes or ROI images are extracted from the compressed video stream according to a preset period. A quality assessment or lightweight target recognition operation is performed on the extracted keyframes or ROI images. When any of the following triggering conditions are met, the quantization parameter offset corresponding to the ROI region is reduced or the inference frequency of the AI ​​inference unit is increased: the confidence level of the compressed target detection decreases by more than a preset confidence threshold; the clarity index of the ROI region decreases by more than a preset clarity threshold; the number of target feature points is less than a preset feature point number threshold; text recognition, license plate recognition, or instrument recognition fails; the reliability of continuous frame target tracking decreases by more than a preset reliability threshold. When the device load exceeds a preset load threshold, the device temperature exceeds a preset temperature threshold, or the network bandwidth is lower than a preset bandwidth threshold, the inference frequency of the AI ​​inference unit in non-alarm scenarios is reduced. When the target approaches the static monitoring area or the monitoring event status is an alarm state, the inference frequency of the AI ​​inference unit is increased.

[0175] After the video encoding unit completes compression encoding, the central processing unit extracts compressed video frames from the bitstream output buffer for quality evaluation according to a preset period. Preset period Configured by the system. This is a positive integer, expressed in frames, indicating the interval between frame evaluations. The value ranges from one frame every 30 to every 60 frames, with one frame preferably being selected every 50 frames. The current frame's frame number... satisfy When the frame is a keyframe or contains a Region of Interest (ROI), the central processing unit (CPU) triggers a quality assessment process. After triggering the quality assessment process, the CPU first reads the compressed bitstream of the compressed frame from the bitstream output buffer and calls the video decoding unit to perform partial decoding operations on the compressed frame, decoding to the pixel domain to reconstruct the image. The partial decoding operation includes at least entropy decoding, inverse quantization, inverse transform, and intra- or inter-frame prediction compensation steps. The reconstructed image contains complete pixel data, but only the coding units containing the ROI regions are fully decoded to reduce computational overhead. Based on the coding parameters corresponding to the compressed frame, the CPU obtains the spatial location information of each ROI region in the frame, i.e., the target bounding box. and the set of coding units covered by the context protection region.

[0176] The central processing unit (CPU) performs quality assessment or lightweight object recognition operations on the extracted keyframes or Region of Interest (ROI) images. The quality assessment operation quantifies the degree of image quality degradation in the compressed ROI region. The CPU employs a reference-free image quality assessment algorithm to extract various image features from the compressed ROI region and calculate a quality score. This algorithm does not rely on the original reference image and directly assesses quality based on the statistical characteristics of the compressed image. Specifically, the CPU calculates a sharpness index within the ROI region, which reflects the degree of image edge preservation based on gray-level variance or Laplacian variance. Let the pixel gray-level value within the ROI region be... , Integers between 0 and 255, representing coordinates The brightness value and grayscale mean of the pixel. , This represents the total number of pixels within the ROI region. The grayscale variance is the arithmetic mean of all pixel grayscale values. , It is a non-negative real number, representing the degree of dispersion of pixel grayscale values ​​relative to the mean. A larger value indicates stronger image contrast and better clarity; a larger grayscale variance also indicates higher image contrast and better clarity. The Laplacian variance is calculated by applying the Laplacian operator to the ROI region. The variance is then calculated; a larger Laplacian variance indicates sharper image edges and higher resolution. The central processing unit then processes the calculated resolution index. Compared with the preset sharpness threshold Comparison, For the calculated sharpness score, For the preset sharpness scoring threshold, when Below this threshold, it indicates that compression has caused excessive loss of image sharpness. Preset sharpness threshold. According to the system configuration settings, this means that feedback adjustment is triggered when the sharpness index drops by more than 20% to 40%, preferably when it drops by more than 30%. When the sharpness index drops beyond a preset sharpness threshold, it is determined that... This serves as a reference sharpness metric for the same ROI region within an uncompressed video frame. The reference sharpness index is calculated by analyzing the original video frames during system initialization. It serves as a reference for evaluating the sharpness degradation after compression.

[0177] The lightweight target recognition operation employs a simplified lightweight neural network model to re-perform target detection on the compressed ROI region. The lightweight neural network model has fewer parameters and lower computational cost than the main detection model loaded by the AI ​​inference unit, and it runs directly on the central processing unit without consuming the AI ​​inference unit's computational resources. The target detection output of the lightweight target recognition operation includes the target category. Detection confidence and target bounding box , The category labels output by the lightweight model. A real number between 0 and 1 represents the detection confidence of the lightweight model output. The detected target bounding box. The central processing unit outputs the detection confidence score from the lightweight target recognition operation. Detection confidence level of the target before compression Comparison, To compress the detection confidence score output by the AI ​​inference unit before detection, store it in the detection result storage area, and calculate the confidence score decrease rate. , For real numbers, the unit is the confidence level value, when Exceeding a preset threshold indicates that compression significantly reduces the reliability of target detection. Exceeding the preset reliability threshold When the confidence level of target detection decreases beyond a preset confidence threshold after compression, A real number between 0 and 1, representing the maximum allowable decrease in confidence level, and a preset confidence threshold. The value range is from 0.1 to 0.3, preferably 0.2, meaning that feedback adjustment is triggered when the detection confidence after compression decreases by more than 0.2 compared to before compression. The central processing unit also counts the number of feature points within the ROI region. The feature point extraction algorithm uses scale-invariant feature transformation or accelerated robust features to extract local feature points within the ROI region and counts the number of feature points. , A non-negative integer, representing the total number of feature points detected within the ROI region. This value reflects the richness of local texture in the image. Fewer than a preset threshold number of feature points When the number of target feature points is less than a preset feature point number threshold, A positive integer, representing the minimum number of feature points that should be retained within the ROI region. A value lower than this indicates that compression results in excessive loss of image texture information. The value range is from less than 4 to 8 feature points per ROI region, preferably less than 6 feature points. For ROI regions containing text, license plates, or instrument readings, the central processing unit calls the optical character recognition model to perform text recognition, license plate recognition, or instrument reading recognition on the ROI region. When the recognition result returns a failure flag, feedback adjustment is triggered.

[0178] The central processing unit also uses the tracking results output by the target tracking module to evaluate the target tracking reliability across consecutive frames. Residual covariance matrix based on Kalman filter calculate, Let be a 4×4 real symmetric positive definite matrix, representing the covariance of the observation residuals. Its determinant value reflects the uncertainty of the tracking prediction. The tracking reliability is inversely proportional to the determinant value of the residual covariance matrix, i.e. A smaller value indicates higher tracking reliability. The central processing unit calculates the tracking reliability for the current frame. Compared with historical average tracking reliability The comparison results, when The reliability drops below the preset threshold. When the reliability of target tracking in consecutive frames decreases beyond a preset reliability threshold, A real number between 0 and 1, representing the maximum allowable decrease in tracking reliability; exceeding this percentage triggers a feedback adjustment. The value ranges from 15% to 35%, with 25% being preferred. A preset reliability threshold is defined. This means that a feedback adjustment is triggered when the tracking reliability score drops by more than 25%.

[0179] When any of the above triggering conditions is met, the central processing unit (CPU) performs a feedback adjustment operation. The feedback adjustment operation includes at least two adjustment methods: reducing the quantization parameter offset corresponding to the ROI region, or increasing the inference frequency of the AI ​​inference unit. The CPU preferentially uses the method of reducing the quantization parameter offset because this method directly affects the encoding stage and has a fast response speed. The specific operation of reducing the quantization parameter offset is as follows: the CPU adjusts the quantization parameter offset of the encoding unit covered by the ROI region. Reduce the first step length from the current value. , A positive integer, representing the magnitude of the decrease in the quantization parameter offset during a single feedback adjustment. The value range is 1 to 3, preferably 2, which reduces the quantization parameter, decreases the quantization step size, and improves the coding quality in subsequent encoded frames for this region. If the triggering condition persists and the quantization parameter offset has reached the preset minimum limit... hour, This is an integer representing the minimum allowed quantization parameter value, i.e., the maximum encoding quality limit. If the quantization parameter is lowered further after reaching this value, it will exceed the effective range of the encoding standard. The central processing unit will then increase the inference frequency of the AI ​​inference unit to improve the real-time performance and accuracy of target detection. Increasing the inference frequency of the AI ​​inference unit is achieved by adjusting its operating mode, such as switching it from energy-saving mode to high-performance mode, or shortening the inference interval from a preset longer interval to a shorter interval.

[0180] The central processing unit also continuously monitors the system's operating status, including device load, device temperature, and network bandwidth. Device load Provided by the operating system's process scheduler, expressed as a percentage of central processing unit utilization. A real number between 0 and 100, representing the percentage of CPU resources currently being used. The central processing unit reads the current CPU utilization rate from the system file. Device temperature. Provided by a temperature sensor deployed on the surface of the processor chip, This is a real number, expressed in degrees Celsius (°C), representing the current operating temperature of the chip. The central processing unit reads the temperature sensor data via the system management bus. Network bandwidth. Provided by the network interface controller. This is a positive real number, measured in bits per second (bps), representing the available bandwidth of the current network transmission link. The central processing unit obtains this information by reading the current transmission rate from network interface statistics. (The last sentence appears to be incomplete and possibly refers to a different data point.) Exceeding the preset load threshold or equipment temperature Exceeding the preset temperature threshold or network bandwidth Below the preset bandwidth threshold At this time, the central processing unit reduces the inference frequency of the AI ​​inference unit in non-alarm scenarios. A real number between 0 and 100, representing the maximum allowed CPU utilization. A positive real number, in degrees Celsius, representing the maximum permissible operating temperature of the equipment; A positive real number, measured in bits per second, representing the minimum allowable network bandwidth. Preset load threshold. The value ranges from 80% to 95%, preferably 90%. Preset temperature threshold. The value range is 75°C to 85°C, preferably 80°C. Preset bandwidth threshold. The current network bandwidth is lower than the target bit rate. Triggered at 1.2 to 1.5 times the target bitrate, preferably below the target bitrate. The inference frequency of the AI ​​inference unit is triggered when the frequency is 1.3 times higher than that in non-alarm scenarios. The specific operation method for reducing the inference frequency of the AI ​​inference unit in non-alarm scenarios is as follows: For non-alarm targets whose monitored event status is outside the area or near the area, the operating frequency of the AI ​​inference unit is reduced from the current frequency to 50% to 70% of the current frequency to release computing resources and reduce power consumption and heat generation; for alarm targets whose monitored event status is entering the area, staying within the area, or leaving the area, the corresponding AI inference frequency remains unchanged to ensure perception accuracy in alarm scenarios. When the target approaches the static monitoring area or the monitored event status is alarm, the central processing unit increases the inference frequency of the AI ​​inference unit. The increase in inference frequency is one to three times, preferably two times, that is, maintaining the AI ​​inference unit at twice the frequency of non-alarm scenarios to perform detection and tracking on alarm targets, thereby improving target perception accuracy and response speed in alarm scenarios. The central processing unit records the quality assessment results, triggering conditions, and corresponding feedback adjustment operations in the system log for system maintenance personnel to analyze and optimize system configuration parameters.

[0181] S10: Output the compressed video stream and metadata.

[0182] In this step, the metadata includes target identifier, target category, target confidence level, ROI grid diagram or ROI region coordinates generated based on the spatiotemporal importance map, monitoring event type, monitoring event occurrence time, encoding strategy version number, target bitrate, and actual number of encoded bits. The ROI grid diagram or the monitoring event type is embedded as supplementary enhancement information of the compressed video stream into the additional data field of the compressed video stream, or saved as a custom user data field in a container format, or output synchronously with the compressed video stream as an independent metadata stream, or written to the storage medium as a database record.

[0183] After the video encoding unit completes the compression encoding of the current frame, the compressed video bitstream is stored as a continuous byte stream in the bitstream output buffer of shared physical memory. Simultaneously, the central processing unit collects structured metadata associated with the current encoded frame from various storage areas of the shared physical memory. Each component of the metadata has a clear origin: target identifier... The globally unique identifier assigned by the target tracking module in step S3; target category The detected target information is derived from the AI ​​inference unit output in step S2; target confidence. The detection confidence score is derived from the AI ​​inference unit output in step S2; the ROI grid map or the ROI region coordinates generated based on the spatiotemporal importance map are derived from the spatiotemporal importance map generated in step S6. The ROI grid diagram is a spatiotemporal importance map. The output value is higher than the preset output threshold. The binary grid diagram obtained after marking the positions of the coding units. A real number between 0 and 1, representing the minimum spatiotemporal importance threshold required to classify a coding unit as a Region of Interest (ROI). Coding units below this threshold are not marked as ROIs. The value range is from 0.3 to 0.6, preferably 0.5; the monitoring event type comes from the monitoring event status determined in step S4; the monitoring event occurrence time comes from the collection timestamp when the event first triggers the entry state or stays in the area. The encoding strategy version number is the version identifier of the encoding strategy in the current system configuration, which is loaded by the system configuration management module when the system starts; target bitrate. and actual number of encoded bits These are respectively derived from the target bitrate configured in step S7 and the statistical value of the actual number of encoded bits fed back by the video encoding unit.

[0184] The central processing unit serializes and organizes the aforementioned metadata according to a preset data structure format. The serialization process arranges the fields in the metadata into a continuous byte stream in a predetermined order, adding a field identifier and field length information before each field to facilitate parsing of the metadata at the decoding end. The serialized metadata data packet and the compressed video stream are associated through frame sequence numbers and acquisition timestamps, ensuring a one-to-one temporal match between the metadata and the corresponding video frames.

[0185] The central processing unit (CPU) supports four output methods for synchronously saving and transmitting ROI grid maps or monitoring event types along with the compressed video stream. The first output method involves embedding metadata into the supplemental enhancement information (SUE) field of the compressed video stream. For the H.264 encoding standard, SUE, as a type of network abstraction layer (NAL) unit, is located after the video coding layer data and before the transport layer. It does not affect the normal decoding process of the decoder, which can choose to read or not read this information. The CPU encapsulates the serialized metadata into one or more SUE units. Each SUE unit contains a unit type identifier, data payload, and payload length. The encapsulated SUE units are inserted into the compressed video stream of the current video frame according to the syntax structure specified by the encoding standard, forming a complete output data packet together with the compressed video stream. For the H.265 encoding standard, SUE is encapsulated in a similar way within the NLA unit, but the specific syntax structure differs. The CPU selects the corresponding encapsulation format based on the encoding standard used by the current video coding unit.

[0186] The second output method involves saving metadata as custom user data fields in a container format. Video container formats such as MP4, AVI, or MKV all support custom user data fields, which are located in the metadata area of ​​the container file and stored alongside the video stream data. The central processing unit writes the serialized metadata into a private data bin or a custom user data bin in the container format, and records the bin's type identifier as a dedicated type code corresponding to the metadata. This allows the encapsulated video file to identify and extract the metadata using this dedicated type code during playback or analysis. This method is suitable for offline storage and analysis scenarios, facilitating subsequent retrieval and event backtracking.

[0187] The third output method involves outputting metadata as an independent metadata stream, synchronously with the compressed video stream. In this method, the central processing unit writes the compressed video stream and metadata data into two independent output channels. The compressed video stream is output through the main channel of the video output interface, while the metadata data is output through the auxiliary channel of the video output interface or a separate network connection. The data in the two channels are synchronized using frame sequence numbers and timestamps. The receiving end, upon receiving the compressed video stream, simultaneously retrieves the corresponding metadata through the auxiliary channel, achieving real-time synchronous transmission of the video stream and metadata. This method is suitable for applications requiring real-time analysis or cloud storage.

[0188] The fourth output method involves writing metadata as database records to the storage medium. The central processing unit (CPU) writes the metadata to the local storage medium or a remote database server in a structured data table format via a database interface. The database table structure includes at least a timestamp field, frame sequence number field, target identifier field, target category field, monitoring event type field, and ROI area coordinate field. Each record corresponds to the metadata of one frame. This method facilitates target retrieval, event statistics, and trajectory analysis through database query statements. The four output methods can be used individually or in combination depending on the application scenario. The CPU reads the configuration parameters of the output method from the system configuration to determine which output method(s) the current system uses.

[0189] The output module reads the compressed video stream from the output buffer via the video output interface and simultaneously obtains the serialized metadata data from the central processing unit (CPU). It then performs data output operations according to the system's configured output method. When the output method is to embed supplementary enhancement information, the output module writes the encapsulated complete data packet to the video output interface. When the output method is to use custom user data fields or independent metadata streams, the output module outputs the compressed video stream and metadata data packet via the video output interface or network interface, respectively. When the output method is to use database records, the output module writes the metadata to the storage medium via the database interface. After completing all output operations, the output module sends an output completion signal to the CPU. The CPU updates the read pointer of the shared buffer, releasing the shared buffer space occupied by the output frames for subsequent video frame writing. Thus, the system completes a full closed-loop processing flow from video acquisition, target detection, event determination, encoding decision, compression encoding to stream output.

[0190] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. The bolded content represents the parameter definitions and application scenarios added or supplemented in this revised version, mainly including: a complete and clear definition of every symbol, variable, and parameter involved in each mathematical formula in S1 to S10, including its data type, value range, physical meaning, calculation source, and functional role in the overall technical solution; and clarification of each threshold parameter (such as...). , , , , , , , , , The specific meanings and typical value ranges of various filter coefficients (such as...) are explained; supplementary information is provided on each filter coefficient (such as...). , , ), control parameters (such as , , , , , The numerical range and adjustment effect of Kalman filtering are explained; the core variables of Kalman filtering, such as state vector, covariance matrix, and gain matrix, are fully described in terms of dimension and mathematical definition; the parameters such as timestamp, time difference, and prediction coordinates in time-domain interpolation extrapolation are clearly defined; and the judgment conditions and corresponding relationships of various types of region classification are clearly defined.

[0191] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0192] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A video compression method, characterized in that, The method is applied to a processor having an AI inference unit and a video encoding unit, and the method includes: Acquire video frames, write the video frames into a shared buffer, and form a monitoring branch and an encoding branch based on the same video frame; The monitoring branch performs image preprocessing on the video frame and then sends it to the AI ​​inference unit for target detection to obtain the detected target. The detected target is tracked, a target trajectory is generated, and a unique target identifier is assigned to each target; Based on the target trajectory and the preset static monitoring area, determine the overlap relationship and movement trend between the target and the static monitoring area, and generate the monitoring event status; The encoding priority of the target is determined based on the state of the monitored event, and the target position in the current video frame to be encoded is predicted based on the target trajectory. A spatiotemporal importance map is generated based on the target location, and the spatiotemporal importance map contains the spatiotemporal importance of each coding unit in the video frame; The spatiotemporal importance map is smoothed in the temporal domain. The quantization parameter offset of each coding unit is generated based on the smoothed spatiotemporal importance map. The final coding quantization parameter of each coding unit is determined based on the quantization parameter offset and the basic quantization parameter. The video encoding unit compresses and encodes the video frames according to the final encoding quantization parameters of each encoding unit to generate a compressed video stream; Video frames are extracted from the compressed video stream for quality assessment, and the encoding parameters are adjusted based on the quality assessment results.

2. The method according to claim 1, characterized in that, The monitoring branch performs image preprocessing on the video frames and then sends them to the AI ​​inference unit for target detection, specifically including: The monitoring branch scales the video frame to generate a low-resolution monitoring image, and performs color space conversion and pixel value normalization on the low-resolution monitoring image. The preprocessed low-resolution monitoring image is sent to the AI ​​inference unit, which performs target detection, instance segmentation, key point recognition, or abnormal state recognition operations, and outputs the detected target information. The detection target information includes target category, detection confidence level, target bounding box, target service priority, and the acquisition timestamp of the video frame; The target bounding box is a rectangular bounding box or a segmented region outline, and the target service priority is pre-configured according to the target category.

3. The method according to claim 1, characterized in that, Predicting the target position in the current video frame to be encoded based on the target trajectory specifically includes: Obtain the inference timestamp of the AI ​​inference unit, the acquisition timestamp of the video frame, and the encoding timestamp of the video frame to be encoded; When the detected target output by the AI ​​inference unit does not cover the video frame to be encoded, the position coordinates of the detected target in the video frame are interpolated or extrapolated in the temporal domain according to the first time difference between the inference timestamp and the acquisition timestamp and the second time difference between the encoding timestamp and the acquisition timestamp, so as to generate a predicted target region corresponding to the video frame to be encoded. The predicted target region is generated based on at least one of the following information: the target motion speed and direction calculated based on the position difference of the detected target in two adjacent frames, the predicted position obtained by performing Kalman filtering or state estimation on the position coordinates of the detected target, the optical flow field obtained by optical flow calculation on the video frame, the motion vector generated by the video coding unit, and the matching result obtained by feature point matching on adjacent video frames.

4. The method according to claim 1, characterized in that, Based on the target trajectory and the preset static monitoring area, the overlap relationship and movement trend between the target and the static monitoring area are determined, and a monitoring event status is generated, specifically including: Calculate the overlap ratio between the target bounding box and the static monitoring area. The overlap ratio is the ratio of the intersection area of ​​the target bounding box and the static monitoring area to the area of ​​the target bounding box, or the ratio of the intersection area to the area of ​​the static monitoring area, or the ratio of the intersection area to the union area of ​​the target bounding box and the static monitoring area. The monitoring event status is determined based on the comparison result between the overlap ratio and the preset overlap threshold, the positional relationship between the target center point and the boundary of the static monitoring area, the movement direction of the target crossing the boundary of the static monitoring area, and the number of consecutive frames the target stays in the static monitoring area. The monitored event status includes the status outside the area, the status of approaching the area, the status of entering the area, the status of staying within the area, the status of leaving and maintaining the status, and the status of recovery. When the monitoring event status is "entering", the target encoding priority is immediately increased to the first priority level; When the monitored event is in the leave hold state, the target encoding priority gradually decreases from the first priority level to the second priority level within a preset hold duration according to a preset decay rate.

5. The method according to claim 1, characterized in that, Generate a spatiotemporal importance map based on the target location, specifically including: The video frame is divided into multiple coding units, which are macroblocks, coding tree units, or ROI grid units supported by a hardware encoder. For the k-th coding unit in the t-th frame, spatial importance is calculated based on the spatial coverage relationship between the coding unit and the target location, the target category priority, and the context protection region. The spatial importance is the product of the overlap weight between the target bounding box and the coding unit and the target category priority, wherein the overlap weight is calculated based on the distance decay between the coding unit and the center point of the target bounding box. The context protection region is the area formed by expanding the target bounding box outward according to a preset expansion ratio, and the context protection region covers the motion edge region around the target. The spatiotemporal importance is obtained by performing time-domain filtering on the spatial importance; The quantization parameter offset corresponding to the context protection region is between the quantization parameter offset corresponding to the target core region and the quantization parameter offset corresponding to the background region.

6. The method according to claim 1, characterized in that, Temporal smoothing of the spatiotemporal importance map specifically includes: The spatiotemporal importance map is recursively filtered in the time domain to obtain the current frame importance and the previous frame importance of the k-th coding unit in the t-th frame. Compare the importance of the current frame with the importance of the previous frame; When the importance of the current frame is greater than or equal to the importance of the previous frame, the importance of the current frame is smoothed using the first filtering inertia coefficient to obtain the smoothed importance. When the importance of the current frame is less than the importance of the previous frame, the importance of the current frame is smoothed using the second filtering inertia coefficient to obtain the smoothed importance. The first filter inertia coefficient is smaller than the second filter inertia coefficient.

7. The method according to claim 1, characterized in that, The final coding quantization parameters for each coding unit are determined based on the quantization parameter offset and the basic quantization parameters, specifically including: Obtain the preset target bitrate and count the actual number of encoded bits in the encoded video frames; The difference between the target bitrate and the actual number of encoded bits is calculated as the bitrate error, and the basic quantization parameters of the next frame or the next image group are updated based on the bitrate error. The final encoding quantization parameter is the algebraic sum of the basic quantization parameter and the quantization parameter offset, wherein the quantization parameter offset is determined according to the smoothed importance and importance quantization mapping relationship. When the actual number of encoded bits exceeds the target bit rate, the quantization parameter offset of the corresponding region is increased in order of priority: background region, context region, non-alarm target region, and alarm core target region.

8. The method according to claim 1, characterized in that, Video frames are extracted from the compressed video stream for quality assessment, and encoding parameters are adjusted based on the quality assessment results. Specifically, this includes: Keyframes or ROI images are extracted from the compressed video stream according to a preset period, and quality assessment or lightweight target recognition operations are performed on the extracted keyframes or ROI images. When any of the following triggering conditions are met, the quantization parameter offset corresponding to the ROI region is reduced or the inference frequency of the AI ​​inference unit is increased: the confidence of the compressed target detection decreases by more than a preset confidence threshold, the clarity index of the ROI region decreases by more than a preset clarity threshold, the number of target feature points is less than a preset feature point number threshold, text recognition, license plate recognition, or instrument recognition fails, or the reliability of continuous frame target tracking decreases by more than a preset reliability threshold. When the device load exceeds a preset load threshold, the device temperature exceeds a preset temperature threshold, or the network bandwidth is lower than a preset bandwidth threshold, the inference frequency of the AI ​​inference unit in non-alarm scenarios is reduced. When a target approaches the static monitoring area or the monitoring event is in an alarm state, the inference frequency of the AI ​​inference unit is increased.

9. The method according to any one of claims 1 to 8, characterized in that, After the video frames are compressed and encoded, the method further includes: Output the compressed video stream and metadata; The metadata includes target identifier, target category, target confidence level, ROI grid map or ROI region coordinates generated based on the spatiotemporal importance map, monitoring event type, monitoring event occurrence time, encoding strategy version number, target bitrate, and actual number of encoded bits; The ROI grid or the monitoring event type can be embedded as supplementary enhancement information of the compressed video stream into the additional data field of the compressed video stream, or saved as a custom user data field in a container format, or output synchronously with the compressed video stream as an independent metadata stream, or written to the storage medium as a database record.

10. A video compression system, characterized in that, The system for performing the method as described in any one of claims 1 to 9 includes: The video acquisition module is used to acquire video frames, write the video frames into a shared buffer, and form a monitoring branch and an encoding branch based on the same video frame; The AI ​​inference module is set in the monitoring branch and is used to perform target detection after image preprocessing of the video frame to obtain the detected target. The target tracking module is used to track the detected target, generate the target trajectory, and assign a unique target identifier to each target; The event status determination module is used to determine the overlap relationship and movement trend between the target and the static monitoring area based on the target trajectory and the preset static monitoring area, and to generate the monitoring event status. The ROI generation module is used to determine the encoding priority of the target based on the state of the monitored event, predict the target position in the current video frame to be encoded based on the target trajectory, and generate a spatiotemporal importance map based on the target position. The spatiotemporal importance map contains the spatiotemporal importance of each encoding unit in the video frame. A time-domain smoothing module is used to smooth the spatiotemporal importance map in the time domain; The quantization parameter calculation module is used to generate the quantization parameter offset of each coding unit based on the smoothed spatiotemporal importance map, and to determine the final coding quantization parameter of each coding unit based on the quantization parameter offset and the basic quantization parameter. A video encoding module, located in the encoding branch, is used to compress and encode the video frame according to the final encoding quantization parameters of each encoding unit to generate a compressed video stream; The quality feedback module is used to extract video frames from the compressed video stream for quality assessment and adjust the encoding parameters based on the quality assessment results. The output module is used to output compressed video streams and metadata.

Images