A video target recognition method based on multi-model hot switching

Abstract

The application discloses a video target recognition method based on multi-model hot switching, and belongs to the technical field of computer vision and video processing. The method comprises an arbitration module, a switching control module and a feature adaptation and buffer module. The arbitration module generates a switching preparation signal by extracting multi-dimensional indexes such as optical flow mean, local variance, target density and scene confidence in real time through a lightweight channel independent of main reasoning. The switching control module performs atomic replacement of a computation graph pointer in a vertical blanking period, and realizes millisecond-level hot switching with zero frame loss by combining an asynchronous pre-copy and a chasing mechanism of a double buffer. The feature adaptation and buffer module solves tensor shape mismatch between heterogeneous models through a pre-compiled adaptation layer. The application also provides optimization schemes such as multi-index nonlinear fusion decision, zero-copy memory management, local slice focus reasoning and edge-cloud hierarchical unloading, significantly reduces switching delay, guarantees continuous recognition of a video stream, and is suitable for edge computing scenes with limited resources.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video processing technology, specifically a video target recognition method based on multi-model hot switching. Background Technology

[0002] With the deep integration of artificial intelligence technology and edge computing architecture, video object recognition has been widely applied in key areas such as intelligent security, autonomous driving, industrial vision inspection, and edge-cloud collaborative inference. In these complex application scenarios, balancing the detection accuracy of deep learning models with the computational overhead of real-time inference remains a core challenge in computer vision and embedded computing. Video streams are essentially highly dynamic temporal signal sequences. Sudden changes in ambient lighting, fluctuations in target movement speed, and alternating levels of scene congestion require the underlying inference engine to dynamically adjust the computational graph structure to achieve optimal performance output under limited hardware resource constraints. In existing technical practices, three main strategies are typically employed for adapting to dynamic scenes. The first is the static single-model deployment scheme, which loads only a single, fixed-structure neural network model into the device's memory. While this approach performs reasonably well in controlled and stable environments, its inherent structural rigidity becomes apparent under complex conditions: using a high-precision, large model makes it difficult to maintain a real-time frame rate at computationally limited edge devices, easily leading to missed target detections; using a lightweight, small model results in a sharp drop in detection confidence under poor lighting or target occlusion, further increasing the false detection rate. Secondly, the multi-model cold switching scheme dynamically unloads the current model and loads the target model based on scene recognition results. However, the process of loading model files from storage to memory, and the re-initialization of the inference engine for the new computation graph, creates a significant inference stall window, leading to the loss of multiple consecutive frames of video data and severely disrupting the spatiotemporal continuity of target tracking. Finally, the multi-model parallel residency scheme, while avoiding switching latency by running multiple models simultaneously in memory, exhibits a linear increase in memory usage and computational power consumption with the number of models, creating an irreconcilable conflict with the stringent power budgets and limited hardware resources of edge devices.

[0003] However, from a deep technical mechanism analysis, none of the above solutions fundamentally resolve the inherent conflict between the dynamic evolution of model structure and the continuous processing of video streams in video object recognition. The reason lies in the fact that the computation graph structure of mainstream deep learning inference frameworks is determined during the compilation phase. Memory reallocation and operator binding operations typically possess static characteristics, meaning that any modification to the computation graph topology incurs unavoidable time overhead. When processing real-time video streams with extremely high frame rate requirements, this overhead directly translates into switching cavitation, threatening downstream tasks with timestamp breakpoints. Even more challenging is the interface mismatch problem often existing between model branches of different complexities. When the tensor dimension and spatial size of the shared feature map do not match the input of the new branch, existing computation graph management mechanisms struggle to achieve a smooth transition without interrupting inference.

[0004] In addition, existing handover arbitration logic is often highly coupled with the main inference logic, lacking a deep perception of the optical flow field information between video frames and the spatial density of the target, resulting in a significant lag in handover decisions and making it difficult to cope with the mixed handover requirements of global scenes and local high-motion targets.

[0005] In summary, ensuring flexible switching of the computation graph topology while eliminating inference stagnation windows and addressing the real-time issues of cross-branch feature adaptation and high-precision arbitration decisions have become key challenges for those skilled in the art. In-depth exploration of computation graph atomic replacement, double-buffered feature synchronization, and multi-granularity switching collaboration in video stream processing has significant technical value and practical implications for building high-performance, low-power dynamic video recognition systems. Therefore, proposing a multi-model hot-switching mechanism that can guarantee spatiotemporal continuity while achieving feature-level adaptive mapping has become a pressing technical challenge in the fields of computer vision and embedded inference. Summary of the Invention

[0006] The purpose of this invention is to provide a video target recognition method based on multi-model hot switching, so as to solve the problems of high switching latency, large video memory usage, video frame loss during switching, and interface mismatch in the existing technology mentioned in the background art in dynamic scenes.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A video target recognition method based on multi-model hot switching includes the following steps: Step S1: During the system initialization phase, a multi-level computation graph architecture is constructed. The architecture includes a shared backbone network and multiple functional branches with different computational complexities. The shared backbone network is used to extract the basic feature maps of video frames. The functional branches include at least one lightweight branch and at least one high-precision branch. At the same time, for the preset set of branch switching paths, the corresponding feature dimension adaptive adaptation layer is pre-compiled and generated, and the computation graph operators and parameters of the adaptation layer are stored in the compilation cache. Step S2: An asynchronous parallel architecture is adopted. While the arbitration module is running in the main inference link, it independently and in real time extracts measurable arbitration indicators from the input video stream. The arbitration indicators include the inter-frame optical flow field spatial mean, local region variance of optical flow, target spatial density, and scene classification confidence. The arbitration module generates a switching preparation signal based on the arbitration indicators. Step S3: The switching control module receives the switching preparation signal and maintains the first buffer and the second buffer in memory. The first buffer and the second buffer adopt a zero-copy management mechanism. During the vertical blanking period before the input of the next frame of video data, the atomic replacement operation of the computation graph pointer is performed to redirect the execution entry of the inference engine from the current branch pointer to the target branch pointer. During the switching transition period, the previous frame feature map cached in the second buffer is used to perform a catch-up mechanism to achieve a smooth transition between branches. Step S4: The feature adaptation and buffer module detects the tensor shape difference between the feature map output by the shared backbone network and the input interface of the target branch. When a difference exists, a pre-compiled adaptation layer is dynamically inserted. The adaptation layer performs spatial size adjustment and channel number adjustment in sequence to transform the feature map to the shape required by the target branch.

[0008] According to the above technical solution, in step S2, the process of the arbitration module extracting the spatial mean of the inter-frame optical flow field is as follows: calculate the optical flow displacement amplitude of each pixel between the current frame and the previous frame, and then calculate the arithmetic mean of the displacement amplitude of all pixels in the whole image; when the arithmetic mean exceeds the preset global motion threshold, it is determined that a violent global motion has occurred, and a signal to switch to the lightweight branch is triggered.

[0009] According to the above technical solution, the process of the arbitration module extracting the local region variance of optical flow is as follows: the optical flow amplitude map is divided into non-overlapping grids of fixed size, and the standard deviation of the optical flow amplitude in each grid is calculated; grids with standard deviations exceeding the local motion threshold are marked as high motion regions in the binary mask map; the binary mask map is used to guide the target-level switching decision, indicating that the high-precision sub-network only performs enhanced inference on high motion regions.

[0010] According to the above technical solution, the process of the arbitration module extracting the target spatial density is as follows: an ultra-lightweight target detector independent of the main inference link detects the target candidate boxes in the current frame in real time and calculates the ratio of the union area of ​​all candidate boxes to the total area of ​​the image; when the ratio exceeds the preset crowding threshold, a signal to switch to the high-precision branch is triggered.

[0011] According to the above technical solution, in step S3, the catch-up mechanism specifically includes: during steady-state inference, after each frame of inference is completed, the feature map in the current active buffer is asynchronously copied to the inactive buffer, so that the inactive buffer always caches the complete feature map of the most recent frame; when the switching instruction is issued during the vertical blanking period after the end of the Nth frame, the original branch continues to infer the N+1th frame and outputs the result, and the target branch reads the feature map of the Nth frame from the inactive buffer for preheating inference; the preheating inference result is stored in a temporary buffer, and when the N+2th frame is reached, it is back-smoothed and corrected with the output of the original branch in the N+1th frame to update the tracker state.

[0012] According to the above technical solution, in step S2, the arbitration module also adopts a multi-dimensional fusion decision-making mechanism: constructing a nonlinear switching urgency function: in The inter-frame optical flow field spatial mean. This is the average of the sum of the standard deviations of all grid cells exceeding the local motion threshold. For the target spatial density, The highest confidence level for scene classification. , , For reference scale parameters, It is a natural constant. The urgency is then filtered using the leakage integral decision variable, as shown in the following equation: in The leakage coefficient has a value range of (0,1). Let be the decision variable for frame t. Let be the decision variable for frame t-1, with initial values. , Let t be the instantaneous switching urgency of the t-th frame; when A handover preparation signal is triggered when the high threshold is exceeded, only when... A back-cut to the original branch is only allowed when the number of frames continuously falls below the low threshold for more than the stable frame count.

[0013] According to the above technical solution, in step S3, the zero-copy management mechanism of the first buffer and the second buffer includes: the system maintains an atomic variable to indicate the currently active buffer, and flips the atomic variable through an atomic comparison exchange instruction during switching to realize the exchange of the logical address of the buffer without involving actual data movement; at the same time, after each frame of inference is completed, asynchronous DMA copy is started to copy the data of the active buffer to the inactive buffer, and the copy operation is executed overlapping with the inference pipeline of the next frame.

[0014] According to the above technical solution, the method also includes parallel processing of frame-level switching and target-level switching: the global branch is selected by the frame-level switching decision, which is responsible for the inference of the feature map of the whole frame and outputs candidate detection boxes; the local branch extracts feature map slices of high motion regions based on the binary mask map, and after extracting slices by merging connected components and RoI Align, the slices are merged into batch tensors and sent to the high-precision sub-network for batch inference; when the results are fused, the intersection-union ratio of the candidate boxes output by the global branch and the area covered by the binary mask map is calculated, and if it exceeds the threshold, the result of the local branch is used to replace it.

[0015] According to the above technical solution, the method also includes hierarchical offloading and dynamic quantization under edge-cloud collaborative inference: three candidate segmentation points—shallow, middle, and deep—are predefined; edge devices measure network bandwidth, round-trip latency, remaining power, and GPU utilization; and a composite cost function is calculated for each segmentation point. in, Dividing point The corresponding compound cost; Dividing point The amount of data in the feature map; This refers to the current network bandwidth. For network round-trip latency; For edge devices at the split point The amount of computation required; Peak computing power for edge devices; This represents the current GPU utilization rate. Transmission split point The energy consumed by the feature map, and Proportional; Remaining battery power for edge devices; select to enable The smallest split point is used for unloading; dynamic truncation quantization based on information entropy is applied to the intermediate feature maps transmitted, and the feature value distribution is statistically analyzed using a sliding window. The upper and lower bounds of the truncation that minimize entropy loss are selected, and the scaling factor is derived; after the results are returned from the cloud, a confidence-driven soft fusion formula is used: Calculate the weights of the cloud results and update the local tracker status; among which, Weighting for the fusion of results from the cloud; This is a scaling factor that controls the steepness of the S-curve; The detection confidence level corresponding to the results returned from the cloud; This is the corresponding detection confidence score output by the local lightweight branch at the edge.

[0016] According to the above technical solution, the method also includes adaptive Kalman filter parameters during the handover: in the first frame after the handover, the process noise covariance is adjusted to: in, This is the process noise covariance matrix used during the switching period; This is the process noise covariance matrix under normal steady-state conditions. This is the gain factor, which controls the amplification factor of the noise. The sequence number of the frame after the switch; The decay time constant controls the rate of exponential decay.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention solves the cold start stall problem that must be experienced during model switching in existing technologies by decoupling the arbitration, control, and adaptation modules. By using atomic replacement of computation graph pointers combined with a double buffer mechanism, the switching delay is controlled within a single frame interval (less than 33ms for a 30FPS video stream), eliminating switching cavitation and ensuring the temporal integrity of target tracking.

[0018] The pre-compiled feature dimension adaptive adaptation layer introduced in this invention solves the connection failure caused by tensor shape mismatch between branches of heterogeneous models. By pre-processing the spatial and channel mapping relationship offline, it avoids memory fragmentation and computational overhead caused by operator rebinding at runtime, thereby improving the robustness of the system.

[0019] This invention achieves more precise switching control than traditional single-trigger mechanisms by using a multi-dimensional arbitration index of inter-frame optical flow field and target density. In particular, the target-level switching combined with local variance of optical flow enables the system to allocate high-intensity computing power only in necessary areas. In edge-cloud collaborative mode, through dynamic segmentation point negotiation and feature map quantization compression technology, this invention achieves adaptability to fluctuating network environments. The collaborative working mechanism of conservative prediction at the edge and correction via cloud backhaul ensures that the system possesses both the low-latency response characteristics of the edge and the large-scale, high-precision inference capabilities of the cloud, effectively balancing real-time performance and accuracy. Attached Figure Description

[0020] Figure 1 This is a flowchart of a video target recognition method based on multi-model hot switching according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0022] like Figure 1 As shown, the initial design intent of this invention is to address core engineering pain points in video target recognition tasks under high dynamic and resource-constrained environments, such as high latency in model switching, discontinuous inference, chaotic memory management, and interface mismatch between multiple models. By achieving deep decoupling of arbitration, control, and adaptation at the logical and physical levels, this invention can ensure smooth evolution of heterogeneous model branches within milliseconds.

[0023] During system initialization, a multi-level computational graph architecture is pre-constructed. This architecture includes a shared backbone network and multiple functional branches with varying computational complexities. The shared backbone network is used to extract the basic feature maps of video frames, and the functional branches include at least one lightweight branch and at least one high-precision branch. Simultaneously, for a pre-defined set of branch switching paths in actual deployment (this set is selected based on historical switching frequency or manual configuration, with a maximum number of paths set to a preset value, such as no more than 50 common paths), the system pre-compiles and generates corresponding feature dimension adaptive layers, storing the computational graph operators and parameters of these adaptation layers in a pre-defined compilation cache. For uncommon paths, the system employs a just-in-time compilation strategy, asynchronously storing the compilation results in the cache to handle subsequent switching, thereby avoiding combinatorial explosion.

[0024] The arbitration module employs an asynchronous parallel architecture. While the main inference chain runs, it independently and in real-time extracts measurable arbitration metrics from the input video stream to generate a switching preparation signal. "Independent" means that the arbitration module has its own lightweight feature extraction path, which does not share any computing nodes with the main inference chain, only sharing the original video frame input buffer. This ensures that the arbitration process does not interfere with the real-time performance of the main inference. Arbitration metrics include the inter-frame optical flow field spatial mean, local region variance of optical flow, target spatial density, and scene classification confidence.

[0025] Specifically, the process of extracting the spatial mean of the inter-frame optical flow field is as follows: for the input current frame video data Compared with the previous frame of video data The dense optical flow algorithm is used to calculate each pixel. Displacement in the horizontal direction and vertical displacement Subsequently, the spatial average value of pixel displacement amplitude across the entire image area is calculated. Its mathematical expression is: in, and These are the width and height of the video frame, respectively (in pixels). and Each pixel The horizontal and vertical displacements (unit: pixels / frame). This formula directly reflects the average motion intensity of the entire frame. When Exceeding the preset global motion threshold When the arbitration module determines that a violent global motion has occurred in the current scene, it triggers a signal to switch to the lightweight branch to ensure the real-time performance of inference. The value is obtained through offline calibration: on the video sequence representing the scene, the threshold is traversed with a step size of 1 pixel / frame, and the maximum threshold is selected so that the tracking accuracy decreases by no more than 5% after switching and the frame rate remains above the target value.

[0026] Furthermore, the process of extracting the local variance of optical flow is as follows: the calculated optical flow amplitude map is divided into... Non-overlapping grid regions, among which , , and A fixed grid size (e.g., 32×32 pixels) is used to ensure that the number of grids is independent of hardware parallelism and determined solely by the image size. Each grid is computed. Standard deviation of internal optical flow amplitude For satisfying , To preset the local motion threshold, the calibration method is as follows: on a video containing isolated moving targets, a grid (within the minimum standard deviation of 90% recall rate for local high-motion regions) is used, and this grid is applied to a binary mask image. The corresponding region in the image is marked as 1, and the remaining regions are marked as 0. Binary mask image. Used to guide target-level switching decisions, instructing high-precision subnetworks to perform enhanced inference only on high-motion local regions.

[0027] Furthermore, the calculation process for the target spatial density is as follows: obtain the set of all detected target candidate boxes in the current frame. It should be noted that this set of candidate boxes does not originate from the main inference chain, but is generated in real-time by a separate, ultra-lightweight object detector (e.g., SSD based on MobileNetV1, with fewer than 0.5M parameters). This detector is used only for density estimation and does not participate in the main inference. The ratio of the union area of ​​all candidate boxes to the total image area is calculated. : in, For the first 1 target candidate bounding box (defined by the coordinates of the rectangle) The function to calculate the area of ​​a rectangle. and For frame width and height. This formula gives the proportion of the image occupied by the target. When Exceeding the preset congestion threshold (For example, 0.6, based on statistics of densely populated scenarios) When the cumulative distribution function of the value is obtained by taking the 80th quantile, the system triggers a signal to switch to a high-precision branch with stronger feature decoupling capability.

[0028] Furthermore, the scene classification confidence is obtained as follows: a scene classification head is set up in an independent lightweight arbitration path. This classification head takes the optical flow feature map of the arbitration module and the downsampled original frame as input, contains two depthwise separable convolutional layers and a global average pooling layer, and finally outputs a Softmax confidence vector indicating that the current frame belongs to a preset scene category through a fully connected layer. This classification head is completely independent of the shared backbone network of the main inference, its parameter count is controlled within 0.1M, and the inference time is less than 1ms / frame. When the confidence of a specific scene category exceeds 0.9, the arbitration module generates a switching instruction pointing to the scene-specific optimization branch.

[0029] The switching control module receives a switching preparation signal from the arbitration module and manages pointer replacements for computation graph branches. To achieve hot switching, this invention maintains two independent sets of buffers, a first buffer and a second buffer, in memory. During normal inference, the currently active branch writes the feature map extracted from the shared backbone network into the first buffer. Upon receiving a switching instruction, the switching control module does not immediately interrupt the inference of the current frame but marks the instruction as pending.

[0030] During the vertical blanking period before the next frame of video data is input, the switching control module performs an atomic replacement operation on the computation graph pointer, redirecting the execution entry point of the inference engine from the current branch pointer to the target branch pointer. To ensure determinism in a non-real-time operating system, the system adopts the following measures: pre-locking the CPU core of the switching control thread and setting it to a real-time scheduling strategy (such as SCHED_FIFO), placing the detection loop of the vertical blanking period in a busy-wait state, and using a hardware timestamp counter (TSC) for microsecond-level timeout judgment. If the vertical blanking period does not occur within the expected time window (e.g., within 5% of the frame period), the switching is forced to be performed immediately after the current frame is processed, and a jitter log is recorded.

[0031] In the first frame of the handover transition period, the system executes a catch-up mechanism. This mechanism relies on the premise that the second buffer always caches the feature map generated by the most recent complete inference frame. Specifically, during steady-state inference, after each frame of inference is completed, the feature map in the currently used buffer (designated as the first buffer) is copied to the second buffer (asynchronous non-blocking copy). Therefore, when the handover command is issued, the second buffer already contains the feature map of the previous frame (frame N). When the handover occurs during the vertical blanking period after frame N, the transition period is handled as follows: the original branch continues to use the first buffer, extracts features from the newly input frame N+1, and generates the output result. Simultaneously, the target branch reads the feature map of the Nth frame from the second buffer, performs a warm-up inference, and produces the output result. .

[0032] For the final output of the N+1th frame, the system employs a time-aligned fusion strategy: because The Nth frame does not belong to the same time period as the (N+1)th frame and cannot be directly merged. Therefore, the actual merging operation is delayed to the (N+2)th frame. Specifically, at the (N+1)th frame, the system outputs... (Maintaining output continuity); After the target branch completes the warm-up inference of frame N, it stores its result in a temporary buffer; When frame N+2 is input, the target branch has switched to the active branch, and it reads the feature map of frame N+2 from the first buffer (at this time, read and write permissions have been switched, and the first buffer becomes the write area of ​​the target branch), generating... At the same time, the system will temporarily cache the data. and Kalman filtering fusion is performed to correct the output trajectory of frame N+1 (backward smoothing), but this correction does not change the real-time output stream; instead, it is used for tracker state updates. For simplified implementation, another preferred approach is to discard the data directly during the transition period. Only output The target branch takes over completely starting from frame N+2. This simplified approach still guarantees no dropped frames, but sacrifices one frame of high-precision opportunity. This embodiment employs a time-aligned fusion strategy to achieve optimal accuracy. After processing frame N+2, the switching control module swaps the read / write permissions of the first and second buffers, and the system enters the steady-state inference process of the target branch. This double-buffering mechanism ensures that the video stream input is not interrupted during computation graph reconstruction, and the output timestamps remain physically continuous.

[0033] The feature adaptation and buffering module addresses the interface mismatch issue between different model branches. When the tensor shape of the shared feature map is inconsistent with the target branch input interface in terms of channel count or spatial size, this module dynamically inserts a pre-compiled adaptation layer between them. The adaptation layer processing flow is as follows: First, spatial size adjustment is performed, using a bilinear interpolation algorithm to adjust the spatial resolution of the input feature map. , Scale to the resolution required by the target branch. , The interpolation coefficients are calculated using a fixed formula: for the output coordinates , Mapped to input coordinates: Then, the four nearest pixels are taken as a bilinear weight. Following this, channel number adjustment is performed, using a 1×1 convolutional layer to linearly map the feature map, increasing the number of input channels. The number of channels required to transform into the target branch The weight parameters of the 1×1 convolutional layer have been pre-trained and offline quantized during the initialization phase according to the coupling requirements of different branches. Since the adaptation layer has been pre-compiled into a static computation graph and resides in the cache, the overhead of switching is limited to memory copying of tensor data and operator scheduling, avoiding millisecond-level latency caused by runtime recompilation. As a preferred embodiment of the present invention, the method adopts a parallel processing architecture of frame-level switching and target-level switching to cope with the situation where there are both global scene changes and local high-motion targets in the same frame. The parallel processing architecture includes a global branch and a local branch. The global branch is selected by the frame-level switching decision, is responsible for processing the feature map of the entire video frame, and outputs all candidate detection boxes. The local branch generates a binary mask map based on the local region variance of optical flow. The system locates highly motion-salient regions in the feature map. It extracts only the corresponding feature map slices from these regions and inputs them into a high-precision sub-network for local fine-grained inference. To avoid resource conflicts, the global and local branches share the same inference engine using time-division multiplexing: the system divides the processing of each frame into a global stage and a local stage in time. First, global branch inference is performed, and its output is temporarily stored; then, slices are extracted based on the mask map, and local branch inference is performed. The two stages are separated by a memory synchronization barrier to prevent data contention. Since the area of ​​a local slice is typically less than 30% of the entire image, the increase in total computation is controllable.

[0034] During the results fusion phase, the system computes each candidate box from the global branch output. With binary mask image The Intersection over Union (IoU) of the covered area. If the IoU exceeds 0.5, the high-precision detection result output by the local branch in that area replaces the corresponding result of the global branch; otherwise, the output of the global branch is retained. This mechanism achieves a non-uniform distribution of computing resources in the spatial dimension.

[0035] In a preferred embodiment of the present invention, in an edge-cloud collaborative inference scenario, the method determines the partition points of the computation graph through a dynamic negotiation mechanism. The system predefines... Candidate split point index Each segmentation point corresponds to a layer of different depth in the neural network. Edge devices periodically measure the current network round-trip time (RTT) and their own operating parameters such as battery level and computing power utilization, and encapsulate these parameters into a state vector and report them to the cloud server. The cloud server calculates the optimal segmentation point index under the current operating conditions based on a preset performance mapping matrix. This index is then fed back to the edge device. The edge device, based on... The computation graph prefix before the segmentation point is retained, while the computation graph suffix task after the segmentation point is offloaded to the cloud. When processing the current frame, the edge device performs prefix inference to obtain an intermediate feature map. To reduce transmission bandwidth consumption, the edge device performs compression processing on the intermediate feature map based on 8-bit symmetric quantization, and then sends it to the cloud via the network. While waiting for the cloud to send back the result, the edge device synchronously executes a local lightweight prediction branch, producing a conservative recognition result to ensure real-time response in the first frame. When the cloud returns a high-precision recognition result, the edge device uses a Kalman filter to correct the local recognition trajectory. Specifically, the correction logic is as follows: using the cloud result as the observation value, the state vector of the local tracker is updated, thereby eliminating the drift error that may be generated by the lightweight model at the edge.

[0036] Furthermore, at the specific engineering implementation level, the atomic replacement operation of the computation graph pointer is achieved by inserting a memory barrier into the instruction stream, ensuring that no inference operator reads inconsistent address information before the pointer update is completed. The feature map quantization process uses a linear quantization algorithm calibrated with KL divergence. The quantization scaling factor is determined by pre-statistically calculated feature distribution histograms, thereby compressing the 32-bit floating-point tensor into an 8-bit integer tensor. This reduces network transmission load by 75% while ensuring accuracy loss is less than 1%. As another preferred embodiment of the invention, when processing multi-target tracking tasks, the system also synchronously sends a hot-switching signal to the associated Kalman filter. At the moment of switching, the filter automatically increases the process noise covariance in the state transition matrix. This is to address potential abrupt changes in feature representations that may occur during model switching. After the target branch has been running stably for 3 frames, then... The value returned to normal levels. This engineering detail further enhances the system's trajectory smoothness during the heterogeneous evolution of the model.

[0037] To further demonstrate the non-obviousness and technical superiority of this invention, this embodiment provides an engineering implementation example on a specific hardware platform. The hardware environment uses an NVIDIA Jetson Orin computing platform, and the video input source is a 1080P / 30FPS vehicle monitoring video stream. The system is pre-installed with a shared backbone network based on ResNet-18, with the lightweight branch using YOLOv5-Nano and the high-precision branch using YOLOv5-Large. The global motion threshold of the arbitration module... Set to 15 pixels / frame, local motion threshold Set to 10 pixels / frame, crowding threshold The value was set to 0.55. The feature adaptation layer used pre-compiled TensorRT operators, with 30 common switching paths pre-compiled. During the test, when the vehicle started from a standstill and quickly entered a congested intersection, the arbitration module detected it in frame 124. The resolution quickly increased to 22 pixels per frame, with the target density D reaching 0.62. The switching control module immediately initiated the hot-switching logic, completing the pointer replacement at frame 125 (during the vertical blanking period). Experiments showed that the frame delay during the switching process was only 12ms, far less than the 33ms frame gap. The video output showed no flickering or lag, and the ID of the target detection box remained consistent before and after the switching.

[0038] In summary, the technical solution described in Embodiment 1 solves the cold start stagnation problem that must be experienced during model switching in the prior art by decoupling the arbitration, control and adaptation modules and combining double buffer atomic replacement and pre-compiled adaptation layer, thus realizing lossless continuous inference of video streams.

[0039] Example 2 Based on Embodiment 1 above, this embodiment further refines and expands the video target recognition method based on multi-model hot switching. It focuses on the multi-dimensional fusion decision-making mechanism of arbitration indicators, zero-copy memory management of double buffers, efficient extraction method of local feature slices, and hierarchical unloading and adaptive parameter adjustment in edge-cloud collaborative inference.

[0040] In real-world dynamic scenarios, a single arbitration metric is prone to false triggers. For example, a brief increase in the mean optical flow caused by swaying leaves does not necessarily indicate a need to switch to a high-precision branch. Therefore, this embodiment introduces a fusion decision mechanism based on a nonlinear switching urgency function. This mechanism first maps four original metrics to a unified urgency space: defining the accumulation of local motion intensity exceeding a threshold. For all grids exceeding the local motion threshold The average of the sum of the standard deviations, i.e.:

[0041] in , For the number of grid rows and columns, For grid The standard deviation of the internal optical flow amplitude This is the local motion threshold. This index, compared to simple grid scaling, better reflects the cumulative effect of local motion intensity. Subsequently, the system constructs the following nonlinear switching urgency function. : In the formula, The spatial mean of the optical flow field between frames (pixels / frame); The global motion reference scale parameter (pixels / frame) is used to... Normalized to the sensitive region of the logistic function, the function value corresponding to the typical motion range is kept in the non-saturation region through offline calibration; The local motion intensity exceeding the threshold accumulation (pixels / frame) is defined as above; As a local motion reference scale parameter (pixels / frame), it acts on... similar; The target spatial density (dimensionless); This is the density reference scale parameter (dimensionless), usually set to 1, or adjusted according to typical scenarios; The highest confidence score for scene classification (dimensionless, range of values) ); Given a hyperbolic tangent function, compress the product result to... .

[0042] The physical meaning of this formula is that the three logistic functions (Sigmoid) respectively... and Nonlinear mapping to The interval, and then the normalized density. The product is multiplied by the scene confidence level and then compressed using a hyperbolic tangent function to produce saturation characteristics. This product design ensures that the urgency level only increases significantly when the global motion, local motion, and target density all reach a certain level simultaneously; if only one of them is high while the others are low, the product will approach zero, thus effectively suppressing false triggering caused by noise from a single indicator.

[0043] To avoid frequent switching due to momentary fluctuations, the system does not switch during momentary fluctuations. Instead of a direct response, a decision variable with leakage integral is introduced. : In the formula, For the first Decision variables of a frame (dimensionless, initial values) =0); Leakage coefficient (dimensionless, range of values) (Typical value 0.7), controlling the decay rate of historical information; Let t represent the instantaneous switching urgency of the t-th frame.

[0044] The meaning of this recursive formula is: the current decision variable is a weighted average of the previous frame's decision variable and the current instantaneous urgency; historical information influences the current judgment in an exponentially decaying manner. Exceeding the high threshold When the value is 0.65, the system triggers a handover preparation signal; only when... continuously below the low threshold =0.35 and consistently exceeds the stable frame rate Only after 5 frames is a cutback to the original branch allowed. This stabilization mechanism ensures that the system does not produce ping-pong switching when the scene changes rapidly, while retaining the ability to respond quickly to stress events.

[0045] At the memory management level, to achieve the lowest possible latency during hot switching, this embodiment optimizes the dual-buffer mechanism with zero-copy. During initialization, the system allocates two physically contiguous, identical-sized GPU memory regions based on the maximum possible tensor shape of the shared backbone network output feature map, ensuring that their virtual addresses are aligned with memory page boundaries. The system maintains an atomic variable `active_id` (with a value of 0 or 1). The inference engine reads this variable before writing the feature map in each frame to determine the buffer to be written to. When switching occurs, the switching control module only needs to execute one atomic compare-swap instruction to flip `active_id` from its current value to another value. This operation requires only a few clock cycles on modern CPUs and does not involve any GPU memory copying. To ensure that the second buffer always caches the complete feature map of the most recent frame, the system initiates an asynchronous DMA copy stream after each frame's inference is completed, copying the data from the currently active buffer to the inactive buffer. This copy operation is executed in parallel with the inference pipeline of the next frame, overlapping through dual GPU memory channels, and has no impact on the real-time performance of the main inference. This design of atomic semantic tag swapping plus asynchronous pre-copy minimizes the overhead of the switching operation.

[0046] To efficiently extract local feature slices, this embodiment proposes a continuous slice extraction strategy based on connected component merging and RoIAlign. (Binary mask image) High-motion mesh regions were marked, but these meshes may be discontinuous and irregularly shaped. The system first... Eight-neighbor connected component labeling is performed to obtain several independent highly motion-connected regions. For each region, its bounding rectangle is calculated and expanded outwards by a fixed number of pixels (e.g., 16 pixels) to eliminate boundary truncation effects. Subsequently, the RoIAlign operator is used to extract feature slices corresponding to each expanded rectangle from the global feature map. This operator avoids feature misalignment caused by quantization rounding through bilinear interpolation, achieving sub-pixel level spatial precision for the slices. Since multiple slices may overlap, the system merges all slices into a single batch tensor and feeds it into the high-precision sub-network for inference at once, fully utilizing the parallel computing capabilities of the GPU. The coordinates of the detection boxes output by inference are located in the local coordinate system of the slice. Through inverse coordinate transformation (adding the slice origin offset to the local coordinates), they are mapped back to the original image coordinate system. The entire process maintains floating-point precision, thus achieving a non-uniform distribution of computational resources in the spatial dimension.

[0047] In the edge-cloud collaborative inference scenario, this embodiment further introduces a hierarchical offloading and closed-loop adaptive quantization mechanism, enabling the system to dynamically adjust the offloading strategy based on real-time network and computing resource conditions. The system predefines three typical segmentation points: shallow segmentation point (large feature map size but low local computation), mid-level segmentation point (balance point), and deep segmentation point (small feature map size but high local computation). Edge devices periodically measure network bandwidth. Round-trip delay Remaining battery power and local GPU utilization For each candidate split point The system calculates the following composite cost function: in, Dividing point The corresponding composite cost (unit: time × energy factor, dimensionless for comparison). Dividing point The amount of data in the feature map (in bytes); Current network bandwidth (unit: bytes / second); Network round-trip time (in seconds); For edge devices at the split point The amount of computation required (unit: number of floating-point operations); Peak computing power of edge devices (unit: floating-point operations per second); The current GPU utilization (dimensionless, range of values) ); Transmission split point The energy consumed (in joules) in the feature map, and Proportional; Remaining power of edge devices (unit: joules).

[0048] This formula uses a product form instead of the traditional weighted summation, based on the nonlinear coupling between the various costs: when the bandwidth is extremely low, the transmission time... It is already very large, then multiplied by This will further amplify the punishment; while the exponential term This causes the cost of offloading to rise sharply when the local load is heavy and the remaining battery power is low, guiding the system to choose a shallower split point (i.e., offloading more tasks to the cloud). The system selects to... The smallest split point is used as the actual execution split point, thereby achieving an adaptive optimal trade-off under complex working conditions.

[0049] Meanwhile, to improve quantization accuracy, this embodiment abandons the fixed KL divergence calibration scaling factor and instead maintains a lightweight statistical module at the edge device. This module monitors the distribution characteristics of the feature map to be quantized in real time using a sliding window (e.g., 100 frames) and employs a dynamic truncation method based on information entropy to determine the quantization parameters. Specifically, the system statistically analyzes the histogram of feature values ​​within the statistical window, calculates the information entropy at each candidate truncation threshold, and selects the upper and lower truncation points that minimize entropy loss. Then export the quantization scaling factor. (For symmetric quantization, the adjustment is as follows:) in and These are the lower and upper bounds of the quantization cutoff (with the same units as the eigenvalues). The scaling factor is used for quantization. This method does not require any pre-set distribution assumptions and can adaptively match the actual distribution of the feature map, further reducing accuracy loss compared to fixed KL calibration. Edge devices use the cutoff point or scaling factor as metadata and compress the features. Figure 1 It is then sent to the cloud, where the same parameters are used for inverse quantization during decompression.

[0050] Once the cloud results are returned, the system performs a confidence-driven soft fusion. The fusion weights are determined by mapping the difference between the cloud and local confidence scores using an S-curve. in, The fusion weights for cloud results (dimensionless, range of values) ); The scaling factor (dimensionless, typical value 5) controls the steepness of the S-curve. The detection confidence level (dimensionless, range of values) corresponding to the results returned from the cloud. ); The corresponding detection confidence (dimensionless, range of values) output by the local lightweight branch at the edge. ).

[0051] The meaning of this formula is: when the confidence level in the cloud... Significantly higher than the local confidence level hour, When the value approaches 1, cloud-based results dominate; conversely... Approaching 0, retain the local result. The merged state is updated as follows: in This is the state vector of the local tracker. This is the state vector mapped from cloud observations. This represents the updated state after fusion. This soft fusion strategy avoids trajectory jitter caused by hard handover while incorporating high-precision information from the cloud.

[0052] Furthermore, this embodiment incorporates a mathematical extension for the adaptive Kalman filter parameters during handover. In the first frame after handover, the system calculates the process noise covariance. Adjusted to: in, The process noise covariance matrix used during switching (units matched to state variables). This is the process noise covariance matrix under normal steady-state conditions. This is the gain factor (dimensionless, typical value 10), which controls the factor by which noise is amplified; The sequence number of the frame after the switch (integer, t=0 is the frame in which the switch occurred); This is the decay time constant (unit: frame, typical value 3), which controls the rate of exponential decay.

[0053] This exponential decay model makes the filter more sensitive to observations during switching (quickly adapting to changes in model output), and then gradually returns to a normal smooth level, ensuring both rapid response during switching and avoiding long-term oscillations. This adjustment is triggered synchronously with the switching signal and is written to the filter parameter register immediately after the atomic replacement is completed by the switching control module.

[0054] It is important to emphasize that all methods described in this embodiment do not depend on specific hardware accelerators or operating systems, but are based solely on general deep learning frameworks (such as TensorRT and OpenVINO) and operating system real-time scheduling mechanisms (such as SCHED_FIFO), thus possessing excellent cross-platform portability. The refinements and extensions based on Embodiment 1 further demonstrate the operability and technical superiority of this invention at the engineering implementation level, forming a complete closed-loop system from low-level memory management and instruction flow control to high-level semantic arbitration.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0056] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video target recognition method based on multi-model hot switching, characterized in that: Includes the following steps: Step S1: During the system initialization phase, a multi-level computation graph architecture is constructed. The architecture includes a shared backbone network and multiple functional branches with different computational complexities. The shared backbone network is used to extract the basic feature maps of video frames. The functional branches include at least one lightweight branch and at least one high-precision branch. At the same time, for the preset set of branch switching paths, the corresponding feature dimension adaptive adaptation layer is pre-compiled and generated, and the computation graph operators and parameters of the adaptation layer are stored in the compilation cache. Step S2: An asynchronous parallel architecture is adopted. While the arbitration module is running in the main inference link, it independently and in real time extracts measurable arbitration indicators from the input video stream. The arbitration indicators include the inter-frame optical flow field spatial mean, local region variance of optical flow, target spatial density, and scene classification confidence. The arbitration module generates a switching preparation signal based on the arbitration indicators. Step S3: The switching control module receives the switching preparation signal and maintains the first buffer and the second buffer in memory. The first buffer and the second buffer adopt a zero-copy management mechanism. During the vertical blanking period before the input of the next frame of video data, the atomic replacement operation of the computation graph pointer is performed to redirect the execution entry of the inference engine from the current branch pointer to the target branch pointer. During the switching transition period, the previous frame feature map cached in the second buffer is used to perform a catch-up mechanism to achieve a smooth transition between branches. Step S4: The feature adaptation and buffer module detects the tensor shape difference between the feature map output by the shared backbone network and the input interface of the target branch. When a difference exists, a pre-compiled adaptation layer is dynamically inserted. The adaptation layer performs spatial size adjustment and channel number adjustment in sequence to transform the feature map to the shape required by the target branch.

2. The video target recognition method based on multi-model hot switching according to claim 1, characterized in that: In step S2, the arbitration module extracts the spatial mean of the inter-frame optical flow field by calculating the optical flow displacement amplitude of each pixel between the current frame and the previous frame, and then calculating the arithmetic mean of the displacement amplitudes of all pixels in the entire image. When the arithmetic mean exceeds the preset global motion threshold, a violent global motion is determined to have occurred, triggering a signal to switch to the lightweight branch.

3. The video target recognition method based on multi-model hot switching according to claim 2, characterized in that: The arbitration module extracts the local variance of optical flow as follows: the optical flow amplitude map is divided into non-overlapping grids of fixed size, and the standard deviation of the optical flow amplitude in each grid is calculated; grids with standard deviations exceeding the local motion threshold are marked as high motion regions in the binary mask map; the binary mask map is used to guide the target-level switching decision, indicating that the high-precision sub-network only performs enhanced inference on high motion regions.

4. The video target recognition method based on multi-model hot switching according to claim 3, characterized in that: The arbitration module extracts the target spatial density as follows: an ultra-lightweight target detector, independent of the main inference link, detects the target candidate boxes in the current frame in real time and calculates the ratio of the union area of ​​all candidate boxes to the total area of ​​the image; when the ratio exceeds the preset crowding threshold, a signal to switch to the high-precision branch is triggered.

5. The video target recognition method based on multi-model hot switching according to claim 1, characterized in that: In step S3, the catch-up mechanism specifically includes: during steady-state inference, after each frame of inference is completed, the feature map in the current active buffer is asynchronously copied to the inactive buffer, so that the inactive buffer always caches the complete feature map of the most recent frame; when the switching instruction is issued during the vertical blanking period after the end of the Nth frame, the original branch continues to infer the N+1th frame and outputs the result, while the target branch reads the feature map of the Nth frame from the inactive buffer for pre-warming inference; the pre-warming inference result is stored in a temporary buffer, and when the N+2th frame is reached, it is back-smoothed and corrected with the output of the original branch in the N+1th frame to update the tracker state.

6. The video target recognition method based on multi-model hot switching according to claim 1, characterized in that: In step S2, the arbitration module also employs a multi-dimensional fusion decision-making mechanism: constructing a nonlinear switching urgency function. in The inter-frame optical flow field spatial mean. This is the average of the sum of the standard deviations of all grid cells exceeding the local motion threshold. For the target spatial density, The highest confidence level for scene classification. , , For reference scale parameters; It is a natural constant. The urgency is then filtered using the leakage integral decision variable, as shown in the following equation: in The leakage coefficient has a value range of (0,1). Let be the decision variable for frame t. Let be the decision variable for frame t-1, with initial values. , Let t be the instantaneous switching urgency of the t-th frame; when A handover preparation signal is triggered when the high threshold is exceeded, only when... A back-cut to the original branch is only allowed when the number of frames continuously falls below the low threshold for more than the stable frame count.

7. The video target recognition method based on multi-model hot switching according to claim 1, characterized in that: In step S3, the zero-copy management mechanism for the first and second buffers includes: the system maintains an atomic variable to indicate the currently active buffer; during switching, the atomic variable is flipped through an atomic comparison exchange instruction to realize the exchange of the buffer's logical address without involving actual data movement; at the same time, after each frame of inference is completed, asynchronous DMA copy is started to copy the active buffer data to the inactive buffer, and the copy operation overlaps with the next frame of inference pipeline.

8. The video target recognition method based on multi-model hot switching according to claim 1, characterized in that: The method also includes parallel processing of frame-level switching and target-level switching: the global branch is selected by the frame-level switching decision, and is responsible for the inference of the feature map of the whole frame and outputting candidate detection boxes; The local branch extracts feature map slices of high-motion regions based on the binary mask image. After extracting slices using connected component merging and RoI Align, multiple slices are merged into a batch tensor and fed into the high-precision sub-network for batch inference. When fusing the results, the intersection-union ratio of the candidate boxes output by the global branch and the area covered by the binary mask image is calculated. If it exceeds the threshold, the result of the local branch is used to replace it.

9. A video target recognition method based on multi-model hot switching according to claim 1, characterized in that: The method also includes hierarchical offloading and dynamic quantization under edge-cloud collaborative inference: three candidate segmentation points—shallow, mid-layer, and deep—are predefined. Edge devices measure network bandwidth, round-trip latency, remaining power, and GPU utilization, and calculate a composite cost function for each segmentation point. in, Dividing point The corresponding compound cost; Dividing point The amount of data in the feature map; This refers to the current network bandwidth. For network round-trip latency; For edge devices at the split point The amount of computation required; Peak computing power for edge devices; This represents the current GPU utilization rate. Transmission split point The energy consumed by the feature map, and Proportional; Remaining battery power for edge devices; select to enable The smallest split point is used for unloading; dynamic truncation quantization based on information entropy is applied to the intermediate feature maps transmitted, and the feature value distribution is statistically analyzed using a sliding window. The upper and lower bounds of the truncation that minimize entropy loss are selected, and the scaling factor is derived; after the results are returned from the cloud, a confidence-driven soft fusion formula is used: Calculate the weights of the cloud results and update the local tracker status; among which, Weighting for the fusion of results from the cloud; This is a scaling factor that controls the steepness of the S-curve; The detection confidence level corresponding to the results returned from the cloud; This is the corresponding detection confidence score output by the local lightweight branch at the edge.

10. A video target recognition method based on multi-model hot switching according to claim 1, characterized in that: The method also includes adaptive Kalman filter parameters during handover: in the first frame after handover, the process noise covariance is adjusted to: in, This is the process noise covariance matrix used during the switching period; This is the process noise covariance matrix under normal steady-state conditions. This is the gain factor, which controls the amplification factor of the noise. The sequence number of the frame after the switch; The decay time constant controls the rate of exponential decay.

Images