A Multi-Model Video Stream Fusion Inference Method and System Based on Graphics Processing Unit

Abstract

A multi-model video stream fusion inference method and system based on a graphics processing unit (GPU) relates to the field of image data processing technology. It is applicable to intelligent image data analysis in various scenarios such as industrial visual inspection, intelligent security, traffic monitoring, and energy inspection. The method includes decoding video frames into NV12 / P010 on the GPU, followed by color conversion, scaling, filling, and normalization to generate tensors via a fusion kernel. Each stage is asynchronously connected to the GPU stream and events. The inference engine is loaded after authorization binding and authentication encryption, and enforces consistency constraints. On the device side, the output of the detection head is decoded, sorted, and subjected to category-independent non-maximum suppression, and can be directly overlaid on the NV12 surface. Events are persisted with minimum copy resolution. Session costs are estimated by combining resolution and frame rate, and devices are optimally allocated using normalized scoring and relaxed alignment. This solution achieves low latency, zero-copy visualization, secure deployment, and stable scheduling.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and specifically to a multi-model video stream fusion inference method and system based on a graphics processing unit. Background Technology

[0002] With the widespread adoption of video structuring in fields such as security, industrial quality inspection, and traffic management, multiple heterogeneous detection models (with different input resolutions, thresholds, and business labels) often need to run in parallel from the same video source to achieve multi-target and multi-view recognition. Existing solutions generally suffer from the following limitations:

[0003] 1. Preprocessing link fragmentation: Color space conversion, scaling, letterboxing, and normalization are completed by multiple kernels / multiple memory accesses, resulting in high memory bandwidth and synchronization overhead, and large end-to-end latency;

[0004] 2. Post-processing and visualization transfer: YOLO-type raw head decoding, sorting and non-maximum suppression (NMS) are often performed on the host side or accompanied by large-scale D2H copying; NV12 frame overlay is often done through the CPU image library, which disrupts the zero-copy link;

[0005] 3. Insufficient engine security and consistency: The model engine lacks effective runtime binding and encryption / decryption constraints. The FP16-only pipeline is easily corrupted by unexpected formats, leading to stability and performance issues.

[0006] 4. Coarse multi-path concurrent scheduling: It lacks a strategy for aligning interpretable capacity estimation of GPU / NVDEC / NVENC / video memory with actual measurements, making it prone to overload and jitter.

[0007] 5. Weak event capture capability: Snapshot / event data organization lacks a unique business identifier (GUID, Globally Unique Identifier), is not time-ordered, and lacks cost-saving and regular cleanup.

[0008] Therefore, a general solution is needed that integrates preprocessing and postprocessing in-GPU environments, has engine-safe loading and FP16-only constraints, and provides interpretable resource scheduling and event capture management. Summary of the Invention

[0009] This invention provides a multi-model video stream fusion inference method and system based on a graphics processing unit (GPU), which realizes full-process fusion processing of video streams within the GPU, secure loading and authorization control of the inference engine, and intelligent allocation and event-based management of multi-GPU resources, significantly improving the system's throughput efficiency, security robustness, and scalability.

[0010] A multi-model video stream fusion inference method based on a graphics processing unit includes:

[0011] S1 receives the video stream and decodes it into image frames with color format NV12 or P010 on the graphics processing unit;

[0012] S2, on the graphics processing unit, a single fusion kernel sequentially performs color space conversion, size scaling and aspect ratio-preserving mailbox filling, channel order adjustment and normalization on the image frame to generate a low bit-width linear tensor layout model input, wherein the fusion kernel is a CUDA kernel;

[0013] S3 assigns computational flows to graphics processing units for each stage of decoding, preprocessing, inference, and postprocessing, and uses events to achieve asynchronous connection and overlapping execution across stages.

[0014] S4 performs encryption header determination and key derivation decryption based on authorized sequence on the deep learning inference engine file, loads the inference engine and creates inference context and output buffer with business unique identifier as key, verifies that the engine input / output meets the constraints of half precision and linear layout, sets the device input / output address and enqueues the inference asynchronously.

[0015] S5 performs frame decoding, probability calculation, score sorting, category-independent nonmaximum suppression, and result compaction on the raw output of the detection head on the graphics processing unit to obtain a stable output on the device side, and draws a visual overlay on the NV12 frame plane without the need for host copying.

[0016] S6 encodes the processed frame on the graphics processing unit and pushes it back to the transmission channel. At the same time, it filters out the detection results that meet the threshold using the minimum copy method from device to host, generates an event record with a combination of business unique identifier, timestamp and index, persists it to the key value database, and performs capture throttling on the same business unique identifier within the minimum time interval.

[0017] S7 estimates the cost of the session's graphics processing unit / video decoding / video encoding / video memory based on the video stream resolution, frame rate, and model input size. Combining real-time device operation statistics and preset thresholds, it aligns the aggregated prediction with the actual measurement through a relaxation strategy. Under multiple threshold constraints, it selects and allocates the target graphics processing unit and registers or releases the session's occupancy.

[0018] Furthermore, the fusion kernel in step S2 supports at least one of nearest neighbor interpolation and bilinear interpolation, and may optionally use bilinear sampling based on texture objects; the mailbox-style fill value is derived from the fill RGB value and mean / variance before normalization.

[0019] Furthermore, in step S2, normalization and YUV to RGB matrix transformation are performed on the 8-bit and 10-bit code values ​​of the NV12 and P010 inputs, respectively, and the UV sub-samples are read and interpolated according to a 2×2 grid alignment.

[0020] Furthermore, in step S3, the stream registry indexed by (business unique identifier, stage label) is used to obtain or create the corresponding graphics processing unit stream, and waits for the producer stream's event to complete on the consumer stream through a reused event object.

[0021] Furthermore, in step S4, when the inference engine loads, it forces verification that all input / output tensor types are low-bit-width numerical formats (e.g., half-precision) and tensor formats are linear layouts (e.g., NCHW); otherwise, execution is refused.

[0022] Furthermore, in step S4, the encryption and decryption of the inference engine file adopts a symmetric key obtained from the authorization sequence and salt based on the key derivation function, and uses a certified symmetric encryption algorithm for encryption and decryption. The file header contains at least the magic number, version, salt, random number, certification tag and ciphertext payload length fields.

[0023] Furthermore, in step S4, the authorization sequence is obtained by the authorization module and cached after the first acquisition; when the authorization sequence is empty, decryption is skipped and an alarm is recorded.

[0024] It should be understood that in this invention, "inference engine" and "inference model" are two related but functionally different concepts:

[0025] An inference model refers to the trained deep learning network structure and its weight file, used to define the computational logic and parameters.

[0026] The inference engine is the runtime framework or execution vehicle for executing the model, used to perform operator scheduling, memory management, and tensor computation on the GPU.

[0027] This invention performs authentication, decryption, and binding operations on the inference engine file during the loading phase to ensure the security and consistency of model execution; the model itself can be independently registered or shared as an input asset of the engine, thereby supporting multi-model fusion inference.

[0028] Furthermore, in step S4, the inference context maintains the execution context and output buffer according to the unique identifier of the business, and the size of the output buffer is allocated and reused on the device side as needed based on the shape of the output tensor.

[0029] Furthermore, in step S4, the inference is started asynchronously after the device address of the input / output tensor is bound to the inference engine execution context and associated with the specified graphics processing unit stream.

[0030] Furthermore, in step S5, the number of output channels of the detection head includes four positioning parameters and at least one category probability channel. When the number of categories is not explicitly configured, the number of categories is inferred from the output tensor dimension.

[0031] Furthermore, in step S5, the probability calculation is directly applied when the original value is in the range [0,1], otherwise the Sigmoid function is applied.

[0032] Furthermore, in step S5, the fraction sorting uses device-side radix sorting or equivalent sorting to sort the fraction and index pairs in descending order, and fills the tail with negative infinity to ensure capacity alignment for any excess count.

[0033] Furthermore, in step S5, the non-maximum suppression is a class-independent suppression, which is performed only on the candidate boxes sorted by intersection-union alignment.

[0034] Further, in step S5, the visualization overlay on the Y plane of the NV12 frame is written in vectorized form with 16-byte alignment, and the U / V components are written in pairs with even-numbered columns aligned in the UV plane.

[0035] Furthermore, the event records adopt an encoding method of "business unique identifier | 20-bit zero-padding microsecond timestamp | 4-bit zero-padding index" to ensure that they are arranged in lexicographical order, i.e., in chronological order.

[0036] Furthermore, in step S6, the captured files are organized according to the format "image root directory / business unique identifier / date (YYYYMMDD) / event_timestamp_index". The system supports minimum interval throttling and periodic cleanup strategies to reduce storage redundancy.

[0037] Furthermore, in step S6, the medium video encoder automatically reconstructs the signal when the input resolution changes. The encoder parameters include at least the service unique identifier, output address, device number, low latency switch, target bit rate, frame rate, and encoding format.

[0038] Furthermore, in step S7, the session cost estimation is scaled by resolution bucket and frame rate to obtain the basic overhead, and the video decoding / encoding cost is superimposed on the first model. When there are multiple models, the decoding / encoding cost is not repeatedly included in the subsequent models.

[0039] Furthermore, in step S7, where the aggregated occupancy is aligned with runtime statistics using a monotonically convergent relaxation function: let the predicted value be... Actual measurement is Minimum margin Relaxation factor Then the updated predicted value Or its equivalent piecewise linear / convex scaling form, so that the predicted value gradually approaches the measured value while retaining a margin.

[0040] Further, in step S7, the device selects the maximum value after normalizing the graphics processing unit utilization rate, video memory usage percentage, video decoding and video encoding utilization rates respectively as the score, and selects the device with the smallest score without exceeding their respective thresholds.

[0041] Furthermore, each threshold and scoring strategy can be set via environment variables or configuration parameters, and will fall back to safe default values ​​in case of default or outlier values.

[0042] A multi-model video stream fusion inference system includes: a stream acquisition and decoding module, a fusion preprocessing module, a pipeline scheduling module, an encryption engine loading and inference module, a device-side post-processing and visualization module, an encoding and transmission module, an event persistence and capture module, and a resource scheduling module.

[0043] Each module works collaboratively in the memory of the graphics processing unit device. The inference engine loading module has functions such as engine file recognition, key derivation, authentication and decryption, and context management.

[0044] The resource scheduling module registers, aligns, and optimizes the allocation of aggregated device occupancy under threshold constraints during session creation and release.

[0045] Furthermore, the pipeline scheduling module provides a graphical processing unit stream and event registry based on (business unique identifier, stage label), supports setting default priorities by stage label and reusing event objects for cross-stage synchronization.

[0046] Furthermore, the encryption engine loading and inference module has the ability to link engine file header recognition, key derivation, authentication decryption and engine deserialization, and maintains the execution context and output buffer according to the unique business identifier.

[0047] Furthermore, the post-processing module performs score sorting and category-independent non-maximum suppression on the device side and writes the visualization overlay directly into the Y / UV plane of the NV12 frame.

[0048] Furthermore, the event persistence and capture module stores event metadata in a key-value database. The key encoding and file path organization satisfy time ordering and aggregation by business, and realize capture throttling with unique identifiers for the same business and periodic cleanup by retention days.

[0049] Furthermore, it also includes an interface and authentication module, which provides remote procedure calls or equivalent service interfaces to the outside world, and authenticates requests based on the configured username / password or token. If authentication fails, it returns a failure status and does not execute the target operation.

[0050] Furthermore, it also includes a configuration parsing module, which parses and defaults the configuration file, including at least: case-insensitive image format and unifying "jpeg" to "jpg"; reverting to the same level directory as the image directory when the database path is default; correcting invalid values ​​for query time window, retention days, number of capture threads, and minimum capture interval; and ensuring compatibility with historical field names when standard fields are missing.

[0051] Furthermore, the model engine registration submodule shares the inference engine instance by (device number, model name); when the same session contains multiple model slots, it maintains its preprocessing, postprocessing and result transmission buffers respectively, and writes them to the event log with model tags.

[0052] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the method described above.

[0053] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0054] 1. Preprocessing single-core fusion and device-side post-processing significantly reduce core launches and memory accesses, resulting in a decrease in end-to-end latency.

[0055] 2. Zero-copy visualization allows pushback / recording and structured detection to share the same NV12 frame without increasing transmission overhead.

[0056] 3. Engine files are bound to licenses, and FP16-only is enforced during loading to ensure performance and consistency and reduce the risk of leakage.

[0057] 4. Resource scheduling selects the best-scoring device within the threshold and aligns the aggregated prediction with the actual measurement to reduce jitter and "overload-backoff" oscillations.

[0058] 5. The event capture button design ensures lexicographical order, i.e., time order, which facilitates prefix scanning; the throttling strategy reduces invalid disk writes.

[0059] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0060] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0061] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a flowchart of a multi-model video stream fusion inference method based on a graphics processing unit disclosed in an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of the overall system architecture and data flow disclosed in an embodiment of the present invention (decoding → fusion preprocessing → inference → device-side post-processing → overlay → encoding and backhaul → event persistence);

[0064] Figure 3 This is a schematic diagram of the fusion preprocessing kernel disclosed in an embodiment of the present invention (NV12 / P010 input, mailbox-style / normalized, low bit-width linear tensor output <e.g., half-precision NCHW>, nearest neighbor / bilinear / texture path).

[0065] Figure 4 This is a flowchart of the device-side post-processing process disclosed in an embodiment of the present invention (detection head decoding → sorting → category-independent NMS → compact output → NV12 overlay).

[0066] Figure 5 This is a schematic diagram of CUDA stream and event scheduling disclosed in an embodiment of the present invention (stream / event registry based on GUID + stage tag).

[0067] Figure 6 This is a schematic diagram of GPU resource scheduling scoring and relaxation alignment disclosed in an embodiment of the present invention;

[0068] Figure 7 This is the first part of the pseudocode in Embodiment 3 of the present invention;

[0069] Figure 8 This is the second part of the pseudocode in Embodiment 3 of the present invention. Detailed Implementation

[0070] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0071] The technical problems to be solved by this invention include:

[0072] 1. Reduce the number of kernel scheduling and memory accesses in the inference link, and shorten end-to-end latency.

[0073] 2. Enable device-side decoding and sorting of the original output of the detection head, and direct overlay of NMS and NV12 regardless of category, reducing D2H.

[0074] 3. Secure engine deployment by binding authentication encryption and authorization, and enforce consistency in FP16+NCHW operation.

[0075] 4. Stable multi-GPU resource scheduling is achieved by using session cost estimation, measured relaxation alignment, and threshold scoring.

[0076] 5. Provides persistent event capture capabilities with ordered GUIDs, controllable throttling, and cleanup capabilities.

[0077] This invention proposes the following: On the GPU, a single fusion core converts NV12 / P010 frames into low-bit-width linear tensor layout model inputs (e.g., half-precision, linear layout such as NCHW), and performs mailbox filling and normalization; On the device side, the raw output of the detection head is decoded, scores are sorted, and class-independent NMS is performed, and the detection bounding box is vectorized and drawn on the NV12 plane; The engine file uses authentication encryption based on key derivation functions, is bound to the authorization sequence, and enforces consistency (low-bit-width + linear layout) verification during loading; CUDA streams and events are assigned according to the decoding / preprocessing / inference / postprocessing stages; Resource scheduling uses multi-threshold normalized scores and adopts a "relaxation" strategy to align aggregated predictions with actual measurements; Event capture uses "GUID|20-bit microsecond|4-bit index" as the key, organizes images by date directory, and supports minimum interval throttle and periodic cleanup.

[0078] The specific embodiments of the present invention are as follows:

[0079] Example 1: End-to-end process (S1~S7), as follows Figure 1 As shown:

[0080] S1: Receives the video stream and decodes it into NV12 or P010 frames on the GPU.

[0081] S2: On the GPU, using a single fusion core, sequentially perform color space conversion, size scaling while maintaining aspect ratio (mailbox filling), channel order adjustment, and normalization on the frame to generate input for a low-bit-width linear tensor layout (e.g., half-precision, linear layout such as NCHW), such as... Figure 2 As shown.

[0082] The fusion kernel in step S2 supports at least one of nearest neighbor interpolation and bilinear interpolation, and may optionally use bilinear sampling based on texture objects; the mailbox-style fill value is derived from the fill RGB value and mean / variance before normalization.

[0083] To ensure consistency between geometry and color:

[0084] Mailbox-style scaling Content area offset ;

[0085] target pixels The source coordinate inverse mapping is , The sampling method can be nearest neighbor, bilinear, or texture bilinear.

[0086] Code value normalization and color transformation according to , Then proceed according to the channel. Standardized output;

[0087] In the P010 scenario, the 10-bit code value is first normalized in terms of bit width before the above transformation is applied.

[0088] Parameter Definition Reference Table 1

[0089]

[0090] S3: Assigns CUDA streams to each stage of decoding, preprocessing, inference, and postprocessing, and enables asynchronous connection and overlapping execution of cross-stage dependencies through events; manages streams and events by (business unique identifier GUID, stage label), and supports setting default priorities by label.

[0091] S4: Perform encryption header checks and key derivation decryption based on the authorized sequence on the deep learning inference engine file; load the inference engine and create an inference context and output buffer with GUID as the key; verify that the engine I / O meets the constraints of FP16+ linear NCHW; set the device-side input / output addresses and enqueue them for execution, such as... Figure 4 As shown.

[0092] S5: On the device side, the raw output of the detection head is subjected to frame decoding, score sorting, category-independent non-maximum suppression, and result compaction to obtain a stable output on the device side; and the detection frame can be drawn in a vectorized writing manner on the NV12 frame plane, such as... Figure 3 As shown.

[0093] Center coding , restore to ;

[0094] If the output is normalized coordinates, then first press Inverse normalization, then use the inverse mapping from step S2 back to .

[0095] The category probability is not in the original value Apply Sigmoid at time: IoU is defined as Category-independent NMS with threshold Suppress any constraint that satisfies the conditions of the reserved bounding box. Candidates.

[0096] The sorting phase Sort in descending order; if the capacity is insufficient, add fractions to the end of the list. Fill in the gaps to allow for parallel processing of alignment.

[0097] Parameter Definition Reference Table 2

[0098]

[0099] S6: Encode (overlay-enabled) frames back to the transmission channel on the GPU; simultaneously filter out detection results that meet the threshold using a device-to-host minimum copy method, generate event records with GUID|timestamp|index as keys, persist them to a key-value database, and perform snapshot throttling and periodic cleanup for the same GUID within the minimum time interval, wherein the event key is encoded as follows: ,like Figure 4 , 5 As shown.

[0100] S7: Estimates the GPU / NVDEC / NVENC / memory cost of a session based on video stream resolution / frame rate and model input size. Combined with real-time device statistics and preset thresholds, a relaxation strategy aligns the aggregated prediction with actual measurements. Under multiple threshold constraints, it optimally allocates target GPUs and registers / releases session occupancy. Figure 5 , 6 As shown.

[0101] Frame rate scaling factor It affects the basic cost;

[0102] Aggregate prediction Compared with actual measurement Alignment is achieved using a relaxation function. (when Time to take );

[0103] Equipment selected as Under the premise that none of the items exceed the threshold, the device with the lowest score is selected.

[0104] Parameter Definition Reference Table 3

[0105]

[0106] Example 2: Optional Parameters and Boundaries

[0107] 1. Normalized mean / variance, fill color, confidence threshold, NMS IoU, visual line width, encoding bitrate and frame rate are configurable, and default values ​​are rolled back when values ​​are missing or invalid;

[0108] 2. The number of channels output by the detection head includes 4 positioning parameters and at least one category channel. When the number of categories is not explicitly configured, it is inferred from the output tensor dimension.

[0109] 3. Event capture uses the path "image root directory / business unique identifier / date (YYYYMMDD) / event_timestamp_index.format", and the key uses "GUID|20-bit zero-padding microsecond|4-bit zero-padding index".

[0110] Example 3: System Partitioning

[0111] The system includes: stream acquisition and decoding module, fusion preprocessing module, pipeline scheduling module, encryption engine loading and inference module, device-side post-processing and visualization module, encoding and transmission module, event persistence and capture module, and resource scheduling module;

[0112] Each module works collaboratively within the GPU device's memory. Authorization and encryption are bound together during the loading phase: a 256-bit key is derived from the authorization sequence and salt using HKDF-SHA256, and the inference model is authenticated and encrypted using AES-256-GCM. If the signature verification fails, execution is rejected. Low-width and linear layout constraints are enforced during context creation.

[0113] The encryption engine loading and inference module has functions such as engine file recognition, key derivation, authentication and decryption, and context management; the resource scheduling module registers and aligns the aggregated occupation of devices during session creation and release, and performs optimal allocation under multiple threshold constraints.

[0114] Furthermore, to facilitate understanding of the system's structure and the data flow relationships between modules, each functional module is described below:

[0115] 1. Stream acquisition and decoding module

[0116] The module receives compressed video frames from video input sources (including camera streams, network streams, or file streams) and generates NV12 or P010 format image frames on the graphics processing unit via a hardware decoding unit. The output frames are stored in the device's video memory and include a timestamp and a unique service identifier (GUID) for direct access by the fusion preprocessing module.

[0117] 2. Fusion Preprocessing Module

[0118] This module performs color conversion, scaling, filling, and normalization operations on the GPU using a single fusion core, generating model input data with a low-bit-width linear tensor layout. The output tensor is queued by the pipeline scheduling module to the encryption engine loading and inference module.

[0119] 3. Assembly line scheduling module

[0120] The module is used to establish a stream and event registry based on the unique business identifier and stage label, manage the asynchronous execution and dependency synchronization of each stage, realize the parallel connection of decoding, preprocessing, inference, postprocessing and encoding, and ensure the orderly and efficient flow of data.

[0121] 4. Encryption Engine Loading and Inference Module

[0122] The module is used to receive model input tensors from the fusion preprocessing module, complete the decryption, loading and context creation of the inference engine file, execute inference tasks on the asynchronous stream, and output inference result tensors to the device-side post-processing and visualization module.

[0123] In this system, the "Inference Engine Loading and Inference Module" is mainly responsible for loading and executing the inference engine, while the "Inference Model" is one of the objects and input parameters for its operation.

[0124] The engine's responsibility is to parse the model structure, schedule GPU operators, and manage input and output memory; the model, on the other hand, defines the specific network structure and weights.

[0125] This layered design allows multiple models to share the same inference engine instance, thereby reducing redundant loading overhead and improving GPU utilization.

[0126] 5. Device-side post-processing and visualization module

[0127] The module is used to perform box decoding, sorting and nonmaximum suppression on the inference results at the graphics processing unit device (GPU device) to obtain the final detection results, and directly draw the detection boxes and labels on the video frames to achieve zero-copy visualization output.

[0128] 6. Encoding and Feedback Module

[0129] The module is used to encode and push back the post-processed video frames on the GPU, and at the same time pass the detection result metadata to the event persistence and capture module to realize result recording and business backtracking.

[0130] 7. Event Persistence and Snapshot Module

[0131] The module is used to receive and persistently store detection events, organize captured files according to business unique identifiers and timestamps, support throttling and cleanup strategies, and provide operational statistics to the resource scheduling module.

[0132] 8. Resource Scheduling Module

[0133] The module aggregates the usage of GPU, video encoding / decoding, and video memory, and performs optimal device allocation and session management based on a prediction and measurement alignment strategy to ensure the stability and efficiency of system operation.

[0134] Based on the above modules, the data flow relationship during system operation is as follows:

[0135] The stream acquisition and decoding module outputs decoded frames to the fusion preprocessing module; the fusion preprocessing module generates standardized tensor inputs and sends them to the encryption engine loading and inference module; the inference module outputs result tensors for processing by the device-side post-processing and visualization module; the post-processing module generates overlay frames and detection indexes, which are then encoded and pushed by the encoding and feedback module; the encoding module outputs video streams and event data to the event persistence and capture module; the event persistence and capture module completes recording and throttling cleanup, and feeds back statistical information to the resource scheduling module; the resource scheduling module, in conjunction with the pipeline scheduling module, manages GPU resource allocation in a unified manner, achieving end-to-end asynchronous collaboration and closed-loop scheduling.

[0136] Key process pseudocode (exemplary, not a limitation on the technical solution)

[0137] Figure 7 and 8 The system's pseudocode is shown. The pseudocode is used to help understand the core processing and timing. The variable names and structures are abstract representations and are not limited to specific implementation languages ​​or libraries.

[0138] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0139] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0140] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0141] The steps of the methods or algorithms described in conjunction with the embodiments herein can be directly embodied in hardware, software modules executed by a processor, or a combination thereof. The software modules can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor and storage medium can exist as discrete components in the user terminal.

[0142] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. This software code can be stored in memory units and executed by a processor. The memory units can be implemented within the processor or outside the processor; in the latter case, they are communicatively coupled to the processor via various means, as is well known in the art.

[0143] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that the various embodiments can be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

Claims

1. A multi-model video stream fusion inference method based on a graphics processing unit, characterized in that, Includes the following steps: S1 receives the video stream and decodes it into image frames with color format NV12 or P010 on the graphics processing unit; S2, on the graphics processing unit, a single fusion kernel sequentially performs color space conversion, size scaling and aspect ratio-preserving mailbox filling, channel order adjustment and normalization on the image frame to generate a low bit-width linear tensor layout model input, wherein the fusion kernel is a CUDA kernel; S3 assigns computational flows to graphics processing units for each stage of decoding, preprocessing, inference, and postprocessing, and uses events to achieve asynchronous connection and overlapping execution across stages. S4 performs encryption header determination and key derivation decryption based on authorized sequence on the deep learning inference engine file, loads the inference engine and creates inference context and output buffer with business unique identifier as key, verifies that the engine input / output meets the constraints of half precision and linear layout, sets the device input / output address and enqueues the inference asynchronously. S5 performs frame decoding, probability calculation, score sorting, category-independent nonmaximum suppression, and result compaction on the raw output of the detection head on the graphics processing unit to obtain a stable output on the device side, and draws a visual overlay on the NV12 frame plane without the need for host copying. S6 encodes the processed frame on the graphics processing unit and pushes it back to the transmission channel. At the same time, it filters out the detection results that meet the threshold using the minimum copy method from device to host, generates an event record with a combination of business unique identifier, timestamp and index, persists it to the key value database, and performs capture throttling on the same business unique identifier within the minimum time interval. S7 estimates the cost of the session's graphics processing unit / video decoding / video encoding / video memory based on the video stream resolution, frame rate, and model input size. Combining real-time device operation statistics and preset thresholds, it aligns the aggregated prediction with the actual measurement through a relaxation strategy. Under multiple threshold constraints, it selects and allocates the target graphics processing unit and registers or releases the session's occupancy.

2. The method as described in claim 1, characterized in that, in, The fusion kernel in step S2 supports at least one of nearest neighbor interpolation and bilinear interpolation, and may optionally use bilinear sampling based on texture objects; the mailbox-style fill value is derived from the fill RGB value and mean / variance before normalization, wherein the 8-bit and 10-bit code values ​​of NV12 and P010 inputs are normalized and YUV to RGB matrix transformations are performed respectively, and the UV subsamples are read and interpolated according to a 2×2 grid alignment.

3. The method as described in claim 1, characterized in that, The encryption and decryption of the inference engine file adopts a symmetric key obtained from the authorization sequence and salt based on the key derivation function, and uses a certified symmetric encryption algorithm for encryption and decryption. The file header contains at least the magic number, version, salt, random number, authentication tag and ciphertext payload length fields. The authorization sequence is obtained by the authorization module and cached after the first acquisition. If the authorized sequence is empty, decryption is skipped and an alarm is logged.

4. The method as described in claim 1, characterized in that, The aggregation occupancy and runtime statistics are aligned using a monotonically convergent relaxation function: Let the predicted value be... Actual measurement is Minimum margin Relaxation factor Then the updated predicted value Or its equivalent piecewise linear / convex scaling form, so that the predicted value gradually approaches the measured value while retaining a margin.

5. A multi-model video stream fusion inference system implementing the method as described in any one of claims 1 to 4, characterized in that, include: The module includes: stream acquisition and decoding module, fusion preprocessing module, pipeline scheduling module, encryption engine loading and inference module, device-side post-processing and visualization module, encoding and transmission module, event persistence and capture module, and resource scheduling module. Each module works collaboratively in the memory of the graphics processing unit device. The inference engine loading module is used for engine file recognition, key derivation, authentication and decryption, and context management. The resource scheduling module registers, aligns, and performs optimal allocation under threshold constraints for aggregated device occupancy during session creation and release.

6. The system as described in claim 5, characterized in that, The pipeline scheduling module provides a graphical processing unit stream and event registry indexed by business unique identifier and stage label. It supports setting default priority by stage label and reusing event objects for cross-stage synchronization. The encryption engine loading and inference module is used for the linkage of engine file header recognition, key derivation, authentication decryption and engine deserialization, and maintains execution context and output buffer by business unique identifier.

7. The system as described in claim 5, characterized in that, The event persistence and capture module stores event metadata in a key-value database. The key encoding and file path organization meet the requirements of time ordering and aggregation by business, and realize capture throttling with unique identifiers for the same business and periodic cleanup by retention days.

8. The system as described in claim 5, characterized in that, It also includes an interface and authentication module, which provides remote procedure calls or equivalent service interfaces to the outside world, and authenticates requests based on the configured username / password or token. If authentication fails, it returns a failure status and does not execute the target operation. It also includes a configuration parsing module, which parses and defaults the configuration file, including at least: case-insensitive image format and unifying "jpeg" to "jpg"; if the database path is default, it will fall back to the same level directory as the image directory; it corrects invalid values ​​for query time window, retention days, number of capture threads and minimum capture interval; and it is compatible with historical field names when standard fields are missing.

9. The system as described in claim 5, characterized in that, The model engine registration submodule shares inference engine instances using device number and model name as indexes; when the same session contains multiple model slots, it maintains their preprocessing, postprocessing and result transmission buffers respectively, and writes them to the event log with model tags.

Images