A modular, loosely coupled video acquisition method and device

Abstract

This invention discloses a modular, loosely coupled video acquisition method and apparatus, relating to the field of video processing technology. The method includes: initializing a video processing system, creating a unified system resource management interface to centrally manage video acquisition, encoding, decoding, OSD control, and NPU push modules; based on modular design principles, encapsulating the video acquisition, encoding, decoding, OSD control, and NPU push functions into independent modules, and achieving data interaction between modules through standardized interfaces. This invention, by modularizing the video processing system and providing a unified standardized interface, significantly reduces the coupling of project code. Developers do not need to concern themselves with the underlying logic when calling the interface, ensuring the independence of module functions. Through the modular interface design, developers can quickly integrate platform functions without needing to deeply understand the entire codebase, thereby improving project development efficiency.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a modular, loosely coupled video acquisition method and apparatus. Background Technology

[0002] In existing video processing technologies, traditional video processing systems suffer from highly coupled functional modules. For example, video acquisition, encoding, and decoding modules are tightly linked, meaning that affecting one module often has a ripple effect. Modifying or upgrading one module often requires corresponding adjustments to the code of several other modules. This significantly increases the complexity of system development and maintenance, severely limits the system's scalability, and makes it difficult to quickly adapt to new hardware interfaces, encoding formats, and other requirements. Furthermore, this highly coupled architecture leads to decreased system stability and difficulties in error troubleshooting and repair, greatly reducing the system's flexibility and reliability in practical applications. Therefore, innovative architectural design is urgently needed to address this core pain point. Summary of the Invention

[0003] The purpose of this invention is to provide a modular, loosely coupled video acquisition method and apparatus to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a video acquisition method based on modular low coupling, comprising the following steps:

[0005] The video processing system is initialized, and a unified system resource management interface is created to achieve centralized management of video acquisition, encoding, decoding, OSD control, and NPU push modules;

[0006] Based on the principle of modular design, the video acquisition, encoding, decoding, OSD control and NPU push functions are encapsulated as independent modules, and data interaction between modules is realized through standardized interfaces;

[0007] Each module is configured using structured parameters, which include at least resolution, bitrate, frame rate, and encoding format parameters, enabling dynamic adjustment of video processing parameters.

[0008] A multi-channel video processing mechanism is established to support the simultaneous processing of multiple video streams, and window binding and layout management are performed on the multiple video streams.

[0009] OSD processing is performed on the video stream to enable dynamic configuration and real-time refreshing of subtitle styles and banner displays;

[0010] A push channel for video data to the NPU is constructed to efficiently transmit the preprocessed video data to the NPU for computation and processing;

[0011] The video processing system releases resources by reclaiming the resources of each module through a unified deinitialization interface.

[0012] Preferably, the standardized interface includes:

[0013] The video port creation / destruction interface is used to manage video capture ports for HDMI, SDI, and MIPI interfaces.

[0014] The encoder creation / configuration interface is used to dynamically create and adjust parameters of encoders for H.264, H.265, and AV1 encoding formats.

[0015] Decoder creation interface, used to implement decoding processing of multi-channel video streams;

[0016] The OSD control interface is used to control subtitle refresh and banner display.

[0017] The NPU data push interface is used to control the transmission of video data to the NPU.

[0018] Preferably, the configuration of each module using structured parameters includes at least resolution, bitrate, frame rate, and encoding format parameters, enabling dynamic adjustment of video processing parameters, including:

[0019] The resolution parameters are encapsulated using a preset data structure, supporting dynamic switching between multiple resolutions;

[0020] Intelligent adjustment of bitrate parameters, automatically increasing or decreasing the target bitrate based on motion detection results;

[0021] The encoding format parameters can be configured to support real-time switching between different encoding formats. Frame-level buffering technology is used during the switching process to ensure a smooth transition of the bitstream.

[0022] Preferably, the establishment of the multi-channel video processing mechanism, which supports simultaneous processing of multiple high-definition video streams and performs window binding and layout management for the multiple video streams, includes:

[0023] A priority-based resource scheduling algorithm is adopted, setting a priority level of 0-15 for each video channel to enable high-priority channels to preempt resources from low-priority channels;

[0024] Establish a load balancing mechanism to monitor the load of each encoder in real time, and trigger a stream migration operation when the load exceeds the threshold.

[0025] It supports picture-in-picture, four-screen, and nine-grid preset layouts as well as custom layouts, and enables multi-window display management through window coordinates, size, and video source binding relationships.

[0026] Preferably, the OSD processing of the video stream to achieve dynamic configuration and real-time refresh of subtitle styles and banner display includes:

[0027] It supports independent control of up to 8 OSD layers, with each layer having different priorities, transparency, and update frequency.

[0028] The subtitle style can be dynamically configured, including adjustments to font size, color, and background transparency;

[0029] Implement the drawing of rectangles, circles, and straight lines for marking regions of interest in video footage;

[0030] An animation callback mechanism is used to achieve a dynamic fade-in / fade-out effect for the subtitles.

[0031] Preferably, the construction of the video data push channel to the NPU, which efficiently transmits the preprocessed video data to the NPU for computation, includes:

[0032] Construct a preprocessing pipeline, including a dynamic combination of image denoising, color space conversion, and size normalization processing nodes;

[0033] Supports ROI extraction based on coordinate regions or AI detection, and only pushes data of the region of interest to the NPU;

[0034] Establish an asynchronous inference callback mechanism to receive NPU inference results and process target coordinates, category, and confidence information;

[0035] The inference results of consecutive frames are cached and merged, and the Kalman filter algorithm is used to optimize the target trajectory.

[0036] Preferably, the initialization of the video processing system includes:

[0037] The system resource allocation is completed by calling a unified initialization interface;

[0038] A three-level error code system is established, including system-level, module-level, and application-level error codes, and error context tracking information is configured for each error code. The error context includes at least the module name, function name, error occurrence time, and key input parameters.

[0039] The present invention also provides a modular, loosely coupled video acquisition device, comprising:

[0040] The initialization module is used to initialize the video processing system and create a unified system resource management interface;

[0041] The modular processing module encapsulates video acquisition, encoding, decoding, OSD control, and NPU push functions into independent modules, and enables data interaction between modules through standardized interfaces;

[0042] The parameter configuration module is used to configure each module using structured parameters, which include at least resolution, bitrate, frame rate and encoding format parameters.

[0043] A multi-channel processing module is used to establish a multi-channel video processing mechanism, support the simultaneous processing of multiple video streams, and perform window binding and layout management on the multiple video streams.

[0044] The OSD processing module is used to perform OSD processing on the video stream, enabling dynamic configuration and real-time refreshing of subtitle styles and banner displays;

[0045] The NPU push module is used to build a push channel for video data to the NPU, and efficiently transmit the preprocessed video data to the NPU for computation and processing.

[0046] The resource release module is used to release resources of the video processing system and reclaim resources of each module through a unified deinitialization interface.

[0047] The present invention also provides an electronic device, which is a physical device, comprising:

[0048] The processor and the memory are communicatively connected.

[0049] The memory is used to store at least one executable instruction executed by the processor, which executes the executable instruction to implement the modular, loosely coupled video acquisition method described above.

[0050] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the modular, loosely coupled video acquisition method described above.

[0051] Compared with existing technologies, this modular, loosely coupled video acquisition method and device have the following advantages:

[0052] By modularizing the video processing system, with each module passing data through callback functions and structures and providing a unified, standardized interface, the coupling of project code is greatly reduced. Developers do not need to concern themselves with the underlying logic when calling the interface, ensuring the independence of module functions. The modular interface design enables developers to quickly integrate platform functions without needing to deeply understand the entire codebase, thus improving project development efficiency. The structured parameter configuration and modular architecture allow different projects to select and call relevant modules as needed, without loading irrelevant code, flexibly adapting to various project requirements. Due to the encapsulated design of each functional module, when the platform's underlying structure is upgraded, the main framework code of the project does not need to be modified; only the interface implementation is updated, achieving excellent platform upgrades and compatibility.

[0053] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0054] Figure 1 This is a main flowchart of a modular, loosely coupled video acquisition method provided in an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the structure of a modular, low-coupling video acquisition device provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The method in this embodiment is executed by a terminal, which can be a mobile phone, tablet computer, PDA, laptop or desktop computer, etc. Of course, it can also be other devices with similar functions, and this embodiment does not limit them.

[0059] Please see Figure 1 This invention provides a modular, loosely coupled video acquisition method, comprising:

[0060] Step 100: Initialize the video processing system and create a unified system resource management interface to achieve centralized management of video acquisition, encoding, decoding, OSD control, and NPU push modules.

[0061] Step 100 includes:

[0062] Step 110: Call the unified initialization interface to complete system resource allocation;

[0063] This interface adopts a modular design, which can dynamically allocate system resources such as CPU resources, memory space, and network bandwidth according to different hardware environments and business needs. At the same time, through the resource pre-allocation strategy, sufficient resources are reserved in advance for high-load operations that may occur during video acquisition, so as to ensure the stable operation of the system under high concurrency and large data volume scenarios. In addition, the initialization interface will verify the validity of the allocated resources. If insufficient resources or allocation failure is detected, a resource request compensation mechanism will be triggered immediately to re-request the required resources from the system, ensuring the integrity and availability of resource allocation.

[0064] Step 120: Establish a three-level error code system, including system-level, module-level, and application-level error codes, and configure error context tracking information for each error code. The error context includes at least the module name, function name, error occurrence time, and key input parameters.

[0065] System-level error codes identify major faults at the entire video capture system level, such as system crashes and hardware failures. These error codes have the highest priority; once triggered, the system immediately enters a security protection mode and sends alerts to maintenance personnel via email, SMS, and other channels. Module-level error codes target various functional modules within the video capture system (such as video capture modules, data encoding modules, and network transmission modules). When an anomaly occurs within a module, the corresponding module-level error code is activated, and the module's operating status and key data at the time of the error are recorded, facilitating rapid identification of the root cause by technical personnel. Application-level error codes focus on user interaction and business logic, such as incorrect user input parameters or insufficient operating permissions. Application-level error codes not only record the error context but also generate corresponding user prompts based on the error type, informing users of the problem in a user-friendly manner and guiding them to perform the correct operation.

[0066] Specifically, in existing technologies, unreasonable allocation of system resources often leads to system instability, and inconsistent error handling mechanisms make it difficult to pinpoint problems. This invention, during the system initialization phase, completes system resource allocation by calling a unified initialization interface, `vvplatform_mpp_init`, ensuring that modules such as video acquisition, encoding, decoding, OSD control, and NPU push receive adequate resources during initialization. Simultaneously, a three-level error code system is established, encompassing system-level, module-level, and application-level errors. Each error code is configured with error context tracking information, which at least includes the module name, function name, error occurrence time, and key input parameters. For example, when video port creation fails, the system can quickly locate the failure during the execution of the `vvcpt_video_create_port` function due to incorrect interface parameters, effectively solving the problem of difficult error troubleshooting in existing technologies.

[0067] Step 200: Based on the modular design principle, the video acquisition, encoding, decoding, OSD control and NPU push functions are encapsulated as independent modules, and data interaction between modules is realized through standardized interfaces.

[0068] The standardized interface includes:

[0069] The video port creation / destruction interface is used to manage video capture ports for HDMI, SDI, and MIPI interfaces.

[0070] The encoder creation / configuration interface is used to dynamically create and adjust parameters of encoders for H.264, H.265, and AV1 encoding formats.

[0071] Decoder creation interface, used to implement decoding processing of multi-channel video streams;

[0072] The OSD control interface is used to control subtitle refresh and banner display.

[0073] The NPU data push interface is used to control the transmission of video data to the NPU.

[0074] Specifically, in traditional video processing systems, strong inter-module dependencies and non-standardized data interaction interfaces lead to poor system scalability. This invention, based on modular design principles, encapsulates video acquisition, encoding, decoding, OSD control, and NPU push functions into independent modules, enabling data interaction between modules through standardized interfaces.

[0075] Standardized interfaces include several types: video port creation / destruction interfaces, such as vvcpt_video_create_port, which manages video capture ports for HDMI, SDI, MIPI, and other interfaces, solving the problem of fixed video input interfaces in existing technologies and supporting automatic identification and creation of corresponding ports when devices are connected; encoder creation / configuration interfaces, such as vvcpt_video_create_enc and vvcpt_video_config_enc, which enable dynamic creation and parameter adjustment of encoders for H.264, H.265, AV1, and other encoding formats, overcoming the shortcomings of traditional hard-coded encoder parameters. In practical applications, the encoder's encoding format and bitrate parameters can be switched in real time according to network bandwidth and storage requirements; and decoder creation interfaces, such as vvpl. `ayer_video_create_vdec` enables decoding of multi-channel video streams, handling up to 52 high-definition video streams simultaneously compared to existing technologies, meeting the needs of multi-channel video processing. OSD control interfaces, such as `vvplayer_video_osd_refresh` and `vvcpt_video_start_video_banner`, control subtitle refresh and banner display, supporting dynamic adjustments to subtitle styles and banner content. NPU data push interfaces `video_unbind_to_npu_enable` and `video_unbind_to_npu_disable` enable efficient control of video data transmission to the NPU, solving the problem of inefficient video data push to the NPU.

[0076] Step 300: Configure each module using structured parameters, which include at least resolution, bitrate, frame rate, and encoding format parameters, to achieve dynamic adjustment of video processing parameters.

[0077] Step 300 includes:

[0078] Step 310: The resolution parameters are encapsulated using a preset data structure to support dynamic switching between resolutions such as 4K, 1080P, and 720P.

[0079] The system encapsulates resolution parameters using a preset data structure, supporting dynamic switching between mainstream resolutions such as 4K (3840×2160), 1080P (1920×1080), and 720P (1280×720). In practical applications, the system can intelligently match the optimal resolution mode based on device performance, network bandwidth, and user settings. For example, when network bandwidth is insufficient, it automatically switches from 4K resolution to 1080P to ensure smooth video transmission; when the device enters high-definition recording mode, it seamlessly switches to 4K resolution to achieve high-quality image capture.

[0080] Step 320: Intelligently adjust the bitrate parameter, automatically increasing or decreasing the target bitrate based on the motion detection results;

[0081] When high dynamic scenes (such as sports events and action movie clips) are detected, the system will automatically increase the target bitrate to retain more image details and avoid motion blur; while in low dynamic scenes (such as static monitoring images and conference presentations), the bitrate will be reduced to reduce data redundancy and save storage and transmission resources. Through this mechanism, bandwidth usage can be effectively reduced by 30% to 50% while ensuring video quality.

[0082] Step 330: Configure the encoding format parameters to support real-time switching of different encoding formats. Frame-level buffering technology is used during the switching process to ensure smooth transition of the bitstream.

[0083] During the switching process, frame-level buffering technology is introduced. By caching and reassembling video frames, the bitstream is ensured to transition smoothly, avoiding stuttering, screen tearing, and other phenomena. Specifically, before switching the encoding format, the system will pre-cachise a certain number of video frames. After the new encoding format is initialized, the cached frames will be re-encoded and output according to the new format, thereby achieving a seamless encoding format switching experience for the user.

[0084] Specifically, in existing technologies, video processing parameters are difficult to adjust dynamically, resulting in insufficient system flexibility. This invention uses structured parameters to configure each module, including at least resolution, bitrate, frame rate, and encoding format parameters.

[0085] Regarding resolution parameter configuration, a preset data structure is used for encapsulation, supporting dynamic switching between resolutions such as 4K, 1080P, and 720P. For example, when the input video stream resolution is 2560×1440, which conflicts with the preset value, the system automatically activates the scaling module. Based on the bicubic interpolation algorithm, it dynamically adjusts the output resolution through the vv_scale_set_param interface to achieve adaptive resolution processing. For bitrate parameters, the system can automatically increase or decrease the target bitrate based on motion detection results. When the number of moving objects detected in the video increases and the motion detection threshold exceeds 70%, the main encoder bitrate is automatically increased by 20%, effectively improving video transmission efficiency and avoiding bandwidth waste. In terms of encoding format parameter configuration, real-time switching between different encoding formats is supported. Frame-level buffering technology is used during switching to ensure smooth bitstream transitions and prevent video stuttering.

[0086] Step 400: Establish a multi-channel video processing mechanism to support the simultaneous processing of multiple video streams, and perform window binding and layout management on the multiple video streams.

[0087] Step 400 includes:

[0088] Step 410: A priority-based resource scheduling algorithm is adopted to set a priority level of 0-15 for each video channel, so as to enable high-priority channels to preempt resources of low-priority channels.

[0089] Among them, level 0 is the highest priority and is suitable for critical business scenarios such as security monitoring and live streaming; level 15 is the lowest priority and is suitable for non-real-time business such as background log recording. The algorithm supports a priority preemption mechanism. When a high-priority channel requests resources, the system can interrupt the non-critical tasks of the low-priority channel to release computing, bandwidth and other resources, ensuring the smooth transmission of critical video streams. At the same time, the algorithm has a built-in priority inheritance mechanism to avoid priority inversion problems and ensure system stability.

[0090] Step 420: Establish a load balancing mechanism, monitor the load of each encoder in real time, and trigger a stream migration operation when the load exceeds the threshold.

[0091] A distributed load monitoring network is constructed, which collects core performance indicators such as CPU utilization, memory usage, and I / O throughput of each encoder in real time through a combination of periodic polling and event triggering. When the load of any encoder exceeds a preset threshold (such as 80% full load), the system automatically triggers an intelligent stream migration algorithm. Based on historical load data and current resource utilization, the optimal migration path is calculated through genetic algorithm or simulated annealing algorithm, and some video streams are dynamically migrated to encoder nodes with lower load. At the same time, cache preloading technology is used to reduce screen stuttering during the migration process, ensuring the continuity and stability of video processing.

[0092] Step 430 supports the configuration of preset layouts such as picture-in-picture, four-screen, and nine-grid layouts as well as custom layouts, and realizes multi-window display management through window coordinates, size and video source binding relationship;

[0093] It provides a visual configuration interface, supporting 12 preset layout modes such as picture-in-picture, four-screen split, and nine-grid layout. It also allows users to customize the layout via JSON or visual drag-and-drop. In terms of technical implementation, the system adapts the original resolution of the video source to the target window size through a coordinate mapping algorithm and uses double buffering technology to achieve real-time window refresh. For each display window, parameters such as video source binding, scaling mode (proportional scaling / stretch fill), and transparency adjustment can be configured independently. It also supports setting the hierarchical relationship between windows to meet the needs of diverse business scenarios such as conference monitoring and multi-view live streaming.

[0094] Specifically, existing technologies suffer from uneven resource allocation, stuttering in critical video streams, and fixed layouts when processing multiple video streams. This invention establishes a multi-channel video processing mechanism that supports simultaneous processing of multiple video streams and performs window binding and layout management for the video streams.

[0095] A priority-based resource scheduling algorithm is adopted, assigning priority levels 0-15 to each video channel to enable high-priority channels to preempt resources from low-priority channels. In security monitoring scenarios, alarm video streams are set as high priority, allowing them to prioritize encoder resources when system resources are scarce, ensuring that critical video streams do not experience interruptions. Simultaneously, a load balancing mechanism is established to monitor the load of each encoder in real time. When the load of a channel exceeds a threshold (default 80%), the `vv_scheduler_balance` function is automatically triggered, migrating some streams to idle encoders to prevent overall performance degradation due to uneven encoder load. Furthermore, the system supports preset layouts such as picture-in-picture, four-screen, and nine-grid layouts, as well as custom layout configurations. Layout templates are loaded through the `vvplayer_video_load_layout` interface, or the coordinates, size, and video source binding relationships of each window are specified through the `vvplayer_video_add_window` interface, enabling multi-window display management to meet diverse video display needs in different scenarios.

[0096] Step 500: Perform OSD processing on the video stream to achieve dynamic configuration and real-time refresh of subtitle styles and banner display.

[0097] Step 500 includes:

[0098] Step 510 supports independent control of up to 8 OSD layers, with each layer having different priorities, transparency, and update frequency.

[0099] Through a hierarchical management mechanism, key information layers (such as timestamps and device numbers) can be displayed with high priority and while secondary annotation layers can have their transparency dynamically adjusted as needed. At the same time, users can customize the refresh cycle of each layer according to their content needs to avoid excessive resource consumption.

[0100] Step 520: Dynamically configure the subtitle style, including adjusting the font size, color, and background transparency;

[0101] The system has multiple built-in font libraries, supports stepless scaling from 8pt to 72pt, uses RGB / HEX dual-mode input for color parameters, and provides a visual color wheel for adjustment; background transparency can be previewed in real time via a slider from 0% to 100%, meeting the subtitle display needs of various scenarios such as news live broadcasts and education and training.

[0102] Step 530: Draw shapes such as rectangles, circles, and lines to mark areas of interest in the video frame;

[0103] It provides a standard geometric template library and supports custom border width, color and fill attributes; during the annotation process, the system automatically records the correspondence between graphic coordinates and video frame timestamps, which is convenient for later retrieval and analysis, and is suitable for marking key information in professional fields such as security monitoring and medical imaging.

[0104] Step 540: Use an animation callback mechanism to achieve dynamic effects such as fade-in and fade-out of subtitles;

[0105] The system implements dynamic subtitle effect management based on the animation callback mechanism, supporting more than ten preset animation effects such as fade-in / fade-out, slide-in, scaling and deformation. Users can customize the animation curve (linear / ease-in / ease-out / bouncing, etc.), duration (0.1~10s), and delay trigger time. The system achieves precise synchronization between the animation and video playback progress through the event listener interface, providing users with an immersive information display experience.

[0106] Specifically, existing OSD functions only support static subtitles and lack dynamic adjustment capabilities, failing to meet diverse display needs. This invention performs OSD processing on the video stream, enabling dynamic configuration and real-time refreshing of subtitle styles and banner displays.

[0107] The system supports independent control of up to eight OSD layers, each with different priorities, transparency, and update frequencies. For example, in a medical surgery livestream scenario, the bottom layer can be set as a semi-transparent black overlay background, the middle layers display text information such as surgical instrument parameters, and the top layer is used to annotate graphic information of key surgical areas. The subtitle style can be dynamically configured, including adjusting font size, color, and background transparency, through the `vvplayer_video_osd_set_style` interface. Simultaneously, the system can draw rectangles, circles, lines, and other shapes to annotate regions of interest in the video frame; parameters such as border color, fill color, and line width can be set through the `vvplayer_video_osd_draw_shape` interface. Furthermore, an animation callback mechanism is used to achieve dynamic effects such as subtitle fade-in and fade-out, enhancing the video viewing experience.

[0108] Step 600: Construct a push channel for video data to the NPU to efficiently transmit the preprocessed video data to the NPU for computation.

[0109] Step 600 includes:

[0110] Step 610: Construct a preprocessing pipeline, including a dynamic combination of image denoising, color space conversion, and size normalization processing nodes;

[0111] The image noise reduction stage supports flexible switching between multiple algorithms such as Gaussian filtering and median filtering to adapt to the video noise characteristics in different scenarios; the color space conversion module can efficiently convert between common color spaces such as RGB and YUV to meet the NPU's requirements for input data format; the size normalization processing node is based on a dynamic scaling algorithm, which can ensure that key image information is not lost and adjust the video frame size to the optimal processing specification of the NPU.

[0112] Step 620 supports ROI extraction based on coordinate regions or AI detection, and only pushes data of the region of interest to the NPU;

[0113] In coordinate region mode, users can precisely delineate regions of interest by customizing geometric shapes such as rectangles and polygons; AI detection mode uses a pre-trained target detection model to automatically identify key targets in video frames and extract corresponding regions. Both modes employ data compression and optimization techniques, pushing only the effective data of the region of interest to the NPU, significantly reducing the amount of data transmitted and improving processing efficiency.

[0114] Step 630: Establish an asynchronous inference callback mechanism to receive NPU inference results and process information such as target coordinates, category, and confidence.

[0115] An asynchronous inference callback mechanism is established to receive NPU inference results and process information such as target coordinates, category, and confidence. During asynchronous inference, the system can perform other tasks while waiting for NPU processing, thereby improving resource utilization. When the NPU returns the inference results, the callback function is automatically triggered to calibrate the target coordinates, perform mapping and transformation of the category labels, and perform threshold filtering of the confidence to ensure the accuracy and usability of the output results.

[0116] Step 640: Cache and merge the inference results of consecutive frames, and use the Kalman filter algorithm to optimize the target trajectory;

[0117] The cache adopts a circular queue structure, which can store a certain number of historical frame inference results. The merging process integrates the detection results of the same target in different frames through timestamp matching and spatial location association. The Kalman filter algorithm, based on the target motion model, predicts and corrects the target's position, velocity and other states, effectively smoothing the target trajectory, reducing the jitter and loss of detection results, and providing a stable data foundation for subsequent target tracking and behavior analysis.

[0118] Specifically, in existing technologies, video data lacks effective preprocessing before being pushed to the NPU, resulting in unstable inference results and impacting the performance of deep learning tasks. This invention constructs a push channel for video data to the NPU, efficiently transmitting preprocessed video data to the NPU for computation.

[0119] Before data is pushed, a preprocessing pipeline is constructed, including a dynamic combination of processing nodes such as image denoising, color space conversion, and size normalization. For example, in object detection scenarios, image denoising is first performed through the `vv_npu_preprocess_noise` interface, then BGR to RGB color space conversion is performed using the `vv_npu_preprocess_csc` interface, and finally the image size is normalized to the NPU model input requirements using the `vv_npu_preprocess_resize` interface. ROI extraction based on coordinate regions or AI detection is supported, pushing only the region of interest data to the NPU to reduce data transmission volume. An asynchronous inference callback mechanism is established, receiving NPU inference results through the `npu_inference_callback` callback function and processing information such as target coordinates, category, and confidence. Inference results from consecutive frames are cached and merged, and a Kalman filter algorithm is used to optimize the target trajectory, solving the problems of large fluctuations in inference results and unstable trajectories.

[0120] Step 700: Release resources of the video processing system by reclaiming resources of each module through a unified deinitialization interface.

[0121] Specifically, after the system finishes running, such as during shutdown or module unloading, a unified deinitialization interface is called to execute the resource release process. This interface releases resources in reverse order of module dependencies. First, the NPU data push channel is shut down to release NPU computing resources. Then, video stream acquisition and processing are stopped to release resources such as encoding / decoding threads and video memory buffers. Finally, the system resource management interface is unregistered to complete the unloading of hardware drivers. Through this orderly resource reclamation mechanism, the complete release of system resources is ensured, avoiding memory leaks and resource occupation issues.

[0122] Understandably, after the system finishes running, this invention reclaims the resources of each module through a unified uninitialization interface vvplatform_mpp_uninit, ensuring the reasonable release of system resources, avoiding resource waste and system instability factors, forming a complete closed loop with system initialization, and ensuring the reliable operation of the video acquisition method.

[0123] In this embodiment, by modularizing the video processing system, each module transmits data through callback functions and structures, and provides a unified and standardized interface. This significantly reduces code coupling, allowing developers to call interfaces without needing to understand the underlying logic, thus ensuring module functional independence. The modular interface design enables developers to quickly integrate platform functions without needing to deeply understand the entire codebase, thereby improving project development efficiency. The structured parameter configuration and modular architecture allow different projects to select and call relevant modules as needed, without loading irrelevant code, flexibly adapting to various project requirements. Due to the encapsulated design of each functional module, when the platform's underlying structure is upgraded, the main framework code of the project does not need to be modified; only the interface implementation is updated, achieving excellent platform upgrades and compatibility.

[0124] Based on the above embodiments, such as Figure 2 As shown, the present invention also provides a modularly loosely coupled video acquisition device to support the modularly loosely coupled video acquisition method of the above embodiments. The modularly loosely coupled video acquisition device includes:

[0125] Initialization module 11 is used to initialize the video processing system and create a unified system resource management interface;

[0126] Modular processing module 12 is used to encapsulate video acquisition, encoding, decoding, OSD control and NPU push functions into independent modules, and realize data interaction between modules through standardized interfaces;

[0127] The parameter configuration module 13 is used to configure each module using structured parameters, which include at least resolution, bitrate, frame rate and encoding format parameters.

[0128] The multi-channel processing module 14 is used to establish a multi-channel video processing mechanism, support the simultaneous processing of multiple high-definition video streams, and perform window binding and layout management on the multiple video streams.

[0129] OSD processing module 15 is used to perform OSD processing on the video stream to achieve dynamic configuration and real-time refresh of subtitle style and banner display;

[0130] NPU push module 16 is used to build a push channel for video data to NPU, and efficiently transmit the preprocessed video data to NPU for computation and processing;

[0131] The resource release module 17 is used to release resources of the video processing system and reclaim resources of each module through a unified deinitialization interface.

[0132] In an optional embodiment, the parameter configuration module 13 is further configured to:

[0133] The resolution parameters are encapsulated using a preset data structure, supporting dynamic switching between resolutions such as 4K, 1080P, and 720P; the bitrate parameters are intelligently adjusted, automatically increasing or decreasing the target bitrate based on motion detection results; the encoding format parameters are configured, supporting real-time switching between different encoding formats, and frame-level buffering technology is used during the switching process to ensure a smooth transition of the bitstream.

[0134] In an optional embodiment, the multi-channel processing module 14 is further configured to:

[0135] A priority-based resource scheduling algorithm is adopted, setting a priority level of 0-15 for each video channel to enable high-priority channels to preempt resources from low-priority channels; a load balancing mechanism is established to monitor the load of each encoder in real time, and trigger a stream migration operation when the load exceeds the threshold; it supports the configuration of preset layouts such as picture-in-picture, four-screen, and nine-grid, as well as custom layouts, and realizes multi-window display management through window coordinates, size, and video source binding relationship.

[0136] In an optional embodiment, the OSD processing module 15 is further configured to:

[0137] It supports independent control of up to 8 OSD layers, with each layer having different priorities, transparency, and update frequency; it dynamically configures subtitle styles, including adjusting font size, color, and background transparency; it enables the drawing of rectangles, circles, lines, and other shapes to mark areas of interest in the video frame; and it uses an animation callback mechanism to achieve dynamic effects such as subtitle fade-in and fade-out.

[0138] In an optional embodiment, the NPU push module 16 is further configured to:

[0139] A preprocessing pipeline is constructed, including a dynamic combination of image denoising, color space conversion, and size normalization processing nodes; it supports ROI extraction based on coordinate regions or AI detection, and only pushes data of the region of interest to the NPU; it establishes an asynchronous inference callback mechanism to receive NPU inference results and process information such as target coordinates, category, and confidence; it caches and merges inference results of consecutive frames and uses the Kalman filter algorithm to optimize the target trajectory.

[0140] In an optional embodiment, the resource release module 17 is further configured to:

[0141] The system resource allocation is completed by calling a unified initialization interface; a three-level error code system is established, including system-level, module-level and application-level error codes, and error context tracking information is configured for each error code. The error context includes at least the module name, function name, error occurrence time and key input parameters.

[0142] In this embodiment, by modularizing the video processing system, each module transmits data through callback functions and structures, and provides a unified and standardized interface. This significantly reduces code coupling, allowing developers to call interfaces without needing to understand the underlying logic, thus ensuring module functional independence. The modular interface design enables developers to quickly integrate platform functions without needing to deeply understand the entire codebase, thereby improving project development efficiency. The structured parameter configuration and modular architecture allow different projects to select and call relevant modules as needed, without loading irrelevant code, flexibly adapting to various project requirements. Due to the encapsulated design of each functional module, when the platform's underlying structure is upgraded, the main framework code of the project does not need to be modified; only the interface implementation is updated, achieving excellent platform upgrades and compatibility.

[0143] Furthermore, the modularly low-coupling video acquisition device can run the above-described modularly low-coupling video acquisition method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0144] Based on the above embodiments, such as Figure 3 As shown, the present invention also provides an electronic device, the electronic device comprising:

[0145] The processor 22 includes at least one processor 22, at least one memory 21, a communication interface 23, and a communication bus 24, wherein the processor 22 is communicatively connected to the memory 21.

[0146] In this embodiment, the memory 21 can be implemented in any suitable manner, for example, the memory 21 can be a read-only memory, a hard disk drive, a solid-state drive, or a USB flash drive, etc.; the memory 21 is used to store at least one executable instruction executed by the processor;

[0147] In this embodiment, the processor 22 can be implemented in any suitable manner. For example, the processor 22 can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc.; the processor is used to execute the executable instructions to implement the modular, loosely coupled video acquisition method as described above.

[0148] Based on the above embodiments, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the modular, loosely coupled video acquisition method described above.

[0149] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, equipment, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or equipment, and may be electrical, mechanical, or other forms.

[0152] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0154] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program instructions, such as USB flash drives, portable hard drives, read-only storage servers, random access storage servers, magnetic disks, or optical disks.

[0155] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0156] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0157] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video acquisition method based on modular low coupling, characterized in that, The method comprises the following steps: initializing a video processing system, creating a unified system resource management interface, and realizing centralized management of video acquisition, encoding, decoding, OSD control, and NPU pushing modules; based on the principle of modular design, encapsulating the video acquisition, encoding, decoding, OSD control, and NPU pushing functions into independent modules, and realizing data interaction between the modules through a standardized interface; using structured parameters to configure each module, wherein the structured parameters at least include resolution, code rate, frame rate, and encoding format parameters, and realizing dynamic adjustment of video processing parameters; establishing a multi-channel video processing mechanism, supporting simultaneous processing of multiple video streams, and performing window binding and layout management on the multiple video streams; performing OSD processing on the video stream, realizing dynamic configuration and real-time refreshing of subtitle style and banner display; constructing a video data pushing channel to the NPU, and efficiently transmitting the preprocessed video data to the NPU for calculation and processing; releasing resources of the video processing system, and recycling resources of each module through a unified reverse initialization interface; the standardized interface includes: a video port creation / destroying interface for realizing management of video acquisition ports of HDMI, SDI, and MIPI interfaces; an encoder creation / configuration interface for realizing dynamic creation and parameter adjustment of encoders of H.264, H.265, and AV1 encoding formats; a decoder creation interface for realizing decoding processing of multi-channel video streams; an OSD control interface for realizing control of subtitle refreshing and banner display; an NPU data pushing interface for realizing transmission control of video data to the NPU; wherein the structured parameters at least include resolution, code rate, frame rate, and encoding format parameters, and realizing dynamic adjustment of video processing parameters, including: using a preset data structure to encapsulate the resolution parameter, supporting dynamic switching of multiple resolutions; intelligently adjusting the code rate parameter, and automatically increasing or decreasing the target code rate based on the motion detection result; configuring the encoding format parameter, supporting real-time switching of different encoding formats, and using frame-level buffering technology to ensure smooth transition of code streams during the switching process; wherein the initialization of the video processing system includes: calling a unified initialization interface to complete system resource allocation; establishing a three-level error code system, including system-level, module-level, and application-level error codes, and configuring error context tracking information for each error code, wherein the error context at least includes module name, function name, error occurrence time, and key input parameters.

2. The method for video acquisition based on modular low coupling according to claim 1, characterized in that, the multi-channel video processing mechanism supports simultaneous processing of multiple video streams, and performs window binding and layout management on the multiple video streams, including: using a priority-based resource scheduling algorithm to set 0-15 priority levels for each video channel, realizing resource preemption of high-priority channels to low-priority channels; establishing a load balancing mechanism, and real-time monitoring of the load of each encoder, and triggering a stream migration operation when the load exceeds a threshold value; Supports configuration of preset layouts of picture-in-picture, four-split screen and nine-grid, and custom layouts, and realizes multi-window display management through window coordinates, size and video source binding relationship.

3. The method for video acquisition based on modular low coupling according to claim 1, characterized in that, The OSD processing of the video stream realizes dynamic configuration and real-time refreshing of subtitle style and banner display, including: Supports independent control of up to 8 OSD layers, with different priority, transparency and update frequency set for each layer; Dynamically configures the subtitle style, including adjustment of font size, color and background transparency; Realizes drawing of rectangular, circular and straight-line graphics for labeling of regions of interest in the video picture; Realizes dynamic effects of subtitle fade-in and fade-out through an animation callback mechanism.

4. The modular low-coupling based video capturing method of claim 1, wherein, The push channel of video data to the NPU efficiently transmits preprocessed video data to the NPU for calculation and processing, including: Building a preprocessing pipeline, including dynamic combination of image noise reduction, color space conversion and size normalization processing nodes; Supporting ROI extraction based on coordinate area or AI detection, and pushing only the data of the region of interest to the NPU; Establishing an asynchronous inference callback mechanism to receive NPU inference results and process target coordinates, categories and confidence information; Caching and merging continuous frame inference results, and optimizing target trajectories using Kalman filtering algorithm.

5. A video acquisition device based on modular low coupling, characterized in that, It includes: An initialization module for initializing the video processing system and creating a unified system resource management interface; A modular processing module for encapsulating video acquisition, encoding, decoding, OSD control and NPU pushing functions into independent modules and realizing data interaction between modules through standardized interfaces; A parameter configuration module for configuring each module using structured parameters, including at least resolution, bitrate, frame rate and encoding format parameters; A multi-channel processing module for establishing a multi-channel video processing mechanism, supporting simultaneous processing of multiple video streams, and performing window binding and layout management on the multiple video streams; An OSD processing module for OSD processing of video streams, realizing dynamic configuration and real-time refreshing of subtitle style and banner display; An NPU pushing module for building a push channel of video data to the NPU, efficiently transmitting preprocessed video data to the NPU for calculation and processing; A resource release module for releasing resources of the video processing system through a unified deinitialization interface to complete recovery of resources of each module; The standardized interface includes: A video port creation / destroy interface for managing video acquisition ports of HDMI, SDI and MIPI interfaces; An encoder creation / configuration interface for dynamically creating and adjusting parameters of encoders of H.264, H.265 and AV1 encoding formats; A decoder creation interface for decoding processing of multi-channel video streams; An OSD control interface for controlling subtitle refreshing and banner display; An NPU data pushing interface for controlling transmission of video data to the NPU; The structured parameters for configuring each module include at least resolution, bitrate, frame rate and encoding format parameters, including: The resolution parameter is encapsulated by using a preset data structure, and dynamic switching of multiple resolutions is supported; The code rate parameter is intelligently adjusted, and the target code rate is automatically increased or decreased based on a motion detection result; The encoding format parameter is configured, real-time switching of different encoding formats is supported, and frame-level buffering technology is used in the switching process to ensure smooth transition of the code stream; The initialization of the video processing system includes: A unified initialization interface is called to complete system resource allocation; A three-level error code system is established, including system-level, module-level and application-level error codes, and error context tracking information is configured for each error code, the error context at least including a module name, a function name, an error occurrence time and key input parameters.

6. An electronic device, comprising: The electronic device includes: A processor and a memory, the memory being in communication connection with the processor; The memory is used to store executable instructions executed by the processor, and the processor is used to execute the executable instructions to implement the video acquisition method based on modularization and low coupling according to any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the video acquisition method based on modularization and low coupling according to any one of claims 1 to 4.

Images