Video stream processing method and device, electronic equipment and storage medium
By using a video pipeline architecture for real-time monitoring and independent process management, the problems of fault identification and resource waste in multi-channel video stream processing are solved, achieving efficient and stable video stream processing.
Patent Information
- Application Number
- CN202511739703.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies cannot effectively identify some fault states in multi-channel video stream processing, resulting in a high misjudgment rate, serious resource waste, and an inability to dynamically adjust recovery plans according to system load, leading to rigid resource allocation and the risk of fault propagation.
The system adopts a video pipeline architecture to monitor the frame rate, latency, and buffer status of the video stream in real time. It calls independent processes to manage abnormal branches and merges them into a single independent process when the number of independent processes reaches a threshold, thereby dynamically optimizing system stability and resource utilization.
It achieves efficient processing of multiple video streams, quickly isolates faulty branches, dynamically adjusts the number of processes, optimizes system stability and resource utilization, prevents resource depletion, and ensures continuous and stable operation of the system when some monitoring equipment fails.
Smart Images
Figure CN121509751A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of computer vision and multimedia processing, and particularly relates to a video stream processing method and device, an electronic device and a storage medium. BACKGROUND
[0002] With the rapid development of artificial intelligence and computer vision technology, multi-path video stream real-time processing systems have been widely applied in intelligent security, intelligent transportation, industrial detection and other fields. Traditional multi-path video stream processing methods usually adopt simple connection state checking or heartbeat mechanism, which can only detect complete disconnection and cannot identify partial failure states such as frame rate reduction and delay increase, resulting in high misjudgment rate. In terms of fault recovery mechanism, the existing technology adopts a simple reconnection mechanism, which gives up the video stream after multiple retry failures. This static fault tolerance strategy cannot dynamically adjust the recovery scheme according to the system load, resulting in serious resource waste. In terms of system resource management, the existing technology has problems such as rigid resource allocation and fault propagation risk, which makes it difficult to efficiently process a large number of video streams while ensuring system stability. SUMMARY
[0003] In view of the above deficiencies of the prior art, the present application aims to provide a solution that can realize dynamic optimization management and ensure the continuous and stable operation of the system when some monitoring devices fail.
[0004] In a first aspect, the present application provides a video stream processing method, comprising: obtaining a plurality of video streams of a monitoring device; inputting the plurality of video streams into a video stream pipeline architecture; in response to an abnormality occurring in a branch of the plurality of video streams, calling an independent process for hosting the abnormal branch; in response to the number of independent processes reaching a threshold, merging each independent process into the video stream pipeline architecture.
[0005] Further, in response to an abnormality occurring in a branch of the plurality of video streams, calling an independent process for hosting the abnormal branch comprises: obtaining video stream information of the plurality of video streams; determining the abnormality of each branch of the plurality of video streams according to the video stream information; calling an independent process for hosting the abnormal branch.
[0006] Further, calling an independent process for hosting the abnormal branch comprises: removing the link of the abnormal branch in the video stream pipeline architecture; calling an independent process to push the stream of the abnormal branch to a target address.
[0007] Further, the target address is an address in the video pipeline architecture.
[0008] Further, in response to the number of independent processes reaching a threshold, merging each independent process into the video pipeline architecture, comprising: counting the number of independent processes according to a preset merging period; when the number of independent processes reaches the threshold, merging each independent process into one merged independent process, and inputting the video pipeline architecture.
[0009] Further, the preset merging period and the threshold are configured before acquiring the multi-channel video stream of the monitoring device.
[0010] Further, before calling an independent process to host the abnormal branch, the abnormal branch is also reconnected.
[0011] In a second aspect, the present application provides a video stream processing device, comprising: a video stream acquisition module configured to acquire a multi-channel video stream of a monitoring device; a video stream transmission module configured to input the multi-channel video stream into a video pipeline architecture; a process hosting module configured to, in response to an abnormal branch in the multi-channel video stream, call an independent process to host the abnormal branch; a process merging module configured to, in response to the number of independent processes reaching a threshold, merge each independent process into the video pipeline architecture.
[0012] In a third aspect, the present application provides an electronic device, comprising: a processor; and a memory storing a program, wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the video stream processing method described above.
[0013] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the video stream processing method described above.
[0014] The beneficial effects of the present application are as follows: The video stream processing method provided by the present application can efficiently process multi-channel video streams in a single process, while quickly isolating and independently processing fault branches. The independent processes and the merged independent processes are dynamically generated according to the system load, realizing the conversion from decentralized processing to centralized processing, dynamically optimizing the system stability, prioritizing the overall service availability when the system resources are scarce, and preventing resource depletion. BRIEF DESCRIPTION OF DRAWINGS
[0015] Figure 1 FIG. 1 is a schematic diagram of the application scenario of the video stream processing in the present application.
[0016] Figure 2 A flow chart of a video stream processing method in the present application.
[0017] Figure 3 A schematic diagram of a video stream processing device in the present application. DETAILED DESCRIPTION
[0018] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are intended to be merely illustrative of the present application and not in limitation thereof. It should also be noted that, for the purpose of description, only the parts related to the present application are shown in the drawings rather than all the parts.
[0019] Gstreamer framework is an open-source cross-platform multimedia framework. The framework uses the concept of pipeline to process audio and video streams, breaks media processing tasks into small pieces, flexibly combines them, and builds a pipeline to process the tasks. Data starts from the source of the pipeline, passes through various processing units, and finally reaches the output end of the pipeline.
[0020] Streammux refers to a Gstreamer plugin, such as nvstreammux, which is used to merge multiple video streams into a batch processing unit.
[0021] Streamdemux refers to a Gstreamer plugin, such as nvstreamdemux, which splits the merged video stream according to the stream ID.
[0022] In the related art, scenes such as security monitoring and industrial monitoring that need to simultaneously access and process multiple video streams, the processing scheme of a multi-camera system generally adopts two architectures: Method A (single process merging processing): The uridecodebin of Gstreamer is used as a video source, the streammux component merges multiple video streams into a batch processing unit, after complex logic, the tiler splices multiple video pictures into one picture, and pushes it to the same address.
[0023] Method B (multi-process independent processing): Each camera independently creates a Gstreamer pipeline, directly calls nvinfer and nvtracker to process data, and then pushes it to the respective target address.
[0024] The existing technology methods A and B have the following defects: Resource waste and performance bottlenecks: In Method A, if the data stream of a certain camera is abnormal (such as interruption or delay) during the merging process, the entire pipeline needs to be paused for repair, which will cause other normal branches to be interrupted. In Method B, because each camera's independent process occupies additional memory and CPU resources, it is difficult to scale to a large-scale camera cluster.
[0025] Dynamic adjustment is difficult: When adding or deleting cameras, method A requires reconfiguring the pipeline structure, which poses a risk of service interruption; although method B can be operated independently, there is no limit to the number of processes, which can easily cause system overload.
[0026] Insufficient fault isolation: Method A cannot isolate faulty branches, and a single point of failure affects the whole system; although Method B isolates faults, it lacks a mechanism for merging and optimizing, resulting in low resource utilization over long-term operation.
[0027] This application provides a specific implementation example of a video stream processing application scenario, such as... Figure 1 As shown, this application scenario includes a monitoring device 101 and a processing terminal 102. The monitoring device 101 captures video of the target site, with each monitoring device generating a video stream. All video streams can be aggregated into multiple video streams, which are then transmitted to the processing terminal 102 for processing. The processing terminal 102 processes these multiple video streams. The monitoring device 101 can be a camera, and the processing terminal 102 can be a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0028] The video stream processing method provided in this application embodiment can be executed by the video stream processing device provided in this application embodiment, or by a computer device integrating the video stream processing method. The video stream processing device can be implemented in hardware or software. The computer device can be... Figure 1 The processing terminal 102 shown is shown.
[0029] The first aspect of this application provides a video stream processing method, the process of which is as follows: Figure 2 As shown, it includes: S201 acquires multiple video streams from the monitoring equipment; S202 is a multi-stream video pipeline architecture; S203 responds to an exception occurring in one branch of a multi-video stream by calling a separate process to manage the exception branch. S204 responds when the number of independent processes reaches a threshold and merges all independent processes into the video pipeline architecture; S201 acquire the multi-path video stream of the monitoring device.
[0030] The monitoring device refers to an electronic device capable of shooting images or recording videos, which can be, for example, the monitoring device 101 shown in the figure. Each monitoring device corresponds to a branch and a path of video stream, and the video streams corresponding to multiple monitoring devices can be aggregated as a multi-path video stream. Figure 1
[0031] The monitoring device is in communication connection with the processing terminal, and the processing terminal processes the video stream of the monitoring device. The monitoring device is an example of a camera, and the monitoring device is in communication connection with the processing terminal through RTSP or USB protocol. The multi-path video stream corresponds to multiple monitoring devices. When acquiring the multi-path video stream of the monitoring device, the present application provides a monitoring device source layer, which is an input layer of the multi-path video stream. The monitoring device is connected to the monitoring device source layer, and the processing terminal is connected to the monitoring device source layer, thereby realizing the communication connection between the monitoring device and the processing terminal.
[0032] The monitoring device source layer can selectively access one or more monitoring devices with RTSP protocol and one or more monitoring devices with USB protocol. The specific monitoring devices to be selectively accessed can be selected according to the monitoring task. The monitoring device source layer can also modify the configuration of the monitoring device to obtain a multi-path video stream that meets the monitoring task.
[0033] S202 input the multi-path video stream into the video stream pipeline architecture.
[0034] Based on the Gstreamer framework, the present application provides a video stream pipeline architecture (Pipeline), which includes: input layer→treammux (merge multi-path stream)→nvinfer (AI inference)→nvtracker (target tracking)→nvosd (screen overlay display)→streamdemux (split stream)→queue (buffer)→nvvideoconvert (post-processing format)→capsfilter (set video format)→push to target address. In the video stream pipeline architecture, streammux and streamdemux are cooperatively controlled to ensure seamless switching of video streams.
[0035] The video stream pipeline architecture is configured in the processing terminal 102, which realizes the functions of multi-stream input, batch processing inference, target tracking, result overlay to the final split stream and push stream. Inputting the multi-path video stream into the pipeline architecture can push the multi-path video stream to the target address, thereby realizing normal playing of the pictures shot by the monitoring device.
[0036] S203 In response to an exception in a branch of the multi-path video stream, an independent process is called for the abnormal branch to be managed.
[0037] The video pipeline architecture includes a streamdemux. In the present application, the frame rate, delay, and buffer status of all branches in the multi-channel video stream can be monitored in real time from the streamdemux through a listening function. According to the frame rate, delay, and buffer status of each branch, an abnormality judgment is made on each branch. When one branch exhibits one or more of “frame rate lower than a preset value”, “delay exceeds a preset value”, and “time for which the buffer size exceeds a threshold value exceeds a preset value”, it is determined that the branch is abnormal, and the abnormality judgment is completed.
[0038] In some embodiments, the system-level heartbeat packet can also be used to make an abnormality judgment on all branches in the multi-channel video stream.
[0039] The present application notes that when the video pipeline architecture processes the video stream of an abnormal branch, the video stream of the abnormal branch shares the inference engine and memory buffer with other video streams, which occupies too many GPU resources, and too many abnormal video streams can cause the overall video pipeline architecture to crash. Therefore, the present application proposes the idea of using an independent process to host the abnormal branch. When an independent process is used to host the abnormal branch, the isolation is good and does not affect other video streams.
[0040] In the present application, when one branch in the multi-channel video stream is abnormal, the video stream of the branch is first attempted to be reconnected, and when multiple connection attempts are unsuccessful, the video pipeline architecture removes the link of the branch and starts an independent process to host the abnormal branch. In the processing terminal 102, the main branch of the video pipeline architecture is processed, and the sub-branch of the independent process is pushed to stream, which reduces the number of model loading and saves server resources.
[0041] The present application provides an independent process suitable for hosting an abnormal branch, which includes: an input end: streammux (set output resolution) → nvinfer (AI inference) → nvtracker (target tracking) → nvosd (on-screen display) → nvvideoconvert (post-processing format) → capsfilter (set video format) → push to target address. The target address of the independent process can be consistent with the target address of the video pipeline architecture, or it can be another target address. By using an independent process to host an abnormal branch, the GPU memory usage is effectively reduced, server resources are saved, system stability and resource utilization are optimized, and the video pipeline architecture is prevented from being interrupted due to resource consumption by an abnormal branch.
[0042] S204 merges each independent process to the video pipeline architecture in response to the number of independent processes reaching a threshold value.
[0043] As the monitoring device continues to work, more and more abnormal branches will appear. When an abnormal branch appears, an independent process is isolated and managed. It is easy to understand that more and more independent processes will appear. According to the merge period, the number of independent processes is counted regularly. When the number of independent processes reaches the threshold, the merge logic is triggered, all created independent processes are terminated, the memory, GPU computing resources and file descriptors occupied by them are released, and all independent processes are merged into a merged independent process. In this process, independent processes are dynamically merged according to the merge period to prevent resource fragmentation.
[0044] Further, the merged independent process is reconnected to the streammux of the video pipeline architecture. The video pipeline architecture processes the merged independent process in the same way as other video streams, reducing process fragmentation. After merging, the video stream corresponding to the abnormal branch in the merged independent process no longer enjoys independent resource allocation, but shares computing resources with other video streams in the video pipeline architecture. This process realizes the conversion from decentralized processing to centralized processing, dynamically optimizes system stability, prioritizes overall service availability when system resources are scarce, and prevents resource depletion.
[0045] In this application, before obtaining the monitored multi-channel video stream, the above threshold is configured through the max_single_streams parameter setting interface, and the above merge period is configured through the merge_interval parameter setting interface.
[0046] The application provides an example of a specific process of a video stream processing method in a system, including an initialization stage, a monitoring and isolation stage, and a merging stage.
[0047] Initialization stage: The user configures max_single_streams=5 and merge_interval=10.
[0048] The monitoring device source layer and the video pipeline architecture are started, and the video streams Camera1~10 of the monitoring device are loaded.
[0049] Monitoring and isolation stage: The system continuously detects the frame rate, delay and buffer state of all branches in the multi-channel video stream through a listening function.
[0050] Camera3 is disconnected, and the inference delay of Camera5 is out of limit.
[0051] Stop the video stream pushing of Camera3 and Camera5 in the video pipeline architecture.
[0052] Start the first independent process to manage Camera3, and start the second independent process to manage Camera5.
[0053] The first independent process and the second independent process are input into the video stream pipeline architecture.
[0054] The merging stage: Camera1, Camera6 and Camera9 also appear abnormal, and the system has the first independent process, the second independent process, a third independent process corresponding to Camera1, a fourth independent process corresponding to Camera6 and a fifth independent process corresponding to Camera9.
[0055] The video stream pipeline architecture is paused, and the first independent process, the second independent process, the third independent process, the fourth independent process and the fifth independent process are merged into a sixth independent process.
[0056] The sixth independent process is input into the video stream pipeline architecture, and the video stream pipeline architecture is started.
[0057] The resources of the first independent process, the second independent process, the third independent process, the fourth independent process and the fifth independent process are released.
[0058] The second aspect of the application provides a video stream processing device, as shown in the figure, the video stream processing device can be integrated in an electronic device, and specifically can include a video stream acquisition module, a video stream transmission module, a process hosting module and a process merging module. Figure 3 The video stream processing device can be integrated in an electronic device, and specifically can include a video stream acquisition module, a video stream transmission module, a process hosting module and a process merging module.
[0059] The video stream acquisition module is used to acquire multiple video streams of a monitoring device; The video stream transmission module is used to input the multiple video streams into a video stream pipeline architecture; The process hosting module is used to, in response to an abnormality of a branch of the multiple video streams, call an independent process to host the abnormal branch; The process merging module is used to, in response to the number of independent processes reaching a threshold, merge the independent processes into the video stream pipeline architecture.
[0060] The video stream processing device provided by the embodiments of the application includes a video stream acquisition module, a video stream transmission module, a process hosting module and a process merging module, and the functions of the modules are one-to-one corresponding to the steps of the video stream processing method. For specific explanations of the video stream processing device, and related details and optimization, refer to the specific embodiments of the video stream processing method described above, which will not be repeated here.
[0061] In some embodiments, the application further provides an electronic device, which can be a mobile phone, a computer or a tablet computer, etc., including a memory and a processor, and the memory stores a computer program, which is executed by the processor to implement the video stream processing method in the above embodiments.
[0062] The processor is configured to perform all or part of the steps of the video stream processing method in the above embodiments. The memory is configured to store various types of data, which can include, for example, instructions of any application program or method in the electronic device, and application-related data.
[0063] The processor can be an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic elements, which is configured to perform the steps of the video stream processing method in the above embodiments.
[0064] The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read Only Memory (EPROM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk.
[0065] In some embodiments, the present application also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, and the like, having stored thereon a computer program, which, when executed by a processor, can implement the above method steps. For details of the embodiment process, reference can be made to the above embodiments, which will not be repeated here.
[0066] It should be understood that, although each step in the flowchart of each embodiment of the present application is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least a part of the steps in each embodiment can include a plurality of sub-steps or a plurality of stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least a part of other steps or sub-steps or stages of other steps.
[0067] The above embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as limiting the scope of the patent of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.
[0068] The above are only preferred embodiments of the present application, and are not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims
1. A video stream processing method, characterized in that, include: Acquire multiple video streams from monitoring equipment; The multi-channel video stream is input into the video pipeline architecture; In response to an anomaly occurring in one of the multiple video streams, a separate process is invoked to manage the abnormal branch. In response to the number of independent processes reaching a threshold, the independent processes are merged into the video pipeline architecture.
2. The method according to claim 1, characterized in that, In response to an anomaly occurring in one branch of the multi-channel video stream, a separate process is invoked to manage the abnormal branch, including: Obtain the video stream information of the multiple video streams; Anomaly determination is performed on each branch of the multi-channel video stream based on the video stream information; An independent process is used to manage the exception branch.
3. The method according to claim 2, characterized in that, To manage exception branches, a separate process is invoked, including: Remove the link to the abnormal branch in the video pipeline architecture; A separate process is invoked to push the stream of the abnormal branch to the target address.
4. The method according to claim 3, characterized in that, The target address is the address in the video pipeline architecture.
5. The method according to claim 1, characterized in that, In response to the number of independent processes reaching a threshold, the independent processes are merged into the video pipeline architecture, including: The number of independent processes is counted according to a preset merging cycle; When the number of independent processes reaches a threshold, the independent processes are merged into a single merged independent process and input into the video pipeline architecture.
6. The method according to claim 5, characterized in that, The preset merging period and the threshold are configured before acquiring multiple video streams from the monitoring device.
7. The method according to claim 1, characterized in that, Before calling a separate process to manage the abnormal branch, a reconnection test is performed on the abnormal branch.
8. A video stream processing apparatus, characterized in that, include: The video stream acquisition module is used to acquire multiple video streams from the monitoring device; A video stream transmission module is used to input the multiple video streams into the video pipeline architecture; The process management module is used to respond to an anomaly in one branch of the multi-channel video stream by calling an independent process to manage the abnormal branch. A process merging module is used to merge the independent processes into the video pipeline architecture in response to the number of independent processes reaching a threshold.
9. An electronic device, characterized in that, include: processor; And a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1-7.