A method for processing online video images

Abstract

This invention discloses an online video image processing method, including a middleware platform and a streaming service system. The streaming service system provides network camera streams, local video data, and / or image data as data sources. A plugin library stores different plugins for different stream formats for pulling and transcoding. A plugin selection and loading module dynamically loads plugins from the plugin library for different stream formats. An encoding / decoding and transcoding module encodes, decodes, and transcodes the original stream data, outputting image data in the required format. A data processing module associates and combines the image data with the image structured information output by an AI model into data blocks for use by the AI ​​algorithm task module. A control module receives commands from the middleware platform and controls other modules to pull, encode, decode, and transcode the stream in a plug-in manner to meet the requirements of the data processing module. This invention can adapt to different data sources and has high compatibility.

Description

[Technical Field]

[0001] This invention relates to video surveillance, and more particularly to a method for processing online video images. [Background Technology]

[0002] This invention relates to video coding technology, video encoding and decoding technology, and hardware decoding technology.

[0003] Video encoding: The main function of video encoding is to compress video pixel data (RGB, YUV, etc.) into a video bitstream, thereby reducing the amount of video data. Without compression, video files are typically very large; a 4K image (3840x2160x3) is 24.3MB in its original size. Uncompressed video requires approximately 6Gbps of bandwidth, and a single ultra-high-definition camera generates up to 63TB of raw video daily. Only with video encoding can it be transmitted and stored. Video encoding is one of the most important technologies in audio and video technology. The video bitstream accounts for the vast majority of the total audio and video data. High-efficiency video encoding can achieve higher video quality at the same bitrate.

[0004] Audio / video decoding: This involves decoding compressed video / audio data into uncompressed raw video / audio data. Audio compression standards include AAC, MP3, AC-3, etc., while video compression standards include H.264, MPEG2, VC-1, etc. Decoding is the most important and complex step in the entire system. Through decoding, compressed video data is output as uncompressed color data, such as YUV420P, RGB, etc.; compressed audio data is output as uncompressed audio sampled data, such as PCM data (Pulse Code Modulation).

[0005] Hardware Decoding Technology: NVIDIA GPUs include one or more hardware-based decoders and encoders (separate from the CUDA cores) that provide fully accelerated hardware-based video decoding and encoding for a variety of popular codecs. With decoding / encoding offloaded, the graphics engine and CPU are freed to perform other operations. The GPU hardware accelerator engine for video decoding (called NVDEC) and video encoding (called NVENC) supports real-time video processing, making it suitable for use in transcoding applications, in addition to faster video playback. NVENC enables: 1. Encoding and streaming games and applications with high quality and ultra-low latency without using the CPU. 2. High-quality encoding for archiving, OTT streaming, and web video. 3. Encoding with ultra-low power consumption (watts / stream).

[0006] Traditional online video image processing methods are limited in their decoding capabilities, lack flexibility, and have poor compatibility, making them unable to process data in uncommon video encoding formats. [Summary of the Invention]

[0007] The technical problem to be solved by the present invention is to provide a method for processing online video images that can adapt to different data sources.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is an online video image processing method, including a data processing system, the data processing system including a middle platform and a streaming service system, the streaming service system including a data source, a plugin library, a plugin selection and loading module, an encoding / decoding and transcoding module and a data processing module;

[0009] Data source: Provides network camera streams, local video data, and / or image data;

[0010] Plugin library: It stores different plugins for different bitstream formats, allowing for retrieval and conversion of the bitstream;

[0011] Plugin selection and loading module: The C++ dynamic library loading method is used to dynamically load plugins from the plugin library for different bitstream formats;

[0012] Encoding / decoding and transcoding module: Encodes, decodes, and transcodes the raw bitstream data, and outputs image data in the required format;

[0013] Data processing module: Combines image data and image structured information output by AI model into data blocks, and provides them to AI algorithm task module through corresponding channels;

[0014] Control module: Maintains communication with the middle platform, receives commands from the middle platform, and controls the selection and loading of the streaming service module and the encoding / decoding and transcoding module to pull the bitstream, encode, decode, and transcode it in a plug-in manner to meet the requirements of the data processing module.

[0015] The video image processing method described above includes a disk dumping / transfer module in the streaming service system. This module saves the image data output by the encoding / decoding and transcoding modules to a local disk or transfers it to an external device.

[0016] The video image processing method described above includes plugins such as FFmpeg plugin, Remote plugin, VLC plugin, and SDK plugin in its plugin library.

[0017] The video image processing method described above includes the following steps:

[0018] 401) The user specifies the data source and triggers the task on the task startup page, selects the corresponding plugin, and loads it into the video streaming service; the loaded plugin retrieves the data from the data source.

[0019] 402) After obtaining the data from the data source, the data is encoded and decoded, and converted into image data in BGR or RGB format;

[0020] 403) After preprocessing the image data, it is passed to the AI ​​algorithm task module for use;

[0021] 404) The control module maintains communication with the middleware service.

[0022] The video image processing method described above, prior to step 404, stores the image data to a local disk and / or transfers it to an external device.

[0023] In the video image processing method described above, in step 404, after AI inference based on the pipeline, the data processing module extracts the target's category data, coordinate data, confidence data, and / or feature values ​​from the image data and associates them with the image data for use by the AI ​​algorithm task module.

[0024] In the video image processing method described above, in step 405, the control module sends heartbeat packets and working status data to the middleware, allowing the middleware to understand the keep-alive status and current working status of the streaming service system. When the user discovers a backlog of data to be processed by the AI ​​algorithm task module, they send a command to the control module to reduce the frame rate. The control module then instructs the encoding / decoding and transcoding modules to reduce the current frame rate, thereby reducing the processing data of the AI ​​algorithm task module and achieving a balance.

[0025] In the video image processing method described above, in step 405, the streaming service system sets a priority for data processing. When a user issues a task, they can select the priority of the task. Tasks with higher priority exclusively occupy the GPU core of the data processing module, while tasks with lower priority share one GPU core of the data processing module. The data processing module processes high-priority data in real time and processes low-priority data when the AI ​​algorithm task module is idle.

[0026] The online video image processing method of this invention can adapt to different data sources and has high compatibility. [Image Description]

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 This is a data processing system block diagram of the online video image processing method according to an embodiment of the present invention. [Detailed Implementation]

[0029] like Figure 1 As shown, the online video image processing method of this embodiment includes a data processing system. The data processing system includes a middleware platform and a streaming service system. The streaming service system includes a data source, a plugin library, a plugin selection and loading module, an encoding / decoding and transcoding module, a disk dumping / transfer module, and a data processing module.

[0030] Data source: Provides network camera streams, local video data, and / or image data.

[0031] Plugin library: It stores different plugins for different bitstream formats, which can be pulled and converted.

[0032] Plugin selection and loading module: The C++ dynamic library loading method is used to dynamically load plugins from the plugin library for different bitstream formats.

[0033] Encoding / Decoding and Transcoding Module: The video stream acquired by the plugin may be the raw bitstream, such as H264 / H265 format data. In this case, the raw bitstream data needs to be encoded, decoded, and transcoded to obtain image data in the required format, such as BGR or RGB format image data.

[0034] Disk saving / dumping module: Saves the image data output by the encoding / decoding and transcoding modules to the local disk or dumps it to external devices such as NAS.

[0035] Data processing module: Combines image data and image structured information output by AI model into data blocks, and provides them to the AI ​​algorithm task module through corresponding channels.

[0036] Control module: Maintains communication with the middle platform, keeps a heartbeat, and receives commands from the middle platform to shut down, reduce frame rate, and prioritize; controls the plug-in selection and loading module and the encoding / decoding and transcoding module of the streaming service system to pull the bitstream, encode, decode, and transcode it in a plug-in manner to meet the requirements of the data processing module.

[0037] The plugin library contains plugins for FFmpeg, Remote, VLC, and SDK.

[0038] like Figure 1 As shown, the online video image processing method of this embodiment includes the following steps:

[0039] 1) Run the Chrome browser on a server equipped with an NVIDIA graphics card. Users specify a data source and trigger a task on the browser's task initiation page. When issuing the task, users select and specify the corresponding plugin on the task initiation page, which is then loaded into the video streaming service. Data from the data source is obtained through the loaded plugin.

[0040] If the data source is an IPC network camera, the bitstream data can be retrieved via RTSP. In this case, the plugin will choose to call the plugin library written in the open-source FFmpeg library to obtain the RTSP bitstream. FFmpeg plugins support most data stream formats, such as RTSP streams and HLS streams, and their interfaces are more low-level.

[0041] The VLC plugin's interface supports common RTSP streams.

[0042] The Remote plugin supports local video files and image data.

[0043] The SDKs provided by various manufacturers can maximize the expansion of video streaming capabilities.

[0044] 2) After obtaining the data from the data source through the selected plugin, the data is encoded and decoded, and transcoded into BGR or RGB format image data. The obtained data source cannot be directly used for AI algorithm tasks; it needs to be encoded and decoded into the required image format. For example, data obtained via RTSP is in H264 format, which needs to be decoded into YUV format data according to the H264 encoding method, and then the YUV format data is transcoded into BGR format image data.

[0045] 3) After preprocessing the image data, it is passed to the AI ​​algorithm task module. The image data is also stored on the local disk and transferred to external devices such as NAS for later use in structured video display. After AI inference based on the pipeline, the data processing module extracts information such as target category, coordinates, confidence level, and feature values ​​from the video frame metadata and associates them with the video frames for use by the AI ​​algorithm task module.

[0046] 4) During video streaming and data processing, the control module maintains communication with the middleware service.

[0047] The control module sends heartbeat packets and operational status data to the middleware platform, allowing the platform to understand the keep-alive status and current operational status of the streaming service system. When the video playback on the page is choppy, it indicates a backlog of data to be processed by the AI ​​algorithm task module. When a user discovers a data backlog in the AI ​​algorithm task module, before re-issuing the task, they can readjust the frame rate on the task startup page by sending a command to the control module to reduce the frame rate. The control module then instructs the encoding / decoding and transcoding modules to lower the current frame rate, reducing the amount of data processed by the AI ​​algorithm task module and achieving a balance.

[0048] The streaming service system allows for prioritization of data processing, which users can select when assigning tasks. For example, if a user wants to view live feeds from a specific camera (with camera information pre-configured on the page), they can configure the task assignment parameters on the task page before assigning the task. Before assigning a task, the user can select its priority and then click "Start Task Assignment." High-priority tasks exclusively utilize a GPU core in the data processing module, while low-priority tasks share a single GPU core. The data processing module processes high-priority data immediately, while low-priority data is processed when the AI ​​algorithm task module is idle.

[0049] The online video image processing method of the above embodiments of the present invention can acquire bitstreams from all network cameras on the market that support the RTSP protocol, supports image reception, local video, supports access through camera manufacturer SDKs, and supports user-defined protocol image sources. The present invention is adaptable to different data sources and has high compatibility.

Claims

1. An online video image processing system, characterized in that, It includes a data processing system, which comprises a middle platform, a streaming service system, and a control module. The streaming service system includes a data source, a plugin library, a plugin selection and loading module, an encoding / decoding and transcoding module, and a data processing module. Data source: Provides network camera streams, local video data, and / or image data; Plugin library: It stores different plugins for different bitstream formats, allowing for retrieval and conversion of the bitstream; Plugin selection and loading module: The C++ dynamic library loading method is used to dynamically load plugins from the plugin library for different bitstream formats; Encoding / decoding and transcoding module: Encodes, decodes, and transcodes the raw bitstream data, and outputs image data in the required format; Data processing module: Combines image data and image structured information output by AI model into data blocks, and provides them to AI algorithm task module through corresponding channels; Control module: Maintains communication with the middle platform, receives commands from the middle platform, and controls the selection and loading of the streaming service module and the encoding / decoding and transcoding module to pull the bitstream, encode, decode, and transcode it in a plug-in manner to meet the requirements of the data processing module.

2. The online video image processing system according to claim 1, characterized in that, The streaming service system includes a disk dump / transfer module, which saves the image data output by the encoding / decoding and transcoding modules to the local disk or transfers it to an external device.

3. The online video image processing system according to claim 1, characterized in that, The plugin library contains plugins for FFmpeg, Remote, VLC, and SDK.

4. A method for processing online video images, characterized in that, The online video image processing system applied to claim 1 includes the following steps: 401) The user specifies the data source and triggers the task on the task startup page, selects the corresponding plugin, and loads it into the video streaming service; 402) Data is obtained from the data source through the loaded plugin; 403) After obtaining the data from the data source, the data is encoded and decoded, and converted into image data in BGR or RGB format; 404) The image data is preprocessed and then sent to the AI ​​algorithm task module for use; 405) The control module maintains communication with the middleware service.

5. The video image processing system according to claim 4, characterized in that, Prior to step 404, the image data is stored on a local disk and / or transferred to an external device.

6. The video image processing system according to claim 4, characterized in that, In step 404, after AI inference based on the Pipeline, the data processing module extracts the target's category data, coordinate data, confidence data, and / or feature values ​​from the image data and associates them with the image data for use by the AI ​​algorithm task module.

7. The video image processing system according to claim 4, characterized in that, In step 405, the control module sends heartbeat packets and working status data to the middle platform, allowing the middle platform to understand the keep-alive status and current working status of the streaming service system; when the user finds that the AI ​​algorithm task module has a backlog of data to be processed, it sends a command to the control module to reduce the frame rate. The control module notifies the encoding / decoding and transcoding modules to reduce the current frame rate in order to reduce the data processed by the AI ​​algorithm task module and achieve balance.

8. The video image processing system according to claim 4, characterized in that, In step 405, the streaming service system sets priorities for data processing. When a user issues a task, they can select the priority of the task. High-priority tasks exclusively use the GPU core of the data processing module, while low-priority tasks share one GPU core of the data processing module. The data processing module processes high-priority data immediately and processes low-priority data when the AI ​​algorithm task module is idle.

Images