Online video stream sensitive content detection method and system based on pipeline parallelism

Abstract

The invention discloses an online video stream sensitive content detection method and system based on pipeline parallelism, and relates to the field of video stream content safety automatic detection. The problems that the real-time performance and the invasiveness of detection are difficult to balance in video stream detection and the detection efficiency is low in the prior art are solved. The method comprises the following steps: step 1, capturing online video traffic by a DPDK, and carrying out protocol analysis and frame restoration in a user space; step 2, constructing an assembly line parallel processing architecture, and parallelizing each link assembly line of the detection process of the video frame restored in the step 1; and step 3, based on a CUDA parallel computing technology, in combination with computing and memory access characteristics of an NVIDIA GPU hardware bottom layer, bottom layer implementation and reasoning optimization of the detection model VGG-16 and LSTM are carried out, the parallel computing capability of computing resources is fully utilized, the GPU memory overhead is reduced, and the model reasoning efficiency is improved. The method is suitable for the field of data content security of the video platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video stream content security automatic detection, in particular to an online video stream sensitive content detection method and system based on pipeline parallel. BACKGROUND

[0002] With the continuous development of multimedia technology and the diversification of social information dissemination channels, network video services have penetrated into all aspects of public daily life, showing a vigorous development trend in professional education, cultural promotion, e-commerce, food teaching and comprehensive entertainment, etc., broadening people's information access and enriching the public's spiritual and cultural life. Video platform supervisors need to conduct content review on user-uploaded videos or live video streams to prevent the spread of sensitive content.

[0003] Due to the inefficiency of manual review, this approach is gradually being replaced by computer and artificial intelligence technology-assisted video content detection technology. These technologies mainly rely on machine learning and deep learning algorithms to achieve accurate identification of sensitive content through learning of video content. However, existing research on video sensitive content detection technology mainly focuses on improving detection accuracy, and research on improving model detection efficiency and detection system processing capacity is relatively lacking, failing to fully utilize the parallel computing capabilities of computing resources. In addition, current video content detection solutions usually deploy streaming services and video content detection systems on different servers and connect them in series to reduce the impact of detection services on streaming service performance. However, if video detection is performed by remotely reading video streams after the video streams reach the streaming server, the output of the detection result will lag behind the video stream transmission, resulting in high latency. Conversely, if detection is performed before the video stream reaches the streaming server, and the compliant video stream is forwarded to the streaming server, the detection process is system invasive, and the latency will inevitably affect the video stream, especially the real-time nature of live streaming.

[0004] In combination with cutting-edge technology, the present application balances the real-time and invasiveness of detection to a certain extent by capturing and detecting video streams in the bypass of normal video traffic transmission, and optimizes the performance of multiple links such as data packet capture, frame processing and video frame detection in bypass detection, reducing detection latency. This invention not only provides protection for good network space order and the physical and mental health of the public, but also provides ideas and practical experience for the development of video stream content detection performance optimization. SUMMARY

[0005] The present application aims to solve the problem of how to implement video stream bypass detection in the prior art, and how to optimize the performance of each link in the detection process to achieve a high-throughput, low-latency and low-invasive online video stream sensitive content detection system.

[0006] To achieve the above object, the present application provides the following technical solutions: Scheme one, the present application provides an online video stream sensitive content detection method based on pipeline parallel, the method comprises the following steps: Step 1, using the data plane development kit (Data Plane Development Kit, DPDK) multi-core multi-thread mechanism to capture online video traffic, RTMP protocol analysis and restoration to the original video frame in user space; Step 2, the detection process is divided into several preprocessing, model inference and post-processing nodes, and connected in order, to build a pipeline parallel processing architecture, and the original video frame restored in step 1 is sent into the pipeline architecture to execute the detection process; Step 3, using the compute unified device architecture (Compute Unified Device Architecture, CUDA) parallel computing technology, combined with the calculation and memory access characteristics of the underlying NVIDIA GPU hardware, the underlying implementation of the detection model VGG-16 and LSTM is carried out, the parallel computing capability of the computing resource is fully utilized, and the GPU memory overhead is reduced, and the model inference efficiency is improved.

[0007] Further, there is another preferred embodiment, the traffic capture, RTMP protocol analysis and frame restoration mechanism of step 1 is specifically: Step 101, starting DPDK, using CPU affinity, binding multiple packet capturing threads to different CPU cores to avoid memory access across non-uniform memory access (Non-Uniform Memory Access, NUMA) nodes, and polling the network card multi-queue to capture data packets and copy them to the user space; Step 102, design the RTMP protocol analysis mechanism of user space, analyze the video stream data packet captured in step 101, and extract the RTMP payload; Step 103, using FFmpeg to decode the RTMP payload to restore the original video frame.

[0008] Further, there is another preferred embodiment, the video traffic data packet is captured by using DPDK, and the protocol analysis is carried out in the user space to restore the original video frame. The original video frame and network information and tensor data are encapsulated into a frame data unit. The frame data unit is the basic unit flowing in the subsequent pipeline processing architecture.

[0009] Further, there is another preferred embodiment, the pipeline parallel processing architecture in step 2 is specifically: Step 201, respectively instantiate the preprocessing, model inference, post-processing nodes of VGG-16 and LSTM, and connect the nodes in sequence to build a pipeline architecture; Step 202, design a video detection model: the preprocessing node of VGG-16 crops the video frame into a three-channel 224x224 image and performs normalization processing; the model inference node of VGG-16 is responsible for performing convolution operations of several convolution layers to extract spatial local features of each image; the post-processing node of VGG-16 performs two fully connected operations to integrate the features of a single image extracted by the convolution layer into a 1x4096 vector; the preprocessing node of LSTM collects batches of vectors, combining 20 vectors together to form a 20x4096 matrix each time; the model inference node of LSTM takes the 20x4096 matrix as input, captures the temporal dependency between the features of the consecutive 20 images, detects sensitive behaviors such as fighting and attacking, and calculates a 1x256 vector; the post-processing node of LSTM receives the 1x256 vector and integrates the features into a 1x2 vector through three fully connected layers, and calculates the probability of the 20 consecutive frames containing sensitive content through the Softmax function. At the end of the pipeline architecture, a detection effect echoing node is implemented to write the probability value of the frame sequence containing sensitive content into the original video frame of the frame data unit using the OpenCV tool, and use the cross-platform streaming media application framework GStreamer to echo the detection effect in real time.

[0010] Step 203, design a frame data unit transfer mechanism based on a double-buffered queue to realize the flow of frame data units between nodes in the pipeline architecture and reduce thread synchronization and data copy overhead.

[0011] Further, there is a preferred embodiment, the above-mentioned pipeline parallel processing architecture can pipeline parallelize different operations on consecutive frames. When the previous frame is performing a certain processing step, the subsequent frame does not need to wait for the previous frame to be completely detected before entering the pipeline to start calculation, thereby improving the parallelism of frame processing.

[0012] Further, there is a preferred embodiment, the above-mentioned frame data unit transfer mechanism based on a double-buffered queue can reduce the thread synchronization and data copy overhead of the flow of frame data units between nodes in the pipeline.

[0013] Further, there is a preferred embodiment, the above-mentioned step 3 model inference underlying implementation and optimization specifically: Step 301, based on the CUDA technology, the bottom layer implementation and optimization of the convolution layer of VGG-16, on the basis of the conversion of the convolution operation into matrix block multiplication in Im2col, the data sharing mechanism of thread bundle layout optimization, the weight data rearrangement mechanism of merging memory access, and the multi-stream parallel computing and reduction mechanism are designed, the parallel computing resources are fully utilized, and the GPU memory overhead is reduced; Step 302, based on the CUDA technology, the bottom layer implementation and optimization of LSTM, the weight is preloaded into the shared memory of the thread block or the register of the thread, and the time sequence reuse of the weight is realized in different time steps of the same layer; for the multi-layer LSTM, a multi-layer LSTM parallel thread block cooperation mechanism is designed, the LSTM interlayer pipeline parallelism is realized, and the calculation parallelism is improved; Step 303, based on the CUDA technology, the bottom layer implementation and optimization of the full connection layer, the continuous full connection layer used after LSTM for adjusting the dimension and integrating the features is subjected to calculation and storage overlap optimization, the current layer calculation and the next layer weight loading are overlapped, and the memory loading delay is hidden.

[0014] Scheme two, the online video stream sensitive content detection method based on pipeline parallelism provided by the application can be realized by computer software, therefore, the application also provides an online video stream sensitive content detection system based on pipeline parallelism, the system comprises a storage device, and the storage device is used to execute the online video stream sensitive content detection method based on pipeline parallelism proposed above.

[0015] Scheme three, the application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is run by a processor to execute the online video stream sensitive content detection method based on pipeline parallelism.

[0016] Scheme four, the application further provides a computer device, the device comprises a memory and a processor, and the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the online video stream sensitive content detection method based on pipeline parallelism.

[0017] The beneficial effects of the application are as follows: The system design and performance optimization provided by the application are aimed at the problems of low detection efficiency and difficulty in balancing the real-time performance and invasiveness of the existing video stream sensitive content detection system, and an online video stream sensitive content detection system based on pipeline parallelism is proposed and designed. The performance optimization design is performed on multiple links such as video stream capture and analysis, video frame parallel processing, and model inference, the detection time delay of the video stream sensitive content is reduced, and the real-time performance of the detection is improved while avoiding the invasion to the normal video stream transmission.

[0018] The application is suitable for the field of data content security of video platforms. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 Flowchart of the online video stream sensitive content detection method described in the present application; Figure 2 Overall framework entity diagram of the online video stream sensitive content detection system based on pipeline parallelism described in the present application; Figure 3 Schematic diagram of DPDK multi-core multi-thread flow capture described in the present application; Figure 4 Flowchart of multi-TCP stream processing described in the present application; Figure 5 Schematic diagram of encapsulation format of RTMP data in TCP packet described in the present application; Figure 6 Schematic diagram of pipeline frame processing architecture described in the present application; Figure 7 Structure diagram of video sensitive content detection model combined with VGG-16 and LSTM described in the present application; Figure 8 Schematic diagram of frame data unit transfer mechanism based on double-buffer queue described in the present application; Figure 9 Schematic diagram of Im2col described in the present application; Figure 10 Schematic diagram of data sharing mechanism of thread bundle layout optimization described in the present application; Figure 11 Schematic diagram of weight data rearrangement mechanism of merged memory access described in the present application; Figure 12 Schematic diagram of multi-stream parallel computing and reduction mechanism described in the present application; Figure 13 Schematic diagram of LSTM model weight preloading described in the present application; Figure 14 Schematic diagram of LSTM inter-layer pipeline parallelism described in the present application; Figure 15 Schematic diagram of continuous fully connected layer storage and calculation overlap described in the present application; Figure 16Online video traffic capture performance comparison chart according to the present application; Figure 17 Online video traffic restoration performance comparison chart according to the present application; Figure 18 Pipelined video frame processing architecture performance comparison chart according to the present application; Figure 19 Convolution calculation underlying implementation performance comparison chart according to the present application; Figure 20 LSTM calculation underlying implementation performance comparison chart according to the present application; Figure 21 End-to-end performance comparison chart according to the present application. DETAILED DESCRIPTION

[0021] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made, which are within the scope of protection of the present application.

[0022] Embodiment one, see Figure 1 and Figure 2 This embodiment is described, and this embodiment proposes an online video stream sensitive content detection method based on pipelined parallel, for real-time detection of video stream sensitive content, to prevent the spread of sensitive content, and to reduce system detection delay.

[0023] The method as shown in Figure 1 includes the following steps: Step 1, using DPDK multi-core multi-thread mechanism to capture online video traffic, performing RTMP protocol analysis in user space and restoring to original video frames; Step 2, dividing the detection process into a plurality of preprocessing, model inference and post-processing nodes in a fine-grained manner, and connecting them in order to build a pipelined parallel processing architecture, and sending the original video frames restored in step 1 into the pipelined architecture to execute the detection process; Step 3, using CUDA parallel computing technology, combining the underlying calculation and memory access characteristics of NVIDIA GPU hardware, implementing the detection model VGG-16 and LSTM, fully utilizing the parallel computing capability of the computing resources, and reducing the GPU memory overhead to improve the model inference efficiency.

[0024] In actual application, the present embodiment is based on, for example Figure 2The pipeline parallel-based online video stream sensitive content detection system framework shown solves the problems of low efficiency and difficulty in balancing the real-time and invasiveness of the existing online video stream sensitive content detection system. The embodiment optimizes the performance of multiple links such as video stream capture and analysis, video frame parallel processing, and model inference, reduces the detection delay of video stream sensitive content, and is beneficial to avoiding invasions on normal video stream transmission while improving the real-time of detection.

[0025] Embodiment two, this embodiment is a specific description of the pipeline parallel-based online video stream sensitive content detection method described in embodiment one; Step 1, capture online video traffic using the DPDK multi-core multi-thread mechanism, perform RTMP protocol analysis in the user space, and restore to the original video frame; Specifically, Step 101, start DPDK, use CPU affinity to bind multiple packet capture threads to different CPU cores to avoid cross- NUMA node memory access, and poll the network card multi-queue to capture data packets and copy them to the user space.

[0026] Step 102, design a user space RTMP protocol analysis mechanism to analyze the video stream data packets captured in step 101 and extract the RTMP payload.

[0027] Step 103, use FFmpeg to decode the RTMP payload and restore the original video frame.

[0028] Further, the restored original video frame will be encapsulated with network information and tensor data into a frame data unit. The network information includes the source IP, destination IP, source port, and destination port of the frame belonging to the data packet; and the tensor data is used to store the intermediate processing results in the detection process.

[0029] Step 2, divide the detection process into several preprocessing, model inference, and post-processing nodes in a fine-grained manner, connect them in order, build a pipeline parallel processing architecture, and send the original video frame restored in step 1 into the pipeline architecture to execute the detection process; Specifically, Step 201, instantiate the preprocessing, model inference, and post-processing nodes of VGG-16 and LSTM respectively, connect the nodes in order, and build a pipeline architecture; Step 202, design a video detection model: the preprocessing node of VGG-16 preprocesses the video frame; the model inference node of VGG-16 is responsible for performing model inference operation and extracting the spatial local features of each image; the post-processing node of VGG-16 performs two layers of full connection operation, and integrates the features of a single image extracted by the convolution layer into a 1x4096 vector; the preprocessing node of LSTM collects batches of vectors, and combines 20 vectors together to form a 20x4096 matrix each time; the model inference node of LSTM takes the 20x4096 matrix as input, captures the time sequence dependence between the continuous 20 image features, and calculates the probability of containing sensitive content such as fighting and attack in the 20 continuous frames. At the end of the pipeline architecture, a detection effect echoing node is implemented, which uses OpenCV tools to write the probability value of the frame sequence containing sensitive content into the original video frame of the frame data unit, and uses the cross-platform streaming media application framework GStreamer to echo the detection effect in real time.

[0030] Step 203, design a frame data unit transfer mechanism based on double buffer queue to realize the flow of frame data unit between nodes in the pipeline architecture and reduce thread synchronization and data copy overhead.

[0031] Further, the above-mentioned pipeline parallel processing architecture can pipeline the different operations on the continuous frames in parallel. When the previous frame is performing a certain processing step, the subsequent frame does not need to wait for the previous frame to be completely detected before entering the pipeline to start calculation, thereby improving the parallel degree of frame processing. In addition, the frame data unit transfer mechanism based on double buffer queue can reduce the thread synchronization and data copy overhead of the frame data unit flowing between the nodes in the pipeline.

[0032] Step 3, use CUDA parallel computing technology to combine the calculation and memory access characteristics of NVIDIA GPU hardware bottom layer to implement the bottom layer of detection model VGG-16 and LSTM, fully utilize the parallel computing capability of computing resources, reduce the GPU memory overhead, and improve the model inference efficiency.

[0033] Specifically: Step 301, bottom layer implementation and optimization of VGG-16 convolution layer based on CUDA technology. On the basis of Im2col converting convolution operation into matrix block multiplication, the data sharing mechanism of thread bundle layout optimization, the weight data rearrangement mechanism of merging memory access, and the multi-stream parallel computing and reduction mechanism are designed to fully utilize parallel computing resources and reduce GPU memory overhead.

[0034] Step 302, bottom-layer implementation and optimization of LSTM based on CUDA technology. Preload weights into shared memory of thread blocks or registers of threads, implement time multiplexing of weights in different time steps of the same layer; for multi-layer LSTM, design multi-layer LSTM parallel thread block cooperation mechanism, implement LSTM inter-layer pipeline parallelism, and improve calculation parallelism.

[0035] Step 303, memory and calculation overlap optimization for the continuous fully connected layer after LSTM for adjusting dimensions and integrating features, overlap current layer calculation and next layer weight loading, and hide memory loading delay.

[0036] Further, step 3 described above is bottom-layer implementation and performance optimization combined with the calculation and memory access characteristics of the bottom layer of the NVIDIA GPU hardware, and other brands of GPUs need to be performed according to actual conditions.

[0037] Embodiment three, the online video stream sensitive content detection method based on pipeline parallelism described in any one of the above embodiments can be fully implemented by computer software, therefore, the present embodiment provides an online video stream sensitive content detection system based on pipeline parallelism, the system comprises: a storage device for capturing online video traffic by using the DPDK multi-core multi-thread mechanism, performing RTMP protocol analysis in the user space, and restoring the original video frames; a storage device for dividing the detection process into a plurality of preprocessing, model inference and post-processing nodes in a fine-grained manner, connecting them in order, constructing a pipeline parallel processing architecture, and sending the original video frames restored in step 1 into the pipeline architecture to execute the detection process; a storage device for bottom-layer implementation of the detection model VGG-16 and LSTM by using CUDA parallel computing technology combined with the calculation and memory access characteristics of the bottom layer of the NVIDIA GPU hardware, fully utilizing the parallel computing capability of the computing resources, reducing the GPU memory overhead, and improving the model inference efficiency.

[0038] Embodiment four, the present embodiment proposes a computer device comprising a memory, a processor and an image processing unit, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method and steps described in any one of the above embodiments.

[0039] Embodiment five, the present embodiment proposes a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to execute the method and steps described in any one of the above embodiments.

[0040] Embodiment five, the embodiment proposes a specific embodiment for explaining the above-mentioned embodiment, which is specifically: Referring to Figures 1 to 21 It is explained that the method of the embodiment includes the following steps: The embodiment can effectively detect sensitive content in online video stream, reduce detection delay, and avoid intrusion on normal transmission of video stream.

[0041] The online video stream sensitive content detection system based on pipeline parallelism balances the real-time and intrusion of detection to a certain extent by capturing and detecting video stream in the bypass of normal transmission of video traffic, and optimizes the performance of multiple links such as packet capture, frame processing and video frame detection in bypass detection, reduces detection delay, and is beneficial to improve the real-time of video stream detection.

[0042] In combination Figures 1 to 2 , the embodiment proposes an online video stream sensitive content detection system based on pipeline parallelism, which includes the following steps: Step 1: DPDK captures online video traffic and performs protocol analysis and frame restoration in user space.

[0043] The video traffic capture, analysis and frame restoration are specifically: Step 101, use CPU affinity to bind multiple packet capture threads of DPDK to different CPU cores respectively, and poll multiple queues of the network card respectively, copy the packets in the network card to the user space, as shown in Figure 3 .

[0044] Step 102, start the network card hybrid mode, DPDK captures all packets mirrored by the switch. In the user space, filter out the control plane packets and other application layer protocol packets that are not needed by the system, and only keep the packets whose destination port is consistent with the stream media service running port number. Since the stream media server usually establishes TCP connection with multiple clients, each TCP message is processed differently, and the processing flow is as shown in Figure 4 . The RTMP message is encapsulated in the TCP message in the form of a stream, as shown in Figure 5 . The system analyzes the fifth to seventh bytes of the RTMP header to determine the length of the payload and extract the payload. Figure 5 Various cases of RTMP message encapsulated in TCP message are listed in

[0045] Step 103, analyze the RTMP payload. If the value of the second byte of the RTMP payload is 0, it means that the payload is an Instantaneous Decoder Refresh (IDR) frame carrying metadata, and FFmpeg is used to initialize the decoding context of the current sequence; if the value is 1, it means that the payload is a video frame, and FFmpeg is used to restore the payload to the original video frame.

[0046] Step 2: Construct a pipeline parallel processing architecture to pipeline and parallelize each link of the video frame detection process.

[0047] The pipeline parallel video frame processing architecture is specifically: Step 201, in the pipeline detection architecture, a pipeline represents a complete detection task, such as Figure 6 As shown. The system implements the preprocessing, model inference, and post-processing nodes of VGG-16 and LSTM, respectively, and connects the nodes in order to build a pipeline architecture and pipeline parallelize the processing tasks of each node. For example, when a frame of image is sent to the model inference node and processed on the GPU, the next frame can enter the preprocessing node and be processed by the CPU; Step 202, design a video detection model: the preprocessing node of VGG-16 crops the video frame into a three-channel 224x224 image and performs normalization processing; the model inference node of VGG-16 is responsible for performing convolution operations of several convolution layers to extract spatial local features of each image; the post-processing node of VGG-16 performs two fully connected operations to integrate the features of a single image extracted by the convolution layer into a 1x4096 vector; the preprocessing node of LSTM collects batches of vectors, combining 20 vectors together to form a 20x4096 matrix each time; the model inference node of LSTM takes the 20x4096 matrix as input to capture the temporal dependency between the features of the consecutive 20 images, detect sensitive behaviors such as fighting and attacking, and calculate a 1x256 vector; the post-processing node of LSTM receives the 1x256 vector and integrates the features through three fully connected layers into a 1x2 vector, and calculates the probability of the 20 consecutive frames containing sensitive content through the Softmax function. The overall structure of the model is as shown in Figure 7 At the end of the pipeline architecture, a detection effect echoing node is implemented to write the probability value of the frame sequence containing sensitive content into the original video frame of the frame data unit using the OpenCV tool, and to echo the detection effect in real time using the cross-platform streaming media application framework GStreamer.

[0048] Step 203, design a frame data unit transfer mechanism based on double-buffer queue, realize the flow of frame data unit between nodes in the pipeline architecture, and reduce thread synchronization and data copy overhead, as shown in Figure 8 In single producer-single consumer mode, i.e. single-thread working environment, the working thread of the consumer node can read data without lock operation. In multi-producer-multi-consumer mode, i.e. multi-thread working environment, thread contention between the working thread of the producer node and the working thread of the consumer node is avoided, the lock operation overhead is reduced, and the number of data copying is also reduced.

[0049] Step 3, using CUDA technology, combining the calculation and memory access characteristics of the underlying hardware of NVIDIA GPU, the underlying implementation of detection model VGG-16 and LSTM is carried out, the parallel computing capability of the computing resource is fully utilized, and the GPU memory overhead is reduced, and the model inference efficiency is improved.

[0050] The underlying implementation of the detection model based on CUDA technology is specifically: Step 301, underlying implementation and optimization of the convolutional layer of VGG-16 based on CUDA technology. Im2col is a common convolutional layer calculation optimization method, which converts convolutional operation into matrix block multiplication, which adapts to the calculation characteristics of GPU, as shown in Figure 9 On the basis of Im2col, a data sharing mechanism for thread bundle layout optimization is designed. According to the data access mode of block matrix multiplication, the calculation result of each row in the result matrix is obtained by the multiplication and addition operation of the corresponding input data row and different weight data columns, which shows that when a row block of the result matrix is calculated by multiple threads of the same thread bundle, a row of input data can be reused in multiple block calculations after being loaded from global memory to shared memory without repeated loading. On this basis, according to the analysis of the data block loading frequency, the number of data block loading of all thread bundles in the thread block is actually determined by the sum of the number of rows and the number of columns of the thread bundle layout in the thread block. When the product of the number of rows and the number of columns of the thread bundle layout is constant, reducing the sum of the two can effectively reduce the memory overhead, as shown in Figure 10 The solving formula of the data block loading frequency after thread bundle layout optimization is as follows:

[0051]

[0052]

[0053] In the formula, and are the number of rows and the number of columns of the thread bundle layout, respectively. The number of blocks responsible for calculation in the result matrix for each thread block. Solving the formula, Figure 10 The data sharing mechanism of the thread bundle layout optimization reduces 21 memory loading operations compared to the original sharing mechanism, which can effectively alleviate the performance bottleneck caused by global memory access. Further, a weight data rearrangement mechanism that combines memory access is designed, as shown in Figure 11 Each 16x16 data block is tiled into a 256-dimensional vector, and the rearranged weight matrix is written according to its corresponding position in the original matrix, so that each row of the weight data block is adjacent in physical position, thereby maximizing the memory transaction when the thread bundle accesses the weight data block and improving the utilization of GPU memory bandwidth. Further, a multi-stream parallel computing and reduction mechanism is designed, as shown in Figure 12 The reduce axis of the block matrix multiplication is reasonably divided, and the multiplication and addition operations are performed along different parts of the reduce axis by multiple CUDA streams, and the calculation results are reduced to global memory by atomic addition, thereby maximizing the utilization of GPU parallel computing capability.

[0054] Step 302, based on CUDA technology, the underlying implementation and optimization of LSTM. As shown in Figure 13 Each thread block or thread has its own responsible calculation area, which preloads the weights required for its calculation into its block shared memory or thread register, and realizes the time sequence reuse of weights in different time steps of the same layer; for multi-layer LSTM, a multi-layer LSTM parallel thread block coordination mechanism is designed to realize LSTM inter-layer pipeline parallelism, as shown in Figure 14 Once the calculation of a certain layer at a specific time step is completed, the calculation of the next time step of the same layer and the corresponding time step of the previous layer can be started to improve the calculation parallelism and fully utilize the parallel computing capability of GPU.

[0055] Step 303, based on CUDA technology, the underlying implementation and optimization of continuous fully connected layers. The continuous fully connected layers after LSTM for adjusting dimensions and integrating features are optimized for storage and calculation overlap, as shown in Figure 15 Before starting the calculation of the current fully connected layer, an asynchronous copy SASS (Streaming Assembler) instruction is initiated to the hardware, and the weight of the next fully connected layer is loaded asynchronously, i.e. overlapping the current layer calculation and the next layer weight loading, hiding the memory loading delay.

[0056] The embodiment is directed to real-time detection of online video stream sensitive content, utilizes high-performance network packet processing technology to restore video stream data packets to original video frames, constructs a pipeline parallel video frame processing architecture, and optimizes the detection model at the bottom layer, thereby improving the efficiency of online video stream sensitive content detection, reducing the time delay of each link, and meeting the demand for real-time detection.

[0057] The specific implementation is: (I) Implementation method.

[0058] According to the online video stream sensitive content detection flowchart shown in the embodiment of the present application, mainly includes online video stream capture and processing based on DPDK, pipeline parallel video frame processing, and CUDA-based video detection model bottom layer implementation and optimization. Figure 1

[0059] In the multi-core multi-threaded traffic capture of step 101, the CPU affinity is utilized, the multiple packet capturing threads of DPDK are respectively bound to different CPU cores for running, and each corresponding network card queue is polled, the kernel bypass technology is bypassed to bypass the kernel space, the data packets in the network card are directly copied to the user space, and the context switching and data copying are reduced.

[0060] ​In step 102, video traffic protocol analysis, the network card promiscuous mode is started, and DPDK captures all the data packets mirrored by the switch. In the user space, the system does not need the data packets such as control plane data packets and other application layer protocol data packets, and only the data packets with the destination port consistent with the port number of the streaming service running are reserved. Since the streaming server usually establishes TCP connection with multiple clients, a protocol analysis context is maintained for each TCP flow. For each TCP packet, the TCP flow information thereof is checked. When the data packet of the TCP flow is captured for the first time, it is indicated that the TCP flow is in the initial transmission stage. At this time, the system records the sequence number of the next expected data packet of the TCP flow. When the data packet of the same TCP flow is captured subsequently, the actual sequence number and the expected sequence number recorded are compared. If the two match, the data packet is accepted and the expected sequence number is updated; if the actual sequence number is lower than the expected sequence number, it is indicated that the data packet is a retransmission packet, which is directly discarded; if the actual sequence number is higher than the expected sequence number, it is indicated that the data packet arrives out of order, and the system starts the buffering mechanism to temporarily store the data packet. When the expected sequence number is consistent with the sequence number of the data packet, the data packet is taken out for processing. The RTMP packet is encapsulated in the form of a flow in the TCP packet. A single TCP packet can contain multiple RTMP packets, or an RTMP packet can be distributed in two consecutive TCP packets. This means that the RTMP packet analysis needs to be performed according to the actual length thereof. The system analyzes the fifth to seventh bytes of the RTMP header to determine the length of the payload and extract the payload. When the RTMP header and the payload thereof are encapsulated in the same TCP packet, they can be directly analyzed and extracted; when the RTMP payload is encapsulated in two consecutive TCP packets, the RTMP payload segment of the former TCP packet is copied to the buffer, and the offset position of the next copy is calculated, and then the RTMP payload segment of the latter TCP packet is copied to the offset position, and the complete RTMP payload can be extracted from the buffer; when the three bytes in the RTMP protocol header for identifying the length of the payload are encapsulated in two consecutive TCP packets, the higher bytes at the end of the former TCP packet and the lower bytes at the beginning of the latter TCP packet are shifted and added to obtain the length of the payload of the RTMP packet, and the payload is extracted according to the length.

[0061] In step 103, video frame restoration, the RTMP payload is analyzed. If the value of the second byte of the RTMP payload is 0, it indicates that the payload is an IDR frame carrying metadata, and the initialization of the current sequence decoding context by FFmpeg is used; if the value is 1, it indicates that the payload is a video frame, and the payload is restored to the original video frame by FFmpeg.

[0062] In step 201 of the pipeline parallel frame processing architecture construction, the system respectively implements the preprocessing, model inference and post-processing nodes of VGG-16 and LSTM, connects the nodes in sequence, constructs the pipeline architecture, and pipelines the processing tasks of each node in parallel.

[0063] In step 202 of the video detection model design, the preprocessing node of VGG-16 crops the video frame into a three-channel 224x224 image and performs normalization processing; the model inference node of VGG-16 is responsible for performing convolution operations of several convolution layers to extract spatial local features of each image; the post-processing node of VGG-16 performs two fully connected operations to integrate the features of a single image extracted by the convolution layer into a 1x4096 vector; the preprocessing node of LSTM collects batches of vectors, each time combining 20 vectors together to form a 20x4096 matrix; the model inference node of LSTM takes the 20x4096 matrix as input, captures the time sequence dependency between the continuous 20 image features, detects sensitive behaviors such as fighting and attacking, and calculates a 1x256 vector; the post-processing node of LSTM receives the 1x256 vector and integrates the features into a 1x2 vector through three fully connected layers, and calculates the probability of 20 consecutive frames containing sensitive content through the Softmax function. At the end of the pipeline architecture, a detection effect echoing node is implemented to write the probability value of the frame sequence containing sensitive content into the original video frame of the frame data unit using the OpenCV tool, and use the cross-platform streaming media application framework GStreamer to echo the detection effect in real time.

[0064] In step 203 of the frame data unit transfer mechanism design, a frame data unit transfer mechanism based on double-buffered queues is designed to realize the flow of frame data units between nodes in the pipeline architecture and reduce thread synchronization and data copy overhead. In the single producer-single consumer mode, i.e. in a single-threaded environment, the worker thread of the consumer node can read data without lock operation. In the multi-producer-multi-consumer mode, i.e. in a multi-threaded environment, thread contention between the worker threads of the producer node and the consumer node is avoided, reducing the lock operation overhead and also reducing the number of data copies.

[0065] In the VGG-16 bottom layer implementation and optimization of step 301, on the basis of Im2col, a data sharing mechanism for thread bundle layout optimization is designed. According to the data access mode of block matrix multiplication, the calculation result of each row in the result matrix is obtained by the multiplication and addition operation of the corresponding input data row and different weight data columns, which shows that when a row block of the result matrix is calculated by multiple threads of the same thread bundle, a row of input data can be reused in multiple block calculations after being loaded from the global memory to the shared memory without repeated loading. On this basis, according to the analysis of the data block loading frequency, the number of times of data block loading by all threads in a thread block is actually determined by the sum of the number of rows and the number of columns of the thread bundle layout. When the product of the number of rows and the number of columns of the thread bundle layout is constant, reducing the sum of the two can effectively reduce the memory overhead. The formula for solving the number of data block loading after thread bundle layout optimization is as follows:

[0066]

[0067]

[0068] wherein and are the number of rows and the number of columns of the thread bundle layout, respectively; is the number of blocks calculated by each thread block in the result matrix. Further, a weight data rearrangement mechanism for merging memory access is designed, each 16x16 data block is tiled into a 256-dimensional vector, and the weight data block is written into the rearranged weight matrix according to its corresponding position in the original matrix, so that each row of the weight data block is adjacent in the physical position, thereby maximizing the memory transaction when the thread bundle accesses the weight data block, and improving the utilization of GPU memory bandwidth. Further, a multi-stream parallel computing and reduction mechanism is designed to reasonably divide the reduction axis (reduce axis) of the block matrix multiplication, perform multiplication and addition operations along different parts of the reduction axis by multiple CUDA streams, and reduce the calculation result to the global memory by atomic addition, thereby maximizing the utilization of GPU parallel computing capability.

[0069] In the bottom layer implementation and optimization of LSTM of step 302, each thread block or thread has a region responsible for calculation, which preloads the weights required for calculation into the block shared memory or the register of the thread, and realizes the time sequence reuse of weights in different time steps of the same layer; for multi-layer LSTM, a multi-layer LSTM parallel thread block cooperation mechanism is designed to realize the pipeline parallel of LSTM layers. Once the calculation of a certain layer at a specific time step is completed, the calculation of the next time step of the same layer and the corresponding time step of the previous layer can be started to improve the calculation parallelism and fully utilize the parallel computing capability of GPU.

[0070] In the bottom layer implementation and optimization of the continuous full connection layer of step 303, the continuous full connection layer used after the LSTM for adjusting the dimension and integrating the features is optimized by overlapping storage and calculation. Before starting the calculation of the current full connection layer, the SASS (Streaming Assembler) instruction of asynchronous copying is initiated to the hardware, and the weight of the next full connection layer is loaded asynchronously, that is, the calculation of the current layer and the weight loading of the next layer are overlapped, and the memory loading delay is hidden.

[0071] (II) Experimental verification.

[0072] In the present embodiment, the performance of the proposed online video stream sensitive content detection method and its key modules is mainly focused on evaluation and comparison with related products.

[0073] In the performance test of traffic capture and processing, the pcap packet file containing video traffic is replayed on the link at different rates (1Gbps to 25Gbps), and is captured by DPDK and traditional packet capture technology Libpcap respectively. As shown in Figure 16 , when the sending rate reaches 25Gbps, the packet loss rate of the traditional Libpcap-based packet capture technology reaches 13.18%, while the packet loss rate of the DPDK-based packet capture technology remains 0. Further, the video stream is pushed to the streaming media server at different frame rates (150FPS to 3000FPS), as shown in Figure 17 , the frame restoration rate of the traditional Libpcap-based packet capture decreases from 100% to 88.53%, while the frame restoration rate of the DPDK-based packet capture remains 100%.

[0074] In the performance test of the pipeline frame processing architecture constructed in the present application, the open source video analysis framework VideoPipe is compared. VideoPipe is widely used in the development of video detection platforms such as face recognition, vehicle detection and behavior detection. In the test process, VideoPipe and the pipeline architecture of the system of the present embodiment are equipped with the same VGG-16 and LSTM model, and libtorch is used for model inference. As shown in Figure 18 , due to the reduction of thread synchronization and data copying overhead between pipeline nodes by double buffering queue, the pipeline frame processing architecture of the present application requires less time when processing a specified number of frames, and the throughput is improved by an average of about 9.54%.

[0075] In the CUDA-based detection model bottom layer implementation and optimization of the present application, the general inference optimization framework TVM and the NVIDIA deep learning acceleration library cuDNN are compared. Table 1 lists the four convolutional layers with the most time-consuming in VGG-16.Figure 19 The calculation performance of the application on the four convolution layers is compared with TVM and cuDNN. Compared with TVM, the convolution calculation time delay of the underlying implementation of the application is reduced by an average of about 13.80%; compared with cuDNN, the convolution calculation time delay of the underlying implementation of the application is reduced by an average of about 22.11%. Figure 20 The calculation performance of the application on single-layer and double-layer LSTM is compared with TVM and cuDNN. On single-layer LSTM, compared with TVM, the time delay is reduced by an average of about 18.70%, and compared with cuDNN, the time delay is reduced by an average of about 22.75%. On double-layer LSTM, compared with TVM, the time delay is reduced by an average of about 27.91%, and compared with cuDNN, the time delay is reduced by an average of about 39.04%.

[0076] Table 1

[0077] Finally, the pipeline detection architecture and the detection model underlying implementation of the application are integrated and optimized, and the end-to-end performance test is carried out, as shown in Figure 21 The test results show that the throughput of the application is increased by about 19.56% compared with VideoPipe.

[0078] In the future, the performance of the online video stream sensitive content detection system will be further optimized, especially in the model underlying implementation, which will be adapted to be made in China, combined with the calculation characteristics and memory characteristics of domestic GPU hardware, to improve the model detection efficiency and reduce the model inference time delay, so as to improve the overall throughput of the system.

[0079] In the above description, it should be understood that, Figure 1Any processes or methods described in the flowcharts or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) of the application, and / or that the various embodiments of the application can represent alternative process or method steps that can be implemented by the functions of the disclosed functions, and that such functions can be carried out in substantially the same way as described in the illustrative embodiments, although not necessarily implemented in the same order as illustrated. The processes, methods, or algorithms described in the flowcharts or otherwise described herein represent example architectures, functions, and operations for possible implementations of the various embodiments of the application. In this regard, each block in the flowcharts or described herein can represent a module, segment, or portion of code which comprises one or more executable instructions for implementing the specified logical functions ("application tasks"). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It will also be noted that each block of the flowchart illustrations and / or illustrated in the block diagrams and / or described herein, and combinations of blocks in the flowcharts and / or block diagrams and / or described herein, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0080] Those skilled in the art will understand that the above description is merely illustrative of the preferred embodiments of the application and the various features and / or claims thereof can be combined or integrated in various ways, even if such combinations or integrations are not expressly disclosed in the above description. The scope of the application is therefore not limited to the specific embodiments described above, but only to the scope of the appended claims, even if further modifications or changes to the described embodiments are made. Any modifications or changes to the described embodiments are intended to be within the spirit and scope of the application.

[0081] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such variations and modifications as fall within the scope of the application. It is also to be understood that the application can be practiced by individuals other than the inventor without departing from the scope of the application. Accordingly, the scope of the application is to be interpreted only by the appended claims and their equivalents.

Claims

1. A method for sensitive content detection in online video streaming based on pipelined parallelism, characterized in that, The method comprises the following steps: Step 1: Capture online video traffic by using the multi-core multi-thread mechanism of the data plane development kit, perform RTMP protocol analysis in the user space, and restore the original video frame; Step 2: Divide the detection process into a plurality of preprocessing, model inference and post-processing nodes in a fine-grained manner, connect them in sequence, build a pipeline parallel processing architecture, and send the original video frame restored in step 1 into the pipeline parallel processing architecture to execute the detection process; Step 3: Use the unified computing device architecture parallel computing technology, combine the computing and memory access characteristics of the underlying NVIDIA GPU hardware, and build the underlying of the detection models VGG-16 and LSTM.

2. The pipeline-parallel based online video stream sensitive content detection method according to claim 1, wherein, Step 1 specifically comprises: Step 101: Start the data plane development kit, use CPU affinity to bind a plurality of packet capture threads to different CPU cores, poll the network card multi-queue, capture data packets and copy them to the user space; Step 102: Design an RTMP protocol analysis mechanism in the user space, analyze the video stream data packets captured in step 101, and extract the RTMP payload; Step 103: Decode the RTMP payload by using FFmpeg to restore the original video frame.

3. The pipeline-parallel based online video stream sensitive content detection method according to claim 2, wherein, The original video frame is encapsulated into a frame data unit together with network information and tensor data.

4. The pipeline-parallel based online video stream sensitive content detection method according to claim 3, wherein, The frame data unit is the basic unit that flows in the subsequent pipeline processing architecture.

5. The pipeline-parallel based online video stream sensitive content detection method of claim 1, wherein, Step 2 specifically comprises: Step 201: Instantiate the preprocessing, model inference and post-processing nodes of VGG-16 and LSTM respectively, connect the nodes in sequence, and build a pipeline architecture; Step 202: Design a video detection model: the preprocessing node of VGG-16 crops the video frame into a three-channel 224x224 image and performs normalization processing; the model inference node of VGG-16 is responsible for performing convolution operations of a plurality of convolution layers to extract the spatial local features of each image; The post-processing node of VGG-16 performs two fully connected operations to integrate the features of a single image extracted by the convolution layer into a 1x4096 vector; the preprocessing node of LSTM collects batches of vectors, combines 20 vectors together to form a 20x4096 matrix each time; the model inference node of LSTM takes the 20x4096 matrix as input, captures the time sequence dependency between the continuous 20 image features, detects sensitive behaviors such as fighting and attacking, and calculates a 1x256 vector; the post-processing node of LSTM receives the 1x256 vector, integrates the features into a 1x2 vector through three fully connected layers, and calculates the probability of 20 consecutive frames containing sensitive content through the Softmax function, and finally realizes a detection effect echoing node to write the probability value of the frame sequence containing sensitive content into the original video frame of the frame data unit by using the OpenCV tool, and use the cross-platform streaming media application framework GStreamer to echo the detection effect in real time; Step 203: Design a frame data unit transmission mechanism based on a double-buffer queue to realize the flow of frame data units between nodes in the pipeline architecture.

6. The pipeline-parallel based online video stream sensitive content detection method according to claim 5, wherein, The pipeline parallel processing architecture can pipeline and parallelize different operations on continuous frames.

7. The pipelined parallelism based online video stream sensitive content detection method of claim 1, wherein, Step 3 is specifically: Step 301, based on the CUDA technology, the bottom layer of the convolution layer of VGG-16 is realized and optimized, and on the basis of the conversion of the convolution operation into matrix block multiplication in Im2col, a data sharing mechanism of thread bundle layout optimization, a weight data rearrangement mechanism of merging memory access, and a multi-flow parallel computing and reduction mechanism are designed; Step 302, based on the CUDA technology, the bottom layer of the LSTM is realized and optimized, the weight is preloaded into the shared memory of the thread block or the register of the thread, and the time sequence reuse of the weight is realized in different time steps of the same layer; for the multi-layer LSTM, a multi-layer LSTM parallel thread block cooperation mechanism is designed to realize the pipeline parallel of the LSTM layers; Step 303, based on the CUDA technology, the bottom layer of the full connection layer is realized and optimized, the continuous full connection layer used after the LSTM for adjusting the dimension and integrating the features is stored and calculated overlapped, the current layer calculation and the next layer weight loading are overlapped, and the hidden memory loading delay is hidden.

8. A system for sensitive content detection in online video streaming based on pipelined parallelism, characterized in that, The system comprises a storage device for executing the method and steps of claim 1.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is run by a processor to execute the online video stream sensitive content detection method based on pipeline parallelism according to any one of claims 1-7.

10. A computer device, comprising: The device comprises a memory and a processor, and the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the online video stream sensitive content detection method based on pipeline parallelism according to any one of claims 1-7.

Images