Multi-thread dynamic video transmission technology based on bird repelling application

Abstract

The invention provides a multi-thread dynamic video transmission technology based on bird repelling application, which is characterized in that real-time transcoding and transmission of multiple paths of high-definition videos can be completed, multi-thread dynamic video transmission is completed in the same window in a man-machine interaction mode, and the video transmission technology adopts coding terminal cache optimization and congestion control algorithm optimization. According to the coding terminal cache optimization, a specific data structure is applied to cache design, the utilization rate of a cache region is increased, and the data circulation speed is increased; according to the congestion control algorithm optimization, by changing a slow start threshold value, the data volume is changed from exponential growth to linear growth, the data fluctuation in the network is reduced, and the probability of packet loss retransmission is reduced; the QT multi-thread technology is adopted, the data transmission efficiency among multiple threads is improved, the data circulation speed is increased, software flash quit caused by memory overflow is avoided, the system delay time is shortened, a client side can conduct multi-channel video transmission at the same time, and the function requirement for real-time multi-channel video transmission under the complex scene of bird repelling application is met.

Description

Technical Field

[0001] This technical invention involves application fields such as aerospace, airport security, and intelligent transportation. It is suitable for early warning or bird repelling at civil airports, and prevention of bird strike incidents at airports or other scenarios. Specifically, it is a QT multi-threaded dynamic video transmission technology. Background Art

[0002] In recent years, with the rapid development of aviation, the probability of bird strike accidents around the world has shown a clear upward trend. Major bird strike accidents occur from time to time, causing serious loss of life and property, and have attracted great attention from relevant departments of various countries. The frequent occurrence of bird strike accidents is a key and difficult problem for aviation stations and civil aviation airports. At present, most solutions for bird strikes use technical means such as sound waves, light beams, chemistry, and biology, such as gas cannons, sonic bird repellers, laser bird repellers, bird repellers, bird repellers, drone bird repellers, and bird repellers. Although it can have a certain effect, the lack of detection and identification devices, the inability to warn and evaluate the effect of repellers, the lack of intelligent algorithms, the single repeller means, the lack of pertinence, the lack of big data analysis and deep learning, etc., make the bird repeller fail to achieve the expected effect.

[0003] A professional R&D team has been formed to prevent bird strike incidents at airports. The team inherits the previous research results of related technologies such as radar detection of flying birds in complex backgrounds, optoelectronic and infrared identification, bird characteristics research, big data statistical analysis, artificial intelligence edge computing, deep learning, sound intensity and laser bird repellent technology, and adopts an "open and modular" design concept. It can be flexibly used according to airport deployment requirements, and has comprehensive prevention functions such as "intelligent detection and perception → data fusion processing → artificial intelligence edge computing → sound and light multi-energy repellent → repellent effect evaluation → machine deep learning". We have developed bionics and optoacoustic-electric integration, artificial intelligence airport detection, perception and bird repellent equipment, which can be used for military and civil airports around the clock and in all regions to perform detection and remote early warning of bird flocks and drones, and intelligently and targetedly implement efficient bird repellent, reduce personnel participation, reduce bird repellent costs, and improve identification and repellent efficiency. It has important theoretical and practical research significance for flight safety in the aviation industry.

[0004] With the advent of the digital network era, video transmission plays an increasingly important role in society. It covers the fields of radio and television, security monitoring, medical health, intelligent transportation, etc. In autonomous driving technology, a large number of on-board cameras continue to generate massive amounts of video data for road condition identification; in the field of security, application scenarios such as highway monitoring, building monitoring, and factory monitoring have generated great demand for multi-channel high-definition video real-time transmission. There are many factors that affect the communication quality of real-time communication equipment. In addition to hardware platform conditions, operating system performance, and the adopted communication protocol, the data transmission software at the application layer is also a key factor. The application layer software used for data transmission in the current real-time communication system has different implementation methods according to different application scenarios. Although there is no unified design pattern, they all face a series of common challenges such as concurrent program design, real-time data transmission guarantee, and software performance optimization. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the existing technology and provide a multi-threaded dynamic video transmission technology based on bird-repelling application.

[0006] The present invention is realized through the following technical solutions:

[0007] A multi-threaded dynamic video transmission technology based on bird-repelling application is proposed. Its characteristics are that it can complete the real-time transcoding and transmission of multiple high-definition videos, realize multi-threaded dynamic video transmission in the same window by human-computer interaction, and realize the function of local storage. The video transmission technology adopts encoding terminal cache optimization and congestion control algorithm optimization. The encoding terminal cache optimization applies specific data structure to cache design to improve the utilization rate of the cache area and speed up the data circulation speed; the congestion control algorithm optimization changes the slow start threshold value, and the data volume changes from exponential growth to linear growth, thereby reducing data fluctuations in the network and reducing the probability of packet loss and retransmission; QT's multi-threading technology is used to realize multi-threaded dynamic video transmission in the same window, improve the data transmission efficiency between multiple threads, speed up the data circulation speed, avoid software crash caused by memory overflow, reduce system delay time, and enable the client to perform multi-channel video transmission at the same time, realizing the functional requirements of real-time multi-channel video transmission in complex scenarios of bird-repelling application. The system function flow chart is shown in the figure. Figure 1 shown.

[0008] The encoding terminal cache optimization is characterized in that after the encoding terminal encodes the video data using the X265 encoder, it does not send it directly through the TCP protocol, but first stores specific data in the cache area, and under the premise of a normal network, continuously takes out data from the cache area for sending, and releases the memory occupied by the data that has been sent for caching subsequent data. The encoding terminal uses a circular FIFO (FIRST INFIRST OUT) data structure in the cache area for optimization, which has the advantage of ensuring that the data in the cache area can be queued and dequeued in order. Based on the first-in-first-out feature of the queue, the used cache can be quickly cleared to avoid blocking. In multi-threaded scheduling, the concurrency capability of the software and the operating efficiency of the system are improved, which plays a role in reducing transmission delays.

[0009] The congestion control algorithm optimization is characterized in that when packet loss and retransmission occur, the slow start threshold ssthresh and the congestion window cwnd are adjusted, and these two parameters determine the amount of data transmitted. The coding terminal reduces the slow start process time by reducing the slow start threshold ssthresh, speeding up the algorithm to enter the congestion avoidance phase. After entering the congestion avoidance phase, the data volume changes from exponential growth to linear growth, thereby reducing data fluctuations in the network, increasing transmission stability, and reducing the probability of packet loss and retransmission, thereby reducing transmission delay.

[0010] The multi-threading technology optimization is characterized in that a thread pool is created and time-consuming tasks are processed in a subclass. After the processing is completed, the processing result is returned to the main thread through a specific method. After the main thread obtains the result, it can perform corresponding operations to update the UI interface. The advantage is that it does not require the asynchronous message transmission combination of UI thread communication + QueuedConnection, can automatically manage the creation and destruction of threads, avoids the loss of frequent creation and destruction of threads, improves thread utilization and performance, and ensures thread safety.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] (1) The encoding terminal cache technology of the present invention does not directly send data through the TCP protocol, but first stores the data in the cache area. Under the premise of normal network, the data is continuously taken out from the cache area for sending, and the memory occupied by the data that has been sent is released for caching subsequent data. The encoding terminal uses a circular FIFO (FIRSTINFIRST OUT) data structure in the cache area for optimization. The advantage is that the data in the cache area can be queued and dequeued in order. Based on the first-in-first-out feature of the queue, the used cache can be quickly cleared to avoid blocking. In multi-threaded scheduling, the concurrency capability of the software and the operating efficiency of the system are improved, which plays a role in reducing transmission delay.

[0013] (2) The congestion control algorithm of the present invention determines whether congestion occurs by packet loss and retransmission. The coding terminal reduces the slow start process time by reducing the slow start threshold ssthresh, speeding up the algorithm to enter the congestion avoidance phase. After entering the congestion avoidance phase, the data volume changes from exponential growth to linear growth, thereby reducing data fluctuations in the network, increasing transmission stability, and reducing the probability of packet loss and retransmission, thereby reducing transmission delay.

[0014] (3) The multi-threading technology of the present invention uses a thread pool to add tasks to the queue during processing, and then automatically starts these tasks after creating a thread, and processes time-consuming tasks in a subclass. After the processing is completed, the processing results are returned to the main thread through a specific method. After the main thread obtains the result, it can perform corresponding operations to update the UI interface. There is no need to use the asynchronous message transmission combination of UI thread + QueuedConnection. The processing efficiency is improved based on asynchronous communication, the loss of frequent thread creation and destruction is avoided, the thread utilization and performance are improved, and thread safety is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present technical invention is further described below with reference to the accompanying drawings and embodiments.

[0016] Figure 1 It is a software system functional flow chart

[0017] Figure 2 It is the encoder principle of H.264 / AVC

[0018] Figure 3 It is the decoder principle of H.264 / AVC

[0019] Figure 4 There are three sampling formats of YUV

[0020] Figure 5 This is a ring FIFO structure diagram

[0021] Figure 6 This is the video sending implementation flow chart DETAILED DESCRIPTION

[0022] The present invention is further described below using drawings and examples. The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0023] In order to meet the low cost and high reliability principles in commercial applications, the system needs to have the lowest possible complexity. Under this premise, its main performance indicators are shown in Table 1. The performance requirements described in Table 1 can be summarized into the following five points:

[0024] (1) Lower system complexity, higher stability and reliability;

[0025] (2) It can complete real-time transcoding and transmission of multiple high-definition videos, support mainstream video coding standards and video resolutions, support mainstream network transmission protocols, and the transcoding and transmission delay is less than or equal to 200ms;

[0026] (3) The system can be accessed through a browser web page, providing a convenient and friendly way to access and configure the device;

[0027] (4) Compatible with network cameras, hard disk recorders and other devices from mainstream manufacturers;

[0028] (5) Video recordings can be stored and played locally.

[0029] Table 1 Main performance indicators of the system

[0030]

[0031] The encoding terminal needs to call the USB camera to obtain the original video data in RGB format. The RGB format video data stream cannot be directly applied to H.264 compression encoding. It needs to be converted into a YUV420 video stream through transcoding, and then compressed and encoded by X.264. The compressed data is transmitted to the server through the TCP protocol, and stored in the local folder for subsequent use. The decoding terminal needs to develop an APP to realize the functional requirements, connect to the client encoding terminal based on the TCP protocol and receive the H.264 data stream, and perform H.264 decoding through FFmpeg. The H.264 encoding structure is as follows: Figure 2 As shown in the figure, its basic process is: predictive coding, transform coding, and entropy coding. Basic framework: predictive coding, transform coding, and entropy coding. In the encoding process, there is no direct way to control the size of the encoded data. It can only be indirectly controlled by adjusting the quantization parameter QP value of the quantization process. The H.264 decoding structure is as follows Figure 3 As shown in the figure, the decoding process is as follows: entropy decoding, inverse quantization, and inverse transformation are performed in sequence in units of macroblocks to obtain residual data, which is then combined with the prediction information in the macroblock to find the decoded reference block, and then the decoded reference block and the residual data of this block are combined to obtain the actual data of this block. After the macroblock is decoded, slices are combined, and slices are combined to form an image. The decoded video data format is the YUV420 format. There are many sampling rates for the U and V components in the YUV format, and the common ones are YUV4:4:4, YUV4:4:2, and YUV4:2:0. Figure 4 shown.

[0032] The conversion formula between RBG format and YUV is shown in the formula:

[0033]

[0034] The video data volume in YUV420 format is large, which is not conducive to the real-time high frame rate refresh display of the control. Therefore, it is necessary to transcode the data stream in YUV420 format into RGB format, and then display it through the Qlabel control after transcoding. The bird-repelling video transmission system has four threads, including one main thread and three sub-threads. The main thread is responsible for the display of the interface, and the three sub-threads are compression encoding thread, network transmission thread and video decoding thread. The specific implementation process is as follows:

[0035] (1) Optimization of encoding terminal cache After encoding video data using the X.265 encoder, it is not sent directly through the TCP protocol. Instead, the data is first stored in the cache area. Under the premise of a normal network, data is continuously taken out from the cache area for sending, and the memory occupied by the sent data is released for caching subsequent data. The premise for this process to run normally is that the cache area is large enough to ensure that data will not be blocked when flowing in the cache area, resulting in slow data transmission. However, blindly increasing the cache is not a wise move, which will cause a waste of memory. The encoding terminal uses a circular FIFO (FIRST INFIRST OUT) data structure in the cache area for optimization. Its advantage is that it can ensure that the data in the cache area is queued and dequeued in order. Based on the first-in-first-out characteristics of the queue, the used cache can be quickly cleared to avoid blocking. In multi-threaded scheduling, the concurrency capability of the software and the operating efficiency of the system are improved, which plays a role in reducing transmission delays.

[0036] A circular FIFO is a data structure formed by connecting queues end to end. The data status in the queue can be quickly determined through pointers. Data enters from the head of the queue. Each time a data is inserted, the whole queue moves back one position. When it moves to the tail, it returns to the head position for processing. This return operation is performed through modulo. In the code implementation, the head q[0] and the tail q[length-1] of the array are connected, and two pointers head and tail are specified to point to the readable position q[x] and the writable position q[y] respectively, where the intervals of x and y are both in [0, length-1]. The key to the circular queue FIFO is to determine whether the queue is empty or full. When the pointer tail catches up with the head, it proves that the queue is full. When the pointer head catches up with the tail, it means that the queue is empty. If the queue is not full, write data. If the queue is not empty, read data. The circular FIFO structure diagram is as follows Figure 5 shown.

[0037] (2) The design uses TCP as the network transmission protocol. When packet loss and retransmission occur, the network transmission congestion control algorithm will adjust the slow start threshold ssthresh and the congestion window cwnd parameters, and these two parameters determine the amount of data transmitted. Congestion control algorithms can be divided according to packet loss and delay. Congestion control algorithms based on packet loss include TCPTahoe, TCPReno, TCPNewReno, etc., and congestion control algorithms based on delay are mainly TCPVegas.

[0038] In the congestion control algorithms of TCPTahoe, TCPReno, and TCPNewReno:

[0039] In the slow start state, the congestion window cwnd increases by one each time the sender receives an ACK until the value of cwnd is greater than or equal to ssthresh:

[0040] cwnd=cwnd+1

[0041] After a period of time, the value of the congestion window cwnd gradually increases. When it is greater than the slow start threshold ssthresh, the congestion avoidance phase is entered:

[0042]

[0043] If packet loss occurs during data transmission, fast retransmission is performed immediately. At the same time, the slow start threshold ssthresh needs to be reduced to cope with the change in data volume. It is usually modified to half of the congestion window cwnd, and then cwnd is set to 1:

[0044] ssthresh=cwnd / 2

[0045] cwnd=1

[0046] TCPReno adds a fast recovery algorithm, which greatly optimizes TCPTahoe. When three duplicate ACKs are received, it is assumed that the network is congested. Then fast retransmission is performed:

[0047] ssthresh = max(2, cwnd / 2)

[0048] cwnd=cwnd / 2

[0049] After fast retransmission, the system does not return to the slow start phase, but enters the fast recovery phase. When n duplicate ACKs (ndup) are transmitted:

[0050] cwnd=ssthresh+ndup

[0051] cwnd=ssthresh

[0052] TCPNewReno continues to optimize fast recovery. Compared with TCPReno and TCPNewReno, when receiving partial ACK, it will not exit the fast recovery phase immediately, but wait until all data packets are received. This allows errors to be corrected in advance when a large number of data packets are lost without waiting for timeout retransmission, thus improving link utilization.

[0053] (3) The decoding terminal is based on QT. QT's unique multi-threading mechanism means that the child thread cannot modify the UI main thread directly, and must redraw the UI interface through the QueuedConnection mechanism. This means that the data transmission process needs to be transferred through multiple intermediate threads, which has low transmission efficiency. The cumbersome data processing process increases system latency. The multi-threading mechanism of the decoding terminal is optimized by using a thread pool. The thread pool is a form of multi-threaded processing. During the processing process, tasks are added to the queue, and then these tasks are automatically started after the thread is created. Time-consuming tasks are processed in the subclass. After the processing is completed, the processing results are returned to the main thread through a specific method. After the main thread obtains the result, it can perform corresponding operations to update the UI interface.

[0054] The advantage of the thread pool mechanism is that it does not require the use of the asynchronous message transmission combination of UI thread + QueuedConnection. It improves processing efficiency based on asynchronous communication, avoids the loss of frequent thread creation and destruction, improves thread utilization and performance, and ensures thread safety. First, multiple cameras at the airport collect video. The process of sending video data is as follows: Figure 6 As shown in the figure, when sending video data, two data buffers are allocated. During the transmission process, three threads are needed to realize the transmission: the acquisition thread collects video data from the video data acquisition interface in real time; the encoding thread cooperates with the acquisition thread to encode the collected data and write the encoded data into the two buffers in turn; the data moving thread moves the data from the two buffers to the interface in turn. When receiving video data, three threads and two data buffers are also needed to match the transmission rate and the playback rate, and the mutex queue is used to control the read and write order of the two buffers.

[0055] The specific implementation method described above provides a detailed explanation of the purpose, technical scheme and beneficial effects of the technical invention. It should be understood that the above is only a specific implementation method of the technical invention and is not used to limit the protection scope of the technical invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the technical invention should be included in the protection scope of the technical invention.

Claims

1. A multi-threaded dynamic video transmission technology based on bird-repelling application is proposed, which is characterized by It can complete the real-time transcoding and transmission of multiple high-definition videos, use human-computer interaction to realize multi-threaded dynamic video transmission in the same window, and can realize the local storage function. The video transmission technology adopts encoding terminal cache optimization and congestion control algorithm optimization. The encoding terminal cache optimization applies specific data structures to cache design to improve the utilization rate of the cache area and speed up the data circulation speed; the congestion control algorithm optimization changes the slow start threshold value, and the data volume changes from exponential growth to linear growth, thereby reducing data fluctuations in the network and reducing the probability of packet loss and retransmission; QT's multi-threading technology is used to realize multi-threaded dynamic video transmission in the same window, improve the data transmission efficiency between multiple threads, speed up data circulation, avoid software crashes caused by memory overflow, reduce system delay time, and enable the client to perform multi-channel video transmission at the same time, realizing the functional requirements of real-time multi-channel video transmission in complex scenarios of bird-repelling applications.

2. According to claim 1, the encoding terminal cache optimization is characterized in that after the encoding terminal uses the X265 encoder to encode the video data, it is not sent directly through the TCP protocol, but first stores specific data in the cache area, and under the premise of normal network, continuously takes out data from the cache area for sending, and releases the memory occupied by the data that has been sent for caching subsequent data. The encoding terminal uses a circular FIFO (FIRST INFIRST OUT) data structure in the cache area for optimization, which has the advantage of ensuring that the data in the cache area can be queued and dequeued in order. Based on the first-in-first-out feature of the queue, the used cache can be quickly cleared to avoid blocking. In multi-threaded scheduling, the concurrency capability of the software and the operating efficiency of the system are improved, which plays a role in reducing transmission delay.

3. According to claim 1, the congestion control algorithm optimization is characterized in that When packet loss and retransmission occur, the slow start threshold ssthresh and congestion window cwnd will be adjusted, and these two parameters determine the amount of transmitted data. The coding terminal reduces the slow start process time by lowering the slow start threshold ssthresh, speeding up the algorithm to enter the congestion avoidance phase. After entering the congestion avoidance phase, the data volume changes from exponential growth to linear growth, thereby reducing data fluctuations in the network, increasing transmission stability, and reducing the probability of packet loss and retransmission, thereby reducing transmission delay.

4. According to claim 1, the multi-threading technology optimization is characterized by By creating a thread pool and processing time-consuming tasks in the subclass, the processing results are returned to the main thread through specific methods after the processing is completed. After the main thread gets the result, it can perform corresponding operations to update the UI interface. The advantage is that it does not require the asynchronous message transmission combination of UI thread communication + QueuedConnection, and can automatically manage the creation and destruction of threads, avoiding the loss of frequent creation and destruction of threads, improving thread utilization and performance, and ensuring thread safety.

Images