Expressway unmanned aerial vehicle inspection video self-adaptive transmission system

Abstract

The invention provides a highway unmanned aerial vehicle inspection video adaptive transmission system, and relates to the technical field of unmanned aerial vehicle video transmission, and the system comprises an unmanned aerial vehicle video collection end which is used for collecting video data of a to-be-detected region and carrying out the video coding and packaging of the video data; and sending the to-be-transmitted data to a streaming media server side through a 5G signal base station, monitoring the network state of the environment where the unmanned aerial vehicle video acquisition end is located to obtain monitoring data, and adjusting the video code rate of a video encoder of the unmanned aerial vehicle video acquisition end according to the monitoring data to obtain the video data of the unmanned aerial vehicle. And the ground station video receiving end receives the to-be-transmitted data forwarded by the streaming media server end in real time, and performs video decoding and storage operation on the to-be-transmitted data. According to the invention, the problems of poor remote video transmission stability and poor video image quality caused by a dynamic and complex mobile network environment of an unmanned aerial vehicle video transmission system in highway inspection operation in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) video transmission, and in particular to an adaptive transmission system for unmanned aerial vehicle (UAV) inspection videos of highways. Background Art

[0002] Traditional drone video transmission links mostly use radio communication methods. The image transmission data links of domestic consumer drones are mainly concentrated in the 2.4GHz and 5.8GHz frequency bands, namely the industrial, scientific and medical dedicated frequency bands, referred to as the ISM frequency band. The Interim Measures for the Administration of Radio Frequency of Civil Unmanned Aerial Vehicles clearly require that the data transmission power of drones in the 2.4GHz and 5.8GHz frequency bands shall not exceed 10mW and 25mW, and shall not cause spectrum interference to other frequency bands. This specification leads to low efficiency of drone video transmission, which can only achieve 1080p video transmission within a range of up to 5 kilometers, and the video quality and real-time performance cannot be guaranteed. With the emergence of the fifth-generation mobile communication technology (5G), with its significant advantages of high speed and low latency, the field of audio and video communication has officially entered the era of ultra-high-definition transmission. In recent years, 5G technology has attracted much attention and has developed rapidly. At present, my country has built a total of 3.189 million 5G base stations, which has laid the foundation for the widespread application of 5G technology. Compared with the previous generation of mobile communication technology, 5G technology has a wider network coverage and data transmission performance has been improved by nearly 100 times. Its peak rate can reach 20Gbps and the round-trip delay is only 1ms. It is suitable for use as a drone video transmission link that is sensitive to real-time.

[0003] Although 5G technology has overcome the technical difficulties that prevented wireless transmission of ultra-high-definition video in the past, it still faces the common pain point problem in mobile network communication methods, namely, the dynamic change of network bandwidth. Usually, external factors such as the geographical location, flight altitude, base station switching and weather of the drone will have a great impact on the network environment, and network congestion is very likely to occur, resulting in freezes, picture loss or poor image quality in the returned video. Therefore, it is necessary to optimize the video transmission process of the drone for the dynamically changing network environment. The commonly used method is to dynamically adjust the video transmission rate according to the changes in the network environment. With the radiance of machine learning technology in various fields of today's society, it also provides new ideas for the bit rate control method in streaming video transmission. Through machine learning, the changing trend of the network environment can be accurately judged, and the appropriate bit rate adjustment strategy can be made in time, which can effectively alleviate a series of adverse effects caused by network bandwidth fluctuations during real-time video transmission, thereby ensuring the remote real-time video transmission quality of drones in highway inspection application scenarios, and laying the foundation for the effective implementation of drone autonomous inspection work.

[0004] At present, in order to improve the performance of drone video transmission systems, major drone manufacturers around the world are committed to researching more efficient drone remote image transmission technology. Take DJI as an example. The most advanced digital image transmission solution OcuSync 3.0Plus (O3+) can achieve 60 frames of 1080p resolution video image transmission at a maximum distance of 15km under ideal conditions. In order to optimize the stability of video transmission, O3+ adopts 2.4 / 5.8GHz dual-band automatic switching technology, which can automatically select the best frequency band according to the environment, and perfectly utilizes the advantages of strong penetration of the 2.4GHz band and fast transmission speed of the 5.8GHz band. However, according to the experimental data provided by DJI, the efficiency and stability of video transmission of drone equipment equipped with the O3+ solution will be greatly reduced when working in an environment with interference or obstruction such as the periphery of a city, and its actual effective image transmission distance is only 3 to 9 kilometers. In order to meet the specific needs of certain industries, DJI has successively launched industry application version drone equipment that can be equipped with 4G LTE image transmission enhancement modules. By connecting to the 4G mobile network on the basis of the original image transmission solution, it overcomes the problem of stable transmission of video images when the digital image transmission signal is interfered with or exceeds the communication range. This type of drone usually automatically switches between the currently available 4G mobile network, 2.4GHz and 5.8GHz radio frequency bands according to the flight environment of the equipment to maintain the stability of the drone control signal and the image transmission signal. Although this solution breaks through the distance limitation of the traditional drone image transmission system, it can only transmit 1080p resolution video images remotely, and there is no flexible video traffic control strategy designed for video transmission in a dynamic 4G mobile network environment. Once the available bandwidth of the 4G mobile network drops sharply, it is very easy to cause network congestion, resulting in problems such as drone video transmission failure or low quality of returned video, which is unacceptable in highway inspection operations. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a highway drone inspection video adaptive transmission system, which solves the problem of poor remote video transmission stability and poor video image quality caused by the dynamic and complex mobile network environment in the prior art of drone video transmission systems during highway inspection operations.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A highway drone inspection video adaptive transmission system, comprising:

[0008] The drone video acquisition end, streaming media server end and ground station video receiving end are connected in sequence;

[0009] The drone video acquisition end is used to collect video data of the area to be tested and perform video encoding and packaging processing on the video data, obtain the data to be transmitted and send the data to be transmitted to the streaming media server end through the signal base station, and monitor the network status of the environment in which the drone video acquisition end is located to obtain monitoring data, and adjust the video bit rate of the video encoder of the drone video acquisition end according to the monitoring data. The streaming media server end is used to receive the data to be transmitted, and transmit the data to be transmitted through the network firewall to the ground station video receiving end. The ground station video receiving end is used to perform data unpacking and video decoding processing on the data to be transmitted, obtain video data, and store and process the video data.

[0010] Preferably, the drone video acquisition terminal includes:

[0011] A network monitoring module, an adaptive bit rate control module, and an image acquisition module, a video encoding module, a data encapsulation module, a video streaming module, and a video storage module connected in sequence;

[0012] The image acquisition module is used to acquire video data, the video encoding module is used to encode the video data to obtain encoded data, the data encapsulation module is used to encapsulate the encoded data to obtain data to be transmitted, the video streaming module is used to transmit the data to be transmitted to the streaming media server through a signal base station, the network monitoring module is used to monitor the network status of the environment in which the drone video acquisition end is located to obtain monitoring data, and the adaptive bit rate control module is used to control the video bit rate of the video encoding module according to the monitoring data, and dynamically adjust the optimal output bit rate of the video encoding module.

[0013] Preferably, the network monitoring module includes:

[0014] Network information acquisition submodule and evaluation module;

[0015] The network information acquisition submodule is used to obtain real-time network information through the RTCP protocol and send the real-time network information as a state parameter characterizing the network conditions to the intelligent agent to obtain initial data. The evaluation module is used to provide reward feedback to the intelligent agent based on the QoE evaluation model as a reward function and update the state space and action space, wherein the three evaluation indicators of the QoE evaluation model include: video quality, video smoothness and video delay.

[0016] Preferably, the expression of the state space is:

[0017]

[0018] Among them, di, ji, li, ri represent the network transmission delay, network jitter, packet loss rate and actual video bit rate of the previous sampling period respectively, i = 1, 2..8, i represents the sampling order.

[0019] Preferably, the QoE model expression of the video quality is:

[0020]

[0021] Among them, Q is the video quality.

[0022] Preferably, the QoE model expression of the video smoothness is:

[0023]

[0024] Where S is the video smoothness.

[0025] Preferably, the QoE model expression of the video delay is:

[0026]

[0027] Where D is the video delay.

[0028] Preferably, the input of the neural network of the UABR model is a parameter matrix with a dimension of 4 and a length of 8.

[0029] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0030] The present invention provides an adaptive transmission system for highway drone inspection videos. The present invention improves the video data transmission link adopted by the traditional drone video transmission system, solves the problem of limited transmission distance in the existing drone video transmission scheme through 5G mobile network communication mode and streaming media video transmission technology, and realizes drone video transmission beyond visual range; proposes a bit rate adaptive control algorithm suitable for the drone video transmission system, which intelligently adjusts the output bit rate of the video encoder by real-time sensing of the mobile network status of the drone, thereby avoiding network congestion caused by changes in mobile network bandwidth, thereby affecting the stability of video transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1A schematic diagram of the structure of a highway drone inspection video adaptive transmission system provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the deep reinforcement learning model structure of the UABR algorithm provided in an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the neural network structure of the UABR algorithm provided in an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the L3VTP simulation system model structure provided by an embodiment of the present invention;

[0036] Figure 5 A schematic diagram of an entropy value curve provided by an embodiment of the present invention;

[0037] Figure 6 A schematic diagram of a reward value curve provided by an embodiment of the present invention;

[0038] Figure 7 A schematic diagram showing a comparison between target bit rates and actual bandwidths of different algorithms provided in an embodiment of the present invention;

[0039] Figure 8 A schematic diagram of bandwidth estimation error between UABR and GCC on a validation set provided by an embodiment of the present invention;

[0040] Fig. 9 A schematic diagram of the QoE indicators of UABR and GCC on a validation set provided by an embodiment of the present invention;

[0041] Fig.10 A schematic diagram of QoE probability distribution on a validation set provided by an embodiment of the present invention;

[0042] Fig.11 A schematic diagram of the hardware framework of the drone video acquisition terminal provided by an embodiment of the present invention;

[0043] Fig.12 A schematic diagram of a multi-threaded design of a video acquisition terminal provided by an embodiment of the present invention;

[0044] Fig.13 A schematic diagram of a circular queue buffer provided by an embodiment of the present invention;

[0045] Fig.14 A schematic diagram of the V4L2 video acquisition process provided by an embodiment of the present invention;

[0046] Fig.15 A schematic diagram of the HEVC video encoding process provided by an embodiment of the present invention;

[0047] Fig.16A schematic diagram of the RTSP video streaming program flow provided by an embodiment of the present invention;

[0048] Fig.17 A schematic diagram of UABR model transplantation provided in an embodiment of the present invention;

[0049] Fig.18 A schematic diagram of the RKNN-Toolkit2 conversion model process provided by an embodiment of the present invention;

[0050] Fig.19 A schematic diagram of the RKNN model deployment process provided by an embodiment of the present invention;

[0051] Fig. 20 A schematic diagram of the RTSP streaming media protocol interaction process provided by an embodiment of the present invention;

[0052] Fig.21 A schematic diagram of a data packet forwarding process provided by an embodiment of the present invention, wherein: Fig.21 (a) is a schematic diagram of the receiving process. Fig.21 (b) is a schematic diagram of the data packet sending process;

[0053] Fig. 22 A schematic diagram of a VLC video receiving process provided by an embodiment of the present invention;

[0054] Fig.23 A schematic diagram of PSNR analysis of 300 frames of video provided by an embodiment of the present invention;

[0055] Fig.24 A schematic diagram of SSIM analysis of 300 frames of video provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] like Figure 1 As shown, the present invention provides a highway drone inspection video adaptive transmission system, comprising:

[0059] The drone video acquisition end, streaming media server end and ground station video receiving end are connected in sequence;

[0060] The drone video acquisition end is used to collect video data of the area to be tested and perform video encoding and packaging processing on the video data, obtain the data to be transmitted and send the data to be transmitted to the streaming media server end through the 5G signal base station, and monitor the network status of the environment in which the drone video acquisition end is located to obtain monitoring data, and adjust the video bit rate of the video encoder of the drone video acquisition end according to the monitoring data. The streaming media server end is used to receive the data to be transmitted, and transmit the data to be transmitted through the network firewall to the ground station video receiving end. The ground station video receiving end is used to perform data unpacking and video decoding processing on the data to be transmitted, obtain video data, and store and process the video data.

[0061] Specifically, the drone video acquisition terminal mainly completes the tasks of image acquisition, video encoding and packaging, and video streaming. It uses the 5G mobile network as the data transmission link and sends the RTSP video stream to the streaming server through the signal base station. At the same time, the bit rate adaptive algorithm monitors the network status of the drone's environment in real time through the RTCP protocol, and intelligent decisions can match the video bit rate of the current network conditions, thereby optimizing the video transmission quality. The streaming server deployed in the cloud completes the remote forwarding task of the RTSP video stream, and transmits the video stream to the ground station receiving end after filtering through the network firewall. The ground station receives the video stream forwarded by the streaming server in real time, plays the video images taken by the drone in real time after data unpacking, video decoding and other links, and stores the received video locally for backup management.

[0062] Furthermore, the drone video acquisition terminal includes:

[0063] A network monitoring module, an adaptive bit rate control module, and an image acquisition module, a video encoding module, a data encapsulation module, a video streaming module, and a video storage module connected in sequence;

[0064] The image acquisition module is used to acquire video data, the video encoding module is used to encode the video data to obtain encoded data, the data encapsulation module is used to encapsulate the encoded data to obtain data to be transmitted, the video streaming module is used to transmit the data to be transmitted to the streaming media server through a signal base station, the network monitoring module is used to monitor the network status of the environment in which the drone video acquisition end is located to obtain monitoring data, and the adaptive bit rate control module is used to control the video bit rate of the video encoding module according to the monitoring data, and dynamically adjust the optimal output bit rate of the video encoding module.

[0065] Specifically, in the actual application scenario of highway inspection, the mobile network bandwidth of the drone will be in a dynamic change process with multiple factors. Timely adjusting the video bitrate of the encoder according to the state of the network environment can effectively avoid the risk of video buffer overflow, and is also an effective way to improve the stability of video transmission and reduce video delay. The present invention transforms the bitrate control problem in the real-time video transmission scenario of drones into a reinforcement learning model, and proposes a drone video bitrate adaptive algorithm (UAVAdaptive Bitrate, UABR) based on network perception.

[0066] Furthermore, the network monitoring module includes:

[0067] Network information acquisition submodule and evaluation module;

[0068] The network information acquisition submodule is used to obtain real-time network information through the RTCP protocol and send the real-time network information as a state parameter characterizing the network conditions to the intelligent agent to obtain initial data. The evaluation module is used to provide reward feedback to the intelligent agent based on the QoE evaluation model as a reward function and update the state space and action space, wherein the three evaluation indicators of the QoE evaluation model include: video quality, video smoothness and video delay.

[0069] The UAV video bitrate adaptation algorithm interacts with the environment to obtain input status, establishes a reasonable QoE model and bitrate selection space, and uses deep reinforcement learning to explore the best bitrate control strategy. The intelligent agent outputs the video bitrate that can match the current network environment to the encoder according to the learning strategy. The deep reinforcement learning model of the UABR algorithm is as follows: Figure 2 shown.

[0070] The UABR model uses drones as an interactive environment. The network monitoring module deployed on the drone obtains real-time network information by obtaining RTCP packets fed back from the ground station when receiving videos, and sends them to the agent as state parameters representing network conditions, and uses the QoE evaluation model as a reward function for reward feedback. At the same time, the agent will select an appropriate bitrate value as the action output based on the learning strategy, and the video encoder uses this bitrate as the target value for video encoding. This paper selects a time granularity of 1s to update the bitrate. While updating the bitrate, it feeds back the QoE evaluation parameters to the agent and updates the state space, and then continues to make the next round of bitrate decisions.

[0071] The UABR algorithm defines the input state of the agent as the real-time network state information obtained by the network monitoring module through the RTCP packet in the past 1 second, including four dimensions: network transmission delay, network jitter, packet loss rate, and actual video bit rate. Among them, each RTCP packet contains 8 network parameters corresponding to RTP packets, so the state space can be expressed by formula (1). The expression of the state space is:

[0072]

[0073] Among them, di, ji, li, ri represent the network transmission delay, network jitter, packet loss rate and actual video bit rate of the previous sampling period respectively, i = 1, 2..8, i represents the sampling order.

[0074] Since the dimensions of the state parameters of each dimension are different, directly sending the state matrix into the neural network is likely to cause unstable learning or low learning efficiency. Therefore, in reinforcement learning, each element of the state space is usually normalized to avoid unforeseen interference of singular data on the learning process. The method for normalizing the state parameters is shown in formula (2).

[0075]

[0076] UABR defines the output action as the bit rate value to be selected by the video encoder at the next moment, that is, to determine the bit rate range {A min ,A max} a specific discrete value within A min With A max The choice needs to strike a balance between network resources and video image quality. In order to determine a reasonable bit rate range for algorithm research, the present invention selects a road video collected on-site at Section 1408B of G22 Expressway as a test object. By analyzing the image quality before and after video encoding at different bit rates, it is finally determined that the action space of UABR can be defined as a discrete set of available bit rate experience values: {5000, 10000, 15000, 20000, 25000, 30000} kbps as alternatives for video bit rates. The output strategy of UABR is the probability distribution matrix corresponding to each alternative bit rate under the input state at time t, and finally the bit rate value with the largest probability is taken as action a. t Output, that is, the video bit rate r to be selected in the state space at time t t At this point, the encoder updates the bit rate configuration and sets r t The target bit rate is used for video encoding.

[0077] The reward function can directly reflect the quality of the bitrate selection strategy, and is consistent with the goal of the ABR algorithm to improve the QoE index, so the QoE evaluation model can be used as the reward function. This algorithm focuses on three evaluation indicators of the real-time video transmission system when constructing the QoE model, namely video quality Q, video smoothness S, and video delay D.

[0078] The video bit rate is positively correlated with the data size of the video frame. Videos with high bit rates usually have higher image quality. Therefore, the video quality Q can be regarded as the average value of the actual video bit rates fed back by N RTP packets in the past time granularity. When the network environment is poor, the bit rate needs to be reduced to avoid network congestion. Conversely, the bit rate needs to be increased to reduce bandwidth waste. The QoE model of video quality Q is shown in formula (3).

[0079]

[0080] Video smoothness S is the amplitude of the switching bitrate of continuous video clips, which can be reflected by the absolute value of the difference between the video bitrates of two adjacent RTP packets. The smaller the value of S, the smaller the fluctuation range of the video quality Q. Smooth video quality helps to improve the user's sensory experience. The QoE model of video smoothness S is shown in formula (4).

[0081]

[0082] Video delay D is the average of the video transmission delays fed back by the past N RTP packets. The smaller the video delay, the smoother the video playback, and vice versa. The QoE model of video delay D is shown in formula (5).

[0083]

[0084] Combining the above three evaluation indicators, the present invention defines the overall QoE model of the UABR algorithm as:

[0085]

[0086] Among them, N represents the length of the state sequence of the same dimension in the state space, and α, β, and λ are the weight factors of video quality Q, video smoothness S, and video delay D. After multiple sets of experimental comparisons, a set of better weight factor combinations was finally determined, that is, the values ​​of α, β, and λ are 8, 2, and 13 respectively.

[0087] Proximal Policy Optimization (PPO) is a gradient optimization algorithm based on the Actor-Critic framework. In order to overcome the problem of large learning step size leading to model convergence difficulties when updating the gradient of previous reinforcement learning policies, PPO improves the objective function so that the policy update is always in a more beneficial direction than the old policy. In addition, the PPO algorithm uses importance sampling to process the action distribution and uses the samples generated by the old policy to update the policy parameters, which improves the utilization efficiency of sample data. The loss function is shown in formula (7).

[0088]

[0089] in, is the ratio of the new and old strategies, A(s t ,a t ) is the advantage function, as shown in formula (8).

[0090] A(s t ,a t )=r t +γV(s t+1 )-V(s t ) (8)

[0091] The PPO algorithm introduces two variants: proximal policy optimization penalty (PPO-penalty) and proximal policy optimization clipping (PPO-clip). The objective function of PPO-penalty is shown in formula (9).

[0092]

[0093] The value of the penalty coefficient β changes with the KL divergence. If it exceeds the KL maximum target value, the value of β is increased to update the parameters in the direction of reducing the KL divergence, and vice versa.

[0094]

[0095] The loss function of PPO-clip is shown in formula (10). By clipping the boundaries, the update step is limited to the range of (1-ε, 1+ε), which limits the difference between the new and old strategies and effectively improves the problems of high sampling difficulty and difficult model convergence. The relatively smooth strategy update means that the span of the output action is smaller, which helps to improve the smoothness of video playback.

[0096] Considering that this algorithm will eventually be deployed on drones, in order to reduce the amount of model calculations and eliminate the impact of hardware platform resource constraints, UABR selected a relatively lightweight two-layer fully connected neural network for feature extraction. The neural network structure is as follows: Figure 3As shown in the figure. The input of the neural network is a parameter matrix with a dimension of 4 and a length of 8. The normalized state matrix is ​​first flattened to obtain 32 eigenvectors, which are then sent to the neural network for learning. Since the PPO algorithm adopts the Actor-Critic framework, the network model consists of two parts: the Actor network and the Critic network. The Actor network contains an input layer, two cascaded fully connected layers FC1 and FC2, and a 1*6 output layer.

[0097] The number of neurons in the fully connected layers FC1 and FC2 is 128 and 64 respectively, and the activation function uses the LeakyReLU function. The output layer uses the Softmax activation function, and the 6 output nodes correspond to the probability distribution of each video bitrate option, and the video bitrate with the highest probability is taken as the action output. In addition, the structure of the Critic network is basically the same as that of the Actor network, but the output of the Critic network is a scalar value, which is only used to judge the pros and cons of the Actor network for the current action selection strategy, while the Actor network will refer to the value of value for subsequent gradient updates.

[0098] The idea of ​​adaptive bitrate control is to approximately match the video transmission bitrate with the actual network bandwidth, and balance the relationship between video quality and network status without wasting network bandwidth, so as to ensure the quality of video transmission and improve the quality of user experience (Quality of Experience, QoE). Therefore, the optimization goal of the adaptive bitrate algorithm of the present invention is to obtain the maximum QoE return. The core idea of ​​the UABR algorithm is to estimate the optimal bitrate that can be selected at the next moment by monitoring the real-time network status of the drone, and adjust the target output bitrate of the video encoder in time to avoid network congestion at the source.

[0099] Thus, the QoE performance is optimized. Table 1 UABR algorithm training process

[0100]

[0101] The training process of the UABR algorithm is shown in Table 1. After completing the parameter initialization of the Actor network and the Critic network, the UABR algorithm receives the normalized network state parameter S transmitted by the network monitoring module. t , and use the current policy network to generate the corresponding action probability distribution, and then interact with the environment to collect experience data samples. After completing a round of sample collection, start calculating the advantage function A of each set of input state and output action t, calculate the policy gradient according to the probability distribution of the advantage function and the old policy, and limit the update amplitude of the policy network through the PPO-clip objective function clipping strategy to maintain training stability. After repeatedly executing the above process, stop training until the policy reaches a convergence state or the reward value is as large as possible and tends to be stable.

[0102] Training the UABR algorithm in a real video transmission system requires additional server resources, and collecting the required data samples will take a lot of time, and the learning efficiency is relatively low, so it cannot meet the training requirements. Therefore, this paper chooses to study and implement the UABR algorithm on the L3VTP real-time video simulation platform. The Python programming language is used to build an L3VTP simulator environment to simulate the video transmission process of the drone to complete the algorithm training. The system model of the L3VTP simulation platform is as follows: Figure 4 shown.

[0103] In an actual real-time streaming video transmission system, the terminal device usually collects video images in real time, and the original video stream is continuously generated by the encoder to generate video frames, which are then encapsulated into data packets corresponding to the transmission protocol for network transmission. The server transfers the bitstream data packets and finally completes the real-time decoding and playback of the video images at the receiving end. The L3VTP simulation system describes the video acquisition, encoding and bitstream encapsulation process of the sender using the video fragmentation track, while the network transmission process of the video stream is described by the network bandwidth track. The video bitrate during the video transmission process is adjusted through the bitrate control algorithm module, and finally the transmission and playback process of the dynamic video source is simulated by combining the trajectory data of the video fragmentation and the network bandwidth.

[0104] After multiple sets of experimental tests, the optimal parameter settings for UABR algorithm training are finally determined as shown in Table 2.

[0105] Table 2 Training parameter settings

[0106]

[0107]

[0108] In order to improve the algorithm's exploration ability when learning in a variety of network environments and avoid the problem of local optimality in the loss function, UABR introduces a weighted entropy regularization term in the loss function of the Actor network. Under the same environment configuration, three different random number seeds are used for training, and the mean and standard deviation statistics of three sets of parallel experimental data are calculated. Finally, the change curves of entropy value and reward value during model training are plotted. Figure 5 , Figure 6 shown.

[0109] The larger the entropy value corresponding to the strategy, the greater the distribution difference between the predicted value and the true value of each action under the current strategy, and the result of the model prediction cannot be determined. Figure 5 , Figure 6 It can be seen that when the training reaches 25,000 rounds, the entropy value drops to a lower level, and the reward value at this time also tends to converge, indicating that the uncertainty of the model is reduced, and it can take appropriate action selection strategies according to complex input states to obtain the best reward. Therefore, the neural network model trained in the 25,000th round is selected as the best model.

[0110] In order to carry out the performance evaluation of the UABR algorithm, this paper selected the GCC algorithm, which is currently widely used in many video live broadcast platforms, for comparative experiments. GCC is a congestion control algorithm designed for the WebRTC protocol proposed by Google. It deploys packet loss rate-based and delay-based network bandwidth estimation modules at the video sender and receiver respectively. It controls the video sending rate by combining the bandwidth estimation results of the two. It is a rate adaptive algorithm based on rule-based protocols. The original rate control module of L3VTP was replaced by the UABR and GCC algorithms respectively, and tested under the same environmental configuration conditions. Finally, the actual performance of the two algorithms was statistically analyzed to obtain the comparison data of the target bit rate output by each algorithm and the actual network bandwidth. A 600-second data segment was captured as shown in the figure. Figure 7 shown.

[0111] from Figure 7 The results show that since the GCC algorithm uses a fixed rule to estimate the network bandwidth, it exhibits a "slow rise, fast fall" control mode. When the network status is poor, the target bit rate of the GCC algorithm will drop rapidly. However, during the rising process when the network recovers, the performance of the GCC algorithm is relatively sluggish, and it takes a long time to rise to a normal level. Therefore, in application scenarios with large network fluctuations, it is difficult for the GCC algorithm to ensure that the target bit rate matches the actual network bandwidth, especially in high-speed dynamic networks such as 4G and 5G, which is more obvious and not conducive to the full utilization of network resources. Under the strategy of maximizing QoE evaluation, the UABR algorithm directly drives the model to output a reasonable target bit rate through real-time network status parameters, effectively improving the problems of slow bit rate improvement and inaccurate bandwidth matching of the GCC algorithm.

[0112] Figure 8 , Fig. 9 A 600-second local segment of the UABR algorithm experiment on the validation set is given, showing the network bandwidth estimation error and user experience quality of UABR and the comparison algorithm GCC at each time granularity. The bandwidth estimation error of the UABR algorithm is much smaller than that of the GCC algorithm, and most of the bitrate selection actions obtain better QoE returns.

[0113] Table 3 Performance comparison between UABR and GCC

[0114]

[0115] Table 3 gives the experimental results of the two algorithms on the validation set. The experimental results show that the UABR algorithm can effectively reduce the network prediction error compared with the GCC algorithm, and all performance indicators have been greatly improved. The GCC algorithm is more conservative in bit rate control, and the bit rate recovery speed when the network recovers is very slow. Therefore, when the fluctuation range of the mobile network is large, the GCC algorithm often ushered in the next bit rate attenuation before recovering to the peak rate, which inevitably wastes network resources and the video quality cannot achieve the expected effect. In addition, when the network bandwidth drops sharply, the GCC algorithm cannot immediately drop to a suitable bit rate. At this time, the transmission delay will increase due to channel saturation. On the contrary, the UABR algorithm has significant advantages in mobile networks. It can quickly respond to changes in dynamic networks and adjust the bit rate in time, balancing the utilization of video quality and network conditions, and alleviating network congestion problems to the greatest extent.

[0116] The probability distribution of QoE indicators of UABR and GCC algorithms on the validation set is shown in the figure below: Fig.10 As shown. Since the QoE model established by UABR takes video smoothness into account, when the network bandwidth decreases from the peak rate to the minimum value, the span of the bitrate values ​​output before and after UABR is too large, resulting in an increase in the smoothness penalty term. Therefore, in the interval [-40,55], the QoE distribution probability of the UABR algorithm is higher than that of the GCC algorithm. However, in the high score interval [50,240], the QoE index of UABR is significantly better than that of the GCC algorithm. Statistics show that the average QoE of the UABR algorithm on the validation set is 140.873, which is 27.42% higher than the 110.556 of the GCC algorithm.

[0117] Furthermore, the present invention selects the domestic high-performance embedded processor Rockchip RK3588S as the core processor of the drone video acquisition terminal. The terminal is equipped with a Sony IMX415 camera to collect 4K video images, and the 5G mobile network communication is realized through the Quectel RG200U module. A large-capacity MicroSD card is installed to store the video stream for local backup, and relevant debugging modules are reserved for development work. The hardware framework of the drone video acquisition terminal is as follows: Fig.11 shown.

[0118] The terminal uses the Sony IMX415 camera of the MIPI CSI protocol to collect ultra-high-definition video images with a resolution of 3840x2160 and 30 frames. The kernel driver V4L2 of the Linux device controls the camera to capture the original video stream in YUV420 format. The video processing unit integrated in the RK3588S performs real-time encoding and compression of the YUV original video stream in HEVC format, and then pushes the compressed h265 video stream package to the streaming media server. At the same time, the MicroSD storage device is used to back up the generated h265 video stream so that the video data can be restored when special needs arise. The drone video acquisition terminal consists of six parts: the RK3588S embedded development platform, the IMX415 camera, the RG200U mobile communication module, two sets of IPEX antennas and a 5V power supply.

[0119] Furthermore, the drone video acquisition terminal is the core part of the present invention, which needs to undertake the tasks of video acquisition, encoding compression, bit stream transmission and reasoning of the neural network model. It also needs to consider the protocol adaptation issues of the camera module and the 5G communication module. Therefore, the selection of the embedded development platform is particularly important.

[0120] RK3588S is an 8-core 64-bit high-performance embedded processor based on ARMv8 architecture launched by Rockchip of China. It has a main frequency of up to 2.4GHz and supports LPDDR4 memory technology and eMMC storage technology with higher data transmission rate. RK3588S adopts 8nm process and integrates high-performance GPU and NPU internally, with excellent performance in graphics processing and neural network computing. In addition, it has a video processor (VPU) that supports 8K video encoding and decoding capabilities, and a 48M pixel image signal processor (ISP), which provides strong support for embedded devices in the field of video development.

[0121] The camera model used in the system is IMX415-AAQR-C. IMX415 is an 8-megapixel 4K ultra-clear camera that uses the MIPI CSI high-speed transmission protocol and supports video capture at up to 30fps@3864*2192 resolution. In addition, it uses a Sony 1 / 2.8-inch active pixel solid-state CMOS sensor with a 119.8° wide-angle field of view and f2.5 aperture. The module size is only 31*29mm, which meets the application requirements of the present invention.

[0122] The 5G communication module is used to achieve Internet access for drone video acquisition terminals, and transmit the encoded and compressed video stream to the streaming media server through the Internet, thereby realizing the remote drone video transmission task. The 5G module model used in the system is Quectel RG200U-CNV1.

[0123] RG200U is a 5G module designed for IoT applications. It supports both NSA and SA networking modes and is backward compatible with 4G / 3G networks. The theoretical data transmission rate in 5G SA mode can reach: 1Gbps uplink and 2Gbps downlink. It is worth mentioning that RG200U adopts the packaging format of Mini PCIe protocol, which reduces the module volume to 30.7*50.95*5.3mm, and only 2 antennas are needed to achieve high-speed 5G network communication, which can further reduce the volume and weight of the drone video acquisition terminal of the present invention.

[0124] The drone video acquisition terminal mainly completes the acquisition, encoding and streaming of video images. It transplants the UABR algorithm to adaptively control the video bit rate output by the encoder, uses the Linux device kernel driver V4L2 to control the camera to capture the original video stream in YUV420 format, and the video processing unit integrated in RK3588S performs real-time encoding and compression of the YUV video stream in HEVC format. Finally, the compressed h265 video stream is packaged and pushed to the streaming media server.

[0125] The drone video acquisition terminal needs to complete three tasks at the same time: video acquisition and encoding, bitstream encapsulation and streaming, and network status monitoring and bitrate control. In order to achieve multi-task concurrency to speed up the system response, the present invention adopts a multi-threaded development method in the application layer of the embedded board, and divides the task functions into three threads: VideoEncoding, StreamingTransmission and AdaptiveBitrate. Fig.12 shown.

[0126] When the Linux system is powered on, the terminal creates three threads for execution. The VideoEncoding thread is responsible for the acquisition and real-time encoding of video images. It compresses the input YUV format raw video into an h265 video stream through HEVC encoding and stores it on the local MicroSD device for video backup. The StreamingTransmission thread completes the task of encapsulating the h265 video stream into the RTP data packet, and then pushes the stream to the streaming media server with the help of FFmepg using the RTSP protocol, thereby completing the remote transmission task of the drone video image. The AdaptiveBitrate thread feeds back the video receiving end.

[0127] The network status information in the RTCP packet is passed to the UABR algorithm as the input parameter of the model. The output video bit rate will be used as the target bit rate adopted by the encoder at the next moment, thereby realizing adaptive bit rate control in a dynamic network environment.

[0128] There are speed differences in data transmission between threads of a multi-threaded program, and the amount of video stream data produced by the VideoEncoding thread is usually large, which is prone to data loss and overload problems. Therefore, a circular data buffer is used between the VideoEncoding thread and the StreamingTransmission thread of the drone video acquisition end to balance the speed differences of data transmission between multiple threads to ensure stable operation of the system. Fig.13 shown.

[0129] The circular queue buffer consists of an array of fixed length and two pointers dedicated to reading and writing data. The read pointer points to the starting position of the memory space where the buffer stores data, while the write pointer points to the end of the memory space where the data is stored. Each time the circular buffer writes or reads data, the write pointer or read pointer moves to the next position until the two pointers meet again to start a new round of data caching. Since the circular data buffer uses a fixed-size array to store data, the working mode of recycling storage space can effectively avoid unnecessary system overhead caused by dynamic memory allocation and memory fragmentation caused by memory release. For real-time video transmission systems, the use of a circular data buffer can significantly reduce the synchronization overhead between multi-threaded concurrent tasks, thereby reducing video transmission latency.

[0130] The drone terminal uses the IMX415 camera as an input device to collect video images. V4L2 is a video device driver framework dedicated to the Linux system. It realizes the driving of the underlying device by providing a set of standardized video device application program interfaces. The program flow of the present invention using V4L2 for video acquisition is as follows: Fig.14 shown.

[0131] V4L2 will first call the open() function to open the / dev / video11 camera device in the system root directory, query the relevant information of the video11 device and the video format information it supports, and set the capture parameters of the video image through the ioctl() function, such as the output format, image size and other configuration parameters. The VIDIOC_REQBUFS command is used to apply to the driver for the buffer space in the kernel for storing video frames, and the number of buffers is set to 4. Then, the length and address information of each buffer are obtained through VIDIOC_QUERYBUF, and the kernel buffer is mapped to the user space using the mmap() function. After that, the frame buffer is sent to the kernel queue to prepare for video acquisition. When the frame buffer is ready, the video stream capture is started through the VIDIOC_STREAMON command, and the frame buffer is written with video data. The VIDIOC_DQBUF operation is used to read the video data in the obtained frame buffer, and the video frame is written to the file and stored through the write() function. Then, the VIDIOC_QBUF command is used to send the buffer back to the kernel queue, thereby realizing the reuse of the frame buffer. When the video acquisition is completed, call VIDIOC_STREAMOFF to stop the video acquisition, use the munmap() function to release the memory mapping of each frame buffer, and finally close the camera device through the close(fd) function.

[0132] HEVC is a new generation of high-efficiency video codec standard, which introduces more advanced image coding technology and can efficiently compress ultra-high-definition video with 4K resolution. The RK3588S processor used in the present invention integrates two independent video processing units (VPUs) internally, which can complete hardware video encoding and decoding of up to 8K@60fps H.265, and is suitable for real-time encoding of ultra-high-definition video on embedded mobile terminals. In addition, Rockchip provides a dedicated MPP media processing platform for multimedia applications of its chips, which shields the complex logic processing related to the underlying chip and improves the user's development efficiency by providing a unified MPI media processing interface. Therefore, the video hardware encoding task of the present invention is completed based on Rockchip's RKMPP library. The HEVC video encoding process is as follows: Fig.15 shown.

[0133] MPP first initializes the device through the test_ctx_init() function, completes the reading of the camera device and the setting of the buffer, configures the encoder parameters in the test_mpp_enc_cfg_setup() function, including setting the video resolution, frame rate, bit rate, and GOP structure, and then allocates memory space for storing video frames and encoding packets to the encoding task. After the preparation is completed, the encoding task begins, loops to determine whether the input device has new video frames, and reads frame data from the camera through the camera_source_get_frame() function. Call the mpp_frame_init() function to configure the various attributes of the video frame for initialization, then fill the read frame data into the frame buffer, pass the frame data to the encoder through mpi->encode_put_frame(ctx,frame) for HEVC frame layer encoding, and finally call mpi->encode_get_packet(ctx,&packet) to obtain the encoded packet and write it to the file for output. If the camera no longer generates new video frames, release the acquired memory resources and close the encoder to end the encoding task.

[0134] The video streaming module of the present invention is implemented based on the open source multimedia processing framework FFmpeg, and completes the video transmission task of real-time video stream from the drone terminal to the streaming media server through the RTSP real-time transmission protocol. The RTSP video streaming program flow based on FFmpeg is as follows: Fig.16 shown.

[0135] Before pushing the video, use the avformat_network_init() function to initialize the FFmpeg library, complete the registration of the encoder, protocol, etc., and configure the network port and target IP parameters of the RTSP push stream. Then use the avformat_open_input() function to open the RTSP input stream, write the input stream data into the structure AVFormatContext, and call the avformat_find_stream_info() function to obtain the input video stream, then use the avformat_new_stream() function to create the output stream, and use the avcodec_parameters_copy() function to copy the input stream data to the output stream. Then use the avio_open() function to open the RTSP output stream URL, use the avformat_write_header() function to write the output stream file header, use av_read_frame() to read a frame of video data and store it in the AVPacket structure, then call the av_interleaved_write_frame() function to send the video data stored in the AVPacket to the server, and start reading the next frame of data. When there is no new input stream to write, the av_write_trailer() function writes the tail of the output stream file, and finally the avformat_free_context() function ends the push and releases the FFmpeg resources.

[0136] The UABR algorithm proposed in this paper adopts a relatively lightweight neural network structure. The trained neural network model is now transplanted to the embedded mobile terminal. The RK3588S development platform has 16GB LPDDR4 memory and 128GB eMMC flash memory onboard, and uses the Ubuntu 22.04 version of the system. Since the RK3588S integrates a 6TOPS NPU, it can be used to accelerate the neural network reasoning tasks on embedded devices. The transplantation process of the UABR algorithm is as follows: Fig.17 shown.

[0137] The UABR algorithm uses the TensorFlow framework to train the neural network model on the PC. To deploy it on the mobile board, you need to first use the RKNN-Toolkit2 tool officially provided by Rockchip to convert the trained TensorFlow neural network model into an RKNN model, and then use the RKNN API interface to call the NPU to accelerate the model reasoning. The process of RKNN-Toolkit2 for neural network model conversion is as follows: Fig.18 shown.

[0138] Before model conversion, use the RKNN() constructor to initialize the RKNN object, configure the entry parameters verbose and verbose_file to "True" and ". / build.log" respectively to specify that the log information generated when building the model will be written into the build.log file. Then configure the conversion parameters of the neural network model through the rknn.config() interface, specify the target_platform of the RKNN model as "RK3588", the quantization parameter algorithm quantized_algorithm selects "normal", the quantization type quantized_dtype is "INT8", etc. After the configuration is completed, load the specified tensorflow neural network model as input, and after RKNN-Toolkit2 builds the target rknn model according to the expected configuration, complete the model conversion task and use rknn.release() to release the memory resources occupied by the RKNN object.

[0139] After the model conversion is completed, the RKNN model is deployed on the RK3588S embedded platform used in this project with the help of RKNN-Toolkit Lite2. The deployment process of the RKNN model is as follows: Fig.19 shown.

[0140] RKNN-Toolkit Lite2 first creates an RKNNLite object to initialize the operating environment of the RKNN-ToolkitLite2 SDK, then calls the load_rknn() interface to import the converted RKNN neural network model, and initializes the model runtime environment through the init_runtime() function. After all preparations are completed, the input network state parameter matrix is ​​inferred by calling the inference interface, and the inference result is returned, that is, the target video bit rate value to be updated by the encoder at the drone video acquisition end of this project.

[0141] Furthermore, the streaming media server acts as a data transmission bridge between the drone video acquisition end and the ground station receiving end, completing the video transmission task from the drone to the ground station through the RTSP protocol on UDP, and the RTCP data packet transmission task for the ground station to feedback network status information to the drone. In order to achieve the transmission of video from the drone to the ground station without distance restrictions when the drone conducts highway inspections, the present invention uses a cloud server as the carrier of the streaming media server, and performs the video stream data packet sending and receiving work through the public network IP address of the cloud server.

[0142] This system builds a streaming media server on the Alibaba Cloud server. First, create a Linux system instance with version Ubuntu 22.04. The resource configuration is a 2-core CPU, 2GB memory and 100Mbps bandwidth network. Its public IP address is 47.109.197.143. The default listening control port 554 of the RTSP protocol and the UDP port required for sending and receiving RTP / RTCP data packets are opened.

[0143] ZLMediaKit is a high-performance streaming media server open source project developed based on C++. It supports common operating systems and multiple mainstream streaming media transmission protocols. Its advantages such as cross-platform and low latency have made it widely used in the field of video transmission. Therefore, this invention chooses to use ZLMediaKit as the streaming media server platform. The steps to build ZLMediaKit are as follows:

[0144] Step 1: Get the ZLMediaKit project source code and store it in the user root directory of the cloud server instance. Enter the ZLMediaKit directory and execute "git submodule update --init" in the terminal to update the source code module.

[0145] Step 2: Install third-party dependent libraries openssl, ffmpeg, sdl, avcodec and avutil, create a build directory under the ZLMediaKit / directory, generate the Makefile file through cmake, and then execute "make" to compile the project.

[0146] Step 3: Open the config.ini file in the ZLMediaKit / release / linux / Debug directory, modify the streaming server listening port, and configure the corresponding configuration parameters under the RTSP / RTP protocol.

[0147] Step 4: Enter the ZLMediaKit / release / linux / Debug directory and execute ". / MediaServer" in the terminal with root privileges to run the streaming media server.

[0148] The present invention adopts RTSP protocol to realize streaming media video transmission. The interaction process between the video receiving end and the streaming media server of RTSP protocol is as follows: Fig. 20 shown.

[0149] First, the ground station video receiver sends OPTIONS to the streaming media server to query the RTSP functions and methods supported by the server. The server responds to the request and replies with a list of all currently available RTSP methods. Then the receiver sends DESCRIBE to the server to request streaming media description information. The server responds to the receiver by replying the SDP file, informing the receiver of the streaming media encoding and decoding method, video frame rate and other information. Then the receiver sends a SETUP request to establish a session with the server. After the server determines the conversation port, it establishes an RTSP connection with the receiver. When the server receives the PLAY request, it starts to transmit RTP / RTCP data packets. The video stream is encapsulated in RTP data packets for transmission, and the RTCP data packets mainly contain data statistics of the RTP data packets during the transmission process. For example, the RR message provides information such as RTP packet loss rate, packet arrival time and delay jitter. The RTP / RTCP data packets will continue to be transmitted during the RTSP connection until the receiver issues a TEARDOWN command to end the conversation and disconnect the RTSP connection, completing a streaming video transmission process.

[0150] According to the design goal of this system, the video stream encapsulated by the drone video acquisition end needs to be pushed to the streaming media server, and then the ground station pulls the video stream for decoding and viewing. At the same time, the streaming media server also needs to transmit the RTCP monitoring information fed back by the ground station to the drone video acquisition end as the input parameter of the adaptive bit rate control algorithm. Therefore, the program flow of implementing the RTP / RTCP data packet forwarding function on the ZLMediaKit server is as follows: Fig.21 shown.

[0151] In ZLMediaKit, the RTSP client and server are connected through Socket sockets. First, use the Socket::createSocket() function to create a new socket object. After setting the client's URL address, use RtspPlayer() to establish a connection, and the setupSocket() function is responsible for initializing the socket. After confirming that the connection is successfully established through the onConnect() function, the client sends a SETUP request to the server, and the server calls the RtspSession::onRtpTransport() function to process the request and prepare transmission parameters. The client then sends a PLAY request to the server, and starts receiving RTP / RTCP data packets until the server calls the RtspSession::onPlay() function in response to the PLAY request, and ends the listening thread when the data packets stop being pushed.

[0152] Furthermore, this system designs the ground station video receiving end application based on the open source project VLC multimedia framework. The program workflow of VLC receiving RTSP video stream is as follows: Fig. 22 As shown. Before receiving the video stream, VLC must first create an RTSP streaming transmission URL, then send a DESCRIBE request to the streaming media to obtain the media description, and then send SETUP and PLAY requests to establish a streaming media session connection until the server responds to the request and successfully establishes the RTP / RTCP connection and then starts to continuously receive RTP data packets. VLC uses FFmpeg integrated in libVLC to decode the video, and then renders and plays the decoded data. When there is no new video stream to be received, VLC closes the RTP / RTCP session connection and ends the task of pulling the video stream from the streaming server.

[0153] Furthermore, the UAV inspection video adaptive transmission system proposed in the present invention realizes the complete video transmission requirements from video image acquisition, encoding, transmission, decoding to playback, and intelligently adjusts the target video bit rate of the video encoder through the UABR algorithm deployed on the terminal platform, thereby alleviating the problem of unstable video transmission caused by changes in mobile network bandwidth during UAV highway inspection operations, thereby optimizing the video transmission quality.

[0154] The ground station video receiver uses VLC MediaPlayer to pull the video stream forwarded by the ZLMediaKit streaming server. The URL address is rtsp: / / 47.109.197.143:554 / live / test. After VLC successfully pulls the stream, it decodes and plays the video.

[0155] In order to intuitively understand the overall video transmission delay of the system, when performing video acquisition, the camera acquisition area is aligned with the online stopwatch timer in the running state. By comparing the stopwatch time displayed in the video received by VLC with the actual stopwatch time, the video transmission delay parameter can be obtained by taking the difference between the two. After multiple rounds of testing, it was found that the overall video transmission delay of the system is about 800 milliseconds, which can meet the expected video delay performance indicators.

[0156] In order to test the image quality difference between the video image reconstructed by the ground station video receiving end and the original video image by the drone video acquisition end, the system considers the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM). PSNR reflects the information difference between the reconstructed image and the original image, while SSIM integrates the human eye's perception of the image and reflects the similarity between the reconstructed image and the original image. The PSNR and SSIM index curves of the first 300 frames of the test video are shown in the figure below. Fig.23 and Fig.24 shown.

[0157] The specific method of the present invention is: using VLC at the video receiving end to save the received video stream locally for standby, downloading the video image backed up by the video sending end through the SSH connection mode, and then aligning the original video with the first frame image of the reconstructed video to ensure that the nth frame image of the two videos is one-to-one corresponding, respectively intercepting the first 300 frames of images in the two test videos as test objects, and then using the FFmpeg tool to perform PSNR and SSIM analysis on the reconstructed video and the original video, and outputting the analysis results in the form of logs.

[0158] from Fig.23 and Fig.24 According to the test results of the reconstructed video quality shown in , since the video encoder at the drone video acquisition end adopts a bit rate adaptive mechanism, the image quality difference between the reconstructed video and the original video also changes with the complexity of the video picture and the dynamic changes of the network environment. Specifically, the PSNR and SSIM indicators fluctuate within a small range within the acceptable range, but overall the average PSNR of the first 300 frames of video images has reached 36.39557, and the average SSIM index has reached 0.93294, which comprehensively shows that the video reconstructed at the ground station receiving end is at a good level in image quality compared with the original video, that is, there is a certain degree of distortion in the process of encoding compression and remote transmission, but for the remote streaming video transmission of ultra-high-definition video, the drone video adaptive transmission system proposed in the present invention has achieved relatively good results in the image quality of the reconstructed video.

[0159] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0160] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A highway drone inspection video adaptive transmission system, characterized in that: include: The drone video acquisition end, streaming media server end and ground station video receiving end are connected in sequence; The drone video acquisition end is used to collect video data of the area to be tested and perform video encoding and packaging processing on the video data, obtain the data to be transmitted and send the data to be transmitted to the streaming media server end through the 5G signal base station, and monitor the network status of the environment in which the drone video acquisition end is located to obtain monitoring data, and adjust the video bit rate of the video encoder of the drone video acquisition end according to the monitoring data. The streaming media server end is used to receive the data to be transmitted, and transmit the data to be transmitted through the network firewall to the ground station video receiving end. The ground station video receiving end is used to perform data unpacking and video decoding processing on the data to be transmitted, obtain video data, and store and process the video data.

2. The highway drone inspection video adaptive transmission system according to claim 1 is characterized in that: The drone video collection terminal includes: A network monitoring module, an adaptive bit rate control module, and an image acquisition module, a video encoding module, a data encapsulation module, a video streaming module, and a video storage module connected in sequence; The image acquisition module is used to acquire video data, the video encoding module is used to encode the video data to obtain encoded data, the data encapsulation module is used to encapsulate the encoded data to obtain data to be transmitted, the video streaming module is used to transmit the data to be transmitted to the streaming media server through a signal base station, the network monitoring module is used to monitor the network status of the environment in which the drone video acquisition end is located to obtain monitoring data, and the adaptive bit rate control module is used to control the video bit rate of the video encoding module according to the monitoring data, and dynamically adjust the optimal output bit rate of the video encoding module.

3. The highway drone inspection video adaptive transmission system according to claim 2 is characterized in that: The network monitoring module includes: Network information acquisition submodule and evaluation module; The network information acquisition submodule is used to obtain real-time network information through the RTCP protocol and send the real-time network information as a state parameter characterizing the network conditions to the intelligent agent to obtain initial data. The evaluation module is used to provide reward feedback to the intelligent agent based on the QoE evaluation model as a reward function and update the state space and action space, wherein the three evaluation indicators of the QoE evaluation model include: video quality, video smoothness and video delay.

4. The highway drone inspection video adaptive transmission system according to claim 3 is characterized in that: The expression of the state space is: Among them, di, ji, li, ri represent the network transmission delay, network jitter, packet loss rate and actual video bit rate of the previous sampling period respectively, i = 1, 2..8, i represents the sampling order.

5. The highway drone inspection video adaptive transmission system according to claim 4 is characterized in that: The QoE model expression of the video quality is: Among them, Q is the video quality.

6. The highway drone inspection video adaptive transmission system according to claim 4 is characterized in that: The QoE model expression of the video smoothness is: Where S is the video smoothness.

7. The highway drone inspection video adaptive transmission system according to claim 4 is characterized in that: The QoE model expression of the video delay is: Where D is the video delay.

8. The highway drone inspection video adaptive transmission system according to claim 3 is characterized in that: The input of the neural network of the UABR model is a parameter matrix with a dimension of 4 and a length of 8.

Images