Video optimization method and system based on wireless transmission

Abstract

The invention discloses a video optimization method and system based on wireless transmission, and belongs to the technical field of image communication. According to the scheme, the self-adaptive bandwidth allocation effect is achieved through signal intensity monitoring, various optimization modes are adopted, the problems of image distortion and untimely operation in wireless transmission can be effectively solved, and the user experience can be remarkably improved especially in the fields of cloud games, telemedicine, unmanned aerial vehicle control and the like with extremely high real-time performance requirements. According to the method, a prediction algorithm is combined, data for decoding can be obtained more quickly through comparison and correction of an actual frame and a predicted frame, and perception loss is relieved by adopting modes of inter-frame interpolation, frame freezing, background filling and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image communication, and particularly relates to a video optimization method and system based on wireless transmission. BACKGROUND

[0002] In the process of wireless transmission, it takes time for signals to pass between devices, and this transmission delay can cause a series of visual and operational problems: 1. Image distortion: In the process of wireless transmission, data packets may be lost or delayed due to interference, bandwidth limitations or network congestion. Video data is usually transmitted in the form of a sequence of frames, and when some frames are lost or delayed, the receiving end may not be able to reconstruct the complete picture in time, resulting in picture tearing, stuttering or mosaic phenomenon. In addition, in order to make up for the lost data, the system may use error concealment technology, but this may cause the picture quality to decrease, such as color shift, image blur and other distortion phenomena.

[0003] 2. Inoperable in time: For remote control, real-time interaction applications such as remote medical treatment, unmanned aerial vehicle operation or cloud gaming, transmission delay directly affects the real-time performance of operation. User input instructions need to be transmitted to the device end through the wireless network, and then the feedback results are transmitted back to the user. If the transmission delay is too long, it will cause the operation and visual feedback to be out of sync, such as a few hundred milliseconds of delay in the flight of an unmanned aerial vehicle may cause misjudgment of the path, or in cloud gaming, the player presses the jump key, but the character does not jump for a long time, which greatly reduces the user experience, and even causes safety risks.

[0004] In general, transmission delay not only affects visual quality, but also weakens the accuracy and sensitivity of operation, especially in high-demand real-time application scenarios, these problems are more prominent. SUMMARY

[0005] To solve the problems existing in the prior art, the application discloses a video optimization method and system based on wireless transmission, which optimizes from multiple aspects and proposes intelligent scheduling and adaptive strategies to optimize wireless network protocol, improve bandwidth utilization, reduce data packet loss and delay (ACK).

[0006] To achieve the above purpose, the technical scheme of the application is as follows: The application provides a video optimization method based on wireless transmission, comprising the following steps: Step 1, collecting video data at the wireless device end; Step 2, the wireless device end combines dynamic parameters to determine whether the frame rate currently used for transmission needs to be adjusted, and when it is determined that adjustment is needed, the frame group structure is simplified and configured in the encoder, and if adjustment is not needed, the current encoder configuration output is maintained; Step 3, the wireless device end Schottky detection circuit acquires the current image transmission signal strength and judges whether the signal can be used for image transmission signal transmission. If the current signal strength does not meet the requirement, the gain adjustment or channel switching action is performed based on the adaptive allocation method realized by the Schottky diode detection circuit, and the step is repeated until it is judged to be passed. Step 4, the wireless device end Schottky detection circuit obtains the current channel modulation state and judges whether it can be kept in a high-order modulation state for transmission. When it is judged that it cannot be kept in a high-order modulation state, it will be changed to a low-order modulation, and the image transmission signal continues to be transmitted, and the encoder is required to adjust the dynamic frame rate, and step 3 is re-executed; when the channel resource is sufficient, the high-order modulation state is kept to transmit the image transmission signal to the ground; Step 5, the wireless device end Schottky detection circuit acquires the current data transmission signal strength and judges whether it meets the requirement. If it does not meet the requirement, the gain adjustment or channel switching action is performed based on the adaptive allocation method realized by the Schottky diode detection circuit, and the step is repeated until it is judged to be passed to send the data transmission signal to the ground; the ground returns data to the wireless device end; Step 6, the current actual frame data of the image transmission signal transmitted by the wireless device end is compared and corrected with the frame data predicted according to the last frame, the corrected data is decoded and rendered, and output to the screen; the corrected image data is input into the prediction algorithm, and the prediction algorithm outputs the next frame prediction data.

[0007] Further, in step 2, when the dynamic parameter changes reach a pre-set condition, the frame rate for transmission is adjusted.

[0008] Further, in step 3, the adaptive allocation method realized by the Schottky diode detection circuit includes: when the interference signal is too large, the FPGA modifies the transmission frequency point to avoid the interference section; when the radio frequency signal is too small, the gain of the VGA or LNA is modified, and when the adjusted signal is still too small, the FPGA adjusts the encoding parameters.

[0009] Further, in step 4, the condition that cannot keep in a high-order modulation state is that the signal is distorted.

[0010] Further, in step 6, the image transmission data transmitted by the wireless device end to the ground is stored in the cache area after demodulation, and the next frame prediction data output by the prediction algorithm is transmitted to the cache area, and the current actual frame data and the frame data predicted according to the last frame are extracted from the cache area.

[0011] Further, the data transmission signal returned by the wireless device end is input into the flight control system, and the flight control system provides variables to the prediction algorithm.

[0012] A wireless transmission-based video optimization system comprises a wireless device end and a flight control system, wherein the wireless device end comprises a wireless transmission module, the wireless transmission module is used for buffering video data collected based on a network and a sensor, and after the wireless transmission module evaluates the number transmission channel and the image transmission channel by using an adaptive allocation method, the wireless transmission module adopts a suitable channel or gain, a suitable channel modulation state, and selects a suitable encoding and transmission mode; the flight control system receives data transmitted by the wireless transmission module based on a remote controller receiving end, and performs comparison and calibration, image processing, prediction, image rendering on the data and finally outputs.

[0013] The beneficial effects of the present application are: The present application achieves the effect of adaptive bandwidth allocation through signal strength monitoring, and can effectively alleviate the problems of image distortion and untimely operation in wireless transmission by using various optimization methods, especially in fields such as cloud gaming, remote medical treatment, and unmanned aerial vehicle control, which have extremely high real-time requirements, and can significantly improve user experience. The present application combines a prediction algorithm, and through comparison and correction of actual frames and predicted frames, data for decoding can be obtained more quickly, and inter-frame interpolation, frame freezing, background filling and other methods are used to slow down the perceptual loss. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 It is a schematic diagram of channel transmission of an unmanned aerial vehicle and a remote controller.

[0015] Figure 2 It is a schematic diagram of an electronic module of an unmanned aerial vehicle for hardware improvement.

[0016] Figure 3 It is a schematic diagram of channel adaptive allocation logic realized based on a Schottky diode detection circuit.

[0017] Figure 4 It is a schematic diagram of a wireless transmission-based video optimization method provided by the present application. DETAILED DESCRIPTION

[0018] The technical solutions provided by the present application will be described in detail below with specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0019] The technical solutions provided by the present application are realized based on the following optimization strategies: 1. Network optimization step, comprising: An edge computing method is used to process data at an edge node close to the user, thereby reducing the delay of going back and forth to the core network. For example, cloud gaming or remote control applications can quickly render pictures on a local server and transmit them back to the terminal.

[0020] Adopt network slice mode. Based on 5G technology, network slicing can be supported, independent virtual networks can be divided according to application requirements, dedicated resources can be provided for low-latency and high-bandwidth applications, and network congestion risks can be reduced.

[0021] Adopt adaptive bandwidth management method, dynamically adjust transmission rate, adjust video encoding quality and frame rate in real time according to network state, avoid delay or data loss caused by burst traffic.

[0022] Taking a common model of unmanned aerial vehicle operation as an example, during flight, the unmanned aerial vehicle and the remote controller usually establish two sets of wireless communication: one is a data transmission signal for controlling and monitoring the state of the unmanned aerial vehicle. The channel takes WIFI in the 2.4GHz frequency domain as an example. The other is a picture transmission signal for transmitting real-time pictures. The channel takes WIFI in the 5GHz frequency domain as an example. WIFI signals are prone to signal fluctuations during transmission, such as Figure 1 As shown, there are two kinds of channel fluctuations. The present application achieves the effect of adaptive bandwidth allocation through signal strength monitoring.

[0023] In order to realize adaptive bandwidth allocation, wireless device optimization is needed, and Figure 2 The middle box part, that is, a coupler, a Schottky secondary hanging detection circuit and a control circuit are added to form an automatic gain control system. When the received power is detected to be small, the VGA gain will be increased first, so that the transmitted power becomes larger, and when the maximum transmitted power is reached, the received gain can be increased again. The detection circuit of the Schottky diode mainly consists of a matching circuit, a Schottky diode and a filter. Due to the internal structure of the Schottky diode and the low junction capacitance, the switching speed of the Schottky diode is much faster than that of the PN diode, and the reverse recovery time of the Schottky diode is very short. The response time of the filter is related to the cutoff frequency, which is also in the order of nanoseconds. Therefore, the present application can respond in advance in real time and faster, avoiding the situation that when the unmanned aerial vehicle encounters interference, the received signal becomes weak, the delay is between tens of milliseconds and hundreds of milliseconds, and the video appears to be stuck.

[0024] The logic of channel adaptive allocation realized based on the Schottky diode detection circuit is as shown in Figure 3 The radio frequency signal input detection circuit outputs a voltage, which is judged by a threshold value. When the interference signal is too large, the FPGA modifies the transmission frequency point to avoid the interference section. When the signal is too small, the gain of the VGA or LNA is modified. When the signal is still too small after the gain adjustment, the FPGA adjusts the encoding parameters to strengthen the signal.

[0025] ​The detection circuit is real-time updated, the received signal is amplified by the LNA, then passes through the coupler, the power output by the coupler enters the Schottky diode detection circuit, the output voltage of the detection circuit enters the control circuit, the control circuit judges the signal size at this time, and adjusts the gain size of the VGA and the LNA, and after configuration, it is looped back to the FPGA again until the threshold meets the requirements.

[0026] According to the above control logic, the application can analyze the wireless channel quality in real time based on the signal, avoid interference frequency bands in advance, or temporarily switch to a low code rate when interference suddenly increases to ensure smooth transmission.

[0027] The signal acquisition process at the UAV end is as follows: (1) Video acquisition and encoding: the UAV camera captures the picture and compresses the video data through the encoder.

[0028] (2) Signal strength monitoring: real-time detection of wireless signal strength, such as RSSI (Received Signal Strength Indicator) or signal-to-noise ratio.

[0029] (3) Adaptive controller decision: according to the signal strength threshold, decide the code rate and resolution adjustment strategy: when the signal is strong, maintain high code rate, high resolution and high frame rate (such as 1080p / 60fps), when the signal is weak, use low code rate, reduce resolution and frame rate, reduce bandwidth occupation (such as 720p / 30fps).

[0030] (4) Encoding parameter adjustment: according to the decision result, dynamically modify the encoder parameters, and reset the code rate, resolution and frame rate.

[0031] (5) Video stream transmission: transmit real-time video stream through wireless module to ensure smooth picture as much as possible in weak signal area.

[0032] 2. Video encoding optimization Low-latency encoding technology: use efficient encoders such as H.265, AV1, etc. to greatly reduce code rate under the same quality. And open low-latency mode (ACK time), communication protocol delay, reduce encoding and decoding processing time.

[0033] Forward error correction (FEC): add redundant data during transmission, even if some data packets are lost, the receiving end can recover the complete picture through error correction algorithm, reduce distortion caused by packet loss. Common methods such as Raptor codes, LDPC, etc.

[0034] Layered encoding: divide the video stream into base layer and enhancement layer, transmit only the base layer when the network condition is not good, ensure the minimum quality and smoothness; gradually increase the image details after the network recovers.

[0035] 3. Transmission protocol optimization QUIC Protocol: The QUIC protocol based on UDP supports faster connection establishment and data recovery, making it more suitable for real-time interaction scenarios than traditional TCP.

[0036] RTSP / RTMP Optimization: Optimizing the buffering strategy for real-time audio and video protocols, reducing unnecessary buffering time, while avoiding high latency caused by excessive accumulation.

[0037] Traffic Priority Scheduling: In multi-stream data transmission, critical control instructions and core video frames are given priority to ensure that operation instructions and key images are delivered quickly.

[0038] 4. Device-side Optimization Hardware Acceleration: Utilize specialized decoding chips (such as GPU, FPGA) to accelerate video processing speed and reduce device-side processing delay.

[0039] Preloading and Buffering Strategy: In stable network environments, pre-buffer some data to balance delay and picture continuity, avoiding severe lag caused by network jitter. Adjust the buffer size on the remote control side based on flight speed and network status. Reduce cache depth during high-speed flight to reduce latency, and increase cache during low-speed hovering to improve picture stability. Use a preset speed threshold to distinguish between high and low speed flight.

[0040] Intelligent Latency Compensation: In remote control scenarios, use image algorithms to render possible pictures in advance. On the remote controller side, combine sensor data and flight attitude to predict the next frame of picture and render possible images in advance to avoid image lag caused by network delay and reduce the operator's perception delay.

[0041] 5. Intelligent Scheduling and Adaptive Strategy Monitor network status through sensor data returned by the drone, such as increasing flight speed and poor network speed, to provide variables for the prediction algorithm of the display picture and determine whether picture compensation is needed. The prediction algorithm can be trained using public AI models.

[0042] Dynamic Frame Rate Adjustment: In low-latency scenarios, automatically reduce the transmission frequency of non-critical frames, retaining only key action frames to reduce transmission burden.

[0043] Through these multi-level optimization methods, image distortion and operation delay in wireless transmission can be effectively alleviated, especially in fields such as cloud gaming, remote medical treatment, and unmanned vehicle control, which have extremely high real-time requirements, significantly improving user experience.

[0044] Based on the above optimization strategy, the application provides a wireless transmission-based video optimization system, which comprises a wireless device end (a drone) and a ground flight control system. The wireless device end comprises a wireless transmission module, which is used to buffer video data collected based on a network and a sensor, adopt the aforementioned channel adaptive allocation method to evaluate the channel, adopt the adaptive allocation method of the image transmission channel to monitor and adjust the signal, and select a suitable encoding and transmission mode. The flight control system receives data transmitted by the drone transmission module based on a remote controller receiving end, compares and calibrates the data, performs image processing, prediction, image rendering, and finally outputs.

[0045] The application provides a wireless transmission-based video optimization method, the flow of which is shown in Figure 4 The method comprises the following steps: Step 1: The wireless device end (in this example, a drone) receives a video stream. Step 2: The wireless device end combines speed sensors, vertical sensors, network states and other dynamic parameters to determine whether the current frame rate for transmission needs to be adjusted. If the change range of some dynamic parameters is large, and conditions such as acceleration, large-scale rotation of the fuselage, network fluctuation and the like are encountered, it is determined that adjustment is needed, and the frame group structure is simplified and configured in the encoder. If the current dynamic parameter change is not obvious, the current encoder configuration output H.265 compression encoding is maintained. The prerequisite for frame rate adjustment is that the change of which dynamic parameter (channel state, flight state, light change, power and the like) reaches a certain range requirement. The adjustment condition should be preset.

[0046] Step 3: The wireless device end Schottky detection circuit can obtain the current signal strength, and judge whether the current signal can be used for image transmission signal transmission through monitoring. If the current signal strength does not meet the requirement, for example, it suddenly decreases or is interrupted, gain adjustment or channel switching action will be performed based on the adaptive allocation logic (for details, refer to the logic of the aforementioned channel adaptive allocation realized based on the Schottky diode detection circuit), and the monitoring and judgment through the Schottky detection circuit are returned until it is determined to be passed. This step should be performed regularly.

[0047] Step 4: The wireless device end Schottky detection circuit can obtain the current channel modulation state and determine whether it can be kept in a high-order modulation state for transmission by the FPGA. If the signal is distorted due to the occupation of channel resources, it is determined to be a low-order modulation. At this time, the image transmission signal continues to be sent, and the encoder is required to re-adjust the dynamic frame rate to ensure the continuity of the current image transmission, and steps 3 and 4 are re-performed. When the channel resources are sufficient, the high-order modulation state is kept to transmit the image transmission signal to the ground remote controller end. This step should be performed regularly.

[0048] Step 5, the data transmission signal sent from the wireless device end to the ground, the signal strength determines the synchronization step 3 in the picture transmission channel strength determination. The ground returns data to the wireless device end contains the necessary state of the unmanned aerial vehicle such as sensor data, periodic patrol instructions, for the remote control end to carry out flight control analysis and prediction.

[0049] Step 6, the picture transmission data sent from the wireless device end to the ground, after demodulation will first arrive at the cache area, the cache area contains at least 4 frame picture signal. Including the last frame, the predicted frame, the actual frame (the original data received from the unmanned aerial vehicle) and the predicted next frame. The current actual frame and the predicted frame data according to the last frame are compared and compensated, and the data is corrected. The corrected data is decoded and rendered to the screen. The corrected image data is also input to the prediction algorithm, which can predict the next frame data. The next frame data is transmitted to the cache area. The prediction algorithm as an AI involved model (can be trained by existing public model), through the comparison and correction of actual frame and predicted frame, can obtain the data for decoding faster, and adopts the methods such as frame interpolation, frame freezing and background filling to reduce the perceptual loss. The data transmission signal returned by the wireless device end (unmanned aerial vehicle) is input to the flight control system, and the prediction algorithm is provided with variables based on the network state.

[0050] It should be noted that the above content only illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. For ordinary skilled persons in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which fall within the scope of protection of the claims of the present application.

Claims

1. A method for video optimization based on wireless transmission, the method comprising: It comprises the following steps: Step 1: Collecting video data at the wireless device end; Step 2: When the frame rate for transmission needs to be adjusted according to the dynamic parameters at the wireless device end, simplifying the frame group structure in the encoder, or continuing the current encoder configuration output if no adjustment is needed; Step 3: If the current image transmission signal strength does not meet the transmission requirements, adjusting the gain or switching the channel based on the adaptive allocation method realized by the Schottky diode detection circuit, repeating this step until it is determined to be passed; Step 4: If the current channel modulation state cannot be maintained at a high-order modulation state, it becomes a low-order modulation, continues to send the image transmission signal, and at the same time proposes the demand for re-adjusting the dynamic frame rate to the encoder, and re-executes step 3; when the channel resources are sufficient, maintain the high-order modulation state to transmit the image transmission signal to the ground; Step 5: If the current data transmission signal strength does not meet the transmission requirements, adjusting the gain or switching the channel based on the adaptive allocation method realized by the Schottky diode detection circuit, repeating this step until it is determined to be passed to send the data transmission signal to the ground; the ground returns data to the wireless device end; Step 6: Comparing and correcting the current actual frame data of the image transmission data sent by the wireless device end with the predicted frame data according to the last frame, rendering the decoded corrected data, and outputting to the screen; the corrected image data is input into the prediction algorithm, and the prediction algorithm outputs the next frame prediction data.

2. The wireless transmission based video optimization method of claim 1, wherein, In step 2, the frame rate for transmission is adjusted when the dynamic parameters change to reach the pre-set conditions.

3. The wireless transmission based video optimization method of claim 1, wherein, The adaptive allocation method based on the Schottky diode detection circuit includes: when the interference signal is too large, the FPGA modifies the transmission frequency point to avoid the interference section; when the radio frequency signal is too small, the gain of the VGA or LNA is modified, and when the adjusted signal is still too small, the FPGA adjusts the encoding parameters.

4. The wireless transmission based video optimization method of claim 1, wherein, In step 4, the condition that cannot be maintained at a high-order modulation state is that the signal is distorted.

5. The wireless transmission based video optimization method of claim 1, wherein, In step 6, the image transmission data sent by the wireless device end to the ground is stored in the buffer area after demodulation, and the next frame prediction data output by the prediction algorithm is transmitted to the buffer area. The current actual frame data and the predicted frame data according to the last frame are both extracted from the buffer area.

6. A wireless transmission based video optimization system comprising a wireless device end and a flight control system, characterized in that, The wireless device end for realizing the video optimization method based on wireless transmission in any one of claims 1-5 comprises a wireless transmission module for buffering the video data collected based on the network and sensors, adopting an adaptive allocation method to evaluate the data transmission and image transmission channels, adopting a suitable channel modulation state, selecting a suitable encoding and transmission mode; the flight control system receives the data transmitted by the wireless transmission module based on the remote controller receiving end, and compares and calibrates the data, processes the image, predicts, renders the image, and finally outputs.

Images