An unmanned aerial vehicle video intelligent encoding method and related device

Abstract

The application discloses a kind of unmanned vehicle video intelligent encoding method and related device, it is related to video encoding technical field, the method includes: to IMU data and height information are preprocessed;To the IMU data after pre-processing is weighted fusion and data mapping, obtain motion factor, segmented linear compensation analysis is carried out based on the height information after pre-processing, obtain height factor;Rate analysis is carried out based on motion factor and height factor, obtain target code rate;To the video to be encoded is canny edge detection and edge density analysis, obtain edge density information, determine IMU motion energy using motion energy model based on the IMU data after pre-processing;Determine key frame insertion information based on edge density information and IMU motion energy, and encoder is based on target code rate and key frame insertion information and is encoded to the video to be encoded processing.The application has significantly improved encoding efficiency and precision, while guaranteeing real-time, reduce bandwidth occupancy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, in particular to a UAV video intelligent coding method and related device. BACKGROUND

[0002] With the rapid development of UAV technology, real-time video transmission is increasingly widely used in the fields of aerial photography, inspection, security, etc. of UAV. At the same time, the UAV video stream coding is also facing challenges such as complex dynamic environment and large network bandwidth fluctuation. For UAV video coding, the traditional fixed code rate control and fixed key frame interval strategy cannot balance the quality and transmission efficiency, especially in high-speed motion or height change, single-dimensional coding optimization is easy to lead to code rate waste or image quality degradation.

[0003] At the same time, in the prior art of video coding, scene change detection based on RGB space is sensitive to light, and the method based on motion estimation has high computational complexity and is difficult to run in real time on embedded devices. Therefore, an intelligent coding scheme that integrates multi-dimensional information is needed to realize dynamic code rate analysis and adaptive key frame insertion, so as to improve the robustness and efficiency of video transmission. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, and the present application provides a UAV video intelligent coding method and related device, which significantly improves the coding efficiency and accuracy, reduces the bandwidth occupation while ensuring real-time.

[0005] In order to solve the above technical problems, the present application provides a UAV video intelligent coding method, which comprises:

[0006] Collecting inertial measurement unit (IMU) data and height information, and preprocessing the IMU data and height information to obtain preprocessed IMU data and height information;

[0007] Weighted fusion and data mapping are performed on the preprocessed IMU data to obtain a motion factor, and segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor;

[0008] Code rate analysis is performed based on the motion factor and the height factor to obtain a target code rate;

[0009] Canny edge detection and edge density analysis are performed on the to-be-coded video to obtain edge density information, and IMU motion energy is determined based on the preprocessed IMU data using a motion energy model;

[0010] Key frame insertion information is determined based on the edge density information and the IMU motion energy, and the encoder performs coding processing on the to-be-coded video based on the target code rate and the key frame insertion information.

[0011] Optionally, the pre-processing of the IMU data and the height information to obtain pre-processed IMU data and pre-processed height information comprises:

[0012] The IMU data is subjected to low-pass filtering processing to obtain pre-processed IMU data, and the height information is processed based on a sliding window average method to obtain pre-processed height information.

[0013] Optionally, the pre-processed IMU data is subjected to weighted fusion and data mapping to obtain a motion factor, and segmented linear compensation analysis is performed based on the pre-processed height information to obtain a height factor, comprising:

[0014] The pre-processed IMU data is subjected to vectorization processing to obtain an acceleration vector and an angular velocity vector;

[0015] The acceleration vector and the angular velocity vector are subjected to weighted fusion to obtain a weighted fusion value, and the weighted fusion value is mapped to [0.5, 1.5] using an S-shaped curve to obtain a motion factor, and the expression of the motion factor is:

[0016] ,

[0017] wherein, is the motion factor, Clamp is a function of limiting data between a specified minimum value and a maximum value, a is the acceleration vector, and g is the angular velocity vector;

[0018] Segmented linear compensation analysis is performed based on a segmented linear model using the pre-processed height information to obtain a height factor.

[0019] Optionally, the code rate analysis based on the motion factor and the height factor to obtain a target code rate comprises:

[0020] The motion factor and the height factor are input into a code rate calculation model to obtain a target code rate, and the expression of the code rate calculation model is:

[0021] ,

[0022] wherein, FinalBitrate is the final code rate, Clamp is a function of limiting data between a specified minimum value and a maximum value, Bbase is a base code rate, HeightFactor is the height factor, is the motion factor.

[0023] Optionally, the canny edge detection and edge density analysis of the to-be-encoded video to obtain edge density information, and the determination of IMU motion energy based on the pre-processed IMU data using a motion energy model, comprise:

[0024] graying the frames of the video to be encoded to obtain grayed frames, the expression of the graying being:

[0025] ,

[0026] wherein, I is the gray value of the image pixel, I R is the R value of the image pixel, I G is the G value of the image pixel, I B is the B value of the image pixel;

[0027] performing Gaussian blur processing on the grayed frames to obtain Gaussian-blurred grayed frames, the expression of the Gaussian blur processing being:

[0028] ,

[0029] wherein, I blurred is the grayed image after Gaussian blur processing, I gray is the grayed image, G is the Gaussian kernel, and * is the convolution operation, is the standard deviation;

[0030] performing adaptive histogram equalization processing on the Gaussian-blurred grayed frames to obtain equalized frames;

[0031] performing double-threshold detection, morphological closing operation and center region processing on the equalized frames to obtain corresponding image edge information;

[0032] calculating the image edge density based on the image edge information;

[0033] determining the acceleration vector and the angular velocity vector between two consecutive frames based on the preprocessed IMU data, and calculating the acceleration difference vector based on the acceleration vector between two consecutive frames and the angular velocity difference vector based on the angular velocity vector between two consecutive frames;

[0034] determining the IMU motion energy based on the acceleration difference vector and the angular velocity difference vector using a motion energy model.

[0035] Optionally, the expression of the image edge density is:

[0036] ,

[0037] wherein, D edge is the image edge density, N edge is the total number of edge pixels in the image edge information, W is the image width, and H is the image height.

[0038] Optionally, the expression of the acceleration difference vector is:

[0039] ,

[0040] wherein, is an acceleration difference vector, a[n] is an acceleration vector of a current frame, and a[n-1] is an acceleration vector of a previous frame;

[0041] An expression of the angular velocity difference vector is:

[0042] ,

[0043] wherein, is an angular velocity difference vector, is an angular velocity vector of a current frame, is an angular velocity vector of a previous frame;

[0044] An expression of the motion energy model is:

[0045] ,

[0046] wherein, is an IMU motion energy, is an acceleration difference vector, is an angular velocity difference vector.

[0047] Optionally, the determining the key frame insertion information based on the edge density information and the IMU motion energy comprises:

[0048] determining an edge density change rate based on the edge density information, and determining the key frame insertion information based on a comparison result of the edge density change rate and a first preset threshold, a comparison result of the IMU motion energy and a second preset threshold, and a comparison result of a continuous frame number of the video to be encoded and a third preset threshold.

[0049] In addition, the present application also provides a kind of unmanned plane video intelligent coding device, the device comprises:

[0050] data preprocessing module: for gathering inertial measurement unit IMU data and height information, and pre-processing IMU data and height information, obtains pre-processed IMU data and height information;

[0051] factor acquisition module: for the weighted fusion and data mapping of pre-processed IMU data, obtains motion factor, carries out segmented linear compensation analysis based on pre-processed height information, obtains height factor;

[0052] code rate analysis module: for based on the motion factor and height factor carries out code rate analysis, obtains target code rate;

[0053] a motion energy analysis module, configured to perform Canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and determine IMU motion energy based on the preprocessed IMU data using a motion energy model;

[0054] a video encoding module, configured to determine key frame insertion information based on the edge density information and the IMU motion energy, and perform encoding processing on the video to be encoded based on the target code rate and the key frame insertion information.

[0055] In addition, the present application also provides an electronic device comprising a processor and a memory, wherein the memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory to enable the electronic device to perform the unmanned aerial vehicle video intelligent encoding method.

[0056] In the embodiment of the present application, the preprocessed IMU data is weighted and fused and data mapped to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, and the IMU data and the height information are fused to perform code rate analysis, which can adapt to high-speed maneuvering of the unmanned aerial vehicle or high-low air switching scenarios and guarantee stability in dynamic scenarios. Canny edge detection and edge density analysis are performed on the video to be encoded to obtain edge density information, IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, key frame insertion information is determined based on the edge density information and the IMU motion energy, and the image edge density and the IMU motion energy are fused to analyze insertion of the key frame, which avoids misjudgment caused by light interference and reduces loss of picture quality caused by motion blur. The encoder performs encoding processing on the video to be encoded based on the target code rate and the key frame insertion information, which significantly improves encoding efficiency, guarantees real-time performance, and reduces bandwidth occupation. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0058] Figure 1 is a flowchart of the unmanned aerial vehicle video intelligent encoding method in the embodiment of the present application;

[0059] Figure 2 is a flowchart of the unmanned aerial vehicle video intelligent encoding method in another embodiment of the present application;

[0060] Figure 3is a structural composition schematic diagram of the unmanned aerial vehicle video intelligent coding device in the embodiment of the present application.

[0061] Figure 4 is a structural composition schematic diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0063] Embodiment one

[0064] Please refer to Figure 1 , Figure 1 is a flow schematic diagram of the unmanned aerial vehicle video intelligent coding method in the embodiment of the present application, and the method comprises:

[0065] S11: collecting inertial measurement unit (IMU) data and height information, and pre-processing the IMU data and the height information to obtain pre-processed IMU data and height information;

[0066] In the specific implementation process of the present application, the pre-processing of the IMU data and the height information to obtain the pre-processed IMU data and the height information comprises: low-pass filtering processing of the IMU data to obtain the pre-processed IMU data, and processing of the height information based on a sliding window average method to obtain the pre-processed height information.

[0067] Specifically, the IMU data in the current flight environment is collected by an inertial measurement unit (IMU) sensor built in the unmanned aerial vehicle, the inertial measurement unit comprises an accelerometer and a gyroscope, the IMU data comprises acceleration data and angular velocity data, and the height information in the current flight environment is collected by an altimeter built in the unmanned aerial vehicle. The IMU data is subjected to low-pass filtering processing, a 10Hz low-pass filter can be used for low-pass filtering processing of the IMU data to obtain the pre-processed IMU data, and the height information is processed based on a sliding window average method, the sliding window average method is used for reducing noise interference of data, and the average value of a plurality of sampling points is taken to offset the noise, the sampling points of the sliding window can be set to five, and the pre-processed height information is obtained.

[0068] S12: weighted fusion and data mapping are performed on the pre-processed IMU data to obtain a motion factor, and segmented linear compensation analysis is performed based on the pre-processed height information to obtain a height factor;

[0069] In the implementation of the present application, the preprocessed IMU data is weighted and fused, and data mapping is performed to obtain a motion factor, and segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, which includes: vectorizing the preprocessed IMU data to obtain an acceleration vector and an angular velocity vector; the acceleration vector and the angular velocity vector are weighted and fused to obtain a weighted fusion value, and the weighted fusion value is mapped to [0.5, 1.5] using an S-shaped curve to obtain a motion factor, and the expression of the motion factor is:

[0070] ,

[0071] wherein, is the motion factor, Clamp is a function of limiting data between specified minimum and maximum values, a is the acceleration vector, and g is the angular velocity vector; segmented linear compensation analysis is performed based on the segmented linear model using the preprocessed height information to obtain a height factor.

[0072] Specifically, the preprocessed IMU data is vectorized, i.e. the acceleration data and the angular velocity data of the preprocessed IMU data are vectorized to obtain an acceleration vector and an angular velocity vector. The acceleration vector and the angular velocity vector are weighted and fused to obtain a weighted fusion value, the acceleration vector and the angular velocity vector are weighted and fused according to a weight of 7:3, and the weighted fusion value is mapped to [0.5, 1.5] using an S-shaped curve, the function of the S-shaped curve is a nonlinear mapping function for mapping data to the required interval to obtain a motion factor, and the expression of the motion factor is:

[0073] ,

[0074] wherein, is a function of limiting data between specified minimum value and maximum value, a is an acceleration vector, g is an angular velocity vector, the output is limited in [0.5, 1.5] range, the acceleration vector is assigned 70% weight, the angular velocity vector is assigned 30% weight, when the motion factor is calculated, the 30% weight of the angular velocity vector needs to be multiplied by a sensitivity coefficient of 0.1. Based on the segmented linear model, the segmented linear compensation analysis is carried out by using the preprocessed height information, the segmented linear model is a three-segment linear model, when the height information is lower than 30 meters, that is, the unmanned aerial vehicle is in low altitude, the height factor keeps the full code rate coefficient as 1.0, when the height information is in the range of 30-60 meters, that is, the unmanned aerial vehicle is in medium altitude, the code rate coefficient of the height factor is linearly decreased to 0.8, when the height information is more than 60 meters, that is, the unmanned aerial vehicle is in high altitude, the code rate coefficient of the height factor is fixed as 0.8, the corresponding height factor is matched by the preprocessed height information.

[0075] S13: performing code rate analysis based on the motion factor and the height factor to obtain a target code rate;

[0076] In the specific implementation process of the application, the code rate analysis based on the motion factor and the height factor to obtain a target code rate comprises: inputting the motion factor and the height factor into a code rate calculation model to obtain the target code rate, and the expression of the code rate calculation model is:

[0077] ,

[0078] wherein, FinalBitrate is a final code rate, Clamp is a function of limiting data between specified minimum value and maximum value, Bbase is a base code rate, HeightFactor is a height factor, is a motion factor.

[0079] Specifically, the motion factor and the height factor are input into a code rate calculation model to obtain a target code rate, and the expression of the code rate calculation model is:

[0080] ,

[0081] wherein, FinalBitrate is a final code rate, Clamp is a function of limiting data between specified minimum value and maximum value, Bbase is a base code rate, HeightFactor is a height factor, is a motion factor, the finally obtained target code rate is limited in the range of 1.0-8.0 Mbps, and after the base code rate is multiplied by the height factor and the motion factor, the product can be subjected to first-order lag filter processing, and the expression of the first-order lag filter processing is:

[0082]

[0083]

[0084] wherein, is a first-order lag filter processed value, is a filter coefficient, is a current sampling value, is a last filter output value, is a sampling time interval, is a time constant, which can be set to 0.5 seconds, and it needs to be explained that when the high-speed motion of the UAV lasting more than 3 seconds is detected, that is, the acceleration of the UAV is greater than 1.5g, the burst mode of the UAV is started, and the upper limit of the code rate is temporarily increased by 20%, and the processor temperature is detected, and if the processor temperature exceeds 75℃, the weight of the angular velocity vector in the motion factor is gradually reduced until it is zero.

[0085] S14: performing canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and determining IMU motion energy based on the preprocessed IMU data using a motion energy model;

[0086] In the specific implementation process of the application, the canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and the determination of the IMU motion energy based on the preprocessed IMU data using a motion energy model, comprises: performing gray scale processing on each frame of image in the video to be encoded to obtain each frame of gray scale image, and the expression of the gray scale processing is:

[0087] ,

[0088] wherein, is the gray value of the image pixel, is the R value of the image pixel, is the G value of the image pixel, is the B value of the image pixel; performing Gaussian blur processing on each frame of gray scale image to obtain each frame of gray scale image after Gaussian blur processing, and the expression of the Gaussian blur processing is:

[0089] ,

[0090] wherein, Iblurred is the gray scale image after Gaussian blur processing, Igray is the gray scale image, G is the Gaussian kernel, and * is the convolution operation, is a standard deviation; performing adaptive histogram equalization on each frame of the Gaussian blurred gray image to obtain an equalized image; performing double threshold detection, morphological closing operation and center region processing on each equalized image to obtain corresponding image edge information; calculating image edge density based on the image edge information; determining an acceleration vector and an angular velocity vector between two consecutive frames based on the preprocessed IMU data, and calculating an acceleration difference vector based on the acceleration vector between two consecutive frames and calculating an angular velocity difference vector based on the angular velocity vector between two consecutive frames; and determining IMU motion energy based on the acceleration difference vector and the angular velocity difference vector using a motion energy model.

[0091] Further, the expression of the image edge density is:

[0092] ,

[0093] wherein Dedge is the image edge density, Nedge is the total number of edge pixels in the image edge information, W is the image width, and H is the image height.

[0094] Further, the expression of the acceleration difference vector is:

[0095] ,

[0096] wherein, is the acceleration difference vector, a[n] is the acceleration vector of the current frame, and a[n-1] is the acceleration vector of the previous frame.

[0097] The expression of the angular velocity difference vector is:

[0098] ,

[0099] wherein, is the angular velocity difference vector, is the angular velocity vector of the current frame, is the angular velocity vector of the previous frame.

[0100] The expression of the motion energy model is:

[0101] ,

[0102] wherein, is the IMU motion energy, is the acceleration difference vector, is the angular velocity difference vector.

[0103] Specifically, the camera built-in the unmanned aerial vehicle collects aerial video, the aerial video is the to-be-encoded video, each frame of image in the to-be-encoded video is subjected to gray processing, a corresponding weight is assigned to a red green blue (RGB) value of each pixel of each frame of image to determine a gray value, the gray value is subjected to gray processing to obtain each frame of gray image, an expression of the gray processing is:

[0104] ,

[0105] wherein, is a gray value of an image pixel, is an R value of the image pixel, is a G value of the image pixel, is a B value of the image pixel. Each frame of gray image is subjected to Gaussian blur processing to obtain each frame of gray image subjected to Gaussian blur processing, an expression of the Gaussian blur processing is:

[0106] ,

[0107] wherein, Iblurred is the gray image subjected to Gaussian blur processing, Igray is the gray image, G is a Gaussian kernel, and * is a convolution operation, is a standard deviation. Each frame of gray image subjected to Gaussian blur processing is subjected to adaptive histogram equalization processing, the adaptive histogram equalization processing is used to improve a contrast of each frame of image, retain image details, each frame of gray image subjected to Gaussian blur processing is divided into a plurality of region images, the image is divided according to a preset size, a histogram of each region image is calculated, each region image is subjected to equalization processing according to the histogram, each region image subjected to equalization processing is merged to obtain each frame of equalization image. Each frame of equalization image is subjected to double-threshold detection, morphological closing operation and center region processing, the double threshold can be set to 50 and 150, a high threshold is used to identify a strong edge of the image, a low threshold is used to identify a weak edge of the image, the morphological closing operation adopts a 3*3 rectangular kernel and is used to connect a broken edge of the image, in order to improve real-time performance, 20% pixels of the image periphery are removed and only the center region is processed, the resolution can be reduced to 720p for calculation, and finally an edge map of each frame of equalization image is output, that is, corresponding image edge information is obtained. The image edge density is calculated based on the image edge information, the total number of edge pixels in the edge map is counted to calculate the edge density, and an expression of the image edge density is:

[0108] ,

[0109] Wherein, Dedge is image edge density, Nedge is total number of edge pixels in image edge information, W is image width, and H is image height. Acceleration vector and angular velocity vector between two continuous frames are determined based on preprocessed IMU data, and acceleration difference vector is calculated based on the acceleration vector between two continuous frames, and the expression of the acceleration difference vector is:

[0110] ,

[0111] Wherein, is the acceleration difference vector, a[n] is the acceleration vector of the current frame, a[n-1] is the acceleration vector of the previous frame, angular velocity difference vector is calculated based on the angular velocity vector between two continuous frames, and the expression of the angular velocity difference vector is:

[0112] ,

[0113] Wherein, is the angular velocity difference vector, is the angular velocity vector of the current frame, is the angular velocity vector of the previous frame. IMU motion energy is determined based on the acceleration difference vector and the angular velocity difference vector by using a motion energy model, and the expression of the motion energy model is:

[0114] ,

[0115] Wherein, is the IMU motion energy, is the acceleration difference vector, is the angular velocity difference vector.

[0116] S15: Determine key frame insertion information based on the edge density information and the IMU motion energy, and the encoder encodes the to-be-encoded video based on the target code rate and the key frame insertion information.

[0117] In the specific implementation process of the application, determining the key frame insertion information based on the edge density information and the IMU motion energy comprises: determining the edge density change rate based on the edge density information, and determining the key frame insertion information based on the comparison result of the edge density change rate and the first preset threshold, the comparison result of the IMU motion energy and the second preset threshold, and the comparison result of the number of continuous frames of the to-be-encoded video and the third preset threshold.

[0118] Specifically, the edge density change rate is determined based on the edge density information, the edge density change rate is calculated through the edge density information between continuous frames, and the key frame insertion information is determined based on a comparison result of the edge density change rate and a first preset threshold, a comparison result of the IMU motion energy and a second preset threshold, and a comparison result of the number of continuous frames of the video to be encoded and a third preset threshold, the first preset threshold can be set to 12%, the second preset threshold can be set to 2.0, and the third preset threshold can be set to 90. When the edge density change rate is greater than the first preset threshold or the IMU motion energy is greater than the second preset threshold or the number of continuous frames is greater than the third preset threshold, the moment is taken as the insertion moment of the key frame, that is, the key frame insertion information. The key frame insertion information and the target code rate are transmitted to the encoder, the encoder encodes the video to be encoded based on the target code rate and the key frame insertion information, the encoder adjusts the encoding code rate of the video to the target code rate in real time, and the key frame insertion processing is performed on the video to be encoded according to the key frame insertion moment. It should be noted that, in order to prevent false triggering, a maximum of two forced key frame insertions are allowed in the same second. Through dynamic adjustment of the encoding, the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene can be adapted, and through the key frame insertion, the unmanned aerial vehicle jitter interference can be excluded and the motion blur problem can be improved.

[0119] In the embodiment of the application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, so that the code rate analysis is performed by fusing the IMU data and the height information, the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene can be adapted, and the stability in the dynamic scene is ensured. The canny edge detection and edge density analysis are performed on the video to be encoded to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, and the key frame insertion information is determined based on the edge density information and the IMU motion energy, so that the insertion of the key frame is analyzed by fusing the image edge density and the IMU motion energy, false judgment caused by light interference is avoided, and the loss of picture quality caused by motion blur is reduced. The encoder encodes the video to be encoded based on the target code rate and the key frame insertion information, the encoding efficiency is significantly improved, the real-time performance is ensured, and the bandwidth occupation is reduced.

[0120] Embodiment two

[0121] Please refer to Figure 2 , Figure 2 is a flowchart of an intelligent video encoding method for an unmanned aerial vehicle in another embodiment of the application, the method comprising:

[0122] S201: Collect inertial measurement unit (IMU) data and height information, and preprocess the IMU data and the height information to obtain preprocessed IMU data and height information;

[0123] S202: Perform weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and perform segmented linear compensation analysis based on the preprocessed height information to obtain a height factor;

[0124] S203: Perform code rate analysis based on the motion factor and the height factor to obtain a target code rate;

[0125] S204: Perform grayscale processing on each frame of image in the video to be encoded to obtain each frame of grayscale image, and perform Gaussian blur processing on each frame of grayscale image to obtain each frame of grayscale image after Gaussian blur processing;

[0126] S205: Perform adaptive histogram equalization processing on each frame of grayscale image after Gaussian blur processing to obtain each frame of equalized image;

[0127] S206: Perform double-threshold detection, morphological closing operation and center region processing on each frame of equalized image to obtain corresponding image edge information, and calculate image edge density based on the image edge information;

[0128] S207: Determine the acceleration vector and the angular velocity vector between two consecutive frames based on the preprocessed IMU data, calculate the acceleration difference vector based on the acceleration vector between two consecutive frames, and calculate the angular velocity difference vector based on the angular velocity vector between two consecutive frames;

[0129] S208: Determine the IMU motion energy based on the acceleration difference vector and the angular velocity difference vector using a motion energy model;

[0130] S209: Determine the key frame insertion information based on the edge density information and the IMU motion energy, and perform encoding processing on the video to be encoded based on the target code rate and the key frame insertion information.

[0131] In the embodiment of the present application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, the IMU data and the height information are fused to perform code rate analysis, which can adapt to the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene, and ensure the stability in the dynamic scene. The canny edge detection and edge density analysis are performed on the to-be-encoded video to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data by using a motion energy model, the key frame insertion information is determined based on the edge density information and the IMU motion energy, and the insertion of the key frame is analyzed by fusing the image edge density and the IMU motion energy, which avoids the misjudgment caused by the light interference and reduces the loss of picture quality caused by the motion blur. The to-be-encoded video is encoded by the encoder based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, reduces the bandwidth occupation while ensuring the real-time performance.

[0132] Embodiment three

[0133] Please refer to Figure 3 , Figure 3 is a structural composition schematic diagram of the unmanned aerial vehicle video intelligent encoding device in the embodiment of the present application, the device comprises:

[0134] The data preprocessing module 31 is configured to collect inertial measurement unit (IMU) data and height information, and preprocess the IMU data and the height information to obtain preprocessed IMU data and height information.

[0135] The factor obtaining module 32 is configured to perform weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and perform segmented linear compensation analysis based on the preprocessed height information to obtain a height factor.

[0136] The code rate analysis module 33 is configured to perform code rate analysis based on the motion factor and the height factor to obtain a target code rate.

[0137] The motion energy analysis module 34 is configured to perform canny edge detection and edge density analysis on the to-be-encoded video to obtain edge density information, and determine IMU motion energy based on the preprocessed IMU data by using a motion energy model.

[0138] The video encoding module 35 is configured to determine key frame insertion information based on the edge density information and the IMU motion energy, and an encoder performs encoding processing on the to-be-encoded video based on the target code rate and the key frame insertion information.

[0139] In the specific implementation process of the present application, the specific implementation mode of the device item can refer to the implementation mode of the above-mentioned method item, which will not be described here.

[0140] In the embodiment of the present application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, the IMU data and the height information are fused to perform code rate analysis, which can adapt to the high-speed maneuvering or high-low air switching scene of the unmanned aerial vehicle and guarantee the stability in the dynamic scene. The canny edge detection and edge density analysis are performed on the to-be-encoded video to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, the key frame insertion information is determined based on the edge density information and the IMU motion energy, the image edge density and the IMU motion energy are fused to analyze the insertion of the key frame, the misjudgment caused by the light interference is avoided, and the picture quality loss caused by the motion blur is reduced. The encoder encodes the to-be-encoded video based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, reduces the bandwidth occupation while guaranteeing the real-time performance.

[0141] Embodiment four

[0142] Please refer to Figure 4 , Figure 4 is a structural composition schematic diagram of an electronic device in the embodiment of the present application.

[0143] The embodiment of the present application further provides an electronic device, as shown in Figure 4 , the electronic device comprises a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and executable on the processor 43. Those skilled in the art can understand that Figure 4The electronic device shown does not constitute a limitation on all devices, and can include more or fewer components than shown, or combine some components. The memory 41 can be used to store computer programs 42 and various functional modules, and the processor 43 runs the computer programs 42 stored in the memory 41 to perform various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both the internal memory and the external memory. The internal memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. The external memory can include a hard disk, a floppy disk, a ZIP disk, a USB disk, a magnetic tape, etc. The processor 43 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip processor, or the processor 43 can be any conventional processor, etc. The processor and the memory disclosed in the present application include but are not limited to these types of processors and memories. The processor and the memory disclosed in the present application are only examples and are not limited.

[0144] As an embodiment, the electronic device includes one or more processors 43, a memory 41, and one or more computer programs 42, wherein the one or more computer programs 42 are stored in the memory 41 and configured to be executed by the one or more processors 43, and the one or more computer programs 42 are configured to perform the unmanned aerial vehicle video intelligent encoding method in any one of the above embodiments. For specific implementation process, please refer to the above embodiments, which will not be repeated here.

[0145] In the embodiment of the present application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, the IMU data and the height information are fused to perform code rate analysis, which can adapt to the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene, and guarantee the stability in the dynamic scene. The canny edge detection and edge density analysis are performed on the to-be-encoded video to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, the key frame insertion information is determined based on the edge density information and the IMU motion energy, and the insertion of the key frame is analyzed by fusing the image edge density and the IMU motion energy, which avoids the misjudgment caused by the light interference and reduces the loss of picture quality caused by the motion blur. The to-be-encoded video is encoded by the encoder based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, reduces the bandwidth occupation while ensuring the real-time performance.

[0146] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0147] In addition, the above describes in detail the unmanned aerial vehicle video intelligent encoding method and related device provided by the embodiments of the present application, and the principle and implementation manner of the present application are described by using specific examples in this paper, and the above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A method for intelligent video coding of unmanned aerial vehicle, characterized in that, The method comprises: Collecting inertial measurement unit (IMU) data and height information, and preprocessing the IMU data and the height information to obtain preprocessed IMU data and height information; Performing weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and performing segmented linear compensation analysis on the preprocessed height information to obtain a height factor; Performing code rate analysis based on the motion factor and the height factor to obtain a target code rate; Performing Canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and determining IMU motion energy based on the preprocessed IMU data using a motion energy model; Determining key frame insertion information based on the edge density information and the IMU motion energy, and performing encoding processing on the video to be encoded by an encoder based on the target code rate and the key frame insertion information. 2.The method of claim 1, wherein, The preprocessing of the IMU data and the height information to obtain preprocessed IMU data and height information comprises: Performing low-pass filtering on the IMU data to obtain preprocessed IMU data, and processing the height information based on a sliding window average method to obtain preprocessed height information. 3.The method of claim 1, wherein, The weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and the segmented linear compensation analysis on the preprocessed height information to obtain a height factor comprise: Performing vectorization processing on the preprocessed IMU data to obtain an acceleration vector and an angular velocity vector; Performing weighted fusion on the acceleration vector and the angular velocity vector to obtain a weighted fusion value, and mapping the weighted fusion value to [0.5, 1.5] using an S-shaped curve to obtain a motion factor, the expression of the motion factor being: ， wherein, is the motion factor, Clamp is a function that limits data to be between specified minimum and maximum values, a is the acceleration vector, and g is the angular velocity vector. Performing segmented linear compensation analysis on the preprocessed height information based on a segmented linear model to obtain a height factor.

4. The method of claim 1, wherein, The code rate analysis based on the motion factor and the height factor to obtain a target code rate comprises: Inputting the motion factor and the height factor into a code rate calculation model to obtain a target code rate, the expression of the code rate calculation model being: ， where FinalBitrate is the final bitrate, Clamp is a function that limits data to be between a specified minimum and maximum value, Bbase is the base bitrate, HeightFactor is a height factor, is the motion factor.

5. The method of claim 1, wherein, The Canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and the determination of IMU motion energy based on the preprocessed IMU data using a motion energy model comprise: Performing grayscale processing on each frame of image in the video to be encoded to obtain each frame of grayscale image, the expression of the grayscale processing being: ， wherein, is a gray value of an image pixel, is an R value of an image pixel, is a G value of an image pixel, is a B value of an image pixel; Performing Gaussian blur processing on each frame of grayscale image to obtain each frame of grayscale image after Gaussian blur processing, the expression of the Gaussian blur processing being: ， Wherein, Iblurred is the gray image after Gaussian blur processing, Igray is the gray image, G is the Gaussian kernel, and * is the convolution operation, is the standard deviation; Performing adaptive histogram equalization processing on each frame of grayscale image after Gaussian blur processing to obtain each frame of equalized image; Performing double-threshold detection, morphological closing operation and center region processing on each frame of equalized image to obtain corresponding image edge information; Calculating image edge density based on the image edge information; Determining the acceleration vector and the angular velocity vector between two consecutive frames based on the preprocessed IMU data, calculating the acceleration difference vector based on the acceleration vector between two consecutive frames, and calculating the angular velocity difference vector based on the angular velocity vector between two consecutive frames; determine an IMU motion energy based on the acceleration difference vector and the angular velocity difference vector.

6. The method of claim 5, wherein, An expression of the image edge density is: ， where Dedge is the image edge density, Nedge is the total number of edge pixels in the image edge information, W is the image width, and H is the image height.

7. The method of claim 5, wherein, An expression of the acceleration difference vector is: ， wherein, is the acceleration difference vector, a[n] is the acceleration vector of the current frame, and a[n-1] is the acceleration vector of the previous frame. An expression of the angular velocity difference vector is: ， wherein is the angular velocity difference vector, is the angular velocity vector of the current frame, is the angular velocity vector of the previous frame; An expression of the motion energy model is: ， wherein, is the IMU motion energy, is the acceleration difference vector, is the angular velocity difference vector.

8. The method of claim 1, wherein, The key frame insertion information is determined based on the edge density information and the IMU motion energy, including: An edge density change rate is determined based on the edge density information, and the key frame insertion information is determined based on a comparison result of the edge density change rate and a first preset threshold, a comparison result of the IMU motion energy and a second preset threshold, and a comparison result of a continuous frame number of the video to be encoded and a third preset threshold.

9. An unmanned aerial vehicle video intelligent encoding apparatus, characterized in that, The apparatus includes: A data preprocessing module is configured to collect inertial measurement unit (IMU) data and height information, and to preprocess the IMU data and the height information to obtain preprocessed IMU data and height information. A factor obtaining module is configured to perform weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and to perform segmented linear compensation analysis based on the preprocessed height information to obtain a height factor. A code rate analysis module is configured to perform code rate analysis based on the motion factor and the height factor to obtain a target code rate. A motion energy analysis module is configured to perform Canny edge detection and edge density analysis on a video to be encoded to obtain edge density information, and to determine an IMU motion energy based on the preprocessed IMU data using a motion energy model. A video encoding module is configured to determine key frame insertion information based on the edge density information and the IMU motion energy, and to perform encoding processing on the video to be encoded based on the target code rate and the key frame insertion information.

10. An electronic device comprising a processor and a memory, wherein The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory to cause the electronic device to perform the unmanned aerial vehicle video intelligent encoding method according to any one of claims 1 to 8. The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory to cause the electronic device to perform the unmanned aerial vehicle video intelligent encoding method according to any one of claims 1 to 8.

Images