Unmanned aerial vehicle video intelligent coding method and related device

Abstract

The invention discloses an unmanned aerial vehicle video intelligent coding method and a related device, and relates to the technical field of video coding, and the method comprises the steps: carrying out the preprocessing of IMU data and height information; performing weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and performing piecewise linear compensation analysis based on the preprocessed height information to obtain a height factor; performing code rate analysis based on the motion factor and the height factor to obtain a target code rate; performing canny edge detection and edge density analysis on the to-be-coded video to obtain edge density information, and determining IMU motion energy by using a motion energy model based on the preprocessed IMU data; and key frame insertion information is determined based on the edge density information and the IMU motion energy, and the encoder performs encoding processing on the video to be encoded based on the target code rate and the key frame insertion information. According to the invention, the coding efficiency and precision are obviously improved, and the bandwidth occupation is reduced while the real-time performance is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video coding technology, in particular to a UAV video intelligent coding method and related device. BACKGROUND

[0002] With the rapid development of UAV technology, real-time video transmission is increasingly widely used in the fields of aerial photography, inspection, security and the like of UAV. Meanwhile, UAV video stream coding is also facing challenges such as complex dynamic environment and large network bandwidth fluctuation. For UAV video coding, the traditional fixed code rate control and fixed key frame interval strategy cannot balance the quality and transmission efficiency, especially in high-speed motion or height change, single-dimensional coding optimization easily leads to code rate waste or image quality degradation.

[0003] Meanwhile, in the prior art of video coding, scene change detection based on RGB space is sensitive to light, and the method based on motion estimation has high computational complexity and is difficult to run in real time on embedded devices. Therefore, an intelligent coding scheme integrating multi-dimensional information is needed to realize dynamic code rate analysis and adaptive key frame insertion, so as to improve the robustness and efficiency of video transmission. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art, and provides a UAV video intelligent coding method and related device, which significantly improves the coding efficiency and accuracy, reduces the bandwidth occupation while ensuring real-time.

[0005] To solve the above technical problems, the present application provides a UAV video intelligent coding method, which comprises: Collecting inertial measurement unit (IMU) data and height information, and preprocessing the IMU data and height information to obtain preprocessed IMU data and height information; Weighted fusion and data mapping are performed on the preprocessed IMU data to obtain a motion factor, and segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor; Code rate analysis is performed based on the motion factor and the height factor to obtain a target code rate; Canny edge detection and edge density analysis are performed on the video to be coded to obtain edge density information, and IMU motion energy is determined based on the preprocessed IMU data using a motion energy model; Key frame insertion information is determined based on the edge density information and the IMU motion energy, and the encoder performs coding processing on the video to be coded based on the target code rate and the key frame insertion information.

[0006] Optionally, the preprocessing of the IMU data and the height information to obtain the preprocessed IMU data and the height information comprises: The IMU data is low-pass filtered to obtain pre-processed IMU data, and the height information is processed based on a sliding window average method to obtain pre-processed height information.

[0007] Optionally, the pre-processed IMU data is weighted fused and data mapped to obtain a motion factor, and the pre-processed height information is used for segmented linear compensation analysis to obtain a height factor, including: The pre-processed IMU data is vectorized to obtain an acceleration vector and an angular velocity vector; The acceleration vector and the angular velocity vector are weighted fused to obtain a weighted fusion value, and the weighted fusion value is mapped to [0.5, 1.5] using an S-shaped curve to obtain a motion factor, and the expression of the motion factor is: , wherein, is the motion factor, Clamp is a function of limiting data between a specified minimum value and a maximum value, a is the acceleration vector, and g is the angular velocity vector; The pre-processed height information is used for segmented linear compensation analysis based on a segmented linear model to obtain a height factor.

[0008] Optionally, the motion factor and the height factor are used for code rate analysis to obtain a target code rate, including: The motion factor and the height factor are input into a code rate calculation model to obtain a target code rate, and the expression of the code rate calculation model is: , wherein, FinalBitrate is the final code rate, Clamp is a function of limiting data between a specified minimum value and a maximum value, Bbase is a base code rate, HeightFactor is the height factor, is the motion factor.

[0009] Optionally, the pre-processed IMU data is used to determine IMU motion energy based on a motion energy model, and the encoded video is subjected to canny edge detection and edge density analysis to obtain edge density information, including: Each frame of image in the encoded video is subjected to grayscale processing to obtain each frame of grayscale image, and the expression of the grayscale processing is: , wherein, is a grayscale value of an image pixel, is an R value of the image pixel, is a G value of the image pixel, is a B value of the image pixel. Gaussian blur processing is performed on each frame of the grayscale image to obtain each frame of the grayscale image after Gaussian blur processing, and an expression of the Gaussian blur processing is as follows: , where Iblurred is the grayscale image after Gaussian blur processing, Igray is the grayscale image, G is a Gaussian kernel, and * is a convolution operation, is a standard deviation; Adaptive histogram equalization processing is performed on each frame of the grayscale image after Gaussian blur processing to obtain each frame of the equalized image; Double-threshold detection, morphological closing operation and center region processing are performed on each frame of the equalized image to obtain corresponding image edge information; Image edge density is calculated based on the image edge information; Acceleration vectors and angular velocity vectors between two consecutive frames are determined based on the preprocessed IMU data, and an acceleration difference vector is calculated based on the acceleration vectors between two consecutive frames, and an angular velocity difference vector is calculated based on the angular velocity vectors between two consecutive frames; IMU motion energy is determined based on the acceleration difference vector and the angular velocity difference vector using a motion energy model.

[0010] Optionally, an expression of the image edge density is as follows: , where Dedge is the image edge density, Nedge is the total number of edge pixels in the image edge information, W is the image width, and H is the image height.

[0011] Optionally, an expression of the acceleration difference vector is as follows: , where is the acceleration difference vector, a[n] is the acceleration vector of the current frame, and a[n-1] is the acceleration vector of the previous frame. An expression of the angular velocity difference vector is as follows: , where is the angular velocity difference vector, is the angular velocity vector of the current frame, is the angular velocity vector of the previous frame. An expression of the motion energy model is as follows: , where is the IMU motion energy, is the acceleration difference vector, The angular velocity difference vector.

[0012] Optionally, the determining the key frame insertion information based on the edge density information and the IMU motion energy comprises: determining an edge density change rate based on the edge density information, and determining the key frame insertion information based on a comparison result of the edge density change rate and a first preset threshold, a comparison result of the IMU motion energy and a second preset threshold, and a comparison result of a continuous frame number of the video to be encoded and a third preset threshold.

[0013] In addition, the present application also provides a kind of unmanned vehicle video intelligent coding device, the device comprises: data preprocessing module: for collecting inertial measurement unit IMU data and height information, and pre-processing IMU data and height information, obtains pre-processed IMU data and height information; factor acquisition module: for pre-processed IMU data is weighted fusion and data mapping, obtains motion factor, based on pre-processed height information carries out segmented linear compensation analysis, obtains height factor; code rate analysis module: for based on the motion factor and height factor carries out code rate analysis, obtains target code rate; motion energy analysis module: for carrying out canny edge detection and edge density analysis to video to be encoded, obtains edge density information, based on pre-processed IMU data utilizes motion energy model to determine IMU motion energy; video coding module: for based on the edge density information and the IMU motion energy determines key frame insertion information, and encoder is based on the target code rate and key frame insertion information to video to be encoded carries out encoding processing.

[0014] In addition, the present application also provides a kind of electronic equipment, the electronic equipment includes processor and memory, the memory is used to store instruction, the processor is used to call the instruction in the memory, so that the electronic equipment executes above-mentioned unmanned vehicle video intelligent coding method.

[0015] In the embodiment of the present application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, and the IMU data and the height information are fused to perform code rate analysis, which can adapt to high-speed maneuvering of the unmanned aerial vehicle or high-low air switching scenarios and guarantee stability in dynamic scenarios. Canny edge detection and edge density analysis are performed on the to-be-encoded video to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, the key frame insertion information is determined based on the edge density information and the IMU motion energy, and the insertion of the key frame is analyzed by fusing the image edge density and the IMU motion energy, which avoids misjudgment caused by light interference and reduces the loss of picture quality caused by motion blur. The encoder encodes the to-be-encoded video based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, reduces the bandwidth occupation while ensuring real-time performance. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a flowchart of the unmanned aerial vehicle video intelligent encoding method in the embodiment of the present application; Figure 2 is a flowchart of the unmanned aerial vehicle video intelligent encoding method in another embodiment of the present application; Figure 3 is a structural composition diagram of the unmanned aerial vehicle video intelligent encoding device in the embodiment of the present application; Figure 4 is a structural composition diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0019] Embodiment one Please refer to Figure 1 , Figure 1is a flowchart of an unmanned aerial vehicle video intelligent coding method in an embodiment of the present application, the method comprising: S11: collect inertial measurement unit (IMU) data and height information, and pre-process the IMU data and the height information to obtain pre-processed IMU data and height information; In the specific implementation process of the present application, the pre-processing of the IMU data and the height information to obtain the pre-processed IMU data and the height information comprises: low-pass filtering the IMU data to obtain pre-processed IMU data, and processing the height information based on a sliding window average method to obtain pre-processed height information.

[0020] Specifically, the IMU data in the current flight environment is collected by the inertial measurement unit (IMU) sensor built in the unmanned aerial vehicle, the inertial measurement unit includes an accelerometer and a gyroscope, the IMU data includes acceleration data and angular velocity data, and the height information in the current flight environment is collected by the altimeter built in the unmanned aerial vehicle. The IMU data is processed by low-pass filtering, a 10Hz low-pass filter can be used to process the IMU data by low-pass filtering to obtain pre-processed IMU data, and the height information is processed based on a sliding window average method, the sliding window average method is used to reduce noise interference of data, and the average value of a plurality of sampling points is taken to offset noise, the sampling points of the sliding window can be set to five, and pre-processed height information is obtained.

[0021] S12: weighted fusion and data mapping of the pre-processed IMU data to obtain a motion factor, and segmented linear compensation analysis based on the pre-processed height information to obtain a height factor; In the specific implementation process of the present application, the pre-processing of the IMU data and the height information to obtain the pre-processed IMU data and the height information comprises: vectorization processing of the pre-processed IMU data to obtain an acceleration vector and an angular velocity vector; weighted fusion of the acceleration vector and the angular velocity vector to obtain a weighted fusion value, and mapping of the weighted fusion value to [0.5, 1.5] using an S-shaped curve to obtain a motion factor, the expression of the motion factor being: , wherein, is the motion factor, Clamp is a function of limiting data between a specified minimum value and a maximum value, a is the acceleration vector, and g is the angular velocity vector; segmented linear compensation analysis based on a segmented linear model using the pre-processed height information to obtain a height factor.

[0022] Specifically, the preprocessed IMU data is subjected to vectorization processing, that is, the acceleration data and the angular velocity data of the preprocessed IMU data are subjected to vectorization processing to obtain an acceleration vector and an angular velocity vector. The acceleration vector and the angular velocity vector are subjected to weighted fusion to obtain a weighted fusion value, the acceleration vector and the angular velocity vector are subjected to weighted fusion according to a weight of 7:3, and the weighted fusion value is mapped into [0.5, 1.5] by using an S-shaped curve, the function of the S-shaped curve is a nonlinear mapping function for mapping data into a required interval to obtain a motion factor, and the expression of the motion factor is: , wherein, is the motion factor, Clamp is a function of limiting data between a specified minimum value and a maximum value, a is the acceleration vector, g is the angular velocity vector, the output is limited in the range of [0.5, 1.5], the acceleration vector is assigned a weight of 70%, the angular velocity vector is assigned a weight of 30%, and the 30% weight of the angular velocity vector needs to be multiplied by a sensitivity coefficient of 0.1 when the motion factor is calculated. The segmented linear compensation analysis is performed based on a segmented linear model by using the preprocessed height information, the segmented linear model is a three-segment linear model, when the height information is lower than 30 meters, that is, the unmanned aerial vehicle is in low altitude, the height factor keeps a full code rate coefficient of 1.0, when the height information is in the range of 30-60 meters, that is, the unmanned aerial vehicle is in medium altitude, the code rate coefficient of the height factor linearly decreases to 0.8, and when the height information is higher than 60 meters, that is, the unmanned aerial vehicle is in high altitude, the code rate coefficient of the height factor is fixed at 0.8, and the corresponding height factor is matched by the preprocessed height information.

[0023] S13: performing code rate analysis based on the motion factor and the height factor to obtain a target code rate; In the specific implementation process of the application, the code rate analysis based on the motion factor and the height factor to obtain a target code rate includes: inputting the motion factor and the height factor into a code rate calculation model to obtain the target code rate, and the expression of the code rate calculation model is: , wherein, FinalBitrate is the final code rate, Clamp is a function of limiting data between a specified minimum value and a maximum value, Bbase is a base code rate, HeightFactor is a height factor, is the motion factor.

[0024] Specifically, the motion factor and the height factor are input into a code rate calculation model to obtain a target code rate, and the expression of the code rate calculation model is: , wherein FinalBitrate is the final bitrate, Clamp is a function that limits data to be between a specified minimum and maximum value, Bbase is the base bitrate, HeightFactor is the height factor, The final target bitrate is limited to the range of 1.0-8.0 Mbps as the motion factor, and after multiplying the base bitrate by the height factor and the motion factor, the product is subjected to first-order lag filter processing, and the expression of the first-order lag filter processing is:

[0025]

[0026] wherein, is the value after the first-order lag filter processing, is a filter coefficient, is the current sampling value, is the last filter output value, is a sampling time interval, is a time constant, which can be set to 0.5 seconds, and it is necessary to note that when the high-speed motion of the UAV lasting more than 3 seconds is detected, that is, the acceleration of the UAV is greater than 1.5g, the burst mode of the UAV is started, and the upper limit of the bitrate is temporarily increased by 20%, and the processor temperature is detected, and if the processor temperature exceeds 75℃, the weight of the angular velocity vector in the motion factor is gradually reduced until it is zero.

[0027] S14: performing canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and determining IMU motion energy based on the preprocessed IMU data using a motion energy model; In the specific implementation process of the application, the canny edge detection and edge density analysis on the video to be encoded to obtain edge density information, and the determination of the IMU motion energy based on the preprocessed IMU data using a motion energy model, comprises: performing gray scale processing on each frame of image in the video to be encoded to obtain each frame of gray scale image, and the expression of the gray scale processing is: , wherein, is the gray scale value of the image pixel, is the R value of the image pixel, is the G value of the image pixel, is the B value of the image pixel; performing Gaussian blur processing on each frame of gray scale image to obtain each frame of gray scale image after Gaussian blur processing, and the expression of the Gaussian blur processing is: , wherein, Iblurred is the gray image after Gaussian blur processing, Igray is the gray image, G is Gaussian kernel, and * is convolution operation, is standard deviation; performing adaptive histogram equalization processing on each frame of the gray image after Gaussian blur processing to obtain each frame of the equalized image; performing double threshold detection, morphological closing operation and center region processing on each frame of the equalized image to obtain corresponding image edge information; calculating image edge density based on the image edge information; determining acceleration vector and angular velocity vector between two consecutive frames based on the preprocessed IMU data, and calculating acceleration difference vector based on the acceleration vector between two consecutive frames, and calculating angular velocity difference vector based on the angular velocity vector between two consecutive frames; determining IMU motion energy based on the acceleration difference vector and the angular velocity difference vector using a motion energy model.

[0028] Further, the expression of the image edge density is: wherein, Dedge is the image edge density, Nedge is the total number of edge pixels in the image edge information, W is the image width, and H is the image height.

[0029] Further, the expression of the acceleration difference vector is: wherein, is the acceleration difference vector, a[n] is the acceleration vector of the current frame, and a[n-1] is the acceleration vector of the previous frame. The expression of the angular velocity difference vector is: wherein, is the angular velocity difference vector, is the angular velocity vector of the current frame, is the angular velocity vector of the previous frame. The expression of the motion energy model is: wherein, is the IMU motion energy, is the acceleration difference vector, is the angular velocity difference vector.

[0030] ​​​​Specifically, the camera built in the unmanned aerial vehicle collects aerial video, the aerial video is the to-be-encoded video, each frame of image in the to-be-encoded video is subjected to gray processing, a corresponding weight is given to a red green blue (RGB) value of each pixel of each frame of image to determine a gray value, the gray value is subjected to gray processing to obtain each frame of gray image, an expression of the gray processing is: , wherein, is a gray value of an image pixel, is an R value of the image pixel, is a G value of the image pixel, is a B value of the image pixel. Each frame of gray image is subjected to Gaussian blur processing to obtain each frame of gray image subjected to Gaussian blur processing, an expression of the Gaussian blur processing is: , wherein, Iblurred is the gray image subjected to Gaussian blur processing, Igray is the gray image, G is a Gaussian kernel, and * is a convolution operation, is a standard deviation. Each frame of gray image subjected to Gaussian blur processing is subjected to adaptive histogram equalization processing, the adaptive histogram equalization processing is used to improve a contrast of each frame of image, retain image details, each frame of gray image subjected to Gaussian blur processing is divided into a plurality of region images, the image is divided according to a preset size, a histogram of each region image is calculated, each region image is subjected to equalization processing according to the histogram, each region image subjected to equalization processing is merged to obtain each frame of equalization image. Each frame of equalization image is subjected to double-threshold detection, morphological closing operation and center region processing, the double threshold can be set to 50 and 150, a high threshold is used to identify a strong edge of the image, a low threshold is used to identify a weak edge of the image, the morphological closing operation adopts a 3*3 rectangular kernel and is used to connect a broken edge of the image, in order to improve real-time performance, 20% pixels of the image periphery are removed and only the center region is processed, the resolution can be reduced to 720p for calculation, and finally an edge map of each frame of equalization image is output, that is, corresponding image edge information is obtained. Image edge density is calculated based on the image edge information, the total number of edge pixels in the edge map is counted to calculate the edge density, and an expression of the image edge density is: , wherein, Dedge is the image edge density, Nedge is the total number of edge pixels in the image edge information, W is a width of the image, and H is a height of the image. Acceleration vector and angular velocity vector between two continuous frames are determined based on preprocessed IMU data, and an acceleration difference vector is calculated based on the acceleration vector between two continuous frames, and an expression of the acceleration difference vector is: , wherein, is an acceleration difference vector, a[n] is an acceleration vector of a current frame, a[n-1] is an acceleration vector of a previous frame, an angular velocity difference vector is calculated based on an angular velocity vector between two consecutive frames, and an expression of the angular velocity difference vector is: , wherein, is an angular velocity difference vector, is an angular velocity vector of a current frame, is an angular velocity vector of a previous frame. An IMU motion energy is determined based on the acceleration difference vector and the angular velocity difference vector by using a motion energy model, and an expression of the motion energy model is: , wherein, is an IMU motion energy, is an acceleration difference vector, is an angular velocity difference vector.

[0031] S15: determining key frame insertion information based on the edge density information and the IMU motion energy, and encoding the video to be encoded based on the target code rate and the key frame insertion information.

[0032] In the implementation of the present application, the determination of the key frame insertion information based on the edge density information and the IMU motion energy comprises: determining an edge density change rate based on the edge density information, and determining the key frame insertion information based on a comparison result of the edge density change rate and a first preset threshold, a comparison result of the IMU motion energy and a second preset threshold, and a comparison result of a number of consecutive frames of the video to be encoded and a third preset threshold.

[0033] Specifically, the edge density change rate is determined based on the edge density information, the edge density change rate is calculated through the edge density information between continuous frames, and the key frame insertion information is determined based on a comparison result of the edge density change rate and a first preset threshold, a comparison result of the IMU motion energy and a second preset threshold, and a comparison result of the number of continuous frames of the video to be encoded and a third preset threshold, the first preset threshold can be set to 12%, the second preset threshold can be set to 2.0, and the third preset threshold can be set to 90. When the edge density change rate is greater than the first preset threshold or the IMU motion energy is greater than the second preset threshold or the number of continuous frames is greater than the third preset threshold, the moment is taken as the insertion moment of the key frame, that is, the key frame insertion information. The key frame insertion information and the target code rate are transmitted to the encoder, the encoder encodes the video to be encoded based on the target code rate and the key frame insertion information, the encoder adjusts the encoding code rate of the video to the target code rate in real time, and the key frame insertion processing is performed on the video to be encoded according to the key frame insertion moment. It should be noted that, in order to prevent false triggering, a maximum of two forced key frame insertions are allowed in the same second. Through dynamic adjustment of the encoding, the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene can be adapted, and through the key frame insertion, the unmanned aerial vehicle jitter interference can be excluded and the motion blur problem can be improved.

[0034] In the embodiment of the application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor. The fusion of the IMU data and the height information can adapt to the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene and guarantee the stability in the dynamic scene. The canny edge detection and edge density analysis are performed on the video to be encoded to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, and the key frame insertion information is determined based on the edge density information and the IMU motion energy. The fusion of the image edge density and the IMU motion energy can analyze the insertion of the key frame, avoid false judgment caused by light interference, and reduce the loss of picture quality caused by motion blur. The encoder encodes the video to be encoded based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, guarantees the real-time performance, and reduces the bandwidth occupation.

[0035] Embodiment two Please refer to Figure 2 , Figure 2 is a flowchart of an intelligent video encoding method for an unmanned aerial vehicle in another embodiment of the application, and the method comprises: S201: Collecting inertial measurement unit (IMU) data and height information, and preprocessing the IMU data and the height information to obtain preprocessed IMU data and height information; S202: Weighted fusion and data mapping are performed on the preprocessed IMU data to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor; S203: Code rate analysis is performed based on the motion factor and the height factor to obtain a target code rate; S204: Gray processing is performed on each frame of image in the video to be encoded to obtain each frame of gray image, and Gaussian blur processing is performed on each frame of gray image to obtain each frame of gray image after Gaussian blur processing; S205: Adaptive histogram equalization processing is performed on each frame of gray image after Gaussian blur processing to obtain each frame of equalization image; S206: Double-threshold detection, morphological closing operation and center region processing are performed on each frame of equalization image to obtain corresponding image edge information, and image edge density is calculated based on the image edge information; S207: Acceleration vector and angular velocity vector between two continuous frames are determined based on the preprocessed IMU data, acceleration difference vector is calculated based on the acceleration vector between two continuous frames, and angular velocity difference vector is calculated based on the angular velocity vector between two continuous frames; S208: IMU motion energy is determined based on the acceleration difference vector and the angular velocity difference vector by using a motion energy model; S209: Key frame insertion information is determined based on the edge density information and the IMU motion energy, and an encoder encodes the video to be encoded based on the target code rate and the key frame insertion information.

[0036] In the embodiment of the application, weighted fusion and data mapping are performed on the preprocessed IMU data to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, code rate analysis is performed based on the motion factor and the height factor, and IMU data and height information are fused to perform code rate analysis, which can adapt to high-speed maneuvering of the unmanned aerial vehicle or high-low air switching scene and guarantee stability in a dynamic scene. Canny edge detection and edge density analysis are performed on the video to be encoded to obtain edge density information, IMU motion energy is determined based on the preprocessed IMU data by using a motion energy model, key frame insertion information is determined based on the edge density information and the IMU motion energy, and the insertion of the key frame is analyzed by fusing the image edge density and the IMU motion energy, which avoids misjudgment caused by light interference and reduces picture quality loss caused by motion blur. The encoder encodes the video to be encoded based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, guarantees real-time performance and reduces bandwidth occupation.

[0037] Embodiment three Please refer to Figure 3 , Figure 3is a structural component schematic diagram of the unmanned aerial vehicle video intelligent coding device in the embodiment of the present application, the device comprises: The data preprocessing module 31 is configured to collect inertial measurement unit (IMU) data and height information, and preprocess the IMU data and the height information to obtain preprocessed IMU data and height information. The factor acquisition module 32 is configured to perform weighted fusion and data mapping on the preprocessed IMU data to obtain a motion factor, and perform segmented linear compensation analysis based on the preprocessed height information to obtain a height factor. The code rate analysis module 33 is configured to perform code rate analysis based on the motion factor and the height factor to obtain a target code rate. The motion energy analysis module 34 is configured to perform Canny edge detection and edge density analysis on the video to be coded to obtain edge density information, and determine IMU motion energy based on the preprocessed IMU data using a motion energy model. The video coding module 35 is configured to determine key frame insertion information based on the edge density information and the IMU motion energy, and perform coding processing on the video to be coded based on the target code rate and the key frame insertion information.

[0038] In the embodiment of the present application, the specific implementation of the device item can refer to the implementation of the method item described above, which will not be repeated here.

[0039] In the embodiment of the present application, the preprocessed IMU data is subjected to weighted fusion and data mapping to obtain a motion factor, and segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor. Code rate analysis is performed based on the motion factor and the height factor, and the IMU data and the height information are fused to perform code rate analysis, which can adapt to high-speed maneuvering of the unmanned aerial vehicle or high-low air switching scenarios, and ensure stability in dynamic scenarios. Canny edge detection and edge density analysis are performed on the video to be coded to obtain edge density information, and IMU motion energy is determined based on the preprocessed IMU data using a motion energy model. Key frame insertion information is determined based on the edge density information and the IMU motion energy, and the image edge density and the IMU motion energy are fused to analyze the insertion of the key frame, thereby avoiding misjudgment caused by light interference and reducing the loss of picture quality caused by motion blur. The encoder performs coding processing on the video to be coded based on the target code rate and the key frame insertion information, which significantly improves the coding efficiency, reduces the bandwidth occupation while ensuring real-time performance.

[0040] Embodiment four Please refer to Figure 4 , Figure 4 is a structural component schematic diagram of the electronic device in the embodiment of the present application.

[0041] The embodiment of the present application further provides an electronic device, such asFigure 4 As shown, the electronic device includes a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and executable on the processor 43. Those skilled in the art can understand that the processor 43 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip computer or the processor 43 can be any conventional processor. The processor and the memory disclosed in the present application include but are not limited to these types of processors and memories. The processor and the memory disclosed in the present application are only examples and are not limited. Figure 4 The electronic device shown does not constitute a limitation on all devices, and can include more or fewer components than shown, or combine certain components. The memory 41 can be used to store the computer program 42 and various functional modules, and the processor 43 runs the computer program 42 stored in the memory 41 to perform various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both an internal memory and an external memory. The internal memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. The external memory can include a hard disk, a floppy disk, a ZIP disk, a U disk, a magnetic tape, etc. The processor 43 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip computer or the processor 43 can be any conventional processor. The processor and the memory disclosed in the present application include but are not limited to these types of processors and memories. The processor and the memory disclosed in the present application are only examples and are not limited.

[0042] As an embodiment, the electronic device includes one or more processors 43, a memory 41, and one or more computer programs 42, wherein the one or more computer programs 42 are stored in the memory 41 and configured to be executed by the one or more processors 43, and the one or more computer programs 42 are configured to perform the unmanned aerial vehicle video intelligent encoding method in any one of the above embodiments. For specific implementation process, please refer to the above embodiments, which will not be repeated here.

[0043] In the embodiment of the present application, the preprocessed IMU data is weighted and fused and data mapping is performed to obtain a motion factor, segmented linear compensation analysis is performed based on the preprocessed height information to obtain a height factor, and code rate analysis is performed based on the motion factor and the height factor, the IMU data and the height information are fused to perform code rate analysis, which can adapt to the high-speed maneuvering of the unmanned aerial vehicle or the high-low air switching scene, and guarantee the stability in the dynamic scene. The canny edge detection and edge density analysis are performed on the to-be-encoded video to obtain edge density information, the IMU motion energy is determined based on the preprocessed IMU data using a motion energy model, the key frame insertion information is determined based on the edge density information and the IMU motion energy, and the insertion of the key frame is analyzed by fusing the image edge density and the IMU motion energy, which avoids the misjudgment caused by the light interference and reduces the loss of picture quality caused by the motion blur. The to-be-encoded video is encoded by the encoder based on the target code rate and the key frame insertion information, which significantly improves the encoding efficiency, reduces the bandwidth occupation while ensuring the real-time performance.

[0044] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0045] In addition, the above describes in detail the unmanned aerial vehicle video intelligent encoding method and related device provided by the embodiments of the present application, and the principle and implementation manner of the present application are described by using specific examples in this paper, and the above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A method for intelligent encoding of UAV video, characterized in that, The method includes: The system collects inertial measurement unit (IMU) data and altitude information, and preprocesses the IMU data and altitude information to obtain preprocessed IMU data and altitude information. The preprocessed IMU data is weighted and fused and mapped to obtain motion factors. Piecewise linear compensation analysis is performed based on the preprocessed height information to obtain height factors. Based on the motion factor and altitude factor, a bitrate analysis is performed to obtain the target bitrate. Canny edge detection and edge density analysis are performed on the video to be encoded to obtain edge density information. Based on the preprocessed IMU data, the motion energy of the IMU is determined using a motion energy model. Based on the edge density information and IMU motion energy, keyframe insertion information is determined, and the encoder performs encoding processing on the video to be encoded based on the target bitrate and keyframe insertion information.

2. The UAV video intelligent encoding method according to claim 1, characterized in that, The preprocessing of IMU data and altitude information to obtain preprocessed IMU data and altitude information includes: The IMU data is low-pass filtered to obtain preprocessed IMU data. The height information is then processed using the sliding window averaging method to obtain preprocessed height information.

3. The UAV video intelligent encoding method according to claim 1, characterized in that, The preprocessed IMU data is weighted, fused, and mapped to obtain motion factors. Based on the preprocessed height information, piecewise linear compensation analysis is performed to obtain height factors, including: The preprocessed IMU data is vectorized to obtain acceleration vectors and angular velocity vectors; The acceleration vector and angular velocity vector are weighted and fused to obtain a weighted fused value. This weighted fused value is then mapped to the range [0.5, 1.5] using an S-shaped curve to obtain a motion factor. The expression for the motion factor is as follows: ， in, Here, 'a' is the motion factor, 'g' is the function that restricts the data between a specified minimum and maximum value, 'a' is the acceleration vector, and 'g' is the angular velocity vector. Based on the piecewise linear model, piecewise linear compensation analysis is performed using preprocessed height information to obtain the height factor.

4. The UAV video intelligent encoding method according to claim 1, characterized in that, The step of performing bitrate analysis based on the motion factor and height factor to obtain the target bitrate includes: The motion factor and altitude factor are input into the bitrate calculation model to obtain the target bitrate. The expression of the bitrate calculation model is as follows: ， Where FinalBitrate is the final bitrate, Clamp is a function that limits the data to between a specified minimum and maximum value, Bbase is the baseline bitrate, and HeightFactor is the height factor. It is a motion factor.

5. The UAV video intelligent encoding method according to claim 1, characterized in that, The process involves performing Canny edge detection and edge density analysis on the video to be encoded to obtain edge density information. Based on the preprocessed IMU data, the IMU motion energy is determined using a motion energy model, including: Each frame of the video to be encoded is converted to grayscale to obtain grayscale images of each frame. The expression for the grayscale conversion is as follows: ， in, The grayscale value of an image pixel. The R value of the image pixels. The G value of the image pixels. The B value of the image pixel; Gaussian blurring is applied to each frame of grayscale image to obtain the Gaussian blurred grayscale image. The expression for the Gaussian blurring is: ， Where Iblurred is the grayscale image after Gaussian blurring, Igray is the grayscale image, G is the Gaussian kernel, and * represents the convolution operation. Standard deviation; Adaptive histogram equalization is performed on each frame of grayscale image after Gaussian blurring to obtain equalized images of each frame. Perform double threshold detection, morphological closing operation and center region processing on each frame of equalized image to obtain the corresponding image edge information; Calculate image edge density based on image edge information; The acceleration vector and angular velocity vector between two consecutive frames are determined based on the preprocessed IMU data. The acceleration difference vector is calculated based on the acceleration vector between the two consecutive frames, and the angular velocity difference vector is calculated based on the angular velocity vector between the two consecutive frames. The motion energy of the IMU is determined using the motion energy model based on the acceleration difference vector and the angular velocity difference vector.

6. The UAV video intelligent encoding method according to claim 5, characterized in that, The expression for the image edge density is: ， Where Dedge is the image edge density, Nedge is the total number of edge pixels in the image edge information, W is the image width, and H is the image height.

7. The UAV video intelligent encoding method according to claim 5, characterized in that, The expression for the acceleration difference vector is: ， in, Let a[n] be the acceleration difference vector, where a[n] is the acceleration vector of the current frame and a[n-1] is the acceleration vector of the previous frame. The expression for the angular velocity difference vector is: ， in, Let ω be the angular velocity difference vector. This is the angular velocity vector of the current frame. This is the angular velocity vector of the previous frame; The expression for the kinetic energy model is: ， in, For IMU kinetic energy, For acceleration difference vectors, This is the angular velocity difference vector.

8. The UAV video intelligent encoding method according to claim 1, characterized in that, The determination of keyframe insertion information based on the edge density information and IMU motion energy includes: The edge density change rate is determined based on the edge density information, and keyframe insertion information is determined based on the comparison results of the edge density change rate with a first preset threshold, the IMU motion energy with a second preset threshold, and the number of consecutive frames of the video to be encoded with a third preset threshold.

9. A drone video intelligent encoding device, characterized in that, The device includes: Data preprocessing module: Used to collect inertial measurement unit (IMU) data and altitude information, and to preprocess the IMU data and altitude information to obtain preprocessed IMU data and altitude information; Factor acquisition module: used to perform weighted fusion and data mapping on preprocessed IMU data to obtain motion factors, and to perform piecewise linear compensation analysis based on preprocessed height information to obtain height factors; Bitrate analysis module: used to perform bitrate analysis based on the motion factor and height factor to obtain the target bitrate; Motion energy analysis module: used to perform Canny edge detection and edge density analysis on the video to be encoded, obtain edge density information, and determine the IMU motion energy based on the preprocessed IMU data using a motion energy model; The video encoding module is used to determine keyframe insertion information based on the edge density information and IMU motion energy, and the encoder performs encoding processing on the video to be encoded based on the target bitrate and keyframe insertion information.

10. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory, causing the electronic device to execute the UAV video intelligent encoding method as described in any one of claims 1 to 8.

Images