Unmanned aerial vehicle real-time video decoding method and system of lightweight deep learning model

Abstract

The invention relates to an unmanned aerial vehicle real-time video decoding method and system of a lightweight deep learning model, and relates to the technical field of unmanned aerial vehicle video decoding, and the method comprises the steps: receiving a coded video data stream transmitted by an unmanned aerial vehicle; calling a lightweight deep learning model, performing decoding complexity analysis on each coded video frame in sequence, configuring the lightweight deep learning model according to a decoding complexity analysis result, and performing decoding processing on each coded video frame to obtain a plurality of decoded video frames; and outputting and displaying the plurality of decoded video frames in sequence to complete real-time decoding display of the unmanned aerial vehicle video. According to the method, the problems of computing resource waste and key area decoding quality damage caused by unified decoding of the whole video frame in a traditional video decoding method are solved, and efficient utilization of the computing resources of the unmanned aerial vehicle and effective guarantee of the key area decoding quality are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle video decoding, in particular to a real-time unmanned aerial vehicle video decoding method and system based on a lightweight deep learning model. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles in power line inspection, search and rescue tasks, agricultural monitoring and other fields, the resource utilization efficiency of video real-time decoding and the decoding quality of key areas have become important technical requirements to ensure the accurate implementation of core tasks of unmanned aerial vehicles.

[0003] At present, the traditional video decoding method adopts a unified decoding strategy for the entire video frame, without considering the characteristics of limited computing resources and different decoding requirements of different areas in the unmanned aerial vehicle scene, which not only leads to waste of computing resources in simple areas, but also causes decoding quality damage in key areas due to the lack of targeted decoding support, thereby affecting the accuracy of subsequent defect detection and target recognition tasks. SUMMARY

[0004] The present application provides a real-time unmanned aerial vehicle video decoding method and system based on a lightweight deep learning model, which improves the status quo of waste of computing resources and damage to decoding quality in key areas caused by the traditional video decoding method which adopts a unified decoding strategy for the entire video frame.

[0005] The embodiments of the present application disclose the following technical solutions: In a first aspect, the embodiments of the present application provide a real-time unmanned aerial vehicle video decoding method based on a lightweight deep learning model, which comprises: receiving an encoded video data stream transmitted by an unmanned aerial vehicle, wherein the encoded video data stream comprises a plurality of encoded video frames; calling a lightweight deep learning model, sequentially analyzing the decoding complexity of each encoded video frame, configuring the lightweight deep learning model according to the decoding complexity analysis result, and decoding each encoded video frame to obtain a plurality of decoded video frames; sequentially outputting and displaying the plurality of decoded video frames to complete real-time decoding and display of the unmanned aerial vehicle video.

[0006] In a second aspect, the embodiments of the present application provide a real-time unmanned aerial vehicle video decoding system based on a lightweight deep learning model, which comprises: an encoded video data stream receiving module configured to receive an encoded video data stream transmitted by an unmanned aerial vehicle, wherein the encoded video data stream comprises a plurality of encoded video frames; The encoded video frame decoding processing module is used to call a lightweight deep learning model, sequentially perform decoding complexity analysis on each of the encoded video frames, configure the lightweight deep learning model according to the decoding complexity analysis results, and perform decoding processing on each of the encoded video frames to obtain multiple decoded video frames. The decoded video frame output and display module is used to output and display the multiple decoded video frames in sequence, thereby completing the real-time decoding and display of the drone video.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application proposes a real-time video decoding method and system for UAVs using a lightweight deep learning model. By performing step-by-step tasks such as receiving encoded video data streams, analyzing decoding complexity frame by frame, configuring a lightweight deep learning model, decoding differentiated regions, fusing and stitching multi-region results, and outputting and displaying decoded video frames, the method achieves the dual goals of real-time video decoding and quality assurance in key regions for UAVs in scenarios with limited computing resources. First, the system receives a data stream containing multiple encoded video frames from the drone, clarifies the transmission protocol and data format, and monitors transmission stability to ensure data integrity. Next, it extracts the encoded video frames in timestamp order, determines the complexity of each macroblock by extracting macroblock encoding parameters, and divides the region into high, medium, and low decoding complexity categories. Then, it configures a lightweight deep learning model based on the region complexity, constructing a deep decoding branch for high-complexity regions, a standard decoding branch for medium-complexity regions, and a fast decoding branch for low-complexity regions. These three branches are integrated to form a model adapted to the current video frame. Subsequently, the encoded data for each region is input into the corresponding branch for decoding, yielding the decoding results for the three regions. These results are then fused and stitched together based on location identifiers to form a complete decoded video frame. Finally, all decoded video frames are output sequentially in time, completing the real-time decoding and display of the drone video.

[0008] This application's technical solution addresses the problems of wasted computational resources, compromised decoding quality in key areas, and difficulty in adapting to dynamic UAV operation scenarios caused by uniform processing of the entire frame in traditional video decoding. It significantly improves resource utilization efficiency and the accuracy of detail rendering in key areas during UAV video decoding, providing high-quality video data support for UAV missions such as power line defect detection, search and rescue target identification, and crop pest and disease monitoring while ensuring real-time decoding. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating the real-time video decoding method for drones using a lightweight deep learning model provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a real-time video decoding system for a drone using a lightweight deep learning model, as provided in an embodiment of this application.

[0011] The components represented by each number in the attached diagram are explained below: Encoded video data stream receiving module 01, encoded video frame decoding processing module 02, and decoded video frame output display module 03. Detailed Implementation

[0012] This application provides a lightweight deep learning model-based real-time video decoding method and system for UAVs, which addresses the technical problem that traditional video decoding methods in the prior art adopt a uniform decoding strategy for the entire video frame, resulting in a waste of computing resources and easy damage to the decoding quality of key areas in UAV application scenarios.

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0015] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0016] Example 1, as shown in the appendix Figure 1 As shown, this application provides a method for real-time video decoding of drones using a lightweight deep learning model, the method comprising the following steps: S110: Receive the encoded video data stream transmitted by the UAV, the encoded video data stream including multiple encoded video frames; In this embodiment of the application, in application scenarios where computing resources are limited and video quality requirements are differentiated, such as when drones perform power line inspection and search and rescue missions, in order to provide basic data support for subsequent adaptive decoding based on content complexity and to avoid affecting decoding accuracy and real-time performance due to incomplete or delayed data reception, it is necessary to first accurately receive the encoded video data stream transmitted by the drone to ensure that subsequent decoding processing can be carried out based on complete and continuous video frame data.

[0017] Specifically, the transmission protocol and data format of the encoded video data stream must first be clearly defined. During flight, the drone will continuously collect and encode video data. These encoded video data streams need to be transmitted to the ground processing terminal through a wireless communication link. The transmission protocol needs to be adapted to the signal stability requirements of the drone in the mobile scenario, and the data format needs to include key information such as the encoding parameters of the video frame, timestamp, and frame number.

[0018] The timestamp is used to ensure the timing consistency of subsequent video frame processing, and the frame number can help check whether there are frame drops during data transmission, providing data basis for subsequent frame-by-frame decoding.

[0019] Meanwhile, during the reception of encoded video data streams, the stability and integrity of data transmission need to be monitored in real time. Because drone operations are often accompanied by complex environmental interference, such as electromagnetic interference during power line inspections or signal fluctuations caused by terrain obstruction during search and rescue missions, these factors can cause data transmission interruptions or data packet corruption.

[0020] Therefore, it is necessary to monitor the data transmission status and count the number of data packets received, lost, and erroneous per unit time in real time. If the data packet loss rate or error rate is found to exceed the preset threshold (e.g., loss rate > 3%, error rate > 1%), a retransmission request is immediately triggered, and a data retransmission command is sent to the drone to ensure that no critical information is missing from the received encoded video data stream.

[0021] Furthermore, the received encoded video data stream undergoes preliminary parsing and classification for storage. During the parsing process, the data stream needs to be split into individual encoded video frames based on the frame sequence number and timestamp, while extracting basic identification information for each encoded video frame, such as frame type (I-frame, P-frame, B-frame) and encoding resolution.

[0022] When classifying and storing video frames, a video frame storage queue should be established according to the timestamp order to ensure that when the lightweight deep learning model is called up later for decoding complexity analysis, it can be processed frame by frame according to the actual time sequence of video capture, avoiding temporal disorder in the decoded video due to frame order disorder.

[0023] S120: Call up a lightweight deep learning model, perform decoding complexity analysis on each of the encoded video frames in sequence, configure the lightweight deep learning model according to the decoding complexity analysis results, and perform decoding processing on each of the encoded video frames to obtain multiple decoded video frames. In this embodiment of the application, in order to avoid the waste of computing resources and insufficient decoding quality in key areas caused by the traditional unified decoding method, a lightweight deep learning model needs to be invoked to complete the complexity analysis, model adaptation configuration and decoding processing of the encoded video frames in sequence, so as to achieve efficient use of resources and quality assurance in key areas of UAV video decoding.

[0024] Specifically, the first video frame to be processed (i.e., the first encoded video frame) is extracted from the multiple received encoded video frames according to the timestamp order, and it is used as the initial analysis object to carry out decoding complexity analysis.

[0025] During the analysis, it is necessary to extract the macroblock coding parameters of the first encoded video frame, determine the decoding difficulty of each macroblock through parameter features, and then divide the entire video frame into regions with different complexity levels according to the macroblock complexity to form the decoding complexity analysis results of the corresponding frame.

[0026] Furthermore, based on the obtained decoding complexity analysis results, the lightweight deep learning model is configured accordingly. This lightweight deep learning model includes a deep decoder, a standard decoder, and a fast decoder. It is necessary to match the corresponding decoder to each region based on the requirements of different complexity areas, and adjust the number of decoding units called within the decoder to form a dedicated model adapted to the current video frame.

[0027] Furthermore, the configured dedicated model is used to decode the current video frame. The encoded data of different complexity regions are input into the corresponding decoders respectively. After obtaining the decoding results of each region, they are spliced ​​and merged according to the original position to generate a complete decoded video frame.

[0028] Following the above process for processing the first video frame (i.e. the first encoded video frame), complexity analysis, model configuration, and decoding operations are performed sequentially on the remaining encoded video frames to ensure that each video frame can obtain an appropriate decoding scheme based on its own content characteristics. In the end, multiple consecutive and reliable decoded video frames are obtained, providing clear and real-time video data support for the accurate execution of UAV missions.

[0029] Step S120 in the method provided in this application embodiment includes: According to the timestamp order, the first encoded video frame is extracted from the plurality of encoded video frames, and the decoding complexity analysis is performed on the first encoded video frame to obtain the first decoding complexity analysis result; Configure the lightweight deep learning model based on the first decoding complexity analysis result to obtain the first lightweight deep learning model; The first encoded video frame is decoded using the first lightweight deep learning model to obtain the first decoded video frame. Following the method of processing the first encoded video frame to obtain the corresponding first decoded video frame, the remaining encoded video frames are processed sequentially to obtain the corresponding decoded video frames, resulting in multiple decoded video frames.

[0030] In this embodiment of the application, in order to avoid the waste of computing resources caused by the traditional unified decoding method in the drone scenario, and at the same time to ensure the decoding quality of key areas, it is necessary to complete the extraction of encoded video frames, complexity analysis, model configuration and decoding processing in sequence, so as to achieve accurate resource allocation and quality assurance of key areas in drone video decoding.

[0031] Specifically, the first encoded video frame is extracted according to the timestamp order of the encoded video frames. The timestamp order must strictly match the actual timing of the video captured by the drone. For example, if the drone captures video at a rate of 25 frames per second during power line inspection, the corresponding timestamps of the encoded video frames are spaced 40 milliseconds apart. During extraction, the first frame must be selected in this time interval order as the first encoded video frame to avoid frame order disorder causing subsequent video decoding jumps and affecting the judgment capabilities of subsequent tasks such as defect detection and target recognition.

[0032] After extracting the first encoded video frame, a decoding complexity analysis is performed on it to obtain the first decoding complexity analysis result.

[0033] In the method provided in this application embodiment, "performing decoding complexity analysis on the first encoded video frame to obtain a first decoding complexity analysis result" includes: Extract the macroblock coding parameters of the first encoded video frame, and determine the complexity of each macroblock based on the macroblock coding parameters; The first encoded video frame is divided into regions based on the complexity of each macroblock, resulting in a first decoding complexity region, a second decoding complexity region, and a third decoding complexity region. The first decoding complexity region, the second decoding complexity region, and the third decoding complexity region are taken as the first decoding complexity analysis result.

[0034] In this embodiment of the application, in order to avoid the misallocation of computing resources or insufficient decoding quality of key areas due to the inability to distinguish the decoding requirements of different areas within a video frame, it is necessary to clarify the decoding complexity level of each area within the first encoded video frame through macroblock coding parameter analysis and area division, so as to realize the subsequent matching and adaptation decoding strategy for areas with different complexity.

[0035] First, the macroblock coding parameters of the first encoded video frame are extracted. These macroblock coding parameters are the core basis for reflecting the difficulty of macroblock decoding, encompassing information such as the macroblock's coding mode, quantization parameters, and motion vectors.

[0036] Specifically, the smaller the quantization parameter value, the richer the detailed information retained by the macroblock during the encoding process, and the greater the amount of data and computation required for subsequent decoding, resulting in higher complexity. The larger the amplitude of the motion vector, the more intense the motion of the objects in the scene corresponding to the macroblock, and the more complex the motion compensation calculation required during decoding, which also increases the decoding difficulty.

[0037] During the extraction process, it is necessary to ensure that all macroblocks within the first coded video frame are covered to avoid incomplete subsequent complexity judgments due to missing some macroblock parameters. For example, for a video frame with a resolution of 1920×1080, the parameters of each 16×16 or 32×32 macroblock need to be extracted row by row and column by column to form a complete set of macroblock coding parameters.

[0038] After obtaining the macroblock encoding parameters, the complexity of each macroblock is determined based on these parameters. Specifically, a standard for the correspondence between macroblock encoding parameters and complexity needs to be established first. For example, a macroblock is classified as high complexity when the quantization parameter is ≤20 and the motion vector amplitude is ≥10; a macroblock is classified as medium complexity when the quantization parameter is between 21 and 35 and the motion vector amplitude is between 3 and 9; and a macroblock is classified as low complexity when the quantization parameter is ≥36 and the motion vector amplitude is ≤2. Each macroblock is then evaluated according to the above standard to obtain the complexity level corresponding to each macroblock, providing basic data for subsequent region partitioning.

[0039] Furthermore, the first coded video frame is divided into regions based on the complexity of each macroblock. During the division process, adjacent macroblocks with the same complexity level need to be merged to form contiguous regions, avoiding the appearance of isolated macroblock regions.

[0040] Specifically, multiple adjacent high-complexity macroblocks are merged into a single high-complexity region block, multiple adjacent medium-complexity macroblocks are merged into a single medium-complexity region block, and adjacent low-complexity macroblocks are merged into a single low-complexity region block. The resulting first decoding complexity region (high complexity), second decoding complexity region (medium complexity), and third decoding complexity region (low complexity) each contain multiple consecutive region blocks, consistent with the scene characteristics of actual video frames.

[0041] At the same time, after the regions are divided, each region needs to be identified and recorded, including the region's coordinate range and the number of macroblocks it contains. This information will serve as an important basis for subsequent model configuration and decoding result fusion and splicing, ensuring that the decoding data and reconstruction position of each region can be accurately located in the future.

[0042] Finally, the first, second, and third decoding complexity regions, which were divided into regions, were taken as the first decoding complexity analysis result. This result clearly shows the differences in decoding requirements in different regions within the first coded video frame, providing a clear direction for configuring corresponding decoding branches and allocating computing resources based on the region complexity in the future.

[0043] Furthermore, a lightweight deep learning model is configured based on the obtained first decoding complexity analysis results, thereby obtaining the first lightweight deep learning model to achieve differentiated decoding support for different complexity regions of the first coded video frame.

[0044] The method provided in this application embodiment, "configuring the lightweight deep learning model according to the first decoding complexity analysis result to obtain the first lightweight deep learning model" includes: The lightweight deep learning model includes a deep decoder, a standard decoder, and a fast decoder. The deep decoder includes multiple deep decoding units, the standard decoder includes multiple standard decoding units, and the fast decoder includes multiple fast decoding units. A first deep decoding branch is configured based on the first decoding complexity region and the plurality of deep decoding units; a first standard decoding branch is configured based on the second decoding complexity region and the plurality of standard decoding units; and a first fast decoding branch is configured based on the third decoding complexity region and the plurality of fast decoding units. The first deep decoding branch, the first standard decoding branch, and the first fast decoding branch are integrated to obtain the first lightweight deep learning model.

[0045] In this embodiment of the application, in order to enable the lightweight deep learning model to accurately adapt to the decoding requirements of different complexity regions within the first encoded video frame, it is necessary to configure and integrate dedicated decoding branches for different regions based on the first decoding complexity analysis results, so as to achieve differentiated decoding support for each region and meet the dual requirements of efficient resource utilization and video quality assurance in key areas under the drone scenario.

[0046] Specifically, the lightweight deep learning model is designed to adapt to the differentiated decoding needs in drone scenarios. It includes a deep decoder, a standard decoder, and a fast decoder to handle decoding tasks of different complexity regions in video frames, respectively.

[0047] The depth decoder is used to process highly complex regions. It has multiple built-in depth decoding units, each with a sophisticated decoding algorithm, capable of processing video data with rich details and intense motion.

[0048] In addition, the standard decoder is adapted to medium-complexity regions, and the multiple standard decoding units it contains employ a balanced decoding strategy to control computational overhead while ensuring decoding quality.

[0049] Furthermore, the fast decoder targets low-complexity regions by simplifying the algorithm with multiple fast decoding units to complete decoding with minimal computational cost, thereby improving overall decoding efficiency. These three types of decoders and their corresponding decoding units provide the hardware and algorithmic foundation for subsequent branch configurations.

[0050] Furthermore, three types of decoding branches are configured based on the results of the first decoding complexity analysis. First, a first deep decoding branch is configured for the first decoding complexity region (high complexity) and multiple deep decoding units.

[0051] In the method provided in this application embodiment, "configuring a first depth decoding branch according to the first decoding complexity region and the plurality of depth decoding units" includes: Obtain the baseline depth decoding unit call count and baseline depth decoding complexity corresponding to the depth decoder, wherein the baseline depth decoding unit call count is half of the total number of the plurality of depth decoding units; Based on the complexity of each macroblock in the first decoding complexity region, obtain the first average complexity of the first decoding complexity region; The first adjustment coefficient is obtained by calculating the ratio of the first average complexity to the baseline depth decoding complexity. The number of reference depth decoding unit calls is adjusted according to the first adjustment coefficient to obtain the number of first depth decoding unit calls; The first depth decoding branch is formed by calling a number of depth decoding units from the plurality of depth decoding units corresponding to the number of calls to the first depth decoding unit.

[0052] In this embodiment of the application, in order to ensure that the first deep decoding branch can accurately match the decoding requirements of the first decoding complexity region (high complexity), it is necessary to determine the appropriate number of deep decoding unit calls through benchmark parameter setting, complexity calculation and coefficient adjustment, so as to achieve a balance between decoding quality and computational resource consumption in the high complexity region.

[0053] Specifically, the baseline depth decoding unit call count and baseline depth decoding complexity are first obtained for the depth decoder. The depth decoder is a core component in a lightweight deep learning model used to handle decoding tasks in highly complex regions, and it is composed of multiple depth decoding units.

[0054] The method provided in this application embodiment includes the following steps for constructing the "depth decoder": Based on the baseline depth decoding complexity, multiple high-complexity regions of samples are collected, and the encoded data of each high-complexity region of the samples is extracted to construct a high-complexity encoded dataset of samples. Extract the sample decoding results of each sample high-complexity encoded data in the sample high-complexity encoded dataset, and construct a sample high-complexity decoding result set; Obtain a preset number of depth decoding units, and construct a multiple decoding unit architecture based on the preset number of depth decoding units using a deep learning algorithm; Based on the sample high-complexity encoding dataset and the sample high-complexity decoding result set, the multiple decoding unit architectures are trained respectively to generate multiple deep decoding units; The multiple depth decoding units are combined to form the depth decoder.

[0055] Specifically, multiple high-complexity regions were first collected based on a baseline depth decoding complexity. The baseline depth decoding complexity was set to 80 (the quantization value based on macroblock encoding parameter conversion). During collection, regions in the UAV operation scene that met this complexity were selected. The size of each sample region was uniformly 256×256 pixels (corresponding to 16×16 macroblocks), and the number of samples collected was no less than 5000. At the same time, the encoding format of each sample was recorded as H.265, the quantization parameter range was 10-30, and the motion vector amplitude range was 8-20, to ensure that the samples covered diverse high-complexity scenarios and were of uniform specifications.

[0056] Furthermore, after collecting the sample regions, the encoded data of the high-complexity regions of each sample are extracted to construct a sample high-complexity encoded dataset.

[0057] Specifically, during extraction, the NALU data corresponding to the sample region is extracted, including the I-Slice type and reference frame index of the Slice layer, and the intra-prediction mode and transform coefficients of the macroblock layer. It is stored in the structure of "sample ID-encoding format-quantization parameter-motion vector-encoded data block". At the same time, the values ​​of quantization parameter, motion vector and other values ​​are normalized to the 0-1 range to eliminate the impact of the difference in units on model training.

[0058] Furthermore, the sample decoding results of each encoded data in the sample high-complexity encoded dataset are extracted, and then a sample high-complexity decoding result set is constructed.

[0059] Specifically, the sample decoding results are obtained through the HM16.20 offline decoder, and the decoding output is 256×256 pixel data in YUV420 format. During decoding, the reference frame number is set to 1 I-frame and 3 P-frames, and the deblocking filter intensity is 4 levels to ensure the accuracy of the label data. The data is stored in the structure of "sample ID-Y channel data-U channel data-V channel data" and a one-to-one index is established with the sample high-complexity encoded dataset to ensure accurate matching between the input and the label during training.

[0060] Furthermore, a preset number of depth decoding units is obtained, and multiple decoding unit architectures are constructed. Specifically, the preset number of depth decoding units is set to 10, which satisfies the requirements for parallel decoding efficiency while avoiding excessive consumption of drone memory.

[0061] Meanwhile, the decoding unit architecture adopts the improved MobileNetV3 algorithm. The input layer size is 256×256×3, the hidden layer contains a depthwise separable convolutional layer with 6 3×3 convolutional kernels and 2 bottleneck structures with dilation rates of 2 and 4 respectively, the output layer is a 256×256×3 pixel prediction layer, the activation function is ReLU6, the loss function is L1 loss, and the initial weights are initialized using the He normal distribution.

[0062] Furthermore, based on the high-complexity encoded dataset and the high-complexity decoded result set, multiple decoding unit architectures were trained. Training employed end-to-end supervised learning with a learning rate of 0.001, a batch size of 32, and 100 iterations. The Adam optimizer (momentum 0.9) was used for the first 50 iterations, and the SGD optimizer (momentum 0.9, learning rate decay coefficient 0.95) was switched for the last 50 iterations. An early stopping mechanism was also introduced (the process stopped if the validation set loss did not decrease for five consecutive data points).

[0063] In addition, after training, 20% of the total data volume was used as a test set for validation, requiring PSNR (Peak Signal-to-Noise Ratio) ≥38dB and SSIM (Structural Similarity Index) ≥0.92. Ten architectures that met the criteria were selected as deep decoding units, and those that did not meet the criteria were retrained after adjusting the number of convolutional layers.

[0064] Finally, the 10 deep decoding units are combined to form a deep decoder. During the combination process, load balancing scheduling is required to evenly distribute the encoded data of high-complexity regions to each unit; a 100Mbps inter-unit communication interface is set up to support temporary interaction of decoding results; at the same time, fault detection is performed to monitor the computation time of each unit in real time (threshold 50ms). If a unit times out, the task is assigned to a backup unit, ultimately forming a deep decoder with parallel decoding and fault tolerance capabilities.

[0065] Furthermore, after the deep decoder is built, the configuration of the first deep decoding branch needs to be completed step by step based on the core parameters of the decoder and the actual situation of the first decoding complexity region, so as to ensure that the branch can accurately adapt to the decoding requirements of the high complexity region.

[0066] Specifically, the number of reference depth decoding unit calls and the reference depth decoding complexity are obtained first.

[0067] The number of reference depth decoding unit calls is set to half the total number of depth decoding units contained in the depth decoder. For example, if the depth decoder consists of 10 depth decoding units, the number of reference depth decoding unit calls is 5. This setting can avoid the waste of UAV computing resources due to too many initial calls, and also prevent the number from being too small to meet the basic decoding requirements of high-complexity areas.

[0068] In addition, the baseline depth decoding complexity is a pre-set standard value that integrates the macroblock complexity characteristics of common high-complexity scenarios in UAVs, serving as a reference for subsequently judging the actual decoding difficulty of the first decoding complexity region.

[0069] Furthermore, based on the complexity of each macroblock in the first decoding complexity region, the first average complexity of that region is calculated. During the calculation, the complexity values ​​of all macroblocks within the first decoding complexity region are first calculated (derived from macroblock encoding parameter conversion), and then the overall average complexity of the region is obtained using the arithmetic mean method.

[0070] For example, a certain first decoding complexity region contains 20 macroblocks, and the complexity values ​​of each macroblock are 82, 88, 90...85 respectively. The first average complexity is calculated to be 86. This first average complexity can intuitively reflect the overall decoding difficulty of the region.

[0071] Furthermore, the ratio of the first average complexity to the baseline depth decoding complexity is calculated to obtain the first adjustment coefficient. If the first average complexity is higher than the baseline depth decoding complexity, it indicates that the current region's decoding difficulty is greater than that of typical high-complexity scenarios, requiring an increase in the number of decoding unit calls; in this case, the first adjustment coefficient is greater than 1.

[0072] Conversely, if the first average complexity is lower than the baseline depth decoding complexity, it indicates that the current region is relatively easy to decode, and the number of decoding unit calls can be appropriately reduced, with the first adjustment coefficient being less than 1.

[0073] For example, the baseline depth decoding complexity is 80, the first average complexity is 86, and the ratio of the two is 1.075, that is, the first adjustment factor is 1.075 (86 / 80=1.075); if the first average complexity is 75, the ratio is 0.9375, and the adjustment factor is 0.9375 (75 / 80=0.9375).

[0074] Furthermore, the number of calls to the baseline depth decoding unit is adjusted according to the first adjustment coefficient to obtain the number of calls to the first depth decoding unit. Specifically, the baseline call count needs to be multiplied by the first adjustment coefficient, and the result rounded down (ensuring the call count is an integer to meet actual operational requirements).

[0075] For example, when the baseline call count is 5 and the first adjustment factor is 1.075, 5 × 1.075 ≈ 5.375, and the number of calls to the first depth decoding unit after rounding is 5; if the first adjustment factor is 0.9375, 5 × 0.9375 = 4.6875, and the number of calls to the first depth decoding unit after rounding is 5 (if the decimal part of the result is ≥ 0.5, it is rounded up; here, 4.6875 is rounded up to 5. The specific rounding rules need to be determined based on the actual decoding accuracy requirements to ensure that resources are not wasted and quality is not affected).

[0076] Finally, from the multiple depth decoding units of the depth decoder, the number of units corresponding to the number of calls to the first depth decoding unit is called to determine multiple depth decoding call units. These units together constitute the first depth decoding branch, providing continuous and reliable computing power support for the fine decoding of the subsequent first decoding complexity region.

[0077] Furthermore, referring to the configuration method of the first depth decoding branch, the configuration of the first standard decoding branch and the first fast decoding branch are completed respectively.

[0078] Specifically, for the second decoding complexity region (medium complexity), first obtain the baseline standard decoding unit call count and baseline standard decoding complexity corresponding to the standard decoder, then calculate the second average complexity of the second decoding complexity region, compare it with the baseline standard decoding complexity to obtain the second adjustment coefficient, adjust the baseline call count according to the coefficient, and after determining the first standard decoding unit call count, call the corresponding number of units from multiple standard decoding units to form the first standard decoding branch.

[0079] Meanwhile, for the third decoding complexity region (low complexity), based on the baseline fast decoding unit call count and baseline fast decoding complexity of the fast decoder, the ratio of the third average complexity of the third decoding complexity region to the baseline value is calculated to obtain the third adjustment coefficient. After adjustment, the first fast decoding unit call count is determined, and the corresponding number of fast decoding units are called to construct the first fast decoding branch.

[0080] After configuring the three types of decoding branches, the first deep decoding branch, the first standard decoding branch, and the first fast decoding branch need to be integrated to form the first lightweight deep learning model.

[0081] Specifically, during integration, branch data allocation is performed first, and the three types of branches are associated with their corresponding complexity regions in advance. That is, the data of the first decoding complexity region is directed to the first deep decoding branch, the data of the second decoding complexity region is allocated to the first standard decoding branch, and the data of the third decoding complexity region is passed to the first fast decoding branch, so as to ensure that each branch only processes the region encoding data that is suitable and avoid data mismatch.

[0082] Simultaneously, a branch synchronization scheduling mechanism is constructed to monitor the decoding progress of the three branches in real time. If the first deep decoding branch experiences a delay due to processing highly complex data, the scheduling mechanism will temporarily allocate a small amount of spare computing resources to assist the branch, ensuring that the three branches can complete the decoding tasks of their respective regions synchronously, and avoiding the impact of progress differences on the fusion of subsequent results.

[0083] In addition, model parameters need to be stored, recording core configuration parameters such as the number of deep decoding units in the first deep decoding branch, the number of standard decoding units in the first standard decoding branch, and the number of fast decoding units in the first fast decoding branch. When processing other encoded video frames in the future, the parameter template can be directly called and fine-tuned according to the complexity of the new frame, reducing the time spent on repeated configuration, and finally forming the first lightweight deep learning model that can adapt to the current encoded video frame.

[0084] Furthermore, based on the constructed first lightweight deep learning model, the first encoded video frame is decoded to obtain the first decoded video frame.

[0085] The method provided in this application embodiment, "decoding the first encoded video frame using the first lightweight deep learning model to obtain the first decoded video frame," includes: The encoded data of each partition in the first decoding complexity region are input into the first depth decoding branch to obtain the decoding results of each partition in the first decoding complexity region, and thus obtain the first decoding result. The encoded data of each partition in the second decoding complexity region are input into the first standard decoding branch to obtain the decoding results of each partition in the second decoding complexity region, thus obtaining the second decoding result. The encoded data of each partition in the third decoding complexity region are input into the first fast decoding branch to obtain the decoding results of each partition in the third decoding complexity region, and thus obtain the third decoding result. The first decoding result, the second decoding result, and the third decoding result are fused and spliced ​​together to obtain the first decoded video frame.

[0086] In this embodiment of the application, in order to enable the first lightweight deep learning model to accurately decode regions of different complexity within the first encoded video frame, ensuring the decoding details of high-complexity key regions while not wasting computational resources in low-complexity regions, it is necessary to match the corresponding decoding branch according to the region to complete the decoding, and then merge and splice the results of each region to form a complete first decoded video frame, so as to meet the requirements of subsequent UAV missions for video accuracy and real-time performance.

[0087] Specifically, the first decoding complexity region (high complexity) is processed by inputting the encoded data of each partition in the first decoding complexity region into the first depth decoding branch to obtain the decoding results of each partition in the first decoding complexity region, and then summing them up to obtain the first decoding result.

[0088] The method provided in this application embodiment, which involves "inputting the encoded data of each partition in the first decoding complexity region into the first depth decoding branch to obtain the decoding results of each partition in the first decoding complexity region, and obtaining the first decoding result", includes: Determine the first partition from the first decoding complexity region, extract the encoded data of the first partition, and input the encoded data of the first partition into the first depth decoding branch; The encoded data of the first partition is decoded by multiple deep decoding call units in the first deep decoding branch, and multiple first partition unit decoding results are obtained. The decoding results of the plurality of first partition units are fused to obtain the decoding result of the first partition; Following the same method used to process the first partition, the remaining partitions in the first decoding complexity region are processed sequentially to obtain the decoding results of each partition in the first decoding complexity region. The decoding results of each partition in the first decoding complexity region are summarized to obtain the first decoding result.

[0089] Specifically, the first partition is determined from the first decoding complexity region and its encoded data is extracted. The first decoding complexity region typically contains 2-4 partitions, each with a size of 256×256 pixels (corresponding to 16×16 macroblocks). When determining the first partition, the area with higher criticality in the image is selected first. For example, in a drone power line inspection scenario, the area where the power tower insulator is located is prioritized as the first partition.

[0090] At the same time, when extracting encoded data, it is also necessary to completely extract the macroblock encoding information corresponding to the partition, including quantization parameters, motion vectors, intra-frame prediction modes, etc., to ensure that no data is missing before inputting it into the first depth decoding branch.

[0091] Furthermore, after inputting the encoded data of the first partition into the first deep decoding branch, the decoding process is performed by multiple deep decoding call units (e.g., 6) within the branch. The 6 deep decoding call units will perform operations on the encoded data of the same partition in parallel. Each unit outputs one decoding result of the first partition unit based on its own trained decoding model (trained on a high-complexity dataset, PSNR≥38dB, SSIM≥0.92), forming 6 parallel decoding results.

[0092] Furthermore, the decoding results of multiple first partition units are fused to eliminate potential errors in decoding by a single depth decoding call unit, ensuring the accuracy of the first partition decoding results.

[0093] Specifically, the fusion process uses a voting fusion method, which involves counting the values ​​of each pixel in the six results and selecting the value that appears most frequently as the final value for that pixel. For example, if a pixel is 235 in four of the six results and 234 in two, then the final value will be 235.

[0094] If two or more results differ significantly (e.g., pixel value difference exceeds 10), the decoding process for that partition will be retried to avoid significant deviations after fusion.

[0095] Similarly, following the same method used for the first partition, the remaining partitions in the first decoding complexity region are processed sequentially. For example, in a search and rescue scenario, the first decoding complexity region includes a suspected person area (first partition) and a fast-moving object area (second partition). When processing the second partition, its encoded data is extracted and input into the first depth decoding branch. Six depth decoding calling units decode in parallel and output the results. Then, the decoding result of the second partition is obtained through voting fusion. If a third partition exists, the above process is repeated to ensure that the decoding standard and accuracy of each partition are consistent.

[0096] Finally, the decoding results of each partition in the first decoding complexity region are summarized. During the summarization, the position information of each partition needs to be recorded to ensure that it can accurately correspond to the original frame position when fused with the results of other complexity regions later. At the same time, the integrity check of the summarized first decoding result is performed to confirm that there are no missing or duplicate partition results, and then the final first decoding result is formed. This result can clearly present all the key details of the first decoding complexity region.

[0097] Furthermore, the second decoding complexity region is processed in the same way as the first decoding result. The second decoding complexity region contains 2-3 partitions. The encoded data input of each partition consists of a first standard decoding branch composed of 3-5 standard decoding call units (the standard decoding call units are adjusted and called from the 8 standard decoding units of the standard decoder). After each unit decodes in parallel, there is no need for complex voting fusion. The average value of the result is directly taken as the partition decoding result. After processing all partitions in sequence, the results are summarized to obtain the second decoding result, so as to ensure the balance between decoding quality and efficiency in the medium complexity region.

[0098] Furthermore, the third decoding complexity region is processed in the same way as the first decoding result. This third decoding complexity region contains 3-5 partitions. The encoded data of each partition is input into the first fast decoding branch, which consists of 2-4 fast decoding call units (the fast decoding call units are adjusted from the 6 fast decoding units of the fast decoder). The fast decoding units complete the decoding with a simplified algorithm and directly output the decoding results of each partition. The third decoding result is obtained by summarizing all partitions, so as to save the computational resources of the low complexity region to the greatest extent.

[0099] Furthermore, after obtaining the first, second, and third decoding results, the three types of decoding results are fused and spliced ​​together to restore the complete picture of the first encoded video frame.

[0100] In the method provided in this application embodiment, "fusing and splicing the first decoding result, the second decoding result, and the third decoding result to obtain the first decoded video frame" includes: Obtain the position identifiers of the first decoding complexity region, the second decoding complexity region, and the third decoding complexity region; Based on the location identifier, the first decoding result, the second decoding result, and the third decoding result are reconstructed to obtain the first decoded video frame.

[0101] In this embodiment of the application, in order to integrate the decoding results of regions with different complexity into a complete and coherent video frame, it is necessary to first clarify the position identifier of each region, and then reconstruct the position based on the position identifier, so as to ensure that the final output first decoded video frame can retain the details of the high-complexity region and maintain the integrity and coherence of the overall picture.

[0102] Specifically, the location identifiers of the first decoding complexity region, the second decoding complexity region, and the third decoding complexity region are first obtained.

[0103] The location identifiers are the pixel coordinate ranges of each region and its corresponding partition within the original encoded video frame. For example, the first decoding complexity region contains two partitions: the first partition has coordinates of 150-405×150-405, and the second partition has coordinates of 406-661×150-405. The second decoding complexity region contains two partitions with coordinates of 150-405×406-661 and 406-661×406-661, respectively. The third decoding complexity region contains three partitions with coordinates of 0-149×0-816, 0-149×662-816, and 150-661×662-816. These coordinate ranges are recorded synchronously when the first encoded video frame is divided into regions to ensure accurate correspondence between each region and its position in the original frame, providing an accurate basis for subsequent reconstruction.

[0104] Furthermore, after obtaining the location identifier, the location is reconstructed based on the identifier for the first decoding result, the second decoding result, and the third decoding result.

[0105] During the reconstruction process, the first decoding result is first filled into the high-complexity region of the original frame according to the corresponding partition coordinates. For example, the decoded data of the first partition in the first decoding result is filled into the coordinate range of 150-405×150-405, and the data of the second partition is filled into the range of 406-661×150-405. At the same time, the second decoding result is filled into the coordinate range of the medium-complexity region, and the third decoding result is filled into the coordinate range of the low-complexity region.

[0106] When filling, it is necessary to ensure that the data in each area does not overlap or omit anything. At the same time, the edge pixels of adjacent complexity areas should be smoothed. For example, at the junction of the first decoding complexity area and the second decoding complexity area, the average value of 10 edge pixels should be taken as the transition to avoid obvious screen break lines and ensure the overall visual effect is consistent.

[0107] In addition, the integrity of the image needs to be verified during the reconstruction process. During verification, the total number of pixels in the filled video frame is confirmed to be consistent with the number of pixels in the original encoded video frame by counting pixels. At the same time, it is checked whether there are abrupt changes in pixel values ​​at the junctions of each region. If so, the edge smoothing parameters are readjusted until the image transitions naturally.

[0108] By using the above-mentioned location-based reconstruction method, positional deviations during the splicing of results from regions of different complexity are avoided, ensuring the integrity and continuity of the first decoded video frame. At the same time, it takes into account the decoding accuracy and overall decoding efficiency of high-complexity regions, meeting the core requirements of real-time video decoding for UAVs.

[0109] Furthermore, following the method of processing the first encoded video frame to obtain the corresponding first decoded video frame, the remaining encoded video frames are processed sequentially to obtain the corresponding decoded video frames, resulting in multiple decoded video frames.

[0110] During processing, the timestamp order of the encoded video frames must be strictly followed to ensure that the timing of the decoded video is consistent with the original video stream captured by the drone, thus avoiding jumps in the image.

[0111] For example, the first encoded video frame with the earliest timestamp is processed first to obtain the first decoded video frame. Then, subsequent encoded video frames are processed sequentially according to their timestamps. That is, for each encoded video frame to be processed, the process of "decoding complexity analysis (extracting macroblock coding parameters and dividing into high / medium / low complexity regions) - configuring a lightweight deep learning model (constructing corresponding decoding branches) - decoding each region (obtaining three types of decoding results) - fusion and splicing of results (reconstructing the complete video frame based on location identifiers)" is repeated to finally generate the corresponding decoded video frame.

[0112] In addition, the decoding process will optimize efficiency for common features of consecutive frames. For example, if the drone continuously shoots the same type of scene and the complexity distribution of the subsequent encoded video frames is similar to that of the previous frame, the decoding branch configuration parameters of the previous frame can be reused and fine-tuned only according to the slight complexity changes of the current frame to reduce the time spent on repeated calculations and ensure that the efficiency requirements of drone real-time decoding can still be met during batch processing.

[0113] S130: Output and display the multiple decoded video frames sequentially to complete the real-time decoding and display of the drone video.

[0114] In this embodiment of the application, in scenarios where computing resources are limited and the real-time performance of video and key details need to be considered when drones are performing tasks such as power line inspection and search and rescue, in order to enable ground staff to obtain clear video images in a timely manner and accurately identify key information such as power tower defects and suspected personnel, multiple pre-processed decoded video frames need to be output and displayed in sequence to complete the real-time decoding and display of drone video, thus providing support for ground-side task command and analysis.

[0115] Specifically, the output and display adaptation parameters of the decoded video frames are first determined. The output format adopts the RGB color space to ensure that the colors of the picture are consistent with the actual scene; the display resolution matches the original resolution of the decoded video frames to avoid image stretching and distortion; the refresh rate is set to 25-30 frames / second to synchronize with the frame rate of the drone video capture to prevent stuttering.

[0116] At the same time, the decoded video frames are strictly checked according to the timestamp order to ensure that the output order is consistent with the shooting sequence. If a timing abnormality occurs, the output position of the frame is adjusted to avoid the screen from jumping.

[0117] During the output and display process, the display status also needs to be monitored in real time. If the percentage of undisplayed frames exceeds 5%, or if the screen becomes black or distorted, an alarm should be triggered immediately and backup data should be retrieved for re-output to ensure uninterrupted display.

[0118] In addition, it provides simple auxiliary functions for different scenarios, such as overlaying power tower location markers during power inspections, supporting local zoom-in of suspected areas during search and rescue, and adding crop information notes during agricultural monitoring, to help staff obtain key information efficiently.

[0119] For example, when a drone performs agricultural monitoring, the ground processing unit outputs decoded video frames at a resolution of 1280×720 and a frame rate of 25 frames per second in RGB format to the monitoring display. Before output, the timestamp is checked to ensure correct timing; if a frame in the display is distorted, it is immediately replaced with a backup frame. Staff can quickly identify the monitoring area through interface annotations, and when crop abnormalities are detected, clicking on the screen triggers a 2x zoom to clearly observe details of pests and diseases. Ultimately, this achieves real-time, clear display of drone video, assisting in the completion of agricultural monitoring tasks.

[0120] The embodiments of this application, through the specific implementation methods described above, achieve the following technical effects: This application proposes a lightweight deep learning model-based real-time video decoding method for drones. First, it receives an encoded video data stream containing multiple encoded video frames transmitted from the drone, clarifies the transmission protocol and data format, monitors transmission stability to fill in lost or corrupted data, and parses and stores the video frames according to timestamps. Next, it extracts the encoded video frames in timestamp order, extracts macroblock encoding parameters to determine the complexity of each macroblock, merges adjacent macroblocks of the same complexity, and divides the region into high, medium, and low decoding complexity categories. Then, it configures a lightweight deep learning model based on the region complexity, constructing a deep decoding branch for high-complexity regions, a standard decoding branch for medium-complexity regions, and a fast decoding branch for low-complexity regions. These three branches are integrated to form a model adapted to the current video frame. Subsequently, the encoded data of each region is input into the corresponding decoding branch for decoding. High-complexity regions are improved in accuracy through multi-unit parallel decoding and voting fusion, while medium- and low-complexity regions are simplified to control resources. After obtaining the decoding results for the three regions, they are fused and stitched together based on location identifiers to form a complete decoded video frame. Finally, all decoded video frames are output sequentially according to timestamps, completing the real-time, clear decoding and display of the drone video.

[0121] The method provided in this application, through a technical solution of "data reception - complexity analysis - model configuration - differentiated decoding - result fusion - video output," solves the problems of wasted computing resources, insufficient decoding quality in key areas, and difficulty in adapting to computing resource-constrained scenarios caused by the unified processing of all video frames in traditional UAV video decoding. It significantly improves the resource utilization efficiency and the accuracy of detail rendering in key areas of UAV video decoding, ensuring real-time decoding while promoting the practical application of UAVs in resource-constrained scenarios.

[0122] Example 2, as shown in the appendix Figure 2As shown, based on the inventive concept of the real-time video decoding method for UAVs using a lightweight deep learning model provided in Embodiment 1, this application also provides a real-time video decoding system for UAVs using a lightweight deep learning model, specifically including: Encoded video data stream receiving module 01 is used to receive encoded video data stream transmitted by UAV, wherein the encoded video data stream includes multiple encoded video frames; The encoded video frame decoding processing module 02 is used to call a lightweight deep learning model, sequentially perform decoding complexity analysis on each of the encoded video frames, configure the lightweight deep learning model according to the decoding complexity analysis results, and perform decoding processing on each of the encoded video frames to obtain multiple decoded video frames. The decoded video frame output and display module 03 is used to output and display the multiple decoded video frames in sequence to complete the real-time decoding and display of the drone video.

[0123] In one embodiment, the encoded video frame decoding processing module 02 is further configured to: According to the timestamp order, a first encoded video frame is extracted from the plurality of encoded video frames. Decoding complexity analysis is performed on the first encoded video frame to obtain a first decoding complexity analysis result. The lightweight deep learning model is configured according to the first decoding complexity analysis result to obtain a first lightweight deep learning model. The first encoded video frame is decoded using the first lightweight deep learning model to obtain a first decoded video frame. The remaining encoded video frames are processed sequentially to obtain corresponding decoded video frames in the same way as the first encoded video frame is processed to obtain multiple decoded video frames.

[0124] Furthermore, the encoded video frame decoding processing module 02 also includes: Extract the macroblock coding parameters of the first encoded video frame, and determine the complexity of each macroblock based on the macroblock coding parameters; divide the first encoded video frame into regions according to the complexity of each macroblock to obtain a first decoding complexity region, a second decoding complexity region, and a third decoding complexity region; use the first decoding complexity region, the second decoding complexity region, and the third decoding complexity region as the first decoding complexity analysis result.

[0125] Furthermore, the encoded video frame decoding processing module 02 also includes: The lightweight deep learning model includes a deep decoder, a standard decoder, and a fast decoder. The deep decoder includes multiple deep decoding units, the standard decoder includes multiple standard decoding units, and the fast decoder includes multiple fast decoding units. A first deep decoding branch is configured based on a first decoding complexity region and the multiple deep decoding units; a first standard decoding branch is configured based on a second decoding complexity region and the multiple standard decoding units; and a first fast decoding branch is configured based on a third decoding complexity region and the multiple fast decoding units. The first deep decoding branch, the first standard decoding branch, and the first fast decoding branch are integrated to obtain the first lightweight deep learning model.

[0126] Furthermore, the encoded video frame decoding processing module 02 also includes: Obtain the baseline depth decoding unit call count and baseline depth decoding complexity corresponding to the depth decoder, wherein the baseline depth decoding unit call count is half the total number of the plurality of depth decoding units; obtain the first average complexity of the first decoding complexity region based on the complexity of each macroblock in the first decoding complexity region; calculate the ratio of the first average complexity to the baseline depth decoding complexity to obtain a first adjustment coefficient; adjust the baseline depth decoding unit call count according to the first adjustment coefficient to obtain a first depth decoding unit call count; call a number of depth decoding units corresponding to the first depth decoding unit call count from the plurality of depth decoding units to determine a plurality of depth decoding call units, forming the first depth decoding branch.

[0127] Furthermore, the encoded video frame decoding processing module 02 also includes: Based on the baseline depth decoding complexity, multiple high-complexity regions of samples are collected, and the encoded data of each high-complexity region is extracted to construct a high-complexity encoded dataset. The sample decoding results of each high-complexity encoded data in the high-complexity encoded dataset are extracted to construct a high-complexity decoded result set. A preset number of depth decoding units is obtained, and multiple decoding unit architectures are constructed using a deep learning algorithm based on the preset number of depth decoding units. The multiple decoding unit architectures are trained based on the high-complexity encoded dataset and the high-complexity decoded result set to generate multiple depth decoding units. The multiple depth decoding units are combined to form the depth decoder.

[0128] Furthermore, the encoded video frame decoding processing module 02 also includes: The encoded data of each partition in the first decoding complexity region are input into the first depth decoding branch to obtain the decoding result of each partition in the first decoding complexity region, thus obtaining the first decoding result; the encoded data of each partition in the second decoding complexity region are input into the first standard decoding branch to obtain the decoding result of each partition in the second decoding complexity region, thus obtaining the second decoding result; the encoded data of each partition in the third decoding complexity region are input into the first fast decoding branch to obtain the decoding result of each partition in the third decoding complexity region, thus obtaining the third decoding result; the first decoding result, the second decoding result, and the third decoding result are fused and spliced ​​to obtain the first decoded video frame.

[0129] Furthermore, the encoded video frame decoding processing module 02 also includes: A first partition is determined from the first decoding complexity region, the encoded data of the first partition is extracted, and the encoded data of the first partition is input into the first depth decoding branch; the encoded data of the first partition is decoded by multiple depth decoding call units in the first depth decoding branch to obtain multiple first partition unit decoding results; the multiple first partition unit decoding results are fused to obtain the decoding result of the first partition; the remaining partitions in the first decoding complexity region are processed in the same way as the first partition to obtain the decoding results of each partition in the first decoding complexity region; the decoding results of each partition in the first decoding complexity region are summarized to obtain the first decoding result.

[0130] Furthermore, the encoded video frame decoding processing module 02 also includes: Obtain the position identifiers of the first decoding complexity region, the second decoding complexity region, and the third decoding complexity region; based on the position identifiers, reconstruct the positions of the first decoding result, the second decoding result, and the third decoding result to obtain the first decoded video frame.

[0131] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0132] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0133] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application intends to include such modifications and variations.

Claims

1. A lightweight deep learning model-based real-time video decoding method for drones, characterized in that, The method includes: Receive encoded video data stream transmitted by the drone, the encoded video data stream including multiple encoded video frames; A lightweight deep learning model is invoked, and the decoding complexity of each encoded video frame is analyzed sequentially. Based on the decoding complexity analysis results, the lightweight deep learning model is configured to decode each encoded video frame, thereby obtaining multiple decoded video frames. The multiple decoded video frames are output and displayed sequentially to complete the real-time decoding and display of the drone video.

2. The method according to claim 1, characterized in that, Decoding complexity analysis is performed on each of the encoded video frames sequentially, and the lightweight deep learning model is configured based on the decoding complexity analysis results. The encoded video frames are then decoded to obtain multiple decoded video frames, including: According to the timestamp order, the first encoded video frame is extracted from the plurality of encoded video frames, and the decoding complexity analysis is performed on the first encoded video frame to obtain the first decoding complexity analysis result; Configure the lightweight deep learning model based on the first decoding complexity analysis result to obtain the first lightweight deep learning model; The first encoded video frame is decoded using the first lightweight deep learning model to obtain the first decoded video frame. Following the method of processing the first encoded video frame to obtain the corresponding first decoded video frame, the remaining encoded video frames are processed sequentially to obtain the corresponding decoded video frames, resulting in multiple decoded video frames.

3. The method according to claim 2, characterized in that, Decoding complexity analysis is performed on the first encoded video frame to obtain the first decoding complexity analysis result, including: Extract the macroblock coding parameters of the first encoded video frame, and determine the complexity of each macroblock based on the macroblock coding parameters; The first encoded video frame is divided into regions based on the complexity of each macroblock, resulting in a first decoding complexity region, a second decoding complexity region, and a third decoding complexity region. The first decoding complexity region, the second decoding complexity region, and the third decoding complexity region are taken as the first decoding complexity analysis result.

4. The method according to claim 3, characterized in that, Based on the first decoding complexity analysis result, the lightweight deep learning model is configured to obtain the first lightweight deep learning model, including: The lightweight deep learning model includes a deep decoder, a standard decoder, and a fast decoder. The deep decoder includes multiple deep decoding units, the standard decoder includes multiple standard decoding units, and the fast decoder includes multiple fast decoding units. A first deep decoding branch is configured based on the first decoding complexity region and the plurality of deep decoding units; a first standard decoding branch is configured based on the second decoding complexity region and the plurality of standard decoding units; and a first fast decoding branch is configured based on the third decoding complexity region and the plurality of fast decoding units. The first deep decoding branch, the first standard decoding branch, and the first fast decoding branch are integrated to obtain the first lightweight deep learning model.

5. The method according to claim 4, characterized in that, Configure a first depth decoding branch based on the first decoding complexity region and the plurality of depth decoding units, including: Obtain the baseline depth decoding unit call count and baseline depth decoding complexity corresponding to the depth decoder, wherein the baseline depth decoding unit call count is half of the total number of the plurality of depth decoding units; Based on the complexity of each macroblock in the first decoding complexity region, obtain the first average complexity of the first decoding complexity region; The first adjustment coefficient is obtained by calculating the ratio of the first average complexity to the baseline depth decoding complexity. The number of reference depth decoding unit calls is adjusted according to the first adjustment coefficient to obtain the number of first depth decoding unit calls; The first depth decoding branch is formed by calling a number of depth decoding units from the plurality of depth decoding units corresponding to the number of calls to the first depth decoding unit.

6. The method according to claim 5, characterized in that, The construction steps of the depth decoder include: Based on the baseline depth decoding complexity, multiple high-complexity regions of samples are collected, and the encoded data of each high-complexity region of the samples is extracted to construct a high-complexity encoded dataset of samples. Extract the sample decoding results of each sample high-complexity encoded data in the sample high-complexity encoded dataset, and construct a sample high-complexity decoding result set; Obtain a preset number of depth decoding units, and construct a multiple decoding unit architecture based on the preset number of depth decoding units using a deep learning algorithm; Based on the sample high-complexity encoding dataset and the sample high-complexity decoding result set, the multiple decoding unit architectures are trained respectively to generate multiple deep decoding units; The multiple depth decoding units are combined to form the depth decoder.

7. The method according to claim 4, characterized in that, The first encoded video frame is decoded using the first lightweight deep learning model to obtain the first decoded video frame, including: The encoded data of each partition in the first decoding complexity region are input into the first depth decoding branch to obtain the decoding results of each partition in the first decoding complexity region, and thus obtain the first decoding result. The encoded data of each partition in the second decoding complexity region are input into the first standard decoding branch to obtain the decoding results of each partition in the second decoding complexity region, thus obtaining the second decoding result. The encoded data of each partition in the third decoding complexity region are input into the first fast decoding branch to obtain the decoding results of each partition in the third decoding complexity region, and thus obtain the third decoding result. The first decoding result, the second decoding result, and the third decoding result are fused and spliced ​​together to obtain the first decoded video frame.

8. The method according to claim 7, characterized in that, The encoded data of each partition in the first decoding complexity region are input into the first depth decoding branch to obtain the decoding results of each partition in the first decoding complexity region, resulting in the first decoding result, which includes: Determine the first partition from the first decoding complexity region, extract the encoded data of the first partition, and input the encoded data of the first partition into the first depth decoding branch; The encoded data of the first partition is decoded by multiple deep decoding call units in the first deep decoding branch, and multiple first partition unit decoding results are obtained. The decoding results of the plurality of first partition units are fused to obtain the decoding result of the first partition; Following the same method used to process the first partition, the remaining partitions in the first decoding complexity region are processed sequentially to obtain the decoding results of each partition in the first decoding complexity region. The decoding results of each partition in the first decoding complexity region are summarized to obtain the first decoding result.

9. The method according to claim 7, characterized in that, The first decoding result, the second decoding result, and the third decoding result are fused and spliced ​​together to obtain the first decoded video frame, including: Obtain the position identifiers of the first decoding complexity region, the second decoding complexity region, and the third decoding complexity region; Based on the location identifier, the first decoding result, the second decoding result, and the third decoding result are reconstructed to obtain the first decoded video frame.

10. A lightweight deep learning model-based real-time video decoding system for unmanned aerial vehicles, characterized in that, The system is used to perform the method according to any one of claims 1-9, the system comprising: The encoded video data stream receiving module is used to receive the encoded video data stream transmitted by the UAV, wherein the encoded video data stream includes multiple encoded video frames; The encoded video frame decoding processing module is used to call a lightweight deep learning model, sequentially perform decoding complexity analysis on each of the encoded video frames, configure the lightweight deep learning model according to the decoding complexity analysis results, and perform decoding processing on each of the encoded video frames to obtain multiple decoded video frames. The decoded video frame output and display module is used to output and display the multiple decoded video frames in sequence, thereby completing the real-time decoding and display of the drone video.

Images