Unmanned aerial vehicle multi-camera cooperative video decoding and multi-source enhancement method based on dynamic partition

Abstract

The invention discloses an unmanned aerial vehicle multi-camera cooperative video decoding and multi-source enhancement method based on dynamic partition. According to the scheme, the method comprises the following steps: firstly, receiving a layered data stream which is encoded and transmitted to a decoding end, and splitting the data stream into three independent parts; secondly, respectively decoding the three layers of data streams, and reconstructing the data streams by utilizing dynamic partition reconstruction, basic frame reconstruction and ROI (Region of Interest) high-frequency component fusion; then, fusing the three layers of parameters into a complete frame, and performing video frame quality enhancement by utilizing infrared-visible light fusion, depth-guided super-resolution and semantic driving; and finally, outputting the decoded, reconstructed and enhanced data frame. According to the method, through deep cooperation of hierarchical decoding and multi-source enhancement, the coding scheme is strictly compatible, and meanwhile, the application value of the unmanned aerial vehicle video is expanded.

Description

Technical Field

[0001] The present invention relates to the field of drone video data security, and in particular to a drone multi-camera collaborative video decoding and multi-source enhancement method based on dynamic partitioning. Background Art

[0002] Low-altitude economic tourism is becoming a new growth pole for the global cultural and tourism industry. Drone sightseeing, aerial panoramic live broadcast, and AR immersive guided tours are developing rapidly. Real-time high-definition video streaming has become the core medium connecting "aerial perspective" and "ground experience". Its transmission and presentation quality directly determines the immersive experience of tourists and the efficiency of commercial value conversion.

[0003] However, the current low-altitude tourism landscape still faces multiple technical bottlenecks and implementation challenges in delivering immersive drone ride experiences. High bitrate fluctuations in high-dynamic scenes lead to decoding delays and inter-frame distortion, while motion compensation is susceptible to artifacts caused by complex background interference. Degradation factors such as low light and motion blur affect the texture fidelity of super-resolution reconstruction, and depth guidance is susceptible to errors transmitted from sparse depth maps. The conflict between real-time processing requirements and limited computing power is prominent, making it difficult to deploy complex models at the edge. Therefore, we propose a method for collaborative video decoding and multi-source enhancement for drones using multiple cameras based on dynamic partitioning. Summary of the Invention

[0004] To address the problems in the background technology, the present invention provides a method for collaborative video decoding and multi-source enhancement of multiple cameras in a drone based on dynamic partitioning, so as to achieve efficient decoding and real-time enhancement of video streams during an immersive driving experience, including:

[0005] S1: Receives the layered data stream that has been encoded and transmitted to the decoding end, and splits the data stream into three independent parts;

[0006] Furthermore, the data stream splitting process involves: when the receiving end decodes a layered coded video stream, it first parses the base layer, enhancement layer, and auxiliary information layer into independent data packets based on the layer identifier encapsulated in the bitstream header. The quantization coefficients and coding parameters of each layer are then separated through entropy decoding and syntax parsing. The three split data components are then fed into the corresponding inverse quantization module and intra / inter-frame predictor, respectively. Parallel decoding and reconstruction are performed while preserving inter-layer dependencies. Finally, a layer fusion module integrates spatially aligned multi-scale features to support subsequent super-resolution enhancement.

[0007] S2: Decode the three split data streams separately and reconstruct the video using dynamic partition reconstruction, basic frame reconstruction, and ROI high-frequency component fusion;

[0008] Furthermore, decoding and reconstructing the metadata layer's data stream includes extracting encapsulated metadata blocks from the stream, quickly locating the data start position and verifying its integrity using predefined header identifiers. The decoder then parses the dynamic partition table and quantization parameter difference chain to restore the region type and QP mapping table. Furthermore, the decoder simultaneously extracts multi-sensor spatiotemporal parameters to construct a fusion model, combines them with cross-view reference frame indexes to establish camera geometry, and ultimately achieves pixel-level alignment of multi-source data through dynamic inverse quantization control and spatiotemporal compensation.

[0009] Furthermore, the decoding and reconstruction of the data bitstream of the ROI enhancement layer includes: extracting residual coefficients from the bitstream, performing inverse quantization based on the metadata dynamic QP table to restore high-frequency details, superimposing the residuals to the corresponding area of ​​the base frame according to the partition coordinates, combining adaptive filtering to eliminate blocking effects, and achieving detail enhancement through multi-scale residual upsampling fusion, and finally outputting the enhanced area that is temporally and spatially aligned with the metadata.

[0010] Furthermore, the decoding and reconstruction of the base layer data stream includes extracting the standard coding framework compressed stream data from the stream, obtaining motion vectors and prediction modes through entropy decoding, reconstructing the intra-frame prediction blocks, and inverse transforming to restore the low-frequency residual components. Simultaneously, the inverse quantization step size of the core and background regions is differentially adjusted based on the metadata dynamic QP table. The decoder synchronously loads historical image data from the reference frame pool to implement motion compensation, and performs adaptive deblocking filtering on block boundaries to output a base reconstructed frame, providing low-frequency support for the fusion of high-frequency residuals in the enhancement layer.

[0011] S3: Fusion of three-layer data streams into a complete frame, and enhancement of video frame quality using infrared-visible light fusion, depth-guided super-resolution, and semantic-driven methods.

[0012] Furthermore, the use of infrared-visible light fusion to enhance data frame quality includes: spatial registration of multi-source data through a spatiotemporal coordinate conversion model and timestamp synchronization to correct for sensor parallax. Based on decoded and reconstructed infrared thermal radiation and visible light texture, multi-scale decomposition is used to fuse low-frequency energy and high-frequency details. Fusion weights are dynamically assigned using semantic tags to enhance the saliency of high-temperature regions and structural contours. Cross-modal contrast enhancement balances thermal radiation artifacts and visible light noise, synergistically suppressing interfering information, ultimately outputting a fused image that combines thermodynamic features with high-resolution texture.

[0013] Furthermore, the use of depth-guided super-resolution to enhance data frame quality includes: by inputting a low-resolution image and depth map, using depth information to represent the spatial structure of the scene, and guiding the convolutional network to extract multi-scale features. First, the depth map is edge-enhanced and noise-suppressed, and spatial attention weights are constructed to focus on the geometric features of key areas. Then, the depth features are cross-modally fused with the image texture features, and the depth-texture correlation is adaptively matched through deformable convolution, driving the residual network to generate high-frequency details. The decoding stage adopts a progressive upsampling strategy, combined with depth constraints to optimize edge sharpness and structural continuity, suppress artifact generation, and ultimately output a high-resolution image, achieving a balance between target contour enhancement and texture authenticity.

[0014] Furthermore, the semantic-driven technology used to achieve logical consistency in data frames includes: extracting target categories and spatial distribution using a pre-trained semantic segmentation network, and constructing a semantic weight matrix to dynamically control the processing flow. Attention masks are generated to guide multimodal fusion and super-resolution tasks to prioritize key regions, and feature extraction strategies are adaptively adjusted based on the semantic relevance of the knowledge graph. A semantically constrained loss function and adversarial training are introduced to optimize the logical consistency of generated content, ensuring that the output conforms to the physical structure and semantic patterns of real-world scenarios.

[0015] S4: Output decoded, reconstructed and enhanced video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the method flow of the present invention;

[0017] Figure 2 This is a flowchart for decoding and reconstructing a three-layer data stream in the present invention;

[0018] Figure 3 This is a flow chart for data code stream enhancement in the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] See also Figure 1 The present invention provides a method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning, comprising:

[0021] S1: Receives the layered data stream that has been encoded and transmitted to the decoding end, and splits the data stream into three independent parts;

[0022] Furthermore, the data stream splitting process involves: when the receiving end decodes a layered coded video stream, it first parses the base layer, enhancement layer, and metadata layer into independent data packets based on the layer identifier encapsulated in the bitstream header. The quantization coefficients and coding parameters of each layer are then separated through entropy decoding and syntax parsing. The three split data components are then fed into the corresponding inverse quantization module and intra / inter predictor, respectively. Parallel decoding and reconstruction are performed while preserving inter-layer dependencies. Finally, a layer fusion module integrates spatially aligned multi-scale features to support subsequent super-resolution enhancement.

[0023] See also Figure 2 After step S1, the split three-layer data code streams are decoded and reconstructed respectively, and the time-space compensation alignment method is used to improve them.

[0024] Furthermore, the metadata layer extracts the encapsulated metadata blocks from the bitstream, quickly locates the data starting position and verifies its integrity through predefined header identifiers. The decoder unpacks the structured data layer by layer, first reading the dynamic partition table information, decoding the region type identifier and corresponding quantization parameter difference of each coding tree unit in raster scan order, and restoring the complete quantization parameter mapping table through cumulative calculation. The QP interpolation chain formula in the dynamic partition table is as follows:

[0025]

[0026] Among them, QP init (p) is the initial quantization parameter of partition p, ΔQP(p,k) is the QP difference of the kth block in partition p, QP(p,i) is the final QP value of the i-th block in partition p, Clip(x,a,b) limits x to the interval [a,b], QP min , QP max The legal value range for QP is usually defined by the coding standard.

[0027] Synchronously extract the spatiotemporal alignment parameters of multiple sensors, including the parameters of each camera, and construct the spatiotemporal coordinate system conversion model required for multi-source data fusion. The formula is as follows:

[0028]

[0029] Among them, u and v represent two-dimensional coordinate parameters, K int is the internal parameter matrix, R and T are the external parameter rotation and translation matrices, X, Y, Z are the three-dimensional coordinate system parameters, δ distort is the distortion compensation term.

[0030] This process simultaneously analyzes the index information of the reference frame pool and cross-view motion prediction parameters to establish geometric relationships between multiple cameras. The final parsed metadata dynamically controls the inverse quantization strength, regional filtering strategy, and multimodal fusion weights, enabling precise reverse replication of the encoding strategy. Timestamp alignment and spatial coordinate compensation also eliminate spatiotemporal deviations in multi-source data transmission, ensuring pixel-by-pixel precision matching of visible light, infrared, and depth data.

[0031] Furthermore, the base layer parses and extracts the code stream data compressed by the standard coding framework from the code stream, obtains the motion vector and prediction mode through entropy decoding, reconstructs the image blocks generated by intra-frame or inter-frame prediction, and combines inverse quantization and inverse transformation to restore the low-frequency residual components. The entropy decoding formula is:

[0032] p new =p prev +α·(sp prev ) (3)

[0033] Among them, p prev is the binary symbol probability estimate of the current context model before the decoded symbol s, s∈{0,1} is the decoded symbol, and α is the learning rate used for context adaptive probability update.

[0034] During the parsing process, the inverse quantization step sizes of the core and background regions are adjusted differently based on the dynamic QP mapping table and region type identifiers provided by the metadata layer. A fine step size is used in the core region to preserve key structures, while coarse quantization is used in the background region to suppress redundant noise. The decoder simultaneously loads historical image data from the reference frame pool, maintains temporal continuity through motion compensation, and performs adaptive deblocking filtering on block boundaries. The decoder ultimately outputs a base reconstructed frame that is spatiotemporally aligned with the metadata, providing low-frequency support for the fusion of high-frequency residuals in the enhancement layer.

[0035] Furthermore, the ROI enhancement layer extracts the entropy-decoded residual coefficient matrix from the bitstream, performs inverse quantization on each coding unit according to the dynamic QP mapping table provided by the metadata layer, and restores high-frequency residual details by matching the corresponding quantization step size with the region type identifier. The quantization step size matching formula is:

[0036]

[0037] Among them, QP ROI , QP BG Provided by the metadata layer dynamic QP mapping table, β is the background de-stepping expansion factor.

[0038] During the parsing process, the residual data action area is located according to the ROI spatial coordinates recorded in the partition table, and the decoded residual component is superimposed on the base layer reconstructed frame at the pixel level. At the same time, the edge transition area is adaptively filtered to eliminate the blocking effect in combination with the cross-view motion compensation parameters. The formula for bilateral filtering correction is:

[0039]

[0040] Among them, the spatial weight Range weight x, y are the absolute coordinates of the current filter center pixel, i, j are the field offsets with the center pixel (x, y) as the origin, and all pixels within the filter window are traversed.

[0041] In response to the multi-scale residual generation strategy on the encoding side, the parser dynamically loads residual components at different resolution levels, achieves detail enhancement through upsampling and weight fusion, and finally outputs the ROI enhancement area that is spatiotemporally aligned with the metadata, completing the precise conversion from compressed domain residual to pixel domain visual enhancement.

[0042] See also Figure 3 Specifically, the decoded and reconstructed three-layer data code stream enters the multi-source data enhancement module. At the same time, visible light fusion combines spatiotemporal registration with fusion optimization, depth-guided super-resolution combines deep feature fusion with upsampling optimization, and semantic-driven technology combines semantic weights with logical constraints to achieve data code stream enhancement and ultimately output enhanced video frames.

[0043] Furthermore, infrared-visible light fusion realizes multi-source data spatial registration through spatiotemporal coordinate conversion model and timestamp synchronization, corrects sensor parallax, and the coordinate alignment formula is:

[0044]

[0045] Among them, K IR , K RGB is the intrinsic parameter matrix of the infrared / visible light camera, R and T are the external parameter rotation and translation matrices between sensors, X, Y, Z are the three-dimensional space coordinate parameters, u and v are the two-dimensional space coordinate parameters, δ distort is the lens distortion compensation term.

[0046] Based on the decoded and reconstructed infrared thermal radiation and visible light texture, multi-scale decomposition is used to fuse low-frequency energy and high-frequency details. Combined with semantic labels, the fusion weight is dynamically assigned to enhance the saliency of high-temperature areas and structural contours. The multi-scale decomposition and fusion formula is:

[0047]

[0048] Among them, L IR 、L RGBis the infrared / visible light low-frequency component, The weights are based on their local variance σ, The fusion weight of the low-frequency component of visible light RGB, x, y are two-dimensional pixel coordinates, H RGB (x, y) is the high-frequency component of the visible light image at position (x, y), H IR (x,y) is the high-frequency component of the infrared image at position (x,y).

[0049] Through cross-modal contrast enhancement, thermal radiation artifacts and visible light noise are balanced, interference information is synergistically suppressed, and finally a fused image with both thermodynamic features and high-resolution texture is output.

[0050] Furthermore, depth-guided super-resolution uses low-resolution images and depth maps as input, using depth information to represent the spatial structure of the scene and guide the convolutional network to extract multi-scale features. First, the depth map is edge-enhanced and noise-suppressed, and spatial attention weights are constructed to focus on the geometric features of key areas. The spatial attention weight generation formula is:

[0051]

[0052] Where σ(·) is the sigmoid function, k is the size of the local neighborhood, D is the enhanced depth map, and (x, y) is a two-dimensional pixel point.

[0053] The deep features are then cross-modally fused with image texture features, adaptively matching depth-texture correlations through deformable convolutions to drive the residual network to generate high-frequency details. The decoding stage employs a progressive upsampling strategy, combined with depth constraints to optimize edge sharpness and structural continuity, suppress artifacts, and ultimately output a high-resolution image, achieving a balance between object outline enhancement and texture authenticity.

[0054] The upsampling strategy is to gradually increase the resolution in stages, and the formula for each stage is:

[0055]

[0056] Where PixShuffle is a sub-pixel convolution operation that converts the number of channels into spatial resolution; W t is the convolution kernel parameter of the tth stage, I t-1 is the upsampling result of the t-1th stage, α t is the depth-guided weight coefficient, is the nonlinear transformation function of the depth map.

[0057] Furthermore, semantic-driven technology extracts the category labels and spatial distribution of target objects in a scene through a pre-trained semantic segmentation network, constructing a semantic weight matrix to dynamically control the processing flow. Attention masks are generated based on the target semantic features, guiding the model to prioritize key areas for multimodal data fusion and super-resolution reconstruction.

[0058] Combining semantic weights with knowledge graphs to enhance key areas, the attention mask generation formula is:

[0059] A(x,y)=σ(ω semantic (x,y)·MLP(F adj (x,y))) (10)

[0060] Among them, ω semantic (x,y) is the semantic weight of pixel (x,y), MLP is a multi-layer perceptron, and outputs the attention value A∈[0,1], F adj (x,y) is the adjusted feature map, and σ(·) is the sigmoid function.

[0061] By combining the semantic relevance of scenes in the knowledge graph, we adaptively adjust the feature extraction strength and fusion strategy for different categories of objects. We optimize the logical consistency of the generated content through semantic constraint loss functions and adversarial training, ensuring that the output results conform to the physical structure and semantic distribution of the real scene. The segmentation consistency constraint formula is:

[0062]

[0063] Among them, x and y are pixel coordinates, c is the semantic label category, and P c GT is the probability distribution of the true label, ensuring that the enhancement result does not destroy the semantic logic; f θ (I enhanced ) is the feature extraction function of the segmentation network for the enhanced image, and softmax(·) is the normalization function that maps the features into probability distribution.

[0064] S4: Output decoded, reconstructed and enhanced video frames.

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for collaborative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning, characterized in that: include: S1: Receives the layered video data stream that has been encoded and transmitted to the decoding end, and splits the data stream into three independent parts; S2: Decode the three layers of data streams separately and reconstruct the video using dynamic partition reconstruction, basic frame reconstruction, and ROI high-frequency component fusion; S3: Fusion of three-layer data streams into a complete frame, and enhancement of video frame quality using infrared-visible light fusion, depth-guided super-resolution, and semantic-driven methods. S4: Output decoded, reconstructed and enhanced video frames.

2. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 1 is characterized in that: When decoding a layered video stream, the receiver first parses the base layer, enhancement layer, and metadata layer into independent data packets based on the layer identifier encapsulated in the bitstream header. Entropy decoding and syntax parsing are then used to separate the quantization coefficients and coding parameters of each layer. These three data components are then fed into the corresponding inverse quantization module and intra / inter-frame predictor. Parallel decoding and reconstruction are performed while preserving inter-layer dependencies. Finally, a layer fusion module integrates spatially aligned multi-scale features to support subsequent super-resolution enhancement or multimodal analysis tasks.

3. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 1 is characterized in that: The three layers of split data code streams are decoded respectively, and the video code stream is reconstructed by using dynamic partition reconstruction, basic frame reconstruction, and ROI high-frequency component fusion.

4. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 3 is characterized in that: Decoding the metadata layer begins by extracting the encapsulated metadata blocks from the bitstream. Predefined header identifiers are used to quickly locate the data start and verify its integrity. The decoder then parses the dynamic partition table and quantization parameter difference chain to restore the region type and QP mapping table. It then simultaneously extracts multi-sensor spatiotemporal parameters to construct a fusion model, combining them with cross-view reference frame indexes to establish camera geometry. Finally, dynamic inverse quantization control and spatiotemporal compensation are used to achieve pixel-level alignment of multi-source data.

5. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 3 is characterized in that: ROI enhancement layer analysis extracts residual coefficients from the bitstream, performs inverse quantization based on the metadata dynamic QP table to restore high-frequency details, and superimposes the residuals onto the corresponding area of ​​the base frame according to the partition coordinates. Adaptive filtering is combined to eliminate blocking effects, and detail enhancement is achieved through multi-scale residual upsampling and fusion. The final output is an enhanced area that is temporally and spatially aligned with the metadata.

6. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 3 is characterized in that: The base layer extracts the compressed bitstream data from the standard coding framework, obtains motion vectors and prediction modes through entropy decoding, reconstructs intra-frame prediction blocks, and recovers the low-frequency residual components through inverse transforms. The inverse quantization step size for the core and background regions is adjusted differentially based on the metadata dynamic QP table. The decoder simultaneously loads historical image data from the reference frame pool for motion compensation and performs adaptive deblocking filtering on block boundaries to output a base reconstructed frame, providing low-frequency support for high-frequency residual fusion in the enhancement layer.

7. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 1 is characterized in that The three-layer data streams are fused into a complete frame, and the video frame quality is enhanced using infrared-visible light fusion, depth-guided super-resolution, and semantic-driven methods.

8. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 7 is characterized in that: The infrared-visible light fusion method uses a spatiotemporal coordinate conversion model and synchronized timestamps to achieve spatial registration of multi-source data and correct for sensor parallax. Based on decoded and reconstructed infrared thermal radiation and visible light texture, multi-scale decomposition is used to fuse low-frequency energy and high-frequency details. Combined with semantic labels, dynamic fusion weights are assigned to enhance the saliency of high-temperature regions and structural contours. Cross-modal contrast enhancement balances thermal radiation artifacts and visible light noise, synergistically suppressing interfering information. The resulting fused image combines thermodynamic features with high-resolution texture.

9. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 7 is characterized in that: Depth-guided super-resolution uses a low-resolution image and a depth map as input, using depth information to characterize the spatial structure of the scene and guide the convolutional network to extract multi-scale features. First, the depth map is edge-enhanced and noise-reduced, and spatial attention weights are constructed to focus on the geometric features of key areas. The deep features are then cross-modally fused with image texture features, adaptively matching depth-texture correlations through deformable convolutions to drive the residual network to generate high-frequency details. The decoding stage employs a progressive upsampling strategy, combined with depth constraints to optimize edge sharpness and structural continuity, suppress artifacts, and ultimately output a high-resolution image, achieving a balance between object outline enhancement and texture authenticity.

10. The method for cooperative video decoding and multi-source enhancement of multi-cameras of unmanned aerial vehicles based on dynamic partitioning according to claim 7, characterized in that: The semantic-driven technology utilizes a pre-trained semantic segmentation network to extract object categories and spatial distribution, constructing a semantic weight matrix to dynamically control the processing flow. It generates attention masks to guide multimodal fusion and super-resolution tasks, prioritizing key regions for enhancement. It also adaptively adjusts feature extraction strategies based on the semantic relevance of the knowledge graph. A semantically constrained loss function and adversarial training are introduced to optimize the logical consistency of generated content, ensuring that the output conforms to the physical structure and semantic regularities of the real scene. Ultimately, the decoded, reconstructed, and enhanced video frames are output for further video frame processing.

Images