A dual-path map transmission 3D split-screen display method and system for a UAV

Abstract

The application relates to a UAV dual-path map transmission 3D split-screen display method and system, which comprises the following steps: jointly analyzing left-viewpoint video streams and right-viewpoint video streams to generate stereoscopic visual perception metadata associated with video frame timestamps; performing perception quality-oriented hierarchical coding on the left-viewpoint video streams and the right-viewpoint video streams to generate a single-path composite code stream; configuring differentiated data transmission guarantee strategies for data units corresponding to different layers and different level areas in the single-path composite code stream; transmitting the single-path composite code stream with the configured differentiated data transmission guarantee strategies to a ground station through a wireless link; and performing hierarchical decoding and perception quality reconstruction on the received single-path composite code stream in the ground station to output left-viewpoint and right-viewpoint video signals for stereoscopic display. The application has the beneficial effects of significantly improved transmission efficiency, optimized stereoscopic experience quality and comfort, enhanced transmission robustness, intrinsically guaranteed synchronization, and reduced system complexity and cost.

Description

Technical Field

[0001] This invention relates to the field of 3D display technology for unmanned aerial vehicles (UAVs), and more specifically, to a method and system for 3D split-screen display of dual-channel image transmission from UAVs. Background Technology

[0002] With the popularization of drone technology, its role in aerial photography, surveying and mapping, inspection, emergency rescue and film and television production is becoming increasingly prominent. Traditional drone image transmission systems mainly transmit two-dimensional planar video, and operators lack depth perception information, making it difficult to make precise control or situational judgment in complex three-dimensional environments. Therefore, stereoscopic video image transmission technology that can provide immersive spatial information has become an urgent need.

[0003] In existing technologies, facing the industry-wide challenge of immense bandwidth pressure in dual-channel image transmission scenarios, some solutions attempt to calculate depth maps on the drone end and then transmit only one channel of color video and one channel of depth information. However, such solutions typically separate depth estimation from video encoding and transmission strategies. The quality of the depth map and encoding efficiency fail to be integrated with the characteristics of human stereoscopic vision perception, and the transmission process treats all data equally. Moreover, when channel conditions deteriorate, critical depth edge information may be lost due to lack of protection, causing the stereoscopic effect to "collapse" at critical moments, resulting in an uncontrollable user experience. Therefore, there is an urgent need for an end-to-end solution that can coordinate perception, encoding, and transmission to ensure high-quality, high-comfort stereoscopic vision under limited and fluctuating wireless bandwidth. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for dual-channel image transmission 3D split-screen display of UAVs, and to provide a system for dual-channel image transmission 3D split-screen display of UAVs, in view of the above-mentioned defects of the prior art.

[0005] The technical solution adopted by this invention to solve its technical problem is: A method for constructing a dual-path image transmission 3D split-screen display for unmanned aerial vehicles (UAVs) is provided, comprising: Step 1: The binocular camera module mounted on the drone acquires synchronized left-view and right-view video streams; Step 2: Perform joint analysis on the left-viewpoint video stream and the right-viewpoint video stream to generate stereo vision perception metadata associated with video frame timestamps; the stereo vision perception metadata contains spatial importance information of at least two levels based on image content features. Step 3: Based on stereo vision perception metadata, perform perception quality-oriented layered encoding on the left viewpoint video stream and the right viewpoint video stream to generate a single composite bitstream; wherein, the single composite bitstream contains a base layer and at least one enhancement layer. The base layer is used to ensure the continuity of stereo vision and basic depth perception, and the enhancement layer is used to perform differentiated quality enhancement on different levels of spatial importance information indication areas. Step 4: Based on the spatial importance information contained in the stereo vision perception metadata, configure differentiated data transmission guarantee strategies for data units corresponding to different layers and different levels of regions in a single composite bitstream. Step 5: Transmit a single composite stream configured with differentiated data transmission guarantee strategy to the ground station via wireless link; Step 6: The ground station performs layered decoding and perception quality reconstruction on the received single-channel composite bitstream to output left and right viewpoint video signals for stereoscopic display.

[0006] The UAV dual-channel image transmission 3D split-screen display method of the present invention, wherein the generation of stereoscopic visual perception metadata associated with video frame timestamps includes: Perform semantic instance segmentation to obtain the semantic object distribution map of the current frame; Stereo matching is performed on the left-viewpoint video stream and the right-viewpoint video stream to generate a disparity map and the corresponding disparity confidence map; Calculate the spatial gradient map of the disparity map to identify regions of depth discontinuity; Motion estimation is performed on the left-viewpoint or right-viewpoint video stream to generate a motion vector field; The semantic object distribution map, disparity map, disparity confidence map, spatial gradient map, and motion vector field are input into an importance assessment model, which outputs spatial importance information. The importance assessment model is a neural network model trained on a stereo vision perception quality annotation dataset.

[0007] The UAV dual-path image transmission 3D split-screen display method of the present invention, wherein the training objective function of the importance assessment model integrates stereoscopic visual comfort scores based on human subjective testing and objective image quality evaluation indicators.

[0008] The UAV dual-path image transmission 3D split-screen display method of the present invention, wherein the perception quality-oriented hierarchical coding includes: Base layer encoding: After downsampling and frame rate adjustment of the left viewpoint video stream and the right viewpoint video stream, the first set of encoding parameters is used for encoding; Enhancement layer coding: Based on spatial importance information, different coding resources are allocated to different regions in the image, including: Quantization parameter allocation: Assign quantization parameter values ​​lower than those for low importance regions to regions with high importance levels; Inter-frame prediction optimization: For high-importance regions, limit the range of matching blocks they can search in the reference frame, or increase the quality requirements of their reference frames; Rate allocation: Using spatial importance information as a weight map, perceptually weighted rate allocation is performed in the encoder's rate control module.

[0009] The UAV dual-channel image transmission 3D split-screen display method of the present invention wherein the generation of a single-channel composite bitstream is achieved by using a region of interest encoding tool that conforms to H, 266 / VVC, AVS3 or subsequent video encoding standards.

[0010] The UAV dual-path image transmission 3D split-screen display method of the present invention, wherein the configuration of differentiated data transmission guarantee strategies is executed by a cross-layer optimization controller deployed in the UAV communication protocol stack, including: Monitor real-time transmission quality parameters of the wireless link; Based on spatial importance information and the coding layer to which the data unit belongs, a transmission priority identifier is generated for the data packet to be transmitted; Based on the transmission priority identifier, dynamically adjust at least one of the following transmission mechanisms: Redundancy of the forward error correction scheme; The maximum number of retransmissions for automatic retransmission requests and the timeout duration of the retransmission timer; The order in which data packets are scheduled in the send buffer; Robustness level of physical layer modulation and coding schemes.

[0011] The UAV dual-path image transmission 3D split-screen display method of the present invention includes, wherein the layered decoding and perception quality reconstruction comprises: Base layer decoding and reconstruction: Decode the base layer data in a single composite bitstream to obtain the initial left and right viewpoint reconstruction frames; Iterative enhancement of enhancement layers: Based on the spatial importance information obtained from decoding, enhancement layer data is decoded and applied sequentially in order of importance from high to low to enhance the quality of the corresponding regions in the initial left and right viewpoint reconstruction frames. Error concealment handling: When the loss of enhancement layer data in a specific area is detected, based on the principle of stereo consistency, the content is restored by using the decoded information of the corresponding area of ​​the opposite viewpoint and the information of adjacent frames; Stereo consistency post-processing: The final reconstructed left and right viewpoint images are filtered to suppress noise inconsistencies between viewpoints that may be caused by layered coding and transmission.

[0012] A UAV dual-channel image transmission 3D split-screen display system is used to implement the above-mentioned UAV dual-channel image transmission 3D split-screen display method, characterized in that it includes a UAV terminal system and a ground terminal system; The UAV terminal system includes a binocular camera module, an airborne processing module, and an airborne wireless transmission module; the binocular camera module is used to acquire raw stereoscopic video; the airborne processing module is used to generate stereoscopic vision perception metadata, perform perception quality-oriented hierarchical coding to generate a single-channel composite bitstream, and configure differentiated data transmission guarantee strategies; the airborne wireless transmission module is used to transmit the single-channel composite bitstream. The ground terminal system includes: a ground wireless receiving module, a ground processing module, and a 3D display device; the ground wireless receiving module is used to receive a single composite bitstream; the ground processing module is used to perform layered decoding and perceptual quality reconstruction; and the stereoscopic display device is used to present the reconstructed stereoscopic video.

[0013] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the above-described UAV dual-channel image transmission 3D split-screen display method.

[0014] A drone, wherein the drone is equipped with a drone terminal system as described above for a drone dual-channel image transmission 3D split-screen display system.

[0015] The beneficial effects of this invention are as follows: Significantly improved transmission efficiency: By integrating the left and right viewpoint video streams into a single perception-quality-oriented composite bitstream and performing differentiated encoding on different important areas, the bandwidth requirement is effectively reduced by more than 50% while ensuring the core stereoscopic visual experience, or the image quality of key areas is significantly improved under the same bandwidth.

[0016] Optimization of stereoscopic experience quality and comfort: With stereoscopic vision perception metadata as the core, the encoding and transmission resources are guided to tilt towards the areas that have the greatest impact on depth perception and visual comfort (such as the edges of moving objects and areas of abrupt changes in depth), which ensures high-quality stereoscopic viewing from the source and reduces visual fatigue caused by compression or transmission damage.

[0017] Enhanced transmission robustness: The cross-layer optimization controller maps visual importance to transmission priority, realizing dynamic matching between network resources and perception importance; when channel conditions deteriorate, it prioritizes the reliable and low-latency transmission of high-importance data, ensuring that the "skeleton" of stereo vision is not lost, and the system's anti-interference capability is greatly improved.

[0018] Synchronization inherent guarantee: Due to the use of single-channel composite bitstream transmission, the left and right viewpoint information has been precisely synchronized and encapsulated at the encoding end, and the decoding at the receiving end is naturally synchronized, which completely solves the problem of inter-frame and inter-line asynchrony that is difficult to overcome in traditional dual-channel independent transmission.

[0019] Reduced system complexity and cost: The adoption of a single-path encoding and transmission architecture reduces a complete encoding and RF link, which helps to reduce the power consumption, weight and hardware cost of the drone, and improve system integration and reliability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart of a preferred embodiment of the UAV dual-path image transmission 3D split-screen display method of the present invention; Figure 2 This is a block diagram illustrating the principle of a dual-path image transmission 3D split-screen display system for drones, according to a preferred embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0022] A preferred embodiment of the UAV dual-path image transmission 3D split-screen display method of the present invention, such as... Figure 1 As shown, including S01: The binocular camera module mounted on the drone acquires synchronized left-viewpoint and right-viewpoint video streams; S02: Perform joint analysis on the left-viewpoint video stream and the right-viewpoint video stream to generate stereoscopic vision perception metadata associated with video frame timestamps; The stereo vision perception metadata contains spatial importance information at least two levels based on image content features; S03: Based on stereo vision perception metadata, perform perception quality-oriented layered coding on the left viewpoint video stream and the right viewpoint video stream to generate a single composite bitstream; The single-channel composite bitstream includes a base layer and at least one enhancement layer. The base layer is used to ensure the continuity of stereo vision and basic depth perception, while the enhancement layer is used to perform differentiated quality enhancement on regions with different levels of spatial importance information. S04: Based on the spatial importance information contained in the stereo vision perception metadata, configure differentiated data transmission guarantee strategies for data units corresponding to different layers and different levels of regions in a single composite bitstream. S05: A single composite data stream configured with differentiated data transmission guarantee strategy is transmitted to the ground station via a wireless link; S06: The ground station performs layered decoding and perception quality reconstruction on the received single-channel composite bitstream to output left and right viewpoint video signals for stereoscopic display; The beneficial effects of this invention are as follows: Significantly improved transmission efficiency: By integrating the left and right viewpoint video streams into a single perception-quality-oriented composite bitstream and performing differentiated encoding on different important areas, the bandwidth requirement is effectively reduced by more than 50% while ensuring the core stereoscopic visual experience, or the image quality of key areas is significantly improved under the same bandwidth.

[0023] Optimization of stereoscopic experience quality and comfort: With stereoscopic vision perception metadata as the core, the encoding and transmission resources are guided to tilt towards the areas that have the greatest impact on depth perception and visual comfort (such as the edges of moving objects and areas of abrupt changes in depth), which ensures high-quality stereoscopic viewing from the source and reduces visual fatigue caused by compression or transmission damage.

[0024] Enhanced transmission robustness: The cross-layer optimization controller maps visual importance to transmission priority, realizing dynamic matching between network resources and perception importance; when channel conditions deteriorate, it prioritizes the reliable and low-latency transmission of high-importance data, ensuring that the "skeleton" of stereo vision is not lost, and the system's anti-interference capability is greatly improved.

[0025] Synchronization inherent guarantee: Due to the use of single-channel composite bitstream transmission, the left and right viewpoint information has been precisely synchronized and encapsulated at the encoding end, and the decoding at the receiving end is naturally synchronized, which completely solves the problem of inter-frame and inter-line asynchrony that is difficult to overcome in traditional dual-channel independent transmission.

[0026] Reduced system complexity and cost: The adoption of a single-path encoding and transmission architecture reduces a complete encoding and RF link, which helps to reduce the power consumption, weight and hardware cost of the drone, and improve system integration and reliability.

[0027] Preferably, generating stereoscopic vision perception metadata associated with video frame timestamps includes: Perform semantic instance segmentation to obtain the semantic object distribution map of the current frame; Stereo matching is performed on the left-viewpoint video stream and the right-viewpoint video stream to generate a disparity map and the corresponding disparity confidence map; Calculate the spatial gradient map of the disparity map to identify regions of depth discontinuity; Motion estimation is performed on the left-viewpoint or right-viewpoint video stream to generate a motion vector field; The semantic object distribution map, disparity map, disparity confidence map, spatial gradient map, and motion vector field are input into an importance assessment model, which outputs spatial importance information. The importance assessment model is a neural network model trained on a stereo vision perception quality annotation dataset.

[0028] Semantic instance segmentation can employ lightweight real-time models, such as DeepLabV3+MobileNetV3.

[0029] Stereo matching can use the semi-global matching (SGM) algorithm or its hardware-accelerated version to generate a confidence map by calculating the curvature or kurtosis of the matching cost while generating the disparity map.

[0030] The spatial gradient map is obtained by convolving the disparity map with the Sobel or Scharr operator.

[0031] Motion estimation can reuse the motion search module in the video encoder or run a lightweight optical flow network (such as RAFT-Lite).

[0032] The above four feature maps (semantic, disparity, gradient, and motion) will be compared with confidence. Figure 1 Similarly, as a five-channel input, it is fed into a pre-trained convolutional neural network (CNN), such as an improved U-Net structure.

[0033] This CNN model, acting as an "importance assessment model," outputs a single-channel importance heatmap with the same resolution as the input, where each pixel value represents a score indicating the importance of that location to stereo perception.

[0034] Unlike traditional methods that rely solely on brightness, chromaticity, or motion amplitude for simple judgments, this approach integrates semantic, geometric, and dynamic multidimensional information. This enables the generated importance heatmap to accurately pinpoint regions that are truly critical to the stereoscopic visual experience (such as depth perception and comfort), such as the edges of moving objects in the foreground and areas of abrupt changes in depth. This provides a highly reliable basis for subsequent differentiated processing that aligns with human visual characteristics.

[0035] By combining confidence maps, the system can identify and reduce the importance misjudgment weights in areas with low texture, occlusion, and other matching difficulties, thereby improving the robustness of the analysis.

[0036] The generated metadata not only guides the encoding process, but can also be transmitted along with the bitstream as key information, providing a consistent "perception blueprint" for intelligent decoding, error hiding, and post-processing at the receiving end, thus ensuring global consistency across the processing chain.

[0037] The preferred training objective function of the importance assessment model integrates stereo visual comfort scores based on human subjective testing and objective image quality evaluation metrics.

[0038] First, a stereo vision perception quality annotation dataset needs to be constructed. This dataset contains a large number of stereo video clips, each of which has been processed as follows: different types of distortions are artificially introduced (such as different regions, different degrees of compression blur, noise, viewpoint shift, etc.).

[0039] Then, the participants watched these clips in a standardized 3D display environment and rated the overall stereoscopic visual comfort and depth perception clarity of the clips based on their subjective feelings (e.g., using mean opinion score, MOS).

[0040] At the same time, objective quality indicators of the segments are calculated, such as PSNR and SSIM for the left and right viewpoints, and disparity preservation for stereo perception.

[0041] During training, the model's loss function (objective function) is composite, for example: Loss = α * Loss_MOS + β * Loss_PSNR / SSIM + γ * Loss_DisparityConsistency.

[0042] Among them, Loss_MOS is the loss between the importance map predicted by the model and the subjective rating obtained through regression learning, ensuring that the model learns human subjective feelings; Loss_PSNR / SSIM, etc., ensure that the model does not violate the basic image quality rules; Loss_DisparityConsistency, on the other hand, constrains the model to pay more attention to the regions that affect the sense of depth.

[0043] By optimizing the composite loss function through backpropagation, the trained model can automatically learn and quantify the importance of regions that conform to the characteristics of human stereoscopic vision perception.

[0044] By directly incorporating human subjective ratings into the training objective, the model's judgment criteria are strongly correlated with the end-user's visual experience, avoiding the problem of disconnect from subjective feelings that may be caused by pure algorithm evaluation. This fundamentally ensures that the entire technical solution aims to improve user experience. By integrating objective indicators as constraints, the model is prevented from overfitting noise in subjective data, and its output is also reasonable in the traditional sense of image quality. This makes the evaluation results more stable and reliable when the model faces new scenes outside the training set.

[0045] Preferably, the perceived quality-oriented hierarchical coding includes: Base layer encoding: After downsampling and frame rate adjustment of the left viewpoint video stream and the right viewpoint video stream, the first set of encoding parameters is used for encoding; Enhancement layer coding: Based on spatial importance information, different coding resources are allocated to different regions in the image, including: Quantization parameter allocation: Assign quantization parameter values ​​lower than those for low importance regions to regions with high importance levels; Inter-frame prediction optimization: For high-importance regions, limit the range of matching blocks they can search in the reference frame, or increase the quality requirements of their reference frames; Rate allocation: Using spatial importance information as a weight map, perceptually weighted rate allocation is performed in the encoder's rate control module.

[0046] This encoding scheme needs to be implemented on a hardware encoder that supports flexible bitrate control and region encoding (such as one based on the H.266 / VVC standard); Breaking away from the traditional encoding model of "average effort" across all regions, this approach concentrates limited bitrate resources on the areas that contribute most to the stereoscopic experience. While maintaining the total bitrate, it significantly improves the subjective and objective quality of these key areas, resulting in a superior stereoscopic visual experience overall. By implementing stricter inter-frame prediction constraints on highly important areas, it effectively suppresses the propagation and accumulation of compression errors over time, ensuring that these areas remain clear and stable during continuous playback and preventing blurring or blockiness of key objects during movement.

[0047] Preferably, a single-channel composite bitstream is generated using a region-of-interest coding tool that conforms to H, 266 / VVC, AVS3, or subsequent video coding standards.

[0048] Preferably, the differentiated data transmission guarantee strategy is executed by a cross-layer optimization controller deployed in the UAV communication protocol stack, including: Monitor real-time transmission quality parameters of the wireless link; Based on spatial importance information and the coding layer to which the data unit belongs, a transmission priority identifier is generated for the data packet to be transmitted; Based on the transmission priority identifier, dynamically adjust at least one of the following transmission mechanisms: Redundancy of the forward error correction scheme; The maximum number of retransmissions for automatic retransmission requests and the timeout duration of the retransmission timer; The order in which data packets are scheduled in the send buffer; Robustness level of physical layer modulation and coding schemes.

[0049] By directly mapping the "perceived importance" of video content to the "service priority" of network transmission, valuable and fluctuating wireless channel resources can be intelligently allocated to the most important data, reinforcing the "core-preserving, experience-preserving" design principle of the entire system from the transmission level. When encountering interference or signal fading, the system no longer experiences a uniform decline in overall image quality or complete stuttering, but instead prioritizes ensuring the integrity and smoothness of the stereoscopic "skeleton" (base layer and key enhancement layers), sacrificing only the details of secondary background areas. This "graceful degradation" characteristic is crucial for the reliable operation of drones in complex environments.

[0050] Preferably, layered decoding and perceptual quality reconstruction include: Base layer decoding and reconstruction: Decode the base layer data in a single composite bitstream to obtain the initial left and right viewpoint reconstruction frames; Iterative enhancement of enhancement layers: Based on the spatial importance information obtained from decoding, enhancement layer data is decoded and applied sequentially in order of importance from high to low to enhance the quality of the corresponding regions in the initial left and right viewpoint reconstruction frames. Error concealment handling: When the loss of enhancement layer data in a specific area is detected, based on the principle of stereo consistency, the content is restored by using the decoded information of the corresponding area of ​​the opposite viewpoint and the information of adjacent frames; Stereo consistency post-processing: The final reconstructed left and right viewpoint images are filtered to suppress noise inconsistencies between viewpoints that may be caused by layered coding and transmission.

[0051] Decoding in order of importance allows image quality to improve progressively in a way that best suits human visual perception, ensuring users always see the clearest details of the most critical parts first, resulting in a better experience. Simultaneously, this sequential decoding logic is clear and easy to optimize in hardware implementation. The unique "error hiding based on stereoscopic consistency" mechanism fully utilizes the inherent information redundancy in stereoscopic video (left-right viewpoint correlation, temporal correlation), intelligently and seamlessly recovering acceptable content even when some data is lost. This greatly enhances the continuity of the viewing experience under unreliable wireless channels, avoiding annoying pixelation or image freezing. A dual-channel image transmission 3D split-screen display system for unmanned aerial vehicles (UAVs) is provided to implement the aforementioned dual-channel image transmission 3D split-screen display method for UAVs, such as... Figure 2 As shown, it includes a UAV terminal system 110 and a ground terminal system 120; The UAV terminal system 110 includes a binocular camera module 111, an airborne processing module 112, and an airborne wireless transmission module 113; The binocular camera module 111 is used to acquire raw stereoscopic video. It can use two global shutter cameras, which are precisely calibrated and synchronized to acquire synchronized left-viewpoint video streams and right-viewpoint video streams.

[0052] The airborne processing module 112, which is the core of the present invention, is used to generate stereo vision perception metadata, perform perception quality-oriented layered coding to generate a single composite bitstream, and configure differentiated data transmission guarantee strategies.

[0053] The airborne wireless transmission module 113 is used to transmit a single composite code stream. It can be a COFDM or 5G-based image transmission device and is responsible for transmitting the single composite code stream with priority tags through the wireless link 130.

[0054] Ground terminal system 120 includes: The ground wireless receiver module 121 is used to receive wireless signals and demodulate a single-channel composite code stream.

[0055] The ground processing module 122 is used to perform layered decoding and perceptual quality reconstruction; the stereoscopic display device is used to present the reconstructed stereoscopic video.

[0056] A stereoscopic display device 123, such as a 3D monitor or VR headset, receives the reconstructed high-quality left and right viewpoint video signals and displays them in stereoscopic form.

[0057] The beneficial effects of this invention are as follows: Significantly improved transmission efficiency: By integrating the left and right viewpoint video streams into a single perception-quality-oriented composite bitstream and performing differentiated encoding on different important areas, the bandwidth requirement is effectively reduced by more than 50% while ensuring the core stereoscopic visual experience, or the image quality of key areas is significantly improved under the same bandwidth.

[0058] Optimization of stereoscopic experience quality and comfort: With stereoscopic vision perception metadata as the core, the encoding and transmission resources are guided to tilt towards the areas that have the greatest impact on depth perception and visual comfort (such as the edges of moving objects and areas of abrupt changes in depth), which ensures high-quality stereoscopic viewing from the source and reduces visual fatigue caused by compression or transmission damage.

[0059] Enhanced transmission robustness: The cross-layer optimization controller maps visual importance to transmission priority, realizing dynamic matching between network resources and perception importance; when channel conditions deteriorate, it prioritizes the reliable and low-latency transmission of high-importance data, ensuring that the "skeleton" of stereo vision is not lost, and the system's anti-interference capability is greatly improved.

[0060] Synchronization inherent guarantee: Due to the use of single-channel composite bitstream transmission, the left and right viewpoint information has been precisely synchronized and encapsulated at the encoding end, and the decoding at the receiving end is naturally synchronized, which completely solves the problem of inter-frame and inter-line asynchrony that is difficult to overcome in traditional dual-channel independent transmission.

[0061] Reduced system complexity and cost: The adoption of a single-path encoding and transmission architecture reduces a complete encoding and RF link, which helps to reduce the power consumption, weight and hardware cost of the drone, and improve system integration and reliability.

[0062] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the above-described UAV dual-channel image transmission 3D split-screen display method.

[0063] A drone, wherein the drone is equipped with a drone terminal system as described above for a drone dual-channel image transmission 3D split-screen display system.

[0064] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for dual-channel image transmission and 3D split-screen display of a UAV, characterized in that, include: Step 1: The binocular camera module mounted on the drone acquires synchronized left-view and right-view video streams; Step 2: Perform joint analysis on the left-viewpoint video stream and the right-viewpoint video stream to generate stereo vision perception metadata associated with video frame timestamps; the stereo vision perception metadata contains spatial importance information of at least two levels based on image content features. Step 3: Based on stereo vision perception metadata, perform perception quality-oriented layered encoding on the left viewpoint video stream and the right viewpoint video stream to generate a single composite bitstream; wherein, the single composite bitstream contains a base layer and at least one enhancement layer. The base layer is used to ensure the continuity of stereo vision and basic depth perception, and the enhancement layer is used to perform differentiated quality enhancement on different levels of spatial importance information indication areas. Step 4: Based on the spatial importance information contained in the stereo vision perception metadata, configure differentiated data transmission guarantee strategies for data units corresponding to different layers and different levels of regions in a single composite bitstream. Step 5: Transmit a single composite stream configured with differentiated data transmission guarantee strategy to the ground station via wireless link; Step 6: The ground station performs layered decoding and perception quality reconstruction on the received single-channel composite bitstream to output left and right viewpoint video signals for stereoscopic display; The perceived quality-oriented hierarchical coding includes: Base layer encoding: After downsampling and frame rate adjustment of the left viewpoint video stream and the right viewpoint video stream, the first set of encoding parameters is used for encoding; Enhancement layer coding: Based on spatial importance information, different coding resources are allocated to different regions in the image, including: Quantization parameter allocation: Assign quantization parameter values ​​lower than those for low importance regions to regions with high importance levels; Inter-frame prediction optimization: For high-importance regions, limit the range of matching blocks they can search in the reference frame, or increase the quality requirements of their reference frames; Rate allocation: Using spatial importance information as a weight map, perceptually weighted rate allocation is performed in the encoder's rate control module; The hierarchical decoding and perceptual quality reconstruction include: Base layer decoding and reconstruction: Decode the base layer data in a single composite bitstream to obtain the initial left and right viewpoint reconstruction frames; Iterative enhancement of enhancement layers: Based on the spatial importance information obtained from decoding, enhancement layer data is decoded and applied sequentially in order of importance from high to low to enhance the quality of the corresponding regions in the initial left and right viewpoint reconstruction frames. Error concealment handling: When the loss of enhancement layer data in a specific area is detected, based on the principle of stereo consistency, the content is restored by using the decoded information of the corresponding area of ​​the opposite viewpoint and the information of adjacent frames; Stereo consistency post-processing: The final reconstructed left and right viewpoint images are filtered to suppress noise inconsistencies between viewpoints that may be caused by layered coding and transmission.

2. The UAV dual-channel image transmission 3D split-screen display method according to claim 1, characterized in that, The generated stereoscopic vision perception metadata associated with video frame timestamps includes: Perform semantic instance segmentation to obtain the semantic object distribution map of the current frame; Stereo matching is performed on the left-viewpoint video stream and the right-viewpoint video stream to generate a disparity map and the corresponding disparity confidence map; Calculate the spatial gradient map of the disparity map to identify regions of depth discontinuity; Motion estimation is performed on the left-viewpoint or right-viewpoint video stream to generate a motion vector field; The semantic object distribution map, disparity map, disparity confidence map, spatial gradient map, and motion vector field are input into an importance assessment model, which outputs spatial importance information. The importance assessment model is a neural network model trained on a stereo vision perception quality annotation dataset.

3. The UAV dual-channel image transmission 3D split-screen display method according to claim 2, characterized in that, The training objective function of the importance assessment model integrates stereoscopic visual comfort scores based on human subjective testing and objective image quality evaluation indicators.

4. The UAV dual-channel image transmission 3D split-screen display method according to claim 1, characterized in that, A single-channel composite bitstream is generated using a region-of-interest (ROI) coding tool that conforms to H, 266 / VVC, AVS3, or subsequent video coding standards.

5. The UAV dual-channel image transmission 3D split-screen display method according to claim 1, characterized in that, The differentiated data transmission guarantee strategy is executed by a cross-layer optimization controller deployed in the UAV communication protocol stack, including: Monitor the real-time transmission quality parameters of the wireless link; Based on spatial importance information and the coding layer to which the data unit belongs, a transmission priority identifier is generated for the data packet to be transmitted; Based on the transmission priority identifier, dynamically adjust at least one of the following transmission mechanisms: Redundancy of the forward error correction scheme; The maximum number of retransmissions for automatic retransmission requests and the timeout duration of the retransmission timer; The order in which data packets are scheduled in the send buffer; Robustness level of physical layer modulation and coding schemes.

6. A UAV dual-channel image transmission 3D split-screen display system, used to implement the UAV dual-channel image transmission 3D split-screen display method of any one of claims 1 to 5, characterized in that, Including UAV terminal systems and ground terminal systems; The UAV terminal system includes a binocular camera module, an airborne processing module, and an airborne wireless transmission module; the binocular camera module is used to acquire raw stereo video; the airborne processing module is used to generate stereo vision perception metadata, perform perception quality-oriented layered coding to generate a single composite bitstream, and configure differentiated data transmission guarantee strategies. The airborne wireless transmission module is used to transmit a single-channel composite code stream; The ground terminal system includes: a ground wireless receiving module, a ground processing module, and a 3D display device; the ground wireless receiving module is used to receive a single composite bitstream; the ground processing module is used to perform layered decoding and perception quality reconstruction; and the stereoscopic display device is used to present the reconstructed stereoscopic video.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV dual-path image transmission 3D split-screen display method as described in any one of claims 1 to 5.

8. A drone, characterized in that, The drone is equipped with a drone terminal system of the drone dual-channel image transmission 3D split-screen display system as described in claim 6.

Images