High-resolution image generation method based on live video from low-altitude drones

Abstract

This invention discloses a high-definition image generation method based on live video from low-altitude unmanned aerial vehicles (UAVs), belonging to the technical field of high-definition image generation for low-altitude UAVs. The method includes: dividing image regions and performing adaptive grid processing; building a parallel computing framework and using a stitching algorithm to stitch images; adjusting the image grid size in real time and optimizing the image processing algorithm based on real-time feedback data; analyzing flight path data, establishing a predictive model, and modeling future data demands for the region; dynamically optimizing data transmission paths and bandwidth allocation based on image data demands and flight path information; constructing a virtual environment model based on digital twin technology to compensate for image loss caused by environmental changes; performing real-time quality assessment of the image and adjusting the processing strategy in real time; and solving the problems of low image stitching accuracy and inaccurate quality assessment in existing technologies by combining deep learning with image processing techniques that dynamically adjust the grid size.

Description

Technical Field

[0001] This invention relates to the field of high-definition mapping technology for low-altitude unmanned aerial vehicles (UAVs), and more specifically, to a high-definition mapping method based on live video of low-altitude UAVs. Background Technology

[0002] With the continuous development of low-altitude drone technology, low-altitude drones have been widely used in various fields, including environmental monitoring, agricultural spraying, disaster relief missions, and aerial photography. Among them, drone live video technology relies on advanced image sensors, real-time data transmission technology, and image processing algorithms. It can not only acquire high-definition aerial images in real time, but also transmit them to a remote location for processing and viewing within a certain range. Especially when performing large-scale live image stitching and high-precision image synthesis, it is required to process the acquired images efficiently and accurately to meet the dual requirements of image quality and real-time transmission.

[0003] However, existing low-altitude drone live video technology still faces many challenges, especially in image processing and image generation, for example:

[0004] 1. Although modern drones are equipped with high-resolution cameras and powerful data transmission capabilities, real-time image stitching and high-definition image generation still face significant challenges due to limitations in flight speed, environmental conditions, and transmission bandwidth.

[0005] 2. When stitching images, traditional techniques generally rely on static image processing methods, which greatly reduces the accuracy and stability of image stitching in dynamic scenes, making it impossible to achieve the requirements of high-precision and high-quality image formation.

[0006] 3. In addition, existing image quality assessment systems have not been fully optimized for images taken by drones in high dynamic and complex environments. Image restoration and enhancement technologies often face problems such as noise removal and detail restoration, which affect the quality of the final image and its application effect.

[0007] Most importantly, most current systems cannot achieve a dynamic balance between image quality and resource utilization. Especially when transmission bandwidth is limited, the contradiction between image quality and processing speed is still difficult to resolve effectively.

[0008] Based on this, we propose a high-definition image generation method based on live video from low-altitude drones. Summary of the Invention

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] High-definition image generation methods based on low-altitude drone live video include:

[0011] Divide the image region and perform adaptive grid processing;

[0012] A parallel computing framework was built, and a stitching algorithm was used to stitch the images together;

[0013] Adjust the image grid size in real time and optimize the image processing algorithm based on real-time feedback data;

[0014] Analyze flight path data, build predictive models, and model future data needs for the region;

[0015] Based on image data requirements and flight path information, dynamically optimize data transmission paths and bandwidth allocation;

[0016] A virtual environment model is built based on digital twin technology to compensate for image loss caused by environmental changes;

[0017] Perform real-time quality assessment on images and adjust processing strategies in real time;

[0018] Furthermore, the process of dividing the image region and performing adaptive grid processing includes:

[0019] Construct a convolutional neural network, set the input layer, and set the input tensor size to be... The image data, in which, The height of the image. The width of the image. The number of channels in the image;

[0020] We set up a convolutional layer to perform local perception on the input image, extracting local features, including edges and textures. Let the first convolutional layer be the... The convolution operation of a layer is as follows:

[0021] ;

[0022] in, For the first Convolutional kernels of layers; This is the input image for the previous layer; For bias terms; The size of the convolution kernel; through multiple convolutional operations, the network gradually extracts higher-level semantic information, such as texture and object shape, from low-level features, such as edges and corners.

[0023] Pooling layers are used to reduce computation and help extract the maximum local features. Max pooling layers, in particular, retain the most important features by selecting the maximum value in the convolutional image, as shown below:

[0024] ;

[0025] Where is the pixel index within the pooling region; and is the result after the pooling operation.

[0026] After convolution and pooling, a fully connected layer is set to synthesize the extracted features. The fully connected layer learns a vectorized representation to ultimately predict the complexity of the image content, as shown in the following equation:

[0027] ;

[0028] in, For the first Layer output; This is the weight matrix; For activation functions, such as ReLU;

[0029] Images are input into deep convolutional neural networks, which extract features from the images at multiple levels. Each layer analyzes the complexity of the image at different scales. The initial convolutional layers extract basic features of the image through filters, such as edges and corners. These features are crucial for capturing simple, structured content in the image. In subsequent convolutional layers, the network can learn more complex features, such as the shape and texture of objects. These features help the network recognize objects and structures in the image. At higher levels of the network, the network can extract semantic information of the image, such as the relationship between specific objects and the background. These features are closely related to the complexity of the image and can effectively represent the global structure of the image.

[0030] Networks can combine features from different levels to improve the accuracy of image analysis. For example, combining low-level edge information, mid-level texture information, and high-level semantic information can more comprehensively analyze the complexity of an image. For complex regions, the network can use a weighted combination of these features to determine the region's detail requirements.

[0031] By weighting and fusing features at different levels, a comprehensive image complexity metric is obtained. Let the first... The feature map of the layer is The complexity of the image region It can be expressed as the following formula:

[0032] ;

[0033] in, This represents the total number of layers in the convolutional neural network. The weights of the features at each layer; For the first Feature map of the layer; The feature intensity at each location is used; based on this, the image complexity is calculated by combining the features from each layer, and the grid size is adaptively adjusted, as shown in the following formula:

[0034] ;

[0035] in, For the entire image region; For the image at position The first Layer, First The strength of each feature represents the feature extracted from the convolutional neural network; The number of features extracted in each layer; This is a Gaussian weighting function, which controls the calculation accuracy for complex regions of the image. and The parameters of the Gaussian function are used to adjust the influence range of image region complexity and control spatial weighting. The range of values ​​is The higher the complexity, the more computing resources are needed to process that region.

[0036] Furthermore, the construction of the parallel computing framework and the use of a stitching algorithm to stitch the images include:

[0037] Building a parallel computing framework includes:

[0038] Choose the CUDA computing platform, allocate GPU resources, and interact with the GPU through CUDA to accelerate image processing;

[0039] Each region of the image generated by adaptive meshing is divided into N small mesh blocks, and the size of each mesh block is adjusted according to the complexity and resolution of the image;

[0040] Each grid block corresponds to a computing unit. The GPU uses CUDA's thread model to assign computing tasks to threads in each grid block, ensuring that computing tasks can be executed in parallel.

[0041] After each computing unit finishes processing, the GPU summarizes the results and returns them to the host for subsequent image stitching and reconstruction.

[0042] Image data is first transferred from the host memory to the GPU's device memory, where it is processed. After processing, it is returned to the host. Each thread performs specific computational operations, such as image reconstruction and region stitching.

[0043] After the calculation is completed, the results are transferred from GPU memory back to host memory, and the stitched results are merged to obtain the complete image.

[0044] The acceleration formula for parallel computation is as follows:

[0045] ;

[0046] in, The time required for image processing; This refers to the number of processing units in parallel computing. As the number of parallel computing units increases, the processing time... It will decrease;

[0047] The process of merging the stitching results includes: using image fusion technology to reduce the seams between different regions, and seamlessly stitching the images of different regions based on formula (#5);

[0048] After image stitching is completed, the quality of the stitched image is evaluated in real time. The evaluation criteria include edge sharpness and stitching accuracy.

[0049] Based on the evaluation results, computing resources will be allocated preferentially to areas with higher complexity to ensure that these areas can obtain more computing resources and ensure the accuracy of stitching and image quality.

[0050] Through a dynamic optimization mechanism, the system adjusts computing resources in real time based on the evaluation results to optimize the efficiency and quality of image stitching, ensuring that the stitched images meet the requirements of high resolution and high quality.

[0051] The quality assessment formula is:

[0052] ;

[0053] in, To assess the quality of the final stitched image, used to evaluate the stitching effect and sharpness; The gradient of the image reflects the sharpness of the edges; The gradient of the stitched image reflects the stitching accuracy.

[0054] Furthermore, the real-time adjustment of the image grid size and the optimization of the image processing algorithm based on real-time feedback data include:

[0055] The Canny edge detection algorithm is used to identify edge regions in the image, and the gray-level co-occurrence matrix is ​​used to identify textured regions in the image.

[0056] Areas with significant changes in lighting are defined as complex areas, while flat background areas are defined as simple areas.

[0057] Each patch in an image divided into multiple small patches is analyzed, and a feature extraction algorithm is used to obtain the complexity index of the region. The complexity index includes the standard deviation of texture complexity, edge density, and brightness variation within each region.

[0058] Set a threshold to determine the complexity of each region, complete the complexity evaluation of the image region, and mark the image region as complex or simple based on the evaluation results;

[0059] Adjust the grid size, reducing it for complex areas and increasing it for simple areas, to reduce computational burden while ensuring that image details are not lost;

[0060] An image quality assessment system is established using quality assessment formulas.

[0061] Establish quality assessment standards, including:

[0062] Evaluate the edge sharpness and detail retention of the image. Complex areas should have higher edge sharpness and detail retention, while simple areas have lower sharpness requirements.

[0063] Check the transition at the seams to ensure that there are no obvious seams or distortions in the splicing area;

[0064] The image shows reasonable color and brightness distribution across different regions, without abrupt changes.

[0065] The system obtains quality assessment information from the feedback results of image stitching, and adjusts the grid size, computing resource allocation, and processing algorithm in real time based on the feedback.

[0066] Specifically, for areas with poor image quality, the system increases computing resources to perform further fine processing on those areas; for areas with good image quality, it reduces computing resources to improve processing speed until a balance is achieved.

[0067] Furthermore, the analysis of flight path data and the establishment of predictive models to model future regional data demands include:

[0068] Real-time flight path analysis is performed using the aircraft's GPS information, combined with image acquisition information, to predict the expected data demand for the image area, as shown in the following formula:

[0069] ;

[0070] in, The expected data requirements are expressed in data volume. The image region is defined by the flight path. This includes the drone's flight data, such as altitude and speed, which is used to adjust the image area as needed.

[0071] Based on flight paths and real-time data demand forecasting, a data demand forecasting model is established to provide a basis for real-time adjustments to regional data demand, as shown in the following formula:

[0072] ;

[0073] in, To meet the future data needs of the region; The function is a prediction model that predicts future data requirements for a region based on historical and real-time flight path data inputs. The model parameter is the model parameter.

[0074] Combining flight path analysis and data demand forecasting models, a bandwidth forecasting formula is designed, assuming a linear correlation between data demand and transmission bandwidth, as shown in the following formula:

[0075] ;

[0076] in, The predicted bandwidth requirement is expressed in data transfer rate. This is a bandwidth function that adjusts bandwidth requirements based on the flight path, data demand, and the real-time status of the aircraft.

[0077] Furthermore, the dynamic optimization of data transmission path and bandwidth allocation based on image data requirements and flight path information includes:

[0078] By utilizing flight path analysis and image acquisition information, combined with a data demand prediction model, the image data demand in different areas is assessed, and a transmission path is selected. The selection of the transmission path is based not only on the current flight position of the aircraft, but also on the future data demand of the image area, so as to dynamically optimize the data transmission route.

[0079] By combining image data requirements with flight paths and real-time feedback from image processing, the data transmission path can be dynamically adjusted.

[0080] Based on real-time monitoring of aircraft status and network load, the data transmission path is adjusted and optimized as follows:

[0081] ;

[0082] in, The optimal transmission path; Candidate paths;

[0083] Based on the network conditions of the transmission path and the priority of image data, bandwidth allocation is optimized to reduce latency during data transmission, as shown in the following formula:

[0084] ;

[0085] in, To allocate the optimal bandwidth; Candidate bandwidth;

[0086] Real-time monitoring of data traffic and bandwidth usage during data transmission, combined with flight path and network conditions to adjust bandwidth and optimize transmission efficiency, as shown in the following formula:

[0087] ;

[0088] in, This refers to the time for real-time data transmission. This is a real-time monitoring function that represents the impact of real-time data traffic monitoring and scheduling.

[0089] Furthermore, the virtual environment model constructed based on digital twin technology to compensate for image loss caused by environmental changes includes:

[0090] Construct an environment model by taking the aircraft's real-time position and flight data, environmental data, and image data as input, and outputting the environment model as follows:

[0091] ;

[0092] in, This is an environmental model representing the impact of flight path and environmental conditions on image quality; This is an environment model function, representing the impact of environmental changes encountered by the aircraft on the image;

[0093] Based on environmental models and flight data, repair algorithms are used to repair missing and damaged parts of the image, compensating for image loss caused by environmental changes.

[0094] A detail enhancement algorithm based on digital twin technology is used to enhance the details of the restored image. The intensity of image enhancement is automatically adjusted by analyzing environmental data. The image restoration and detail enhancement are defined by the following formula:

[0095] ;

[0096] in, The enhanced image; Input image; For enhancement functions;

[0097] Data from all sensors is fused, and image inpainting algorithms are used to enhance image details and restore images lost due to lighting or weather conditions. The image enhancement and data fusion process is as follows:

[0098] ;

[0099] in, The final enhanced image; For from the Images from various sensors, The weighting coefficient represents the weight of different sensor data.

[0100] Furthermore, the real-time quality assessment of the image and the real-time adjustment of the processing strategy include:

[0101] Using a convolutional neural network model, the quality of each frame of the image is evaluated in real time. Image quality is extracted through multi-scale feature extraction, and the image sharpness, clarity, and noise are comprehensively evaluated.

[0102] During image processing, the image quality is calculated progressively, and the image restoration and enhancement strategies are adjusted based on the evaluation results, as follows:

[0103] ;

[0104] in, Provide real-time image quality scoring; It is a structural similarity index; Peak signal-to-noise ratio;

[0105] Based on the real-time image quality score, different regions in the image are prioritized and sorted according to their quality, with poor quality regions being processed first.

[0106] Based on the current assessment results and computing resource requirements, the allocation of resources is dynamically adjusted, and a parallel computing framework is used to improve processing speed; priority adjustment and resource allocation are represented by formula (#11);

[0107] Real-time evaluation of image quality; adjustment of restoration strategies based on system scores; use of enhancement algorithms to enhance image details; and adjustment of restoration intensity and priority during image processing to ensure that the image is not distorted during enhancement and to avoid over-enhancement as much as possible.

[0108] Based on the image quality assessment results, the enhancement strategy is adjusted by optimizing the algorithm, and the resource utilization in the image restoration process is optimized to ensure the quality and efficiency of the processing results.

[0109] In summary, the present invention has the following beneficial effects:

[0110] By combining deep learning with image processing techniques that dynamically adjust grid size, the problems of low image stitching accuracy and inaccurate quality assessment in existing technologies have been solved.

[0111] By using real-time image quality assessment and dynamic priority adjustment, this scheme can optimize real-time data transmission and processing while ensuring image quality.

[0112] By combining image quality dynamic adjustment technology, this solution can ensure high-quality image output in real-time video transmission, while rationally allocating computing resources, thus avoiding the problems of image quality degradation and processing delay caused by resource bottlenecks in existing technologies. Attached Figure Description

[0113] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0114] Figure 1 This is a flowchart of the high-definition image generation method based on live video from a low-altitude drone, according to the present invention. Detailed Implementation

[0115] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0116] Example:

[0117] The following is in conjunction with the appendix Figure 1 The present invention will be described in further detail below.

[0118] Please see Figure 1 This invention provides a technical solution: a high-definition image generation method based on live video from low-altitude drones, such as... Figure 1 Shown, including:

[0119] Divide the image region and perform adaptive grid processing;

[0120] A parallel computing framework was built, and a stitching algorithm was used to stitch the images together;

[0121] Adjust the image grid size in real time and optimize the image processing algorithm based on real-time feedback data;

[0122] Analyze flight path data, build predictive models, and model future data needs for the region;

[0123] Based on image data requirements and flight path information, dynamically optimize data transmission paths and bandwidth allocation;

[0124] A virtual environment model is built based on digital twin technology to compensate for image loss caused by environmental changes;

[0125] Perform real-time quality assessment on images and adjust processing strategies in real time;

[0126] like Figure 1 As shown, dividing the image region and performing adaptive grid processing includes:

[0127] Construct a convolutional neural network, set the input layer, and set the input tensor size to be... The image data, in which, The height of the image. The width of the image. The number of channels in the image;

[0128] We set up a convolutional layer to perform local perception on the input image, extracting local features, including edges and textures. Let the first convolutional layer be the... The convolution operation of a layer is as follows:

[0129] ;

[0130] in, For the first Convolutional kernels of layers; This is the input image for the previous layer; For bias terms; The size of the convolution kernel; through multiple convolutional operations, the network gradually extracts higher-level semantic information, such as texture and object shape, from low-level features, such as edges and corners.

[0131] Pooling layers are used to reduce computation and help extract the maximum local features. Max pooling layers, in particular, retain the most important features by selecting the maximum value in the convolutional image, as shown below:

[0132] ;

[0133] Where is the pixel index within the pooling region; and is the result after the pooling operation.

[0134] After convolution and pooling, a fully connected layer is set to synthesize the extracted features. The fully connected layer learns a vectorized representation to ultimately predict the complexity of the image content, as shown in the following equation:

[0135] ;

[0136] in, For the first Layer output; This is the weight matrix; For activation functions, such as ReLU;

[0137] Images are input into deep convolutional neural networks, which extract features from the images at multiple levels. Each layer analyzes the complexity of the image at different scales. The initial convolutional layers extract basic features of the image through filters, such as edges and corners. These features are crucial for capturing simple, structured content in the image. In subsequent convolutional layers, the network can learn more complex features, such as the shape and texture of objects. These features help the network recognize objects and structures in the image. At higher levels of the network, the network can extract semantic information of the image, such as the relationship between specific objects and the background. These features are closely related to the complexity of the image and can effectively represent the global structure of the image.

[0138] Networks can combine features from different levels to improve the accuracy of image analysis. For example, combining low-level edge information, mid-level texture information, and high-level semantic information can more comprehensively analyze the complexity of an image. For complex regions, the network can use a weighted combination of these features to determine the region's detail requirements.

[0139] By weighting and fusing features at different levels, a comprehensive image complexity metric is obtained. Let the first... The feature map of the layer is The complexity of the image region It can be expressed as the following formula:

[0140] ;

[0141] in, This represents the total number of layers in the convolutional neural network. The weights of the features at each layer; For the first Feature map of the layer; The feature intensity at each location is used; based on this, the image complexity is calculated by combining the features from each layer, and the grid size is adaptively adjusted, as shown in the following formula:

[0142] ;

[0143] in, For the entire image region; For the image at position The first Layer, First The strength of each feature represents the feature extracted from the convolutional neural network; The number of features extracted in each layer; This is a Gaussian weighting function, which controls the calculation accuracy for complex regions of the image. and The parameters of the Gaussian function are used to adjust the influence range of image region complexity and control spatial weighting. The range of values ​​is The higher the complexity, the more computing resources are needed to process that region.

[0144] In this embodiment, an experiment was designed to verify the advantages of the above embodiment over existing technologies in terms of image stitching quality, data transmission efficiency, and image restoration effect. A low-altitude UAV was selected as the data acquisition device, and data acquisition and image stitching were performed in an urban environment using this UAV. Two sets of control data were set up: the first set used existing traditional image stitching and quality assessment algorithms, and the second set used the method proposed in this embodiment for image stitching, quality assessment, and data transmission optimization. The specific steps are as follows:

[0145] Using a low-altitude drone, model M30, equipped with a high-resolution camera with a resolution of 4096x2160, data was collected under different flight paths and environments. The average altitude of each flight did not exceed 180 meters, the maximum flight speed was 15m / s, and the drone collected 100 frames of images per minute, with each image size being less than 2MB.

[0146] The adaptive network processing method in this embodiment is used to perform region segmentation and gridding processing on the acquired image. The image is divided into multiple small regions, and each region is processed as a unit to optimize processing speed and improve image stitching efficiency. The size of the region segmentation is 960*720 pixels.

[0147] The adaptive network processing method in the above embodiment is used to perform region division and gridding processing on the acquired image. The image is divided into multiple small regions, and each region is processed as a unit to optimize the processing speed and improve the image stitching efficiency. The size of the region division is 64x64 pixels.

[0148] When performing image stitching, traditional methods directly stitch all images together and use low-level features for coarse matching. However, in this embodiment, a deep learning model, convolutional neural network (CNN), is used to analyze the image content, extract high-level feature information of each region, and perform more accurate stitching, avoiding the distortion and seam problems in traditional stitching methods.

[0149] The image quality assessment in this embodiment adopts a method based on SSIM and PSNR, combined with a deep learning model to score the image in multiple dimensions. After the image is stitched together, the quality of the result is assessed in real time.

[0150] The data transmission is optimized using a dynamic bandwidth adjustment algorithm based on flight path analysis and real-time flight data. By analyzing image quality requirements and flight path information, the bandwidth of data transmission is adjusted to ensure a balance between image quality and transmission speed.

[0151] In terms of image restoration, this embodiment adopts deep learning-based image enhancement technology, which combines data from different sensors for restoration and detail enhancement, effectively compensating for the loss caused by environmental factors such as weather and lighting changes, and improving the quality of the final image.

[0152] Comparative Experiment of Image Stitching and Data Transmission Optimization Based on Low-Altitude UAV Live Video (#Table 1)

[0153] ;

[0154] As can be seen from the data in the table above, in terms of image quality, the deep learning model in this embodiment can extract and stitch images more accurately, effectively improving image clarity and detail. Secondly, in terms of processing speed and data transmission efficiency, this embodiment improves the overall performance of the system and reduces processing time and bandwidth pressure through dynamic bandwidth optimization and image restoration and enhancement techniques. Finally, the method in this embodiment not only achieves a breakthrough in image quality but also demonstrates significant advantages in real-time data processing and bandwidth utilization.

[0155] like Figure 1 As shown, the parallel computing framework is built, and the image is stitched together using a stitching algorithm, including:

[0156] Building a parallel computing framework includes:

[0157] Choose the CUDA computing platform, allocate GPU resources, and interact with the GPU through CUDA to accelerate image processing;

[0158] Each region of the image generated by adaptive meshing is divided into N small mesh blocks, and the size of each mesh block is adjusted according to the complexity and resolution of the image;

[0159] Each grid block corresponds to a computing unit. The GPU uses CUDA's thread model to assign computing tasks to threads in each grid block, ensuring that computing tasks can be executed in parallel.

[0160] After each computing unit finishes processing, the GPU summarizes the results and returns them to the host for subsequent image stitching and reconstruction.

[0161] Image data is first transferred from the host memory to the GPU's device memory, where it is processed. After processing, it is returned to the host. Each thread performs specific computational operations, such as image reconstruction and region stitching.

[0162] After the calculation is completed, the results are transferred from GPU memory back to host memory, and the stitched results are merged to obtain the complete image.

[0163] The formula for accelerating parallel computation is:

[0164] ;

[0165] in, The time required for image processing; This refers to the number of processing units in parallel computing. As the number of parallel computing units increases, the processing time... It will decrease;

[0166] The process of merging the stitching results includes: using image fusion technology to reduce the seams between different regions, and seamlessly stitching the images of different regions based on formula (#5);

[0167] After image stitching is completed, the quality of the stitched image is evaluated in real time. The evaluation criteria include edge sharpness and stitching accuracy.

[0168] Based on the evaluation results, computing resources will be allocated preferentially to areas with higher complexity to ensure that these areas can obtain more computing resources and ensure the accuracy of stitching and image quality.

[0169] Through a dynamic optimization mechanism, the system adjusts computing resources in real time based on the evaluation results to optimize the efficiency and quality of image stitching, ensuring that the stitched images meet the requirements of high resolution and high quality.

[0170] The quality assessment formula is:

[0171] ;

[0172] in, To assess the quality of the final stitched image, used to evaluate the stitching effect and sharpness; The gradient of the image reflects the sharpness of the edges; The gradient of the stitched image reflects the stitching accuracy.

[0173] In this embodiment, by building a parallel computing framework and utilizing CUDA and GPU resources for image processing, the computational tasks of each image grid can be accelerated, enabling efficient image stitching. Furthermore, by optimizing the image region in real time and through a reverse mechanism, the image stitching quality is improved, and the allocation of computational tasks is effectively optimized, reducing the computational burden. In addition, by establishing a data demand prediction model, a real-time adjustment scheme can be provided for the data demand of subsequent regions, thereby optimizing the data flow transmission route and bandwidth usage, ensuring unrestricted data transmission during image acquisition, and improving data transmission efficiency.

[0174] like Figure 1 As shown, the image grid size is adjusted in real time, and the image processing algorithm is optimized based on real-time feedback data, including:

[0175] The Canny edge detection algorithm is used to identify edge regions in the image, and the gray-level co-occurrence matrix is ​​used to identify textured regions in the image.

[0176] Areas with significant changes in lighting are defined as complex areas, while flat background areas are defined as simple areas.

[0177] Each patch in an image divided into multiple small patches is analyzed, and a feature extraction algorithm is used to obtain the complexity index of the region. The complexity index includes the standard deviation of texture complexity, edge density, and brightness variation within each region.

[0178] Set a threshold to determine the complexity of each region, complete the complexity evaluation of the image region, and mark the image region as complex or simple based on the evaluation results;

[0179] Adjust the grid size, reducing it for complex areas and increasing it for simple areas, to reduce computational burden while ensuring that image details are not lost;

[0180] An image quality assessment system is established using quality assessment formulas.

[0181] Establish quality assessment standards, including:

[0182] Evaluate the edge sharpness and detail retention of the image. Complex areas should have higher edge sharpness and detail retention, while simple areas have lower sharpness requirements.

[0183] Check the transition at the seams to ensure that there are no obvious seams or distortions in the splicing area;

[0184] The image shows reasonable color and brightness distribution across different regions, without abrupt changes.

[0185] The system obtains quality assessment information from the feedback results of image stitching, and adjusts the grid size, computing resource allocation, and processing algorithm in real time based on the feedback.

[0186] Specifically, for areas with poor image quality, the system increases computing resources to perform further fine processing on those areas; for areas with good image quality, it reduces computing resources to improve processing speed until a balance is reached.

[0187] like Figure 1 As shown, analyzing flight path data and establishing a predictive model to model future regional data needs includes:

[0188] Real-time flight path analysis is performed using the aircraft's GPS information, combined with image acquisition information, to predict the expected data demand for the image area, as shown in the following formula:

[0189] ;

[0190] in, The expected data requirements are expressed in data volume. The image region is defined by the flight path. This includes the drone's flight data, such as altitude and speed, which is used to adjust the image area as needed.

[0191] Based on flight paths and real-time data demand forecasting, a data demand forecasting model is established to provide a basis for real-time adjustments to regional data demand, as shown in the following formula:

[0192] ;

[0193] in, To meet the future data needs of the region; The function is a prediction model that predicts future data requirements for a region based on historical and real-time flight path data inputs. The model parameter is the model parameter.

[0194] Combining flight path analysis and data demand forecasting models, a bandwidth forecasting formula is designed, assuming a linear correlation between data demand and transmission bandwidth, as shown in the following formula:

[0195] ;

[0196] in, The predicted bandwidth requirement is expressed in data transfer rate. This is a transmission bandwidth function that adjusts bandwidth requirements based on the flight path, data demand, and the real-time status of the aircraft.

[0197] like Figure 1 As shown, based on image data requirements and flight path information, dynamically optimizing data transmission paths and bandwidth allocation includes:

[0198] By utilizing flight path analysis and image acquisition information, combined with a data demand prediction model, the image data demand in different areas is assessed, and a transmission path is selected. The selection of the transmission path is based not only on the current flight position of the aircraft, but also on the future data demand of the image area, so as to dynamically optimize the data transmission route.

[0199] By combining image data requirements with flight paths and real-time feedback from image processing, the data transmission path can be dynamically adjusted.

[0200] Based on real-time monitoring of aircraft status and network load, the data transmission path is adjusted and optimized as follows:

[0201] ;

[0202] in, The optimal transmission path; Candidate paths;

[0203] Based on the network conditions of the transmission path and the priority of image data, bandwidth allocation is optimized to reduce latency during data transmission, as shown in the following formula:

[0204] ;

[0205] in, To allocate the optimal bandwidth; Candidate bandwidth;

[0206] Real-time monitoring of data traffic and bandwidth usage during data transmission, combined with flight path and network conditions to adjust bandwidth and optimize transmission efficiency, as shown in the following formula:

[0207] ;

[0208] in, This refers to the time for real-time data transmission. This is a real-time monitoring function that represents the impact of real-time data traffic monitoring and scheduling.

[0209] Building a virtual environment model based on digital twin technology to compensate for image loss caused by environmental changes includes:

[0210] Construct an environment model by taking the aircraft's real-time position and flight data, environmental data, and image data as input, and outputting the environment model as follows:

[0211] ;

[0212] in, This is an environmental model representing the impact of flight path and environmental conditions on image quality; This is an environment model function, representing the impact of environmental changes encountered by the aircraft on the image;

[0213] Based on environmental models and flight data, repair algorithms are used to repair missing and damaged parts of the image, compensating for image loss caused by environmental changes.

[0214] A detail enhancement algorithm based on digital twin technology is used to enhance the details of the restored image. The intensity of image enhancement is automatically adjusted by analyzing environmental data. The image restoration and detail enhancement are defined by the following formula:

[0215] ;

[0216] in, The enhanced image; Input image; For enhancement functions;

[0217] Data from all sensors is fused, and image inpainting algorithms are used to enhance image details and restore images lost due to lighting or weather conditions. The image enhancement and data fusion process is as follows:

[0218] ;

[0219] in, The final enhanced image; For from the Images from various sensors, These are weighting coefficients, representing the weights of different sensor data.

[0220] like Figure 1 As shown, real-time image quality assessment and real-time adjustment of processing strategies include:

[0221] Using a convolutional neural network model, the quality of each frame of the image is evaluated in real time. Image quality is extracted through multi-scale feature extraction, and the image sharpness, clarity, and noise are comprehensively evaluated.

[0222] During image processing, the image quality is calculated progressively, and the image restoration and enhancement strategies are adjusted based on the evaluation results, as follows:

[0223] ;

[0224] in, Provide real-time image quality scoring; It is a structural similarity index; Peak signal-to-noise ratio;

[0225] Based on the real-time image quality score, different regions in the image are prioritized and sorted according to their quality, with poor quality regions being processed first.

[0226] Based on the current assessment results and computing resource requirements, the allocation of resources is dynamically adjusted, and a parallel computing framework is used to improve processing speed; priority adjustment and resource allocation are represented by formula (#11);

[0227] Real-time evaluation of image quality; adjustment of restoration strategies based on system scores; use of enhancement algorithms to enhance image details; and adjustment of restoration intensity and priority during image processing to ensure that the image is not distorted during enhancement and to avoid over-enhancement as much as possible.

[0228] Based on the image quality assessment results, the enhancement strategy is adjusted by optimizing the algorithm, and the resource utilization in the image restoration process is optimized to ensure the quality and efficiency of the processing results.

[0229] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0230] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for high-definition image generation based on live video from low-altitude drones, characterized in that, include: Image regions are divided and adaptive grid processing is performed, including: Construct a convolutional neural network, set the input layer, and set the input tensor size to be... The image data, in which, The height of the image. The width of the image. The number of channels in the image; We set up a convolutional layer to perform local perception on the input image, extracting local features, including edges and textures. Let the first convolutional layer be the... The convolution operation of a layer is as follows: ； in, For the first Convolutional kernels of layers; This is the input image for the previous layer; For bias terms; The size of the convolution kernel; Pooling layers are used to reduce computation and help extract the maximum features of local features. Max pooling layers retain the most important features by selecting the maximum value in the convolutional image. After convolution and pooling, a fully connected layer is set to synthesize the extracted features, as shown in the following formula: ； in, For the first Layer output; This is the weight matrix; The activation function is ReLU; The image is input into a deep convolutional neural network to extract its features; Combining features at different levels can improve the accuracy of image analysis; By weighting and fusing features at different levels, a comprehensive image complexity metric is obtained. Let the first... The feature map of the layer is The complexity of the image region It can be expressed as the following formula: ； in, This represents the total number of layers in the convolutional neural network. The weights of the features at each layer; For the first Feature map of the layer; Feature intensity at each location: ； in, For the entire image region; For the image at position The first Layer, First The intensity of each feature; The number of features extracted in each layer; It is a Gaussian weighted function; and These are the parameters of the Gaussian function; A parallel computing framework was built, and a stitching algorithm was used to stitch images together, including: Choose the CUDA computing platform, allocate GPU resources, and interact with the GPU through CUDA to accelerate image processing; Each region of the image generated by adaptive meshing is divided into N small mesh blocks, and the size of each mesh block is adjusted according to the complexity and resolution of the image; Each grid block corresponds to a computing unit. The GPU uses CUDA's thread model to assign computing tasks to threads in each grid block, ensuring that computing tasks can be executed in parallel. After each computing unit finishes processing, the GPU summarizes the results and returns them to the host for subsequent image stitching and reconstruction. Image data is first transferred from the host memory to the GPU's device memory, where it is processed, and then returned to the host after processing. After the calculation is completed, the results are transferred from GPU memory back to host memory, and the stitched results are merged to obtain the complete image. The acceleration formula for parallel computation is as follows: ； in, The time required for image processing; This refers to the number of processing units in parallel computing. As the number of parallel computing units increases, the processing time... It will decrease; The process of merging the stitching results includes: using image fusion technology to reduce the seams between different regions; After image stitching is completed, the quality of the stitched image is evaluated in real time. Based on the evaluation results, computing resources will be allocated preferentially to regions with higher complexity. Through a dynamic optimization mechanism, the system adjusts computing resources in real time based on evaluation results to optimize the efficiency and quality of image stitching; The quality assessment formula is: ； in, To assess the quality of the final stitched image, used to evaluate the stitching effect and sharpness; The gradient of the image reflects the sharpness of the edges; The gradient of the stitched image reflects the stitching accuracy. Adjust the image grid size in real time and optimize the image processing algorithm based on real-time feedback data; Analyze flight path data, build predictive models, and model future data needs for the region; Based on image data requirements and flight path information, dynamically optimize data transmission paths and bandwidth allocation; A virtual environment model is built based on digital twin technology to compensate for image loss caused by environmental changes; Perform real-time quality assessment on images and adjust processing strategies in real time.

2. The high-definition image generation method based on low-altitude UAV live video according to claim 1, characterized in that, The real-time adjustment of the image grid size and the optimization of the image processing algorithm based on real-time feedback data include: The Canny edge detection algorithm is used to identify edge regions in the image, and the gray-level co-occurrence matrix is ​​used to identify textured regions in the image. Areas with significant changes in lighting are defined as complex areas, while flat background areas are defined as simple areas. Each small patch in an image divided into multiple small patches is analyzed, and a feature extraction algorithm is used to obtain the complexity index of the region. Set a threshold to determine the complexity of each region, complete the complexity evaluation of the image region, and mark the image region as complex or simple based on the evaluation results; Adjust the grid size: decrease the grid size for complex areas and increase the grid size for simple areas. An image quality assessment system is established using quality assessment formulas. Establish quality assessment standards, including: Evaluate the image's edge sharpness and detail retention; Check the transition effect at the splicing points; Check that the color and brightness distribution in different areas of the image is reasonable; The system obtains quality assessment information from the feedback results of image stitching, and adjusts the grid size, computing resource allocation, and processing algorithm in real time based on the feedback.

3. The high-definition image generation method based on low-altitude UAV live video according to claim 2, characterized in that, The analysis of flight path data and the establishment of predictive models to model future regional data needs include: Real-time flight path analysis is performed using the aircraft's GPS information, combined with image acquisition information, to predict the expected data demand for the image area, as shown in the following formula: ； in, The expected data requirements are expressed in data volume. The entire image region is defined, and its size is determined by the flight path. This includes the drone's flight data, such as altitude and speed information, which is used to adjust the image area. Based on flight paths and real-time data demand forecasting, a data demand forecasting model is established to provide a basis for real-time adjustments to regional data demand, as shown in the following formula: ； in, To meet the future data needs of the region; The function is a prediction model that predicts future data requirements for a region based on historical and real-time flight path data inputs. The model parameter is the model parameter. Combining flight path analysis and data demand forecasting models, a bandwidth forecasting formula is designed, assuming a linear correlation between data demand and transmission bandwidth, as shown in the following formula: ； in, The predicted bandwidth requirement is expressed in data transfer rate. This is a function for transmission bandwidth.

4. The high-definition image generation method based on low-altitude UAV live video according to claim 3, characterized in that, The dynamic optimization of data transmission path and bandwidth allocation based on image data requirements and flight path information includes: By utilizing flight path analysis and image acquisition information, combined with a data demand prediction model, the image data demand in different areas is assessed, and transmission paths are selected. By combining image data requirements with flight paths and real-time feedback from image processing, the data transmission path can be dynamically adjusted. Based on real-time monitoring of aircraft status and network load, the data transmission path is adjusted and optimized as follows: ； in, The optimal transmission path; Candidate paths; Based on the network conditions of the transmission path and the priority of image data, bandwidth allocation is optimized to reduce latency during data transmission, as shown in the following formula: ； in, To allocate the optimal bandwidth; Candidate bandwidth; Real-time monitoring of data traffic and bandwidth usage during data transmission, combined with flight path and network conditions to adjust bandwidth and optimize transmission efficiency, as shown in the following formula: ； in, This refers to the time for real-time data transmission. This is a real-time monitoring function that represents the impact of real-time data traffic monitoring and scheduling.

Images