High-definition mapping method based on low-altitude unmanned aerial vehicle live video

Abstract

The invention discloses a high-definition mapping method based on a low-altitude unmanned aerial vehicle live video, and belongs to the technical field of high-definition mapping of low-altitude unmanned aerial vehicles, and the method comprises the steps: dividing an image region, and carrying out the adaptive grid processing; building a parallel computing framework, and splicing the images by adopting a splicing algorithm; adjusting the size of an image grid in real time, and optimizing an image processing algorithm according to real-time feedback data; analyzing flight path data, establishing a prediction model, and modeling data requirements of a future area; dynamically optimizing a data transmission path and bandwidth allocation based on image data requirements and flight path information; a virtual environment model is constructed based on a digital twinning technology, and image loss caused by environment change is compensated; performing real-time quality evaluation on the image, and adjusting a processing strategy in real time; through an image processing technology combining deep learning and dynamic grid size adjustment, the problems of low image splicing precision, inaccurate quality evaluation and the like in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of high-definition mapping of low-altitude unmanned aerial vehicles, and more specifically, it relates to a high-definition mapping method based on live video of low-altitude unmanned aerial vehicles. Background Art

[0002] With the continuous development of low-altitude unmanned aerial vehicle technology, low-altitude unmanned aerial vehicles have been widely used in various fields, including environmental monitoring, agricultural spraying, disaster relief missions, and aerial photography. Among them, the unmanned aerial vehicle live video technology relies on advanced image sensors, real-time data transmission technology, and image processing algorithms, which can not only obtain high-definition aerial images in real time, but also transmit them to a remote end for processing and viewing within a certain range. Especially when performing large-scale live image stitching and high-precision image synthesis, efficient and accurate processing of the collected images is required to meet the dual requirements of image quality and real-time transmission; However, the existing low-altitude unmanned aerial vehicle live video technology in the market still faces many challenges, especially in image processing and mapping. For example: 1. Although modern unmanned aerial vehicles are equipped with high-resolution cameras and powerful data transmission capabilities, due to the limitations of flight speed, environmental conditions, and transmission bandwidth, there are still great difficulties in real-time stitching and high-definition mapping of images; 2. When stitching images, traditional technologies generally rely on static image processing methods, which greatly reduce the accuracy and stability of image stitching in dynamic scenes and cannot meet the requirements of high-precision and high-quality mapping; 3. In addition, the existing image quality assessment system has not been fully optimized for images taken by unmanned aerial vehicles in high-dynamic and complex environments, and image repair and enhancement technologies often face problems such as noise removal and detail restoration, which affect the quality and application effect of the final image; And the most crucial thing is that most current systems cannot achieve a dynamic balance between image quality and resource utilization. Especially in the case of limited transmission bandwidth, the contradiction between image quality and processing speed is still difficult to effectively solve; Based on this, we propose a high-definition mapping method based on live video of low-altitude unmanned aerial vehicles. Summary of the Invention

[0003] To achieve the above object, the present invention provides the following technical solutions: A high-definition mapping method based on live video of low-altitude unmanned aerial vehicles, including: Dividing the image area and performing adaptive grid processing; Building a parallel computing framework and using a stitching algorithm to stitch images; Real-time adjusting the image grid size and optimizing the image processing algorithm according to real-time feedback data; Analyze flight path data, establish a prediction model, and model the data requirements for future regions; Based on the image data requirements and flight path information, dynamically optimize the data transmission path and bandwidth allocation; Build a virtual environment model based on digital twin technology to compensate for image losses caused by environmental changes; Perform real-time quality assessment on the images and adjust the processing strategy in real time; Further, the dividing the image region and performing adaptive grid processing includes: Build a convolutional neural network, set the input layer, and input image data with a tensor size of , where is the height of the image, is the width of the image, is the number of channels of the image; Set the convolutional layer to perform local perception on the input image and extract local features, including edges and textures. Let the convolutional operation of the th layer be the following formula: ; where is the convolutional kernel of the th layer; is the input image of the previous layer; is the bias term; is the size of the convolutional kernel. Through multiple convolutional operations, the network gradually extracts higher-level semantic information, such as textures and object shapes, from low-level features, such as edges and corners; Set the pooling layer to reduce the computational amount and help extract the maximum characteristics of local features. Among them, the max pooling layer retains the most important features by selecting the maximum value in the image after convolution, as shown in the following formula: ; where is the pixel index within the pooling region; is the result after the pooling operation; After convolution and pooling, set the fully connected layer to synthesize the extracted features. The fully connected layer finally predicts the complexity of the image content by learning a vectorized representation, as shown in the following formula: ; where is the output of the th layer; is the weight matrix; is the activation function, such as Relu; Input the image into a deep convolutional neural network to extract features of the image at multiple levels. Each level analyzes the complexity of the image at different scales. Among them, the primary convolutional layer extracts basic features of the image, such as edges, corners, etc., which are crucial for capturing simple and structured content in the image. In subsequent convolutional layers, the network can learn more complex features, such as the shape and texture of objects, which help the network identify objects and structures in the image; at the high level of the network, the network can extract semantic information of the image, such as the relationship between specific objects and the background, etc., which are closely related to the complexity of the image and can effectively represent the global structure of the image; The network can combine features at different levels to improve the accuracy of image analysis. For example, the combination of low-level edge information, middle-level texture information, and high-level semantic information can analyze the complexity of the image more comprehensively. For complex regions, the network can judge the detailed requirements of the region through the weighted combination of these features; Perform weighted fusion on features at different levels to obtain a comprehensive measure of image complexity. Let the feature map of the th layer be , then the complexity of the image region can be expressed as the following formula: ; where, is the total number of layers of the convolutional neural network; is the weight of each layer of features; is the th layer of feature map; is the feature intensity at each position; thus, calculate the image complexity by integrating features of each layer and adaptively adjust the grid size, as shown in the following formula: ; where, is the entire image region; is the intensity of the th layer and the th feature at position in the image, representing the features extracted from the convolutional neural network; is the number of features extracted in each layer; is the Gaussian weighting function, which controls the calculation accuracy of complex regions of the image; and are the parameters of the Gaussian function, which adjust the influence range of the image region complexity and control the spatial weighting; The value range of is

[0004] , the higher the complexity, the more computing resources are required to process this region.

[0004] Further, building the parallel computing framework and stitching images using the stitching algorithm includes: Building the parallel computing framework includes: Select a CUDA computing platform, allocate GPU resources, interact with the GPU through CUDA to accelerate image processing; Divide each region of the image generated by adaptive meshing into N small grid blocks, and adjust the size of each grid block according to the complexity and resolution of the image; Each grid block corresponds to a computing unit, and the GPU distributes computing tasks to the threads of each grid block through the thread model of CUDA to ensure that the computing tasks can be executed in parallel; After each computing unit finishes processing, the GPU aggregates the results and returns them to the host for subsequent image stitching and reconstruction; Image data is first transferred from the host memory to the device memory of the GPU, processed in the device memory, and then returned to the host after processing. Among them, each thread performs specific computing operations, such as image reconstruction, region stitching, etc.; After the calculation is completed, the results are transferred back from the GPU memory to the host memory, and the stitching results are merged to finally obtain a complete image; The acceleration formula for parallel execution of calculations is: ; Among them, is the time required for image processing; is the number of processing units for parallel computing. As the number of parallel computing units increases, the processing time will decrease; Among them, merging the stitching results includes: adopting image fusion technology to reduce the seams between different regions, and seamlessly stitching the images of different regions based on formula (#5); After completing image stitching, the quality of the stitched image is evaluated in real time. The evaluation criteria include edge sharpness, stitching accuracy, etc.; Based on the evaluation results, for regions with higher complexity, computing resources will be preferentially allocated to ensure that these regions can obtain more computing resources and ensure the stitching accuracy and image quality; Through the dynamic optimization mechanism, the system adjusts computing resources in real time according to the evaluation results, optimizes the efficiency and quality of image stitching, and ensures that the stitched image meets the requirements of high resolution and high quality; The quality evaluation formula is: ; Among them, is the quality of the final stitched image, used to evaluate the stitching effect and sharpness; is the gradient of the image, reflecting the sharpness of the edge; is the gradient of the spliced image, reflecting the splicing accuracy.

[0005] Furthermore, the real-time adjustment of the image grid size and the optimization of the image processing algorithm according to the real-time feedback data include: Using the Canny edge detection algorithm to identify the edge regions in the image and using the gray-level co-occurrence matrix to identify the texture complex regions in the image; Defining the regions with obvious illumination changes as complex regions and the flat background regions as simple regions; Analyzing each small block of the image divided into multiple small blocks, and using the feature extraction algorithm to obtain the complexity index of the region. The complexity index includes calculating the texture complexity, edge density, and standard deviation of brightness change within each region; Setting a threshold to judge the complexity of each region, completing the complexity evaluation of the image region, and marking the image region as complex and simple according to the evaluation result; Adjusting the grid size, reducing the grid size for complex regions and increasing the grid size for simple regions, ensuring that the image details will not be lost while reducing the calculation burden; Using the quality evaluation formula to establish an image quality evaluation system; Setting quality evaluation criteria, including: Evaluating the edge sharpness and detail retention of the image. Among them, complex regions should have higher edge sharpness and detail retention, while the clarity requirement for simple regions is lower; Checking the transition effect at the splicing location to ensure that there are no obvious seams or distortions in the splicing region; Checking that the color and brightness distribution in different regions of the image is reasonable without sudden changes; Obtaining quality evaluation information from the feedback results of image splicing, and according to the feedback, the system adjusts the grid size, computing resource allocation, and processing algorithm in real time; Specifically, for the regions with poor image quality, the system will increase the computing resources and perform further fine processing on these regions; for the regions with good image quality, the processing speed will be increased by reducing the computing resources until balance is achieved.

[0006] Furthermore, the analysis of the flight path data and the establishment of a prediction model to model the future regional data requirements include: Using the GPS information of the aircraft to perform real-time flight path analysis, combining with the acquisition information of image acquisition, and predicting the expected data requirements of the image region, as shown in the following formula: ; Wherein, is the expected data requirement, with the unit of data volume; is the image area, and the size of the image area is determined by the flight path; is the flight data of the UAV, including information such as flight altitude and speed, which is used to adjust the requirements of the image area; Based on the flight path and real-time data demand prediction, a data demand prediction model is established to provide a basis for the real-time adjustment of the subsequent regional data demand, as shown in the following formula: ; where, is the data demand of the future area; is the function of the prediction model, which predicts the data demand of the future area based on the historical data and real-time data input of the flight path, and model is the model parameter; Combined with the flight path analysis and the data demand prediction model, a bandwidth prediction formula is designed. It is assumed that the data demand and the transmission bandwidth are linearly related, as shown in the following formula: ; where, is the predicted bandwidth demand, with the unit of data transmission rate; is the transmission bandwidth function, which adjusts the bandwidth demand according to the flight path, data demand and the real-time condition of the aircraft.

[0007] Furthermore, the dynamic optimization of the data transmission path and bandwidth allocation based on the image data demand and flight path information includes: Using the flight path analysis and image acquisition information, combined with the data demand prediction model, evaluate the image data demand of different regions and select the transmission path; The selection of the transmission path is not only based on the current flight position of the aircraft, but also takes into account the data demand of the future image area, so as to dynamically optimize the data transmission route; Use the combined analysis of the image data demand and the flight path, combined with the real-time feedback results of image processing, to dynamically adjust the data transmission path; Combined with the real-time monitored aircraft status and network load situation, adjust the data transmission path and optimize the transmission path, as shown in the following formula: ; where, is the optimal transmission path; is the candidate path; According to the network condition of the transmission path and the priority of the image data, reduce the delay generated during the data transmission process and optimize the bandwidth allocation, as shown in the following formula: ; where, is the optimal bandwidth allocation; is the candidate bandwidth; Monitor the data traffic and bandwidth occupancy during real-time data transmission, adjust the bandwidth in combination with the flight path and network conditions, and optimize the transmission efficiency, as shown in the following formula: ; wherein, is the time of real-time data transmission; is the real-time monitoring function, representing the influence of real-time data traffic monitoring and scheduling.

[0008] Furthermore, constructing a virtual environment model based on digital twin technology to compensate for image loss caused by environmental changes includes: Construct an environment model, input the real-time position and flight data of the aircraft, environmental data and image data, and output the environment model, as shown in the following formula: ; wherein, is the environment model, representing the influence of the flight path and environmental conditions on the image quality; is the environment model function, representing the influence of environmental changes encountered by the aircraft on the image; According to the environment model and flight data, repair the missing and damaged parts of the image through a repair algorithm to compensate for image loss caused by environmental changes; Adopt a detail enhancement algorithm based on digital twin technology to enhance the details of the repaired image, and automatically adjust the intensity of image enhancement by analyzing environmental data. Among them, image repair and detail enhancement are as shown in the following formula: ; wherein, is the enhanced image; is the input image; is the enhancement function; Fuse the data of all sensors, use the image repair algorithm to enhance the details of the image, and restore the loss caused by light irradiation or weather factors. The process of image enhancement and data fusion is as shown in the following formula: ; wherein, is the finally enhanced image; is the image from the th sensor, is the weighting coefficient, representing the weight of different sensor data.

[0009] Furthermore, performing real-time quality assessment on the image and adjusting the processing strategy in real time includes: Using a convolutional neural network model to perform real-time quality assessment on each frame of the image, extract the image quality through multi-scale features, and comprehensively evaluate the clarity, sharpness, and noise of the image; During the image processing process, the quality of the image is gradually calculated, and the image restoration and enhancement strategies are adjusted according to the evaluation results, as shown in the following formula: ; Among them, is the real-time image quality score; is the structural similarity index; is the peak signal-to-noise ratio; Based on the real-time image quality score results, different regions in the image are sorted by priority, sorted by the quality from good to bad, and the regions with poor quality are processed first; Based on the current evaluation results and computational resource requirements, the resource allocation is dynamically adjusted, and a parallel computing framework is used to improve the processing speed; the priority adjustment and resource allocation are represented by formula (#11); For the real-time evaluation of image quality, the restoration strategy is adjusted according to the system score, an enhancement algorithm is used to enhance the details of the image, and the restoration intensity and priority are adjusted during the image processing process to ensure that the image is not distorted during the enhancement process and over-enhancement is avoided as much as possible; Based on the image quality evaluation results, the enhancement strategy is adjusted through an optimization algorithm to optimize the resource utilization in the image restoration process and ensure the quality and efficiency of the processing results.

[0010] In summary, the present invention has the following beneficial effects: By combining deep learning with the image processing technology of dynamically adjusting the grid size, problems such as low image stitching accuracy and inaccurate quality evaluation in the prior art are solved; Through real-time image quality evaluation and dynamic priority adjustment, this solution can optimize real-time data transmission and processing while ensuring image quality; By combining the image quality dynamic adjustment technology, this solution can ensure high-quality output of images in real-time video transmission, and at the same time reasonably allocate computing resources, avoiding problems such as image quality degradation and processing delay caused by resource bottlenecks in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0012] Figure 1 is the flowchart of the high-definition mapping method for live video based on low-altitude UAVs of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0014] Embodiment: The following will further elaborate on the present invention in conjunction with the attached Figure 1 drawings.

[0015] Please refer to Figure 1 , the present invention provides a technical solution: a high-definition mapping method based on live video of low-altitude drones, as Figure 1 shown, including: Dividing the image area and performing adaptive grid processing; Building a parallel computing framework and using a stitching algorithm to stitch images; Real-time adjusting the image grid size, and optimizing the image processing algorithm according to real-time feedback data; Analyzing the flight path data, establishing a prediction model, and modeling the data requirements for future areas; Based on the image data requirements and flight path information, dynamically optimizing the data transmission path and bandwidth allocation; Building a virtual environment model based on digital twin technology to compensate for image loss caused by environmental changes; Performing real-time quality assessment on the image and real-time adjusting the processing strategy; As Figure 1 shown, dividing the image area and performing adaptive grid processing includes: Constructing a convolutional neural network, setting an input layer, and inputting image data with a tensor size of , where is the height of the image, is the width of the image, is the number of channels of the image; Setting a convolutional layer to perform local perception on the input image and extract local features, including edges and textures. Let the convolutional operation of the th layer be the following formula: ; where is the convolutional kernel of the th layer; is the input image of the previous layer; is the bias term; is the size of the convolutional kernel; through multiple convolutional operations, the network gradually extracts higher-level semantic information from low-level features such as edges and corners, such as textures and object shapes; Set the pooling layer to reduce the computational amount and help extract the maximum characteristics of local features. Among them, the max pooling layer retains the most important features by selecting the maximum value in the image after convolution, as shown in the following formula: ; Among them, is the pixel index within the pooling area; is the result after the pooling operation; After convolution and pooling, set the fully connected layer to synthesize the extracted features. The fully connected layer finally predicts the complexity of the image content by learning a vectorized representation, as shown in the following formula: ; Among them, is the output of the th layer; is the weight matrix; is the activation function, such as Relu; Input the image into a deep convolutional neural network to extract the features of the image from multiple levels. Each layer analyzes the complexity of the image at different scales. Among them, the primary convolutional layer extracts the basic features of the image, such as edges and corners, through filters. These features are crucial for capturing simple and structured content in the image. In subsequent convolutional layers, the network can learn more complex features, such as the shape and texture of objects. These features help the network identify objects and structures in the image; at the high level of the network, the network can extract the semantic information of the image, such as the relationship between specific objects and the background. These features are closely related to the complexity of the image and can effectively represent the global structure of the image; The network can combine features at different levels to improve the accuracy of image analysis. For example, the combination of low-level edge information, middle-level texture information, and high-level semantic information can analyze the complexity of the image more comprehensively. For complex regions, the network can judge the detailed requirements of the region through the weighted combination of these features; Perform weighted fusion on features at different levels to obtain a comprehensive measure of image complexity. Let the feature map of the th layer be , then the complexity of the image region can be expressed as the following formula: ; Among them, is the total number of layers of the convolutional neural network; is the weight of each layer of features; is the th layer of feature map; is the feature intensity at each position; thus, the image complexity is calculated by integrating the features of each layer, and the grid size is adaptively adjusted as follows: ; where, is the entire image area; is the intensity of the th layer and the th feature at position in the image, representing the features extracted from the convolutional neural network; is the number of features extracted in each layer; and are the parameters of the Gaussian weighting function, which control the calculation accuracy of the complex regions of the image; The value range of is

[0016] In this embodiment: An experiment was designed to verify the advantages of the above embodiment over the prior art in terms of image stitching quality, data transmission efficiency, and image restoration effect. A low-altitude drone was selected as the data acquisition device for the experiment, and data acquisition and image stitching were carried out using this drone in an urban environment. Two sets of control data were set up. The first set used the existing traditional image stitching and quality assessment algorithms, and the second set used the method proposed in this embodiment for image stitching, quality assessment, and data transmission optimization. The specific steps are as follows: A low-altitude drone, model M30, equipped with a high-resolution camera with a resolution of 4096x2160, was used to collect data at different flight paths and in different environments. The average altitude of each flight did not exceed 180 meters, the maximum flight speed was 15 m / s, the drone collected 100 frames of images per minute, and the image size was within 2 MB; The adaptive network processing method in this embodiment was used to perform regional division and grid processing on the collected images. The images were divided into multiple small regions, and each region was processed as a unit to optimize the processing speed and improve the image stitching efficiency. The size of the regional division was 960*720 pixels; The adaptive network processing method in the above embodiment was used to perform regional division and grid processing on the collected images. The images were divided into multiple small regions, and each region was processed as a unit to optimize the processing speed and improve the image stitching efficiency. The size of the regional division was 64x64 pixels; When performing image stitching, traditional methods directly stitch all images and use low-level features for rough matching. In this embodiment, a deep learning model, a convolutional neural network (CNN), is used to analyze the image content, extract high-level feature information of each region, and perform more accurate stitching, avoiding the distortion and seam problems in traditional stitching methods. The image quality assessment in this embodiment uses a method based on SSIM and PSNR, and combines a deep learning model to score the image in multiple dimensions. After the image stitching is completed, real-time quality assessment is performed on the result. A dynamic bandwidth adjustment algorithm based on flight path analysis and flight real-time data is used to optimize data transmission. By analyzing the image quality requirements and flight path information, the bandwidth of data transmission is adjusted to ensure the balance between image quality and transmission speed. In terms of image restoration, this embodiment adopts an image enhancement technology based on deep learning, combines data from different sensors for restoration and detail enhancement, effectively compensates for the losses caused by environmental factors such as weather and light changes in the image, and improves the quality of the final image. Experimental comparison of image stitching and data transmission optimization based on low-altitude UAV live video (#Table 1) ; Among them, through the data in the above table, it can be found that in terms of image quality, the deep learning model in this embodiment can more accurately extract image features and stitch, effectively improving the clarity and detail performance of the image; secondly, in terms of processing speed and data transmission efficiency, the content of this embodiment improves the overall performance of the system, reduces the processing time and bandwidth pressure through dynamic bandwidth optimization and image restoration enhancement technology; finally, the method in the content of this embodiment not only makes a breakthrough in image quality, but also shows obvious advantages in real-time data processing and bandwidth utilization.

[0017] As Figure 1 shown, building a parallel computing framework and using a stitching algorithm to stitch images includes: Building a parallel computing framework includes: Select a CUDA computing platform, allocate GPU resources, interact with the GPU through CUDA to accelerate image processing; Divide each region of the image generated by adaptive grid processing into N small grid blocks, and the size of each grid block is adjusted according to the complexity and resolution of the image; Each grid block corresponds to a computing unit, and the GPU distributes computing tasks to the threads of each grid block through the thread model of CUDA to ensure that the computing tasks can be executed in parallel; After each computing unit finishes processing, the GPU aggregates the results and returns them to the host for subsequent image stitching and reconstruction; The image data is first transferred from the host memory to the device memory of the GPU, processed in the device memory, and then returned to the host after processing. Among them, each thread performs specific calculation operations, such as image reconstruction, region stitching, etc.; After the calculation is completed, the result is transferred back from the GPU memory to the host memory, and the stitched results are merged to finally obtain a complete image; The acceleration formula for parallel execution of calculations is: ; Among them, is the time required for image processing; is the number of processing units for parallel computing. As the number of parallel computing units increases, the processing time will decrease; Among them, merging the stitched results includes: adopting image fusion technology to reduce the seams between different regions, and seamlessly stitching the images of different regions based on formula (#5); After completing the image stitching, the quality of the stitched image is evaluated in real time. The evaluation criteria include edge sharpness, stitching accuracy, etc.; Based on the evaluation results, for regions with higher complexity, computing resources will be preferentially allocated to ensure that these regions can obtain more computing resources and ensure the stitching accuracy and image quality; Through the dynamic optimization mechanism, the system adjusts the computing resources in real time according to the evaluation results, optimizes the efficiency and quality of image stitching, and ensures that the stitched image meets the requirements of high resolution and high quality; The quality evaluation formula is: ; Among them, is the quality of the final stitched image, used to evaluate the stitching effect and sharpness; is the gradient of the image, reflecting the sharpness of the edge; is the gradient of the stitched image, reflecting the stitching accuracy; In this embodiment, by building a parallel computing framework and using CUDA and GPU resources for image processing, it is possible to accelerate the computing tasks of each image grid, achieve the efficient completion of image stitching, and then through real-time adjustment of the image region and optimization of the feedback mechanism, ensure the improvement of image stitching quality, effectively optimize the allocation of computing tasks, reduce the computing burden. Moreover, in the above embodiment, by establishing a data demand prediction model, it is possible to provide a real-time adjustment plan for the data demand of subsequent regions, thereby optimizing the transmission route of the data stream and the use of bandwidth, ensuring that the data transmission during the image acquisition process is not restricted, and improving the efficiency of data transmission.

[0018] Such as Figure 1As shown, the image grid size is adjusted in real time. According to the real-time feedback data, the optimization of the image processing algorithm includes: Use the Canny edge detection algorithm to identify the edge regions in the image and use the gray-level co-occurrence matrix to identify the texture complex regions in the image; Define the regions with obvious illumination changes as complex regions and the flat background regions as simple regions; Analyze each small block in the image divided into multiple small blocks, and use the feature extraction algorithm to obtain the complexity index of the region. The complexity index includes calculating the texture complexity, edge density and standard deviation of brightness change within each region; Set a threshold to judge the complexity of each region, complete the complexity evaluation of the image region, and mark the image region as complex and simple according to the evaluation results; Adjust the grid size, reduce the grid size for complex regions and increase the grid size for simple regions, while reducing the calculation burden and ensuring that the image details are not lost; Use the quality evaluation formula to establish an image quality evaluation system; Set quality evaluation criteria, including: Evaluate the edge sharpness and detail retention of the image. Among them, complex regions should have higher edge sharpness and detail retention, while the clarity requirements for simple regions are lower; Check the transition effect at the splicing location to ensure that there are no obvious seams or distortions in the splicing region; Check that the color and brightness distributions in different regions of the image are reasonable and there are no sudden changes; Obtain quality evaluation information from the feedback results of image splicing. According to the feedback, the system adjusts the grid size, computing resource allocation and processing algorithm in real time; Specifically, for regions with poor image quality, the system will increase computing resources and perform further fine processing on this region; for regions with good image quality, the processing speed will be increased by reducing computing resources until it reaches balance; Such as Figure 1 As shown, analyze the flight path data, establish a prediction model, and model the data requirements for future regions, including: Use the GPS information of the aircraft to perform real-time flight path analysis, combine with the acquisition information of image acquisition, and predict the expected data requirements of the image region, as shown in the following formula: ; Among them, is the expected data requirement, with the unit of data volume; is the image region, and the size of the image region is determined by the flight path; The flight data of the drone, including information such as flight altitude and speed, is used to adjust the requirements of the image area; Based on the flight path and real-time data requirements prediction, a data requirements prediction model is established to provide a basis for the real-time adjustment of subsequent regional data requirements, as shown in the following formula: ; Among them, is the data requirement of the future area; is the function of the prediction model, which predicts the data requirements of the future area based on the historical data and real-time data input of the flight path, and model is the model parameter; Combined with the flight path analysis and the data requirements prediction model, a bandwidth prediction formula is designed. Assuming that the data demand and the transmission bandwidth are linearly related, as shown in the following formula: ; Among them, is the predicted bandwidth requirement, with the unit of data transmission rate; is the transmission bandwidth function, which adjusts the bandwidth requirement according to the flight path, data demand and real-time status of the aircraft; As Figure 1 shown, based on the image data requirements and flight path information, the dynamic optimization of the data transmission path and bandwidth allocation includes: Using flight path analysis and image acquisition information, combined with the data requirements prediction model, evaluate the image data requirements of different regions and select the transmission path; The selection of the transmission path is not only based on the current flight position of the aircraft, but also takes into account the data demand of the future image area in order to dynamically optimize the data transmission route; Use the combined analysis of image data requirements and flight path, combined with the real-time feedback results of image processing, to dynamically adjust the data transmission path; Combined with the real-time monitored aircraft status and network load conditions, adjust the data transmission path and optimize the transmission path, as shown in the following formula: ; Among them, is the optimal transmission path; is the candidate path; According to the network status of the transmission path and the priority of image data, reduce the delay generated during data transmission and optimize the bandwidth allocation, as shown in the following formula: ; Among them, is the optimal bandwidth allocation; is the candidate bandwidth; Real-time monitor the data traffic and bandwidth occupancy during data transmission, and adjust the bandwidth in combination with the flight path and network conditions to optimize the transmission efficiency, as shown in the following formula: ; Among them, is the time of real-time data transmission; is the real-time monitoring function, indicating the impact of real-time data traffic monitoring and scheduling; Construct a virtual environment model based on digital twin technology to compensate for image loss caused by environmental changes, including: Construct an environment model, input the real-time position and flight data, environmental data, and image data of the aircraft, and output the environment model as follows: ; Among them, is the environment model, indicating the impact of flight path and environmental conditions on image quality; is the environment model function, indicating the impact of environmental changes encountered by the aircraft on the image; According to the environment model and flight data, repair the missing and damaged parts of the image through a repair algorithm to compensate for image loss caused by environmental changes; Adopt a detail enhancement algorithm based on digital twin technology to enhance the details of the repaired image, and automatically adjust the intensity of image enhancement by analyzing environmental data. Among them, image repair and detail enhancement are as follows: ; Among them, is the enhanced image; is the input image; is the enhancement function; Fuse the data of all sensors, use the image repair algorithm to enhance the details of the image, and restore the loss caused by light irradiation or weather factors. The process of image enhancement and data fusion is as follows: ; Among them, is the finally enhanced image; is the image from the th sensor, is the weighting coefficient, indicating the weight of different sensor data; As Figure 1 shown, perform real-time quality assessment on the image and adjust the processing strategy in real time, including: Use a convolutional neural network model to perform real-time quality assessment on each frame of the image, extract image quality through multi-scale features, and comprehensively evaluate the clarity, sharpness, and noise of the image; During the image processing process, gradually calculate the quality of the image and adjust the image repair and enhancement strategies according to the evaluation results, as follows: ; Among them, is the real-time image quality score; is the structural similarity index; is the peak signal-to-noise ratio; Based on the real-time image quality score results, prioritize different regions in the image, prioritize according to the quality from good to bad, and process the regions with poor quality first; Based on the current evaluation results and computational resource requirements, dynamically adjust the resource allocation, and use a parallel computing framework to improve the processing speed; the priority adjustment and resource allocation are represented by formula (#11); For the real-time evaluation of image quality, adjust the restoration strategy according to the system score, use an enhancement algorithm to enhance the details of the image, and adjust the restoration intensity and priority during the image processing to ensure that the image is not distorted during the enhancement process and avoid over-enhancement as much as possible; Based on the image quality evaluation results, adjust the enhancement strategy through an optimization algorithm to optimize the resource utilization during the image restoration process and ensure the quality and efficiency of the processing results.

[0019] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0020] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed.

Claims

1. A high-definition mapping method based on low-altitude UAV live video, characterized in that: include: Divide the image area and perform adaptive grid processing; Build a parallel computing framework and use stitching algorithms to stitch images; Adjust the image grid size in real time and optimize the image processing algorithm based on real-time feedback data; Analyze flight path data and build predictive models to model future regional data needs; Dynamically optimize data transmission paths and bandwidth allocation based on image data requirements and flight path information; Build a virtual environment model based on digital twin technology to compensate for image loss caused by environmental changes; Perform real-time quality assessment of images and adjust processing strategies in real time.

2. The high-definition mapping method based on low-altitude UAV live video according to claim 1 is characterized in that: The dividing of the image area and performing adaptive grid processing comprises: Construct a convolutional neural network, set the input layer, and the input tensor size is image data, where is the height of the image, is the width of the image, is the number of channels of the image; Set up a convolutional layer to perform local perception on the input image and extract local features, including edges and textures. The convolution operation of the layer is as follows: ； in, For the The convolution kernel of the layer; is the input image of the previous layer; is the bias term; is the size of the convolution kernel; Setting the pooling layer reduces the amount of computation and helps extract the maximum characteristics of local features. The maximum pooling layer retains the most important features by selecting the maximum value in the convolution image, as shown in the following formula: ； in, is the pixel index in the pooling area; is the result after the pooling operation; After convolution and pooling, a fully connected layer is set to synthesize the extracted features as follows: ； in, For the The output of the layer; is the weight matrix; is an activation function, such as Relu; Input the image into a deep convolutional neural network to extract the features of the image; Combining features at different levels to improve the accuracy of image analysis; The features at different levels are weighted and fused to obtain a comprehensive image complexity measure. The feature map of the layer is , then the complexity of the image region It is expressed as the following formula: ； in, is the total number of layers of the convolutional neural network; is the weight of each layer feature; For the Feature map of the layer; For each position the characteristic intensity is: ； in, for the entire image area; For images at position The Layer, The strength of a feature; is the number of features extracted in each layer; is the Gaussian weighting function; and are the parameters of the Gaussian function.

3. The high-definition mapping method based on low-altitude UAV live video according to claim 2 is characterized in that: The building of a parallel computing framework and using a stitching algorithm to stitch images includes: Building a parallel computing framework includes: Select the CUDA computing platform, allocate GPU resources, and interact with the GPU through CUDA to accelerate image processing; Each area of ​​the image generated by the adaptive gridding process is divided into N small grid blocks, and the size of each grid block is adjusted according to the complexity and resolution of the image; Each grid block corresponds to a computing unit. The GPU assigns computing tasks to threads of each grid block through the CUDA thread model to ensure that computing tasks can be performed in parallel. After each computing unit has finished processing, the GPU summarizes the results and returns them to the host for subsequent image stitching and reconstruction; The image data is first transferred from the host memory to the GPU's device memory, processed in the device memory, and then returned to the host after processing; After the calculation is completed, the result is transferred from the GPU memory back to the host memory, and the stitching results are merged to finally get a complete image; The acceleration formula for parallel execution calculation is: ； in, The time required for image processing; is the number of processing units for parallel computing. As the number of parallel computing units increases, the processing time will decrease; The merging of the stitching results includes: using image fusion technology to reduce the seams between different areas; After image stitching is completed, the quality of the stitched image is evaluated in real time; Based on the evaluation results, computing resources will be allocated preferentially to areas with higher complexity; Through the dynamic optimization mechanism, the system adjusts computing resources in real time according to the evaluation results to optimize the efficiency and quality of image stitching; The quality assessment formula is: ； in, The quality of the final stitched image is used to evaluate the stitching effect and clarity; is the gradient of the image, reflecting the clarity of the edge; is the gradient of the stitched image, reflecting the stitching accuracy.

4. The high-definition mapping method based on low-altitude UAV live video according to claim 3 is characterized in that: The real-time adjustment of the image grid size and the optimization of the image processing algorithm according to the real-time feedback data include: Use the Canny edge detection algorithm to identify edge regions in the image and use the gray-level co-occurrence matrix to identify regions with complex textures in the image; Areas with obvious lighting changes are defined as complex areas, and areas with flat backgrounds are defined as simple areas; Each small block in the image divided into multiple small blocks is analyzed, and the complexity index of the area is obtained using a feature extraction algorithm; Set a threshold to judge the complexity of each area, complete the complexity evaluation of the image area, and mark the image area as complex or simple according to the evaluation results; Adjust the mesh size, reduce the mesh size for complex areas, and increase the mesh size for simple areas; Use the quality assessment formula to establish an image quality assessment system; Set quality assessment criteria, including: Evaluate the image's edge sharpness and detail preservation; Check the transition effect at the splicing point; Check that the color and brightness distribution in different areas of the image is reasonable; The system obtains quality assessment information from the feedback results of image stitching, and adjusts the grid size, computing resource allocation, and processing algorithm in real time based on the feedback.

5. The high-definition mapping method based on low-altitude UAV live video according to claim 4 is characterized in that: The analysis of flight path data, establishment of a prediction model, and modeling of future regional data requirements includes: The real-time flight path analysis is performed using the GPS information of the aircraft, combined with the collected information of the image acquisition, to predict the expected data demand of the image area, as shown in the following formula: ； in, is the expected data demand, in data volume; is the image area, the size of which is determined by the flight path; The flight data of the drone includes information such as flight altitude and speed, which is used to adjust the image area requirements; Based on the flight path and real-time data demand forecast, a data demand forecast model is established to provide a basis for the real-time adjustment of subsequent regional data demand, as shown in the following formula: ； in, For future regional data needs; It is the function of the prediction model, which predicts the data demand of the future area based on the historical data and real-time data input of the flight path, and model is the model parameter; Combining the flight path analysis and data demand prediction model, a bandwidth prediction formula is designed, assuming that the data demand and transmission bandwidth are linearly related, as shown in the following formula: ； in, is the predicted bandwidth requirement, expressed in data transfer rate; is a function of transmission bandwidth.

6. The high-definition mapping method based on low-altitude UAV live video according to claim 5 is characterized in that: The dynamically optimizing the data transmission path and bandwidth allocation based on the image data demand and the flight path information includes: Using flight path analysis and image acquisition information, combined with data demand prediction models, the image data needs of different areas are evaluated and the transmission path is selected; Use the combined analysis of image data requirements and flight paths, combined with real-time feedback from image processing, to dynamically adjust the data transmission path; Combined with the real-time monitored aircraft status and network load, the data transmission path is adjusted and optimized as follows: ； in, is the optimal transmission path; is the candidate path; According to the network status of the transmission path and the priority of image data, the delay in data transmission is reduced and the bandwidth allocation is optimized as follows: ； in, for optimal bandwidth allocation; is the candidate bandwidth; Monitor the data flow and bandwidth usage during data transmission in real time, adjust the bandwidth based on the flight path and network conditions, and optimize the transmission efficiency, as shown below: ； in, The time for real-time data transmission; is a real-time monitoring function, which represents the impact of real-time data traffic monitoring and scheduling.

7. The high-definition mapping method based on low-altitude UAV live video according to claim 6 is characterized in that: The virtual environment model is constructed based on the digital twin technology to compensate for the image loss caused by the environmental change, including: Construct an environmental model, input the real-time position and flight data of the aircraft, environmental data and image data, and output the environmental model as follows: ； in, For the environmental model; is the environmental model function; Based on the environmental model and flight data, the missing and damaged parts of the image are repaired through the repair algorithm to compensate for the image loss caused by environmental changes; The detail enhancement algorithm based on digital twin technology is used to enhance the details of the repaired image, and the intensity of image enhancement is automatically adjusted by analyzing the environmental data. The image repair and detail enhancement are as follows: ； in, is the enhanced image; is the input image; is the enhancement function; The data from all sensors are fused, and the image details are enhanced using the image restoration algorithm to restore the loss caused by lighting or weather factors. The process of image enhancement and data fusion is as follows: ； in, is the final enhanced image; For the The image of the sensor, is the weighting coefficient.

8. The high-definition mapping method based on low-altitude UAV live video according to claim 6 is characterized in that: The real-time quality assessment of the image and the real-time adjustment of the processing strategy include: Using the convolutional neural network model, the quality of each frame of the image is evaluated in real time, and the image quality is extracted through multi-scale features to comprehensively evaluate the clarity, sharpness, and noise of the image; During the image processing process, the image quality is calculated step by step, and the image restoration and enhancement strategies are adjusted according to the evaluation results, as shown in the following formula: ； in, Score image quality in real time; is the structural similarity index; is the peak signal-to-noise ratio; Based on the real-time image quality scoring results, different areas in the image are prioritized by quality, and areas with poor quality are processed first; Dynamically adjust resource allocation based on current evaluation results and computing resource requirements, and use a parallel computing framework to increase processing speed; Real-time evaluation of image quality, adjusting the restoration strategy according to the system score, using enhancement algorithms to enhance image details, and adjusting the restoration intensity and priority during image processing; Based on the image quality assessment results, the enhancement strategy is adjusted through the optimization algorithm to optimize the resource utilization in the image restoration process.

Images