Panoramic image processing method and device and panoramic unmanned aerial vehicle

Abstract

The application provides a panoramic image processing method and device suitable for a UAV and a panoramic UAV. The method collects multi-view images in real time through a multi-camera array of the UAV, obtains pose data by using an inertial sensor, converts each view image into a first projection image and a second projection image, and performs dynamic target detection and identification on the dynamic target, geometric scene features and target distribution of the second projection image, and divides a differential processing area. A high-precision algorithm is used for a first processing area, and a low-precision simplified algorithm is used for a second processing area, so as to realize power focusing and processing optimization. In the splicing stage, the splicing algorithm is optimized in combination with a dynamic target historical track, and different areas are adaptively processed, so that the quality of a key area is ensured, and the process of a non-key area is simplified to reduce the calculation load. The scheme effectively solves the real-time panoramic splicing problem under the condition that the hardware resources of the UAV are limited, and significantly improves the processing real-time performance and efficiency.

Description

Technical Field

[0001] This application relates to the field of panoramic image processing technology, and in particular to a panoramic image processing method, apparatus, and panoramic drone. Background Technology

[0002] Due to their maneuverability and flexibility, drones are increasingly used in various fields such as geographical surveying, leading to a growing demand for panoramic image capture. However, panoramic image processing algorithms are complex, often involving multiple stages such as image stitching, fusion, and correction, requiring the analysis and processing of large amounts of image data. Drones, limited by hardware constraints in size and power consumption, typically rely on embedded processors for computation, making it difficult to complete complex panoramic image processing tasks quickly. Furthermore, drones are susceptible to factors such as airflow during flight, resulting in instability in image acquisition and further increasing the difficulty of panoramic image processing. Currently, drone-based panoramic image processing suffers from high processing difficulty and poor real-time performance. Summary of the Invention

[0003] In view of the above problems, in order to improve the real-time performance of UAV-based panoramic image processing and reduce the difficulty of UAV processing panoramic images, this application provides a panoramic image processing method, apparatus and panoramic UAV.

[0004] The embodiments of this application disclose the following technical solutions:

[0005] In a first aspect, embodiments of this application provide a panoramic image processing method applied to a drone, the drone including a multi-camera array and multiple inertial sensors, the method comprising:

[0006] The multi-camera array is used to acquire target environment images of the UAV from multiple perspectives in real time, and the pose data of the UAV is acquired through multiple inertial sensors.

[0007] Based on the pose data, dynamic degree analysis and projection transformation are performed on each of the target environment images to generate a first projected coordinate image associated with a low dynamic requirement image and a second projected coordinate image associated with a high dynamic requirement image; the low dynamic requirement image and the high dynamic requirement image are images from multiple frames of the target environment images;

[0008] Dynamic target detection is performed on the second projected coordinate image to determine dynamic target objects within the second projected coordinate image; the dynamic target object is a moving object whose pixel displacement exceeds a preset first threshold.

[0009] Based on the scene features of the second projected coordinate image and the distribution location of the dynamic target object, the region is divided to determine the first processing region and the second processing region within the second projected coordinate image; the processing precision of the first processing region is greater than that of the second processing region.

[0010] Based on the preset processing algorithms associated with the first and second processing regions, and the historical motion trajectory of the dynamic target object in the second projected coordinate image, the second projected coordinate image and the first projected coordinate image are stitched together to generate a panoramic image.

[0011] In one possible implementation, the step of stitching the second projected coordinate image with the first projected coordinate image to generate a panoramic image, based on preset processing algorithms associated with the first and second processing regions and the historical motion trajectory of the dynamic target object in the second projected coordinate image, includes:

[0012] The first processing region in the second projected coordinate image is stitched with high precision using a real-time dynamic detection algorithm and a feature matching algorithm, and the second processing region in the second projected coordinate image is stitched with low precision using a timed sampling detection algorithm and a stitching algorithm to generate a first region stitched image; the first region stitched image includes a third processing region and a fourth processing region, wherein the third processing region is generated by stitching together multiple first processing regions, and the fourth processing region is generated by stitching together multiple second processing regions;

[0013] Based on the historical motion trajectory of the dynamic target object, the motion trajectory of the dynamic target object in a preset future time period is predicted by the Kalman filter algorithm to generate a motion trajectory prediction result.

[0014] Based on the motion trajectory prediction results, the third and fourth processing regions in the first region stitched image are adjusted to generate an adjusted second region stitched image.

[0015] The second region stitched image is subjected to image optimization processing to generate a third region stitched image, and the first projected coordinate image is subjected to image stitching processing that does not include the dynamic target detection to generate a fourth region stitched image.

[0016] The image stitched from the third region and the image stitched from the fourth region are then stitched together to generate the panoramic image.

[0017] In one possible implementation, the motion trajectory prediction result includes: the predicted position of the dynamic target object after a preset time step, and the predicted position is represented by a coordinate range;

[0018] The step of adjusting the region range of the third and fourth processing regions in the first region stitched image based on the motion trajectory prediction result to generate the adjusted second region stitched image includes:

[0019] Based on the predicted position of the dynamic target object after the preset time step, the positional relationship between the dynamic target object and the third processing area and the fourth processing area after the preset time step is determined.

[0020] Based on the predicted position of the dynamic target object and its positional relationship with the third and fourth processing regions, the region ranges of the third and fourth processing regions are adjusted to generate the second region stitched image.

[0021] In one possible implementation, the motion trajectory prediction result includes: motion direction, velocity amplitude, and predicted trajectory;

[0022] The step of adjusting the region range of the third and fourth processing regions based on the predicted position of the dynamic target object and the positional relationship between the third and fourth processing regions to generate the second region stitched image includes:

[0023] If the predicted position of the dynamic target object enters the fourth processing area but does not leave the third processing area, the boundary of the third processing area is extended to the fourth processing area along the motion trajectory extension direction corresponding to the motion direction angle to generate the second region stitched image; wherein, the extension distance is determined by the velocity amplitude and the preset time step.

[0024] If it is determined that the predicted position of the dynamic target object leaves the third processing area, the boundary in the third processing area that is closest to the starting point of the predicted trajectory is shrunk inward; wherein, the shrinkage distance is determined by the pixel distance between the current position of the dynamic target object and the boundary.

[0025] as well as,

[0026] For the region along the predicted trajectory in the fourth processing region, a rectangular sub-region that matches the size of the dynamic target object is included in the adjustment range of the third processing region, so as to adjust the third processing region and the fourth processing region and generate the second region stitched image.

[0027] In one possible implementation, the image optimization processing of the second region stitched image to generate the third region stitched image includes:

[0028] The Gaussian noise reduction algorithm and the median filtering algorithm are used to adjust the noise in the third and fourth processing regions of the stitched image of the second region, respectively, to obtain the adjusted third processing region and the adjusted fourth processing region.

[0029] An edge matching algorithm is used to optimize the stitching boundary between the adjusted third processing region and the adjusted fourth processing region to generate the stitched image of the third region.

[0030] In one possible implementation, the scene features include: image texture complexity, edge density value, and grayscale variance value;

[0031] The process of dividing the scene based on the scene features of the second projected coordinate image and the distribution location of the dynamic target object to determine the first processing region and the second processing region within the second projected coordinate image includes:

[0032] Scene feature analysis is performed on the second projected coordinate image to determine a fifth processing region and a sixth processing region in each second projected coordinate image; the fifth processing region is a region that satisfies at least one of the following three conditions: the image texture complexity is greater than a preset second threshold, the edge density value is greater than a preset third threshold, and the gray-level variance value is greater than a preset fourth threshold; the sixth processing region is a region where the image texture complexity is not greater than the preset second threshold, the edge density is not greater than the preset third threshold, and the gray-level variance value is not greater than the preset fourth threshold.

[0033] as well as,

[0034] Based on the distribution of the dynamic target objects in each second projected coordinate image, a seventh processing region and an eighth processing region are determined in each second projected coordinate image; the seventh region is the region where the density of dynamic targets is greater than a preset fifth threshold, and the eighth region is the region where the speed of dynamic target movement is greater than a preset sixth threshold.

[0035] The fifth processing area, the seventh processing area, and the eighth processing area are designated as the first processing area, and the sixth processing area is designated as the second processing area.

[0036] In one possible implementation, the step of performing dynamic degree analysis and projection transformation on each of the target environment images based on the pose data to generate a first projected coordinate image associated with low dynamic requirement images and a second projected coordinate image associated with high dynamic requirement images includes:

[0037] Dynamic degree analysis is performed on each of the target environment images to determine the dynamic contrast of each target environment image, and based on the dynamic contrast of each target environment image, the high dynamic demand image and the low dynamic demand image among all the target environment images are determined.

[0038] Projection transformation is performed based on the pose data and the dynamic contrast of each of the target environment images to generate the first projected coordinate image and the second projected coordinate image.

[0039] In one possible implementation, the step of performing a projection transformation based on the pose data and the dynamic contrast of each of the target environment images to generate the first projected coordinate image and the second projected coordinate image includes:

[0040] Based on the dynamic contrast of each of the target environment images, the image acquisition resolution of the camera associated with each of the target environment images is determined;

[0041] Based on the pose data and the image acquisition resolution of the corresponding camera, projection transformation is performed on each of the target environment images to determine the first projected coordinate image and the second projected coordinate image.

[0042] Secondly, embodiments of this application provide a panoramic image processing device applied to a drone, the drone including a multi-camera array and multiple inertial sensors, the device comprising:

[0043] The acquisition module is used to acquire target environment images of the UAV from multiple perspectives in real time through the multi-camera array, and to acquire pose data of the UAV through multiple inertial sensors;

[0044] The generation module is used to perform dynamic degree analysis and projection transformation on each of the target environment images based on the pose data, so as to generate a first projected coordinate image associated with a low dynamic requirement image and a second projected coordinate image associated with a high dynamic requirement image; the low dynamic requirement image and the high dynamic requirement image are images in multiple frames of the target environment images;

[0045] The detection module is used to perform dynamic target detection on the second projected coordinate image to determine dynamic target objects within the second projected coordinate image; the dynamic target object is a moving object whose pixel displacement exceeds a preset first threshold.

[0046] The segmentation module is used to segment regions based on scene features of the second projected coordinate image and the distribution location of the dynamic target object, so as to determine a first processing region and a second processing region within the second projected coordinate image; the processing precision of the first processing region is greater than that of the second processing region.

[0047] The stitching module is used to stitch the second projection coordinate image with the first projection coordinate image according to the preset processing algorithms associated with the first processing area and the second processing area, as well as the historical motion trajectory of the dynamic target object in the second projection coordinate image, so as to generate a panoramic image.

[0048] Thirdly, embodiments of this application provide a panoramic drone, which is used to implement any possible panoramic image processing method as described in the first aspect.

[0049] Compared with the prior art, this application has the following beneficial effects: The embodiments of this application provide a panoramic image processing method, device and panoramic drone. First, the multi-camera array on the drone is used to acquire multi-view target environment images in real time. Simultaneously, pose data is acquired through inertial sensors. Based on the pose data, the images from each viewpoint are projected and transformed to a unified projection coordinate system, thereby generating a first projection coordinate image associated with low dynamic demand images and a second projection coordinate image associated with high dynamic demand images. By distinguishing between low dynamic and high dynamic demand images and generating corresponding projection coordinate images, differentiated processing strategies can be adapted for different image characteristics, so that limited computing power can be concentrated on the dynamic and complex areas in the image, thereby improving the real-time processing of panoramic images. Secondly, dynamic target detection accurately identifies dynamic target objects in the second projected coordinate image whose pixel displacement exceeds a preset threshold. Combining scene features such as image texture complexity, edge density, and grayscale variance, as well as the distribution location of dynamic targets, the second projected coordinate image is divided into a first processing region and a second processing region with different processing precision. The first processing region uses a high-precision processing algorithm for key areas with high texture complexity, high dynamic target density, or high speed, while the second processing region uses a low-precision simplification algorithm for low-complexity static areas. This differentiated processing strategy enables the UAV to focus computing resources on key areas, avoiding the waste of computing power caused by uniform high-precision processing of the entire image, and significantly reducing the amount of data processing. Finally, in the image stitching stage, the stitching algorithm is optimized by combining the historical motion trajectory of the dynamic target object in the second projected coordinate image. Adapted preset processing algorithms are used for different processing areas, which not only ensures the image quality of key areas, but also reduces the computational load of the embedded processor by simplifying the processing flow of non-critical areas. This enables the UAV to efficiently complete real-time panoramic image stitching under the condition of limited hardware resources. A complete technology chain is formed from data preprocessing, region division to differentiated algorithm adaptation, which systematically improves the real-time processing and reduces the computational difficulty. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A schematic flowchart of a panoramic image processing method provided in an embodiment of this application;

[0052] Figure 2 A flowchart illustrating a method for generating a first projected coordinate image and a second projected coordinate image, provided in an embodiment of this application;

[0053] Figure 3 A schematic flowchart illustrating a panoramic image generation method provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of a panoramic image processing device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and accompanying drawings. It should be particularly noted that the embodiments described in this application are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0056] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0057] As mentioned earlier, panoramic image processing algorithms are quite complex, often involving multiple stages such as image stitching, fusion, and correction, requiring the analysis and processing of large amounts of image data. However, drones, limited by hardware constraints in size and power consumption, typically rely on embedded processors for computation, making it difficult to complete complex panoramic image processing tasks in a short time. Furthermore, drones are susceptible to factors such as airflow during flight, leading to instability in image acquisition and further increasing the difficulty of panoramic image processing. Currently, drone-based panoramic image processing suffers from high processing difficulty and poor real-time performance.

[0058] This application provides a panoramic image processing method, apparatus, and panoramic drone. First, the drone uses a multi-camera array to acquire multi-view target environment images in real time, and simultaneously acquires pose data through an inertial sensor. Based on the pose data, the images from each viewpoint are projected and transformed to a unified projection coordinate system, thereby generating a first projection coordinate image associated with low dynamic demand images and a second projection coordinate image associated with high dynamic demand images. By distinguishing between low-dynamic and high-dynamic demand images and generating corresponding projection coordinate images, differentiated processing strategies can be adapted for different image characteristics, allowing limited computing power to be concentrated on dynamic and complex areas in the image, thereby improving the real-time processing performance of panoramic images. Secondly, dynamic target detection accurately identifies dynamic target objects in the second projected coordinate image whose pixel displacement exceeds a preset threshold. Combining scene features such as image texture complexity, edge density, and grayscale variance, as well as the distribution location of dynamic targets, the second projected coordinate image is divided into a first processing region and a second processing region with different processing precision. The first processing region uses a high-precision processing algorithm for key areas with high texture complexity, high dynamic target density, or high speed, while the second processing region uses a low-precision simplification algorithm for low-complexity static areas. This differentiated processing strategy enables the UAV to focus computing resources on key areas, avoiding the waste of computing power caused by uniform high-precision processing of the entire image, and significantly reducing the amount of data processing. Finally, in the image stitching stage, the stitching algorithm is optimized by combining the historical motion trajectory of the dynamic target object in the second projected coordinate image. Adapted preset processing algorithms are used for different processing areas, which not only ensures the image quality of key areas, but also reduces the computational load of the embedded processor by simplifying the processing flow of non-critical areas. This enables the UAV to efficiently complete real-time panoramic image stitching under the condition of limited hardware resources. A complete technology chain is formed from data preprocessing, region division to differentiated algorithm adaptation, which systematically improves the real-time processing and reduces the computational difficulty.

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0060] See Figure 1 The figure is a flowchart illustrating a panoramic image processing method provided in an embodiment of this application, specifically including the following steps:

[0061] S101: The multi-camera array is used to acquire target environment images of the UAV from multiple perspectives in real time, and the pose data of the UAV is acquired through multiple inertial sensors.

[0062] In the initial data acquisition stage of UAV panoramic image processing, a multi-camera array and multiple inertial sensors work collaboratively to achieve synchronous real-time acquisition of target environment images and UAV pose data. The multi-camera array typically consists of multiple cameras distributed at different locations on the UAV's fuselage (e.g., front, rear, left, right, top, bottom, etc.). Each camera synchronously acquires images at a preset frame rate, ensuring coverage of the UAV's 360-degree or a specific range of environmental field of view, forming a multi-view sequence of target environment images. The parameters of these cameras, such as focal length, resolution, and field of view, are optimized to ensure image detail clarity and reduce blind spots in subsequent stitching, laying the foundation for the integrity of the panoramic image. Simultaneously, multiple inertial sensors installed inside the UAV, such as accelerometers, gyroscopes, and magnetometers, monitor the UAV's motion status in real time, acquiring pose data including the UAV's three-dimensional position, attitude angles, acceleration, and angular velocity. This data reflects the UAV's dynamic position and attitude changes in space, providing crucial reference information for subsequent image projection transformation.

[0063] It is important to note that the operation of the multi-camera array and the inertial sensor must be strictly synchronized. A unified timestamp or hardware triggering mechanism ensures a one-to-one correspondence between images and pose data acquired at the same moment, avoiding coordinate matching errors caused by time misalignment. The acquired multi-view images carry visual information about the target environment, while pose data records the spatial position and orientation of the UAV during image acquisition. Together, they form the raw data foundation for subsequent processing—pose data serves as the core parameter for coordinate transformation, projecting images acquired by different cameras onto a unified coordinate system based on the UAV, eliminating coordinate deviations caused by differences in viewpoints. The multi-view images are then processed in this unified coordinate system for subsequent dynamic target detection, region segmentation, and stitching. This synchronous acquisition mechanism not only ensures the spatiotemporal consistency of the data but also provides reliable initial data support for the high-precision generation of subsequent panoramic images through multi-sensor fusion, effectively avoiding processing errors caused by insufficient data from a single sensor, and improving the reliability and real-time performance of panoramic image processing from the source.

[0064] S102: Based on the pose data, perform dynamic degree analysis and projection transformation on each of the target environment images to generate a first projected coordinate image associated with a low dynamic requirement image and a second projected coordinate image associated with a high dynamic requirement image; the low dynamic requirement image and the high dynamic requirement image are images from multiple frames of the target environment images.

[0065] In this step, dynamics analysis quantifies the motion degree of dynamic targets in the image scene and the distinction between the target and the background. The aim is to classify multi-frame images into two categories: low-dynamic-demand images and high-dynamic-demand images. For high-dynamic-demand images, there are few dynamic targets, small motion amplitudes, or a high proportion of background in the image scene, resulting in relatively lower requirements for real-time processing accuracy. Conversely, for high-dynamic-demand images, the dynamic targets are significant, motion details are rich, or there are scene changes requiring close attention, demanding higher real-time performance and accuracy. After classifying low-dynamic-demand and high-dynamic-demand images, a unified projected coordinate system based on the UAV is constructed using pose data, and differentiated projection transformations are performed for different image categories. For images with low dynamic requirements, the pose parameters are used to map them from the local coordinate system of each camera to the reference coordinate system of the UAV, generating a first projected coordinate image. This process allows for the use of relatively simplified transformation algorithms to reduce computational overhead while ensuring the integrity of the basic scene information. For images with high dynamic requirements, a second projected coordinate image is generated through more refined pose data calibration and coordinate transformation, ensuring that key information such as the spatial position and motion trajectory of dynamic targets are accurately presented in a unified coordinate system.

[0066] Next, the generation process of the first and second projected coordinate images in step S102 will be described with reference to the accompanying drawings of a specific process embodiment.

[0067] See Figure 2 The figure is a flowchart illustrating a method for generating a first projected coordinate image and a second projected coordinate image according to an embodiment of this application, specifically including the following steps:

[0068] S1021: Perform dynamic degree analysis on each of the target environment images to determine the dynamic contrast of each target environment image, and based on the dynamic contrast of each target environment image, determine the high dynamic demand image and the low dynamic demand image among all the target environment images.

[0069] The dynamic contrast of the target environment image is used to quantify the dynamic degree of a dynamic target in the image scene, as well as the distinguishability of the dynamic target from the image scene. Correspondingly, images with high dynamic requirements are those with dynamic contrast greater than a preset contrast threshold, while images with low dynamic requirements are those with dynamic contrast no greater than the preset contrast threshold.

[0070] Dynamic contrast is a core indicator used to quantify key features of dynamic information in an image scene. Its core components include two dimensions: first, the dynamic degree of the target, i.e., the target's motion characteristics in the scene, such as speed, trajectory complexity, target density, and frequency of change in motion direction, reflecting the inherent complexity of the target's motion; and second, the "distinctiveness" between the dynamic target and the image background, i.e., the degree of difference between the target pixels and the background in visual features such as grayscale, color, texture, and contour, reflecting the visual recognizability of the dynamic target. Incorporating these two dimensions into the definition of dynamic contrast is essentially to comprehensively characterize the intensity of the image's demand on processing algorithms, thereby providing accurate classification criteria for UAVs to achieve efficient image processing with limited computing power.

[0071] The reason for considering dynamic range and discriminability as core elements is that they jointly determine the complexity of image processing and the difficulty of preserving key information. On the one hand, targets with high dynamic range require higher temporal resolution and motion trajectory capture capabilities from algorithms. Insufficient processing accuracy can easily lead to target blurring, trajectory breakage, or missed detection, directly affecting the accuracy of UAV perception of dynamic scenes. On the other hand, dynamic targets with low background discriminability, such as slow-moving objects blending into complex backgrounds or moving targets with colors similar to the environment, require more refined feature extraction and noise reduction processing even if the dynamic range is not high. Otherwise, target loss can easily occur due to noise interference or edge blurring. The combination of these two factors can more comprehensively reflect the "processing priority" of an image. Images with high dynamic range and low discriminability require both capturing details of fast-moving objects and separating difficult-to-identify targets, representing extremely high processing demands. Conversely, images with low dynamic range and high discriminability, even with the presence of dynamic targets (such as slowly moving high-contrast objects), have relatively lower processing difficulty and can be adapted to simplified algorithms.

[0072] By leveraging dynamic contrast, the "dynamic complexity" of an image is transformed into quantifiable processing requirements, enabling UAVs to perceive scene characteristics in real time and proactively allocate computing resources. For images with high dynamic requirements, a refined motion detection and coordinate transformation process is retained to ensure accurate representation of dynamic targets. For images with low dynamic requirements, processing steps are simplified to reduce computational consumption and avoid computational redundancy caused by uniform high-precision processing across the entire image. This step not only lays the data foundation for the differentiated design of subsequent projection transformation, region segmentation, and stitching algorithms, but also, in real-world scenarios where UAV hardware computing power is limited, achieves an optimized balance between dynamic target processing accuracy and overall real-time performance through an "on-demand processing" strategy, significantly improving the efficiency and reliability of panoramic image generation in complex environments.

[0073] S1022: Perform projection transformation based on the pose data and the dynamic contrast of each of the target environment images to generate the first projection coordinate image and the second projection coordinate image.

[0074] In this step, the processing requirements of each frame of image need to be quantified through dynamic contrast, the image acquisition resolution of the corresponding camera needs to be determined accordingly, and then differentiated projection transformation is performed based on pose data to ultimately generate a projected coordinate image adapted to different dynamic scenes. Specifically, this process is achieved through the following two steps:

[0075] Step 1: Determine the image acquisition resolution of the camera associated with each target environment image based on the dynamic contrast of each target environment image.

[0076] Specifically, dynamic contrast, acting as a bridge connecting scene characteristics and processing strategies, plays a core role in driving a dynamic resolution adjustment mechanism. For target environment images with high dynamic contrast, since the scene contains dynamic targets with high-speed movement, low discrimination, or dense interaction, more details need to be preserved to ensure the clear outlines and coherent trajectories of dynamic targets in subsequent stitching. Therefore, the system allocates higher image acquisition resolutions to associated cameras (such as increasing pixel density and optimizing sensor sampling rates) to capture richer texture and motion details at the source. Conversely, images with low dynamic contrast are mainly static backgrounds or low-complexity dynamic targets. Excessively high resolution would only increase the amount of data without significantly improving the processing effect. Therefore, the system automatically reduces the resolution of the corresponding cameras to reduce redundant data while ensuring the integrity of the basic scene information. This resolution adaptive mechanism is not a simple fixed threshold division, but rather achieves stepless adjustment of resolution within a reasonable range through continuous quantization of dynamic contrast values. This allows the resolution configuration to be precisely matched with the dynamic characteristics of the scene, avoiding resource waste or information loss caused by oversampling or undersampling.

[0077] Step 2: Based on the pose data and the image acquisition resolution of the corresponding camera, perform projection transformation on each of the target environment images to determine the first projected coordinate image and the second projected coordinate image.

[0078] After completing the dynamic resolution configuration, projection transformation is performed on the target environment images based on the pose data output in real time by the UAV's inertial sensors. Essentially, this process maps image pixels from different camera coordinate systems to a unified world coordinate system based on the UAV, thereby eliminating geometric distortions and coordinate differences caused by multi-view imaging. For images with high dynamic range requirements, since the corresponding camera resolution has been improved, the images contain more high-frequency details and dense pixel information. Therefore, a more accurate pose calibration model (such as a nonlinear transformation model considering lens distortion parameters) is needed during projection transformation to ensure that the spatial position calculation error of each high-resolution pixel in the unified coordinate system is kept to a minimum. In particular, point-by-point correction is performed on the edge pixels of fast-moving targets to avoid target contour blurring or positional shifts caused by transformation errors.

[0079] Images with low dynamic requirements, due to their reduced resolution and simple scene dynamics, can employ a simplified linear projection model. This skips complex distortion correction steps and directly completes coordinate mapping through translation and rotation parameters in the pose data, significantly reducing matrix computation. The core advantage of this differentiated transformation strategy lies in dynamically binding resolution adjustment with the complexity of the transformation algorithm. High-dynamic scenes use high-resolution input and high-precision transformation to ensure detail preservation, while low-dynamic scenes use low-resolution input and lightweight transformation to reduce computational load. Both achieve coordinate space alignment through a unified reference for pose data, avoiding computational overload caused by uniform high-resolution processing of the entire image and preventing the loss of dynamic target information caused by simple resolution reduction. The resulting first projected coordinate image (corresponding to the low-dynamic requirement image) and second projected coordinate image (corresponding to the high-dynamic requirement image) form a hierarchical scene representation in the UAV reference coordinate system: the former provides a stable background framework and the coarse position of low-complexity dynamic targets, while the latter retains fine dynamic details and high-resolution target outlines, building an efficient and accurate data foundation for subsequent tasks such as image stitching and motion tracking.

[0080] S103: Perform dynamic target detection on the second projected coordinate image to determine the dynamic target object within the second projected coordinate image; the dynamic target object is a moving object whose pixel displacement exceeds a preset first threshold.

[0081] Dynamic targets (i.e., moving objects whose pixel displacement exceeds a preset threshold) are key information carriers in the scene, and their motion directly affects the UAV's environmental perception and decision-making. Therefore, in the process of processing panoramic images based on UAVs, it is necessary to ensure that such dynamic targets are accurately captured and presented in the panoramic image to avoid blurring, ghosting, or loss of targets due to insufficient processing accuracy, which would affect the execution of subsequent tasks. Secondly, due to the limitations of UAV size and power consumption, the computing power of its embedded processor is limited. If a uniform high-precision processing algorithm is used for the entire image, it will lead to computational overload and make it difficult to meet real-time requirements. In practical application scenarios, the area where dynamic targets are located usually only occupies a local part of the image. Applying high-precision processing to these key areas and using low-precision simplification algorithms for static backgrounds or low-dynamic areas can concentrate computing resources on the truly important moving targets, greatly reducing invalid calculations and reducing overall computing power consumption while ensuring the quality of key information processing.

[0082] Therefore, in this step, it is necessary to filter out the target objects that actually undergo significant motion from the second projected coordinate image to provide key information for subsequent differential processing and stitching strategies. Specifically, the detection of dynamic target objects uses "pixel displacement exceeding a preset first threshold" as the quantitative standard for determining moving objects. Its implementation logic is closely integrated with the spatiotemporal consistency characteristics of the projected coordinate image. Since all projected images are unified under the same reference coordinate system, the pixel coordinates directly correspond to the actual positions in physical space. Therefore, the pixel displacement between adjacent frames can more accurately reflect the true motion state of the target object.

[0083] Specifically, in determining dynamic target objects, the first step is to preprocess the projected coordinate images of the continuous time series, reducing the interference of environmental noise on the detection results through operations such as grayscale conversion and noise reduction. Then, classic motion detection algorithms such as inter-frame difference or optical flow are used to calculate the displacement vector of each pixel in adjacent frames. This vector contains horizontal and vertical displacement components, and its magnitude is the quantized value of the pixel displacement. A preset first threshold is used as the benchmark for motion determination, and its value needs to comprehensively consider factors such as the drone's flight speed, camera frame rate, and image resolution. For regions where the pixel displacement exceeds the threshold, target segmentation is further performed using methods such as connected component analysis and contour detection. Discrete moving pixels are clustered into complete dynamic target objects, and their position, size, and direction of motion are extracted, thus completing the selection and determination of dynamic target objects.

[0084] S104: Based on the scene features of the second projected coordinate image and the distribution location of the dynamic target object, the region is divided to determine the first processing region and the second processing region within the second projected coordinate image; the processing accuracy of the first processing region is greater than that of the second processing region.

[0085] The core of this step lies in dividing the second projected coordinate image with high dynamic requirements into regions with different processing precisions by quantitatively analyzing the complexity of the image content and the distribution characteristics of dynamic targets. This allows the UAV to allocate more computing resources to critical areas, achieving a balance between processing efficiency and image quality under the premise of limited hardware computing power. Specifically, the first processing region has a higher processing precision than the second processing region; that is, the first processing region represents the critical areas requiring more computing power, while the second processing region represents non-critical areas that can be analyzed and processed more simply.

[0086] Specifically, the process of dividing the region in this step is implemented through the following two steps:

[0087] Step 1: Perform scene feature analysis on the second projected coordinate image to determine the fifth processing region and the sixth processing region in each second projected coordinate image; the fifth processing region is a region that satisfies at least one of the following three conditions: the image texture complexity is greater than a preset second threshold, the edge density value is greater than a preset third threshold, and the gray-level variance value is greater than a preset fourth threshold; the sixth processing region is a region where the image texture complexity is not greater than the preset second threshold, the edge density is not greater than the preset third threshold, and the gray-level variance value is not greater than the preset fourth threshold.

[0088] as well as,

[0089] Based on the distribution of the dynamic target objects in each second projected coordinate image, a seventh processing region and an eighth processing region are determined in each second projected coordinate image; the seventh region is the region where the density of dynamic targets is greater than a preset fifth threshold, and the eighth region is the region where the speed of dynamic target movement is greater than a preset sixth threshold.

[0090] Step 2: The fifth processing area, the seventh processing area, and the eighth processing area are designated as the first processing area, and the sixth processing area is designated as the second processing area.

[0091] Scene feature analysis focuses on three dimensions: image texture complexity, edge density, and grayscale variance. Texture complexity is calculated using the gray-level co-occurrence matrix to determine the spatial correlation of pixel grayscale values, reflecting the richness of detail in the image, such as the texture variations of plant leaves and building walls. Edge density extracts contour information using edge detection algorithms such as the Canny operator, statistically analyzing the proportion of edge pixels per unit area to reflect the density of object boundaries in the scene, such as road grids and the angular distribution of industrial facilities. Grayscale variance quantifies the uniformity of brightness distribution by calculating the standard deviation of pixel grayscale values ​​within a region, such as unevenly lit shadow areas or solid-color surfaces with uniform material. Regions that meet at least one of the above three indicators exceeding a preset threshold are defined as the "fifth processing region." These regions contain rich details or complex structures, requiring higher feature matching accuracy for image stitching. Conversely, when all three indicators are below the threshold, the region is classified as the "sixth processing region," whose scene content is relatively simple and can accept lower processing accuracy to save computational resources.

[0092] Meanwhile, dynamic target distribution analysis, based on previously detected dynamic target objects, further statistically analyzes the density and speed of dynamic targets within a region. Dynamic target density is defined as the number of moving objects per unit area. When this value exceeds a preset fifth threshold (e.g., busy intersections, crowded areas), the corresponding region is designated as the "seventh processing region." Speed ​​is calculated using the magnitude of the pixel displacement vectors between adjacent frames. When the speed exceeds a preset sixth threshold (e.g., high-speed vehicles, fast-flying birds), it is designated as the "eighth processing region." This classification logic combines the "static complexity" and "dynamic activity" of the scene. That is, the fifth processing region covers static scenes with high information density, while the seventh and eighth processing regions target the clustering and speed of dynamic targets, ultimately accurately identifying key areas requiring high-precision processing (the fifth, seventh, and eighth regions are subsequently merged into the first processing region) and secondary areas that can be simplified (the sixth processing region becomes the second processing region).

[0093] This dual-dimensional region division mechanism avoids over-computation of simple scenes while ensuring the preservation of details in complex texture areas and the continuity of dynamic target trajectories. It enables UAVs to adaptively allocate computing power according to the characteristics of real-time scenes, achieving the optimal balance between image processing accuracy and efficiency under the computing power constraints of embedded hardware. This provides a precise region definition basis for subsequent tasks such as differentiated stitching and dynamic target tracking.

[0094] S105: Based on the preset processing algorithms associated with the first processing area and the second processing area, and the historical motion trajectory of the dynamic target object in the second projected coordinate image, the second projected coordinate image and the first projected coordinate image are stitched together to generate a panoramic image.

[0095] After determining the first processing region requiring high-precision processing and the second processing region requiring low-precision processing in the second projected coordinate image, a preset differential processing algorithm is invoked. By predicting and compensating for the dynamic target motion trajectory, the stitching deviation problem caused by cross-frame motion is solved, and finally, high-quality fusion of the global scene is achieved under computing power constraints.

[0096] Next, the process of generating panoramic images in this step will be described with reference to the accompanying drawings of specific embodiments.

[0097] See Figure 3 The figure is a flowchart illustrating a panoramic image generation method provided in an embodiment of this application, specifically including the following steps:

[0098] S1041: The first processing region in the second projected coordinate image is stitched with high precision using a real-time dynamic detection algorithm and a feature matching algorithm, and the second processing region in the second projected coordinate image is stitched with low precision using a timed sampling detection algorithm and a stitching algorithm to generate a first region stitched image; the first region stitched image includes a third processing region and a fourth processing region, the third processing region is generated by stitching together multiple first processing regions, and the fourth processing region is generated by stitching together multiple second processing regions.

[0099] In implementing this step, the high-precision stitching of the first processing area must meet both real-time and accuracy requirements. A real-time dynamic detection algorithm (such as an optical flow-based or deep learning-based target tracking model) is used to scan the area frequently, capturing real-time positional changes of dynamic targets. Simultaneously, a feature matching algorithm (such as SIFT, ORB, or other sub-pixel-level feature extraction methods) is triggered to perform dense feature point detection and cross-frame matching on image blocks within the area. This processing strategy is particularly suitable for areas with dense dynamic targets or high-speed movement. For example, in traffic intersection scenes, dynamic detection is performed 50 times per second in areas where vehicles are concentrated, ensuring that the feature points of fast-moving vehicles in different projection images are accurately aligned, avoiding target fragmentation or ghosting due to motion blur or matching deviations. For areas with complex textures, high-precision feature matching can retain more detailed information, preventing texture loss or edge misalignment caused by simplified algorithms.

[0100] In contrast, the low-precision stitching in the second processing region prioritizes efficiency, employing a timed sampling detection algorithm to reduce computational frequency. For example, sparse feature detection is performed on this region only once every 5 frames, quickly completing coarse registration using a pre-defined pose transformation matrix and skipping complex global optimization steps. This strategy is suitable for the aforementioned sixth processing region (such as open fields, solid-color walls, and other low-feature-complexity regions), where scene content changes slowly and dynamic targets are minimal, ensuring basic stitching continuity without requiring high-frequency detection.

[0101] The first processing area of ​​multiple second projected coordinate images is stitched together with high precision to form a "third processing area". This area fully preserves the motion trajectory of dynamic targets and the detailed features of complex scenes, ensuring that the position of moving objects is continuous and their shapes are clear in the panoramic image. The second processing area is stitched together with low precision to generate a "fourth processing area". Although its stitching precision is lower than the former, it can still maintain the coherence of the overall image due to the simplicity of the scene, and the computational efficiency is significantly improved.

[0102] S1042: Based on the historical motion trajectory of the dynamic target object, the motion trajectory of the dynamic target object in a preset future time period is predicted by using a Kalman filter algorithm to generate a motion trajectory prediction result.

[0103] Predicting the future motion trajectory of a dynamic target based on its historical motion trajectory is a key step in improving the foresight of a visual processing system. The Kalman filter algorithm, due to its optimal estimation characteristics for dynamic systems in noisy environments, has become the core method for achieving this prediction. This step first integrates the motion data of the dynamic target over historical time periods, including position coordinates, velocity, acceleration, and other state parameters at each moment, forming a multi-dimensional state vector containing both position and velocity dimensions, providing the data foundation for prediction. These historical trajectories are typically correlated using cross-frame target association algorithms (such as the Hungarian algorithm), and denoising is performed to ensure data reliability.

[0104] Kalman filtering operates cyclically through two core steps: prediction and update. In the prediction phase, the algorithm extrapolates the optimal state from the previous moment to the current prediction moment based on a pre-defined motion model (such as a uniform velocity or uniform acceleration model), while also considering the impact of model errors to generate a prior estimate of the target's motion state. In the update phase, it combines the latest observation data (such as the target's detected position in an image) and dynamically fuses the predicted and observed values ​​using Kalman gain to obtain a noise-suppressed posterior estimate, thus optimizing the target's actual motion state. Through multiple iterative extrapolations, it ultimately generates a predicted trajectory for a preset future time period, encompassing the position coordinates and velocity vectors at each future time point.

[0105] S1043: Based on the motion trajectory prediction result, adjust the region range of the third processing region and the fourth processing region in the first region stitched image to generate the adjusted second region stitched image.

[0106] The purpose of this step is to evaluate the potential coverage area changes of dynamic targets in the first region stitched image in real time using the future time period motion trajectory prediction results generated by Kalman filtering. Then, the boundary between the third processing region (high-precision stitching area) and the fourth processing region (low-precision stitching area) is dynamically corrected so that the region division always fits the target motion trend and avoids processing accuracy misalignment or computing power waste caused by static region division.

[0107] Specifically, this process is achieved through the following two steps:

[0108] Step 1: Based on the predicted position of the dynamic target object after the preset time step, determine the positional relationship between the dynamic target object and the third processing area and the fourth processing area after the preset time step.

[0109] Adjusting the range of the third and fourth processing regions based on the predicted position of the dynamic target object after a preset time step is the core step in achieving dynamic matching between processing accuracy and target motion. This process logically begins with the predicted position of the dynamic target object. First, the positional relationship between this predicted position and the third and fourth processing regions in the current stitched image of the first region must be determined—that is, whether the target, after the preset time step, is completely within the third region, partially crosses the boundary between the third and fourth regions, completely enters the fourth region, or is about to leave the image range. This determination of positional relationship is the direct basis for region adjustment. Its core lies in establishing a spatial mapping relationship between the predicted position and the geometric boundaries of the regions (such as the coordinate range of a rectangular region or the vertex set of a polygonal region), and quantifying the coverage degree of the dynamic target object and the region through coordinate calculations.

[0110] Step 2: Based on the predicted position of the dynamic target object and its positional relationship with the third and fourth processing regions, adjust the region range of the third and fourth processing regions to generate the second region stitched image.

[0111] The specific implementation logic of this step is achieved through the following two steps:

[0112] Step 1: If the predicted position of the dynamic target object enters the fourth processing area but does not leave the third processing area, the boundary of the third processing area is extended to the fourth processing area along the motion trajectory extension direction corresponding to the motion direction angle to generate the second region stitched image; wherein, the extension distance is determined by the velocity amplitude and the preset time step.

[0113] When a dynamic target object is predicted to have entered the fourth processing area but has not yet completely left the third processing area, the boundary of the area needs to be dynamically expanded based on the target's movement trend to maintain processing accuracy. The core logic of this process is that when the target partially enters a low-precision area but still intersects with a high-precision area, the boundary of the third processing area is actively expanded towards the fourth processing area along the trajectory extension direction corresponding to its movement direction angle. This ensures that the area the target is still in or about to enter after a preset time step is included in the high-precision processing range. Specifically, the movement direction angle is obtained by fitting the target's historical movement trajectory, reflecting its current main direction of movement (such as due east, northwest, etc.), while the trajectory extension direction is a vector direction formed based on this angle, ensuring that the expansion direction is consistent with the target's actual movement path and avoiding ineffective area expansion.

[0114] The calculation of the extended distance is directly related to the target's motion velocity and time parameters, and is determined by multiplying the velocity amplitude by a preset time step. This calculation is essentially a linear extrapolation based on a uniform motion model, assuming that the target maintains its current velocity amplitude within a preset time period, thereby deriving its theoretical displacement distance within that time period, which serves as the physical measure of boundary expansion. For example, if the target velocity amplitude is 20 pixels / frame and the preset time step is 2 frames, then the extended distance is 40 pixels. The system will expand the boundary of the third processing area by 40 pixels along the direction corresponding to the motion direction angle, so that the fourth processing area part of the target's predicted position and subsequent movement path is included in the third processing area, switching to a high-precision processing mode.

[0115] The core purpose of this expansion operation is to ensure that dynamic targets remain within a sufficiently accurate processing range during cross-regional movement, avoiding problems such as feature loss, trajectory breakage, or stitching misalignment caused by entering low-precision areas. By strictly aligning the direction of motion with the expansion direction, the system can accurately cover the actual movement path of the target, rather than blindly expanding the high-precision area, thus controlling computational power consumption while ensuring processing effectiveness. The generated second-region stitched image thus achieves dynamic matching between the processing area and the target's motion. That is, the high-precision boundary is extended in advance in the direction where the target is about to enter the low-precision area, so that the target is still within the adjusted third processing area after a preset time step, continuously receiving high-frequency feature detection and dense matching, ensuring that its positional accuracy and image stitching quality during cross-regional movement are not negatively affected by the region boundaries. This quantitative expansion strategy based on motion parameters utilizes the foresight of predicted position and achieves precise control of the expansion distance through the quantitative relationship between speed and time, providing support for the real-time performance and accuracy of panoramic image processing in complex scenes.

[0116] Step 2: If it is determined that the predicted position of the dynamic target object leaves the third processing area, the boundary in the third processing area that is closest to the starting point of the predicted trajectory is shrunk inward; wherein, the shrinkage distance is determined by the pixel distance between the current position of the dynamic target object and the boundary.

[0117] as well as,

[0118] For the region along the predicted trajectory in the fourth processing region, a rectangular sub-region that matches the size of the dynamic target object is included in the adjustment range of the third processing region, so as to adjust the third processing region and the fourth processing region and generate the second region stitched image.

[0119] When it is determined that the predicted position of a dynamic target object has left the third processing area, boundary shrinkage and new area inclusion operations must be performed simultaneously to achieve dynamic recovery and forward-looking allocation of high-precision processing resources. The core logic of this process is that after the target completely leaves the third processing area, the inner boundary of the third processing area closest to the starting point of the predicted trajectory (i.e., the last position of the target when leaving the third area) is first shrunk. Here, "trajectory starting point" refers to the last known position of the target in the third area. Selecting the nearest boundary for shrinkage can accurately eliminate invalid high-precision areas where the target has completely withdrawn, thus avoiding computational redundancy. The shrinkage distance is determined by calculating the pixel distance between the target's current position (or trajectory starting point position) and the corresponding boundary. For example, if the position of the target when it leaves is only 15 pixels away from the right boundary of the third area, then shrink 15 pixels inward along that boundary to make the boundary fit the actual coverage area after the target's withdrawal, releasing idle high-precision processing resources outside the original boundary.

[0120] Simultaneously, the areas traversed by the predicted trajectory of the target in the fourth processing region need to be evaluated, and rectangular sub-regions matching the size of the dynamic target should be included in the adjustment range of the third processing region. Specifically, the segment of the fourth processing region that the target will pass through along the predicted trajectory is extracted, and a rectangular sub-region of equal or slightly larger size is generated based on the real-time size of the target (such as the width and height of the circumscribed rectangle), and this sub-region is transferred from the fourth processing region to the third processing region. The essence of this operation is to "lay out" a new high-precision processing zone along the target's movement path after it leaves the original third processing region, ensuring that the target remains within the precision-adapted region during subsequent movement, avoiding a sharp drop in detection accuracy due to entering a low-precision region. For example, if the target size is 50×30 pixels, and the predicted trajectory needs to pass through a certain open area in the fourth region, the system will generate a 55×35 pixel rectangular sub-region along the trajectory, include it in the third processing region, and simultaneously improve the feature detection frequency and matching accuracy of this region.

[0121] Through the coordinated operation of "boundary shrinkage" and "trajectory-alongline inclusion," the high-precision processing area is dynamically updated. On the one hand, it promptly recovers computing power from areas where the target has left (shrinkage distance is accurate to the pixel level, avoiding boundary residue caused by excessive shrinkage). On the other hand, it proactively covers key areas along the target's subsequent movement path (the size of the rectangular sub-region matches the target in real time, preventing computing power waste due to an overly large region or target edge overflow due to an underly small region). The final generated second-region stitched image not only removes outdated parts no longer covered by the target in the third processing region but also constructs a high-precision channel that fits the target's movement trajectory in the fourth processing region, forming a "resources migrate with the target" adjustment mechanism. This ensures that the dynamic target always receives precision-matched processing services when moving across regions, while concentrating computing power on the truly needed areas, effectively improving the overall efficiency of the vision processing system and the stability of target tracking.

[0122] S1044: Perform image optimization processing on the second region stitched image to generate a third region stitched image, and perform image stitching processing on the first projected coordinate image that does not include the dynamic target detection to generate a fourth region stitched image.

[0123] In this step, for the second region stitched image (corresponding to the scene with high dynamic contrast and containing complex dynamic targets in the previous stage), this step is closely focused on the integrity and clarity of the dynamic targets.

[0124] To address the different characteristics of the third and fourth processing regions in the stitched image of the second region, targeted noise reduction algorithms are adopted. The third processing region, due to its high-precision detection requirements for dynamic targets, typically contains rich details and high-frequency features, making it susceptible to Gaussian noise. Therefore, a Gaussian noise reduction algorithm is used, employing a two-dimensional Gaussian kernel function to perform a weighted average of pixels within the region. This smooths Gaussian noise while preserving edge details, ensuring the clarity of target contours and textures. On the other hand, the fourth processing region, being a low-precision region, mainly processes the background or non-critical areas and may be subject to impulse interference such as salt-and-pepper noise. Therefore, a median filtering algorithm is used, replacing each pixel with the median pixel in its neighborhood. This effectively removes isolated noise points while avoiding excessive blurring of the background texture, achieving a balance between computational power and performance.

[0125] After noise reduction, edge optimization is needed at the junction of the third and fourth processing regions to eliminate brightness, contrast, or texture discontinuities caused by dynamic region adjustments. The edge matching algorithm first detects pixel gradient differences at the junction boundary, then divides the image into different scales using multi-resolution pyramid decomposition. It adjusts overall brightness consistency at low-frequency levels and aligns edge details at high-frequency levels. Subsequently, feathering fusion is used to perform a gradual transition on pixels on both sides of the boundary, avoiding abrupt seams. For complex scenes with moving targets crossing the boundary, the algorithm performs motion compensation on dynamic pixels near the boundary based on the predicted target position, ensuring the continuity of the target's contour at the junction boundary and avoiding "tearing" caused by region adjustments.

[0126] The processing of the first projected coordinate image focuses on the characteristics of low dynamic regions, which are mainly static and undisturbed by dynamic regions. With "simplest operation to meet basic needs" as the core, image stitching processing is performed on it with efficiency in mind.

[0127] For image stitching of the first projected coordinate image, dynamic target detection is skipped because dynamic content in low-dynamic areas has no significant impact on the panoramic frame, eliminating the need for additional computational resources for recognition or tracking. Subsequently, a rigid body transformation matrix is ​​calculated based on the UAV's real-time pose data. Leveraging the characteristic of low-dynamic scenes lacking significant nonlinear deformation, the multi-view images are quickly aligned to a unified projected coordinate system using the simplest geometric transformations, achieving geometric matching. Color balancing is achieved through global brightness and contrast mean matching, eliminating the need for pixel-by-pixel or local fine-tuning. The color distribution of static content is stable, and global adjustments are sufficient to ensure color consistency after stitching. Finally, image stitching is directly completed, generating the stitched image for the fourth region.

[0128] S1045: Perform image stitching processing on the third region stitched image and the fourth region stitched image to generate the panoramic image.

[0129] Finally, the stitched images from the third and fourth regions are fused to generate a panoramic image. Using the static base of the fourth region as the spatial reference, the previously shared projection coordinate system is invoked to accurately map the high-dynamic content of the third region to its corresponding position. For example, optimized moving vehicles in the third region will be accurately aligned to their road positions in the fourth region, ensuring spatial consistency between dynamic targets and static backgrounds. Next, the fusion problem at the stitching boundaries is addressed. Because the third region has undergone targeted optimization (such as edge enhancement and motion compensation), while the fourth region is a simplified static background, there may be differences in brightness or texture breaks between the two. Multi-resolution pyramid fusion or edge feathering techniques are then employed. The former decomposes the two regions into different resolution layers and fuses them layer by layer from low to high, prioritizing consistency at the lower layers before addressing transitions at higher layers. The latter applies gradient weighting to the pixels at the stitching boundaries, allowing the edges of dynamic regions to naturally "blend" into the background. The entire process requires no additional complex calculations because the third region has already achieved dynamic detail preservation, and the fourth region has achieved static structural stability; the fusion only needs to solve two core problems: "position alignment" and "boundary transition."

[0130] Thus, the final panoramic image retains the complete framework of the static background in the fourth region while clearly presenting the dynamic target details in the third region, achieving efficient real-time panoramic image processing and stitching under limited hardware resources.

[0131] This application provides a panoramic image processing method. First, a multi-camera array mounted on a drone is used to acquire multi-view target environment images in real time. Simultaneously, pose data is acquired through an inertial sensor. Based on the pose data, the images from each viewpoint are projected and transformed to a unified projection coordinate system, thereby generating a first projection coordinate image associated with low dynamic demand images and a second projection coordinate image associated with high dynamic demand images. By distinguishing between low-dynamic and high-dynamic demand images and generating corresponding projection coordinate images, differentiated processing strategies can be adapted for different image characteristics, allowing limited computing power to be concentrated on dynamic and complex areas in the image, thereby improving the real-time processing performance of panoramic images. Secondly, dynamic target detection accurately identifies dynamic target objects in the second projected coordinate image whose pixel displacement exceeds a preset threshold. Combining scene features such as image texture complexity, edge density, and grayscale variance, as well as the distribution location of dynamic targets, the second projected coordinate image is divided into a first processing region and a second processing region with different processing precision. The first processing region uses a high-precision processing algorithm for key areas with high texture complexity, high dynamic target density, or high speed, while the second processing region uses a low-precision simplification algorithm for low-complexity static areas. This differentiated processing strategy enables the UAV to focus computing resources on key areas, avoiding the waste of computing power caused by uniform high-precision processing of the entire image, and significantly reducing the amount of data processing. Finally, in the image stitching stage, the stitching algorithm is optimized by combining the historical motion trajectory of the dynamic target object in the second projected coordinate image. Adapted preset processing algorithms are used for different processing areas, which not only ensures the image quality of key areas, but also reduces the computational load of the embedded processor by simplifying the processing flow of non-critical areas. This enables the UAV to efficiently complete real-time panoramic image stitching under the condition of limited hardware resources. A complete technology chain is formed from data preprocessing, region division to differentiated algorithm adaptation, which systematically improves the real-time processing and reduces the computational difficulty.

[0132] The following describes a panoramic image processing apparatus provided in an embodiment of this application. The panoramic image processing apparatus described below can be referred to in correspondence with the panoramic image processing method described above.

[0133] See Figure 4 The figure is a schematic diagram of the structure of a panoramic image processing device provided in an embodiment of this application, which specifically includes the following modules:

[0134] The acquisition module 100 is used to acquire target environment images of the UAV from multiple perspectives in real time through the multi-camera array, and to acquire pose data of the UAV through multiple inertial sensors.

[0135] The generation module 200 is used to perform dynamic degree analysis and projection transformation on each of the target environment images based on the pose data, so as to generate a first projected coordinate image associated with a low dynamic requirement image and a second projected coordinate image associated with a high dynamic requirement image; the low dynamic requirement image and the high dynamic requirement image are images in multiple frames of the target environment images;

[0136] The detection module 300 is used to perform dynamic target detection on the second projected coordinate image to determine dynamic target objects within the second projected coordinate image; the dynamic target object is a moving object whose pixel displacement exceeds a preset first threshold.

[0137] The segmentation module 400 is used to segment regions based on scene features of the second projected coordinate image and the distribution location of the dynamic target object, so as to determine a first processing region and a second processing region within the second projected coordinate image; the processing precision of the first processing region is greater than that of the second processing region.

[0138] The stitching module 500 is used to stitch the second projection coordinate image with the first projection coordinate image according to the preset processing algorithms associated with the first processing area and the second processing area, and the historical motion trajectory of the dynamic target object in the second projection coordinate image, so as to generate a panoramic image.

[0139] This application also provides a panoramic drone for implementing the panoramic image processing method in any of the above embodiments.

[0140] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for methods, devices, and panoramic UAVs, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The methods and systems described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0141] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A panoramic image processing method, characterized in that, Applied to a drone, the drone including a multi-camera array and multiple inertial sensors, the method includes: The multi-camera array is used to acquire target environment images of the UAV from multiple perspectives in real time, and the pose data of the UAV is acquired through multiple inertial sensors. Based on the pose data, dynamic degree analysis and projection transformation are performed on each of the target environment images to generate a first projected coordinate image associated with a low dynamic requirement image and a second projected coordinate image associated with a high dynamic requirement image; the low dynamic requirement image and the high dynamic requirement image are images from multiple frames of the target environment images; Dynamic target detection is performed on the second projected coordinate image to determine dynamic target objects within the second projected coordinate image; the dynamic target object is a moving object whose pixel displacement exceeds a preset first threshold. Based on the scene features of the second projected coordinate image and the distribution location of the dynamic target object, the region is divided to determine the first processing region and the second processing region within the second projected coordinate image; the processing precision of the first processing region is greater than that of the second processing region. Based on the preset processing algorithms associated with the first processing area and the second processing area, and the historical motion trajectory of the dynamic target object in the second projected coordinate image, the second projected coordinate image and the first projected coordinate image are stitched together to generate a panoramic image. The step of stitching the second projected coordinate image with the first projected coordinate image to generate a panoramic image, based on the preset processing algorithms associated with the first and second processing regions and the historical motion trajectory of the dynamic target object in the second projected coordinate image, includes: The first processing region in the second projected coordinate image is stitched with high precision using a real-time dynamic detection algorithm and a feature matching algorithm, and the second processing region in the second projected coordinate image is stitched with low precision using a timed sampling detection algorithm and a stitching algorithm to generate a first region stitched image; the first region stitched image includes a third processing region and a fourth processing region, wherein the third processing region is generated by stitching together multiple first processing regions, and the fourth processing region is generated by stitching together multiple second processing regions; Based on the historical motion trajectory of the dynamic target object, the motion trajectory of the dynamic target object in a preset future time period is predicted by the Kalman filter algorithm to generate a motion trajectory prediction result. Based on the motion trajectory prediction results, the third and fourth processing regions in the first region stitched image are adjusted to generate an adjusted second region stitched image. The second region stitched image is subjected to image optimization processing to generate a third region stitched image, and the first projected coordinate image is subjected to image stitching processing that does not include the dynamic target detection to generate a fourth region stitched image. The image stitched from the third region and the image stitched from the fourth region are then stitched together to generate the panoramic image.

2. The method according to claim 1, characterized in that, The motion trajectory prediction result includes: the predicted position of the dynamic target object after a preset time step, and the predicted position is represented by a coordinate range; The step of adjusting the region range of the third and fourth processing regions in the first region stitched image based on the motion trajectory prediction result to generate the adjusted second region stitched image includes: Based on the predicted position of the dynamic target object after the preset time step, the positional relationship between the dynamic target object and the third processing area and the fourth processing area after the preset time step is determined. Based on the predicted position of the dynamic target object and its positional relationship with the third and fourth processing regions, the region ranges of the third and fourth processing regions are adjusted to generate the second region stitched image.

3. The method according to claim 2, characterized in that, The motion trajectory prediction results include: motion direction, velocity amplitude, and predicted trajectory; The step of adjusting the region range of the third and fourth processing regions based on the predicted position of the dynamic target object and the positional relationship between the third and fourth processing regions to generate the second region stitched image includes: If the predicted position of the dynamic target object enters the fourth processing area but does not leave the third processing area, the boundary of the third processing area is extended to the fourth processing area along the motion trajectory extension direction corresponding to the motion direction angle to generate the second region stitched image; wherein, the extension distance is determined by the velocity amplitude and the preset time step. If it is determined that the predicted position of the dynamic target object leaves the third processing area, the boundary in the third processing area that is closest to the starting point of the predicted trajectory is shrunk inward; wherein, the shrinkage distance is determined by the pixel distance between the current position of the dynamic target object and the boundary. as well as, For the region along the predicted trajectory in the fourth processing region, a rectangular sub-region that matches the size of the dynamic target object is included in the adjustment range of the third processing region, so as to adjust the third processing region and the fourth processing region and generate the second region stitched image.

4. The method according to claim 1, characterized in that, The step of performing image optimization processing on the second region stitched image to generate the third region stitched image includes: The Gaussian noise reduction algorithm and the median filtering algorithm are used to adjust the noise in the third and fourth processing regions of the stitched image of the second region, respectively, to obtain the adjusted third processing region and the adjusted fourth processing region. An edge matching algorithm is used to optimize the stitching boundary between the adjusted third processing region and the adjusted fourth processing region to generate the stitched image of the third region.

5. The method according to claim 1, characterized in that, The scene features include: image texture complexity, edge density value, and grayscale variance value; The process of dividing the scene based on the scene features of the second projected coordinate image and the distribution location of the dynamic target object to determine the first processing region and the second processing region within the second projected coordinate image includes: Scene feature analysis is performed on the second projected coordinate image to determine a fifth processing region and a sixth processing region in each second projected coordinate image; the fifth processing region is a region that satisfies at least one of the following three conditions: the image texture complexity is greater than a preset second threshold, the edge density value is greater than a preset third threshold, and the gray-level variance value is greater than a preset fourth threshold; the sixth processing region is a region where the image texture complexity is not greater than the preset second threshold, the edge density value is not greater than the preset third threshold, and the gray-level variance value is not greater than the preset fourth threshold. as well as, Based on the distribution of the dynamic target objects in each second projected coordinate image, a seventh processing region and an eighth processing region are determined in each second projected coordinate image; the seventh processing region is the region where the density of dynamic targets is greater than a preset fifth threshold, and the eighth processing region is the region where the speed of dynamic target movement is greater than a preset sixth threshold. The fifth processing area, the seventh processing area, and the eighth processing area are designated as the first processing area, and the sixth processing area is designated as the second processing area.

6. The method according to claim 1, characterized in that, The step of performing dynamic degree analysis and projection transformation on each of the target environment images based on the pose data to generate a first projected coordinate image associated with low dynamic requirement images and a second projected coordinate image associated with high dynamic requirement images includes: Dynamic degree analysis is performed on each of the target environment images to determine the dynamic contrast of each target environment image, and based on the dynamic contrast of each target environment image, the high dynamic demand image and the low dynamic demand image among all the target environment images are determined. Projection transformation is performed based on the pose data and the dynamic contrast of each of the target environment images to generate the first projected coordinate image and the second projected coordinate image.

7. The method according to claim 6, characterized in that, The step of performing projection transformation based on the pose data and the dynamic contrast of each of the target environment images to generate the first projected coordinate image and the second projected coordinate image includes: Based on the dynamic contrast of each of the target environment images, the image acquisition resolution of the camera associated with each of the target environment images is determined; Based on the pose data and the image acquisition resolution of the corresponding camera, projection transformation is performed on each of the target environment images to determine the first projected coordinate image and the second projected coordinate image.

8. A panoramic image processing device, characterized in that, Applied to drones, the drones including a multi-camera array and multiple inertial sensors, the device includes: The acquisition module is used to acquire target environment images of the UAV from multiple perspectives in real time through the multi-camera array, and to acquire pose data of the UAV through multiple inertial sensors; The generation module is used to perform dynamic degree analysis and projection transformation on each of the target environment images based on the pose data, so as to generate a first projected coordinate image associated with a low dynamic requirement image and a second projected coordinate image associated with a high dynamic requirement image; the low dynamic requirement image and the high dynamic requirement image are images in multiple frames of the target environment images; The detection module is used to perform dynamic target detection on the second projected coordinate image to determine dynamic target objects within the second projected coordinate image; the dynamic target object is a moving object whose pixel displacement exceeds a preset first threshold. The segmentation module is used to segment regions based on scene features of the second projected coordinate image and the distribution location of the dynamic target object, so as to determine a first processing region and a second processing region within the second projected coordinate image; the processing precision of the first processing region is greater than that of the second processing region. The stitching module is used to stitch the second projection coordinate image with the first projection coordinate image according to the preset processing algorithms associated with the first processing area and the second processing area, and the historical motion trajectory of the dynamic target object in the second projection coordinate image, so as to generate a panoramic image. The splicing module is specifically used for: The first processing region in the second projected coordinate image is stitched with high precision using a real-time dynamic detection algorithm and a feature matching algorithm, and the second processing region in the second projected coordinate image is stitched with low precision using a timed sampling detection algorithm and a stitching algorithm to generate a first region stitched image; the first region stitched image includes a third processing region and a fourth processing region, wherein the third processing region is generated by stitching together multiple first processing regions, and the fourth processing region is generated by stitching together multiple second processing regions; Based on the historical motion trajectory of the dynamic target object, the motion trajectory of the dynamic target object in a preset future time period is predicted by the Kalman filter algorithm to generate a motion trajectory prediction result. Based on the motion trajectory prediction results, the third and fourth processing regions in the first region stitched image are adjusted to generate an adjusted second region stitched image. The second region stitched image is subjected to image optimization processing to generate a third region stitched image, and the first projected coordinate image is subjected to image stitching processing that does not include the dynamic target detection to generate a fourth region stitched image. The image stitched from the third region and the image stitched from the fourth region are then stitched together to generate the panoramic image.

9. A panoramic drone, characterized in that, The panoramic drone is used to implement the panoramic image processing method as described in any one of claims 1-7.

Images