Panoramic image real-time splicing algorithm and system based on multi-sensor fusion

Abstract

The invention discloses a panoramic image real-time splicing algorithm and system based on multi-sensor fusion, and particularly relates to the technical field of panoramic image real-time splicing, and the algorithm comprises the following steps: constructing a structured fusion sequence based on multi-source images, postures and position information, optimizing a matching effect through high-density feature extraction and repeated texture recognition, and obtaining a multi-source image fusion sequence; a dynamic foreground and a static background are distinguished by using sparse optical flow so as to improve the visual angle estimation precision, pose fusion optimization is realized in combination with a multi-mode residual error, and the continuity and stability of a spliced image are improved through edge smoothing, brightness tuning and color correction; according to the method, the structured fusion sequence is constructed through multi-source data alignment, so that the data synchronization and splicing stability is improved; identifying repeated regions based on texture direction features, and optimizing feature matching accuracy; and through edge smoothing, brightness harmonizing and color consistency processing, the visual coherence and output quality of the panoramic image are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time panoramic image stitching, and more specifically, to a real-time panoramic image stitching algorithm and system based on multi-sensor fusion. Background Art

[0002] With the widespread deployment of multi-view perception systems, panoramic image real-time stitching algorithms, as a key supporting technology for image enhancement and scene understanding, have played an important role in intelligent transportation, urban security, industrial inspection, mobile robots and other fields. This type of technology usually relies on the collaborative work of multiple image acquisition devices in different locations to spatially align and fuse image frame data from different perspectives, thereby generating a continuous image sequence with a complete field of view. Traditional image stitching methods are mostly based on image feature matching and geometric transformation model solutions, but in actual engineering environments, they still face multiple technical difficulties: the timestamps between image sources are often out of sync, resulting in the accumulation of data alignment errors; repetitive texture areas such as building curtain walls or road bricks are prone to feature mismatching; dynamic occlusions in the foreground will interfere with the perspective estimation of the background area; and differences in imaging characteristics between image sensors may cause sudden changes in boundary brightness and color inconsistencies.

[0003] Taking the example of stitching images of urban intersections from fixed surveillance cameras, image frame data is limited by device synchronization mechanisms, and amplified temporal errors can cause offsets in the image stitching trajectory. Furthermore, these scenes are subject to the dynamic interference of numerous vehicles and pedestrians. Furthermore, due to varying shooting angles and sunlight angles, the brightness and color of the individual camera images vary significantly, resulting in broken edges and unstable fusion in the stitched result, severely impacting subsequent image analysis and target recognition tasks. Therefore, this paper proposes a real-time panoramic image stitching algorithm and system based on multi-sensor fusion to address these issues. Summary of the Invention

[0004] To achieve the above object, the present invention provides the following technical solutions: The real-time panoramic image stitching algorithm based on multi-sensor fusion includes the following steps: The first step is to synchronously acquire image frame data through multiple image acquisition devices at different spatial positions, combine the posture data provided by the inertial measurement device with the position information output by the positioning device, align the multi-source data according to the timestamp, and eliminate data segments with time delays exceeding a preset threshold to form a structured fusion data sequence; The second step is to extract high-density feature points from the grayscale value change areas in the image frame, and use the local texture direction histogram of the image block to build the regional feature context. On this basis, each feature point is associated with the structural characteristics of the region where it is located, and the repetitive texture region is identified based on the spatial distribution pattern, reducing the participation weight of such regional features in the matching; The third step is to extract a set of non-rigid displacement vectors by calculating sparse optical flow between consecutive image frames, separate the foreground dynamic occlusion area and the background static area on the image plane, and estimate the perspective transformation parameters only for the static area to eliminate the influence of the foreground area on the registration process; The fourth step is to construct a multi-frame image residual sequence within the time window. Based on the historical fusion path and the feature drift trend of the current frame image, the spatiotemporal residual values ​​of the corresponding position information of different sensor sources are calculated. The fusion parameters are adaptively updated according to the confidence distribution function to obtain the optimal splicing pose solution. In the fifth step, according to the estimated stitching pose, the image frame data after perspective alignment is smoothed, the brightness overlapping area is blended, and the color consistency is adjusted to generate a continuous panoramic image output. The fusion status is refreshed in real time to ensure the stability of the subsequent frame stitching process.

[0005] In a preferred embodiment, when fusing data collected by multiple image acquisition devices with corresponding inertial measurement devices and positioning devices, a time series residual discrimination method is used to filter out asynchronous data. Specifically, the time interval error between the timestamp sequences of each data source is calculated. When the error exceeds a set synchronization accuracy threshold, all data at that moment is removed from the structured fusion data sequence. To maintain data continuity, when the time difference between two consecutive valid image frames exceeds the maximum inter-frame expected value, the reference pose derived from the two frames of data is inserted into the data structure, and finally a structured fusion data sequence is obtained.

[0006] In a preferred embodiment, the process of high-density feature point extraction includes: calculating the first-order gradient response values ​​in the horizontal and vertical directions for each pixel point in the image frame to construct a gradient intensity map; in the gradient intensity map, selecting the local maximum point as the initial feature point candidate set, and performing non-maximum suppression processing on the feature points in the candidate set whose adjacent distances are less than the set pixel spacing, and retaining the ones with the largest response intensity as valid feature points; finally, a high-density feature point set that is evenly distributed in the grayscale value change area of ​​the image frame and has significant response intensity characteristics is obtained, which provides a basis for subsequent feature context construction and matching.

[0007] In a preferred embodiment, the context construction process performed for each high-density feature point includes: extracting an image block of a fixed size with the feature point as the center, calculating the pixel gradient directional distribution within the image block, and generating a normalized directional histogram as the structural description information of the area; comparing and analyzing the directional histograms corresponding to multiple feature points, and when the similarity between the directional histograms is greater than the similarity threshold and their spatial positions are periodically arranged and distributed, they are identified as repetitive texture areas; setting an inhibition factor for all feature points participating in the matching process in the area to reduce the influence of repeated structural feature points in the feature matching process and improve the discrimination accuracy of feature matching.

[0008] In a preferred embodiment, after completing the identification of repetitive texture areas, in order to control the participation intensity of feature points in the area in feature matching, a corresponding inhibition factor is generated as a score adjustment parameter; the value of the inhibition factor is obtained by inputting the structural repetition calculation result between the context direction histograms of the feature points in the area into the attenuation function, and the attenuation function is a monotonically decreasing function, the input variable is the similarity score between the direction histograms, and the output result is the normalization factor value, which is multiplied by the original feature point confidence to update the matching score, that is, to adjust the participation matching weight; when the similarity between the direction histograms is close to the maximum similarity, the inhibition factor approaches the set minimum value, thereby reducing the influence weight of this type of feature point on the estimation of the registration parameters in the feature matching stage.

[0009] In a preferred embodiment, the sparse optical flow is calculated as follows: between consecutive image frames, the high-density feature points extracted in the second step are used as tracking targets, and the inter-frame displacement of each feature point on the two-dimensional coordinate plane of the image is calculated. The displacement is established based on the constant grayscale assumption of the image, and is minimized by the local photometric consistency criterion to obtain the motion vector of the feature point between consecutive frames; The process of extracting the non-rigid displacement vector set includes: comparing the displacement vector of each feature point with the displacement vectors of its adjacent feature points. If the direction difference is greater than the direction deviation angle threshold and the speed difference exceeds the motion consistency threshold, it is marked as a motion inconsistent point; The set of all motion inconsistent points is processed through a preset region expansion algorithm to generate a foreground dynamic occlusion region mask. The remaining unmarked area is defined as the background static region. Only the feature points in the background static region are retained for the subsequent estimation of the view transformation parameters to eliminate the interference of the foreground dynamic occlusion region on the registration result. The perspective transformation parameter estimation is as follows: perform feature point matching on the feature points retained in the background static area, and establish a candidate set of geometric transformation models based on their coordinate relationship in adjacent image frames, including affine transformation model and homography transformation model; use the least squares error fitting method to solve the parameters of each model, and calculate the reprojection error for each model as the model adaptation score; in the model scoring, select the transformation model with the smallest reprojection error that is less than the set error tolerance threshold as the final perspective transformation parameter solution.

[0010] In a preferred embodiment, constructing a multi-frame image residual sequence within the time window in the fourth step includes: taking the current image frame as a starting point, selecting several preceding image frames as a reference time window, and calculating the Euclidean distance between the predicted position and the actual position of the feature points extracted in the second step in each frame to form a feature point residual sequence; The changing trend of the sequence in the time dimension is extracted as the characteristic drift trend vector; Obtain the pose prediction results of image information, inertial measurement information, and positioning information in the current frame respectively, compare them with the feature drift trend, and calculate the spatiotemporal residual values ​​of the three data sources; The three types of residual values ​​are input into a preset confidence distribution function. The confidence distribution function has a fusion weight allocation mapping relationship, which is based on a class of functions in the monotonically decreasing function family. Its input is the residual value and its output is the normalized weight factor. The three weight factors are used as the participation proportions of the corresponding data sources to calculate the splicing pose solution of the current frame, thereby realizing adaptive fusion control based on residual driving.

[0011] In a preferred embodiment, the processing of the image frame data after the perspective alignment in the fifth step includes the following operations: performing edge smoothing based on Gaussian weighted convolution in the image boundary area to reduce the intensity mutation at the image splicing boundary; The pixel values ​​in the brightness overlap area between image frames are distributed statistically, and local brightness is harmonized through histogram equalization combined with dynamic gamma adjustment. The mean difference between the three-channel color data in the image blocks is sampled and calculated to generate color offset correction parameters, which are used to execute the color mapping function to achieve color consistency adjustment. After completing the output of the current frame image, a status assessment is performed based on the stitching residual trend and image structure integrity indicators. If there is a trend of decreased stability or increased feature drift, the fusion status refresh process is triggered and the fusion weight of the next frame is reset to ensure the stability and continuity of the subsequent image frame stitching process.

[0012] In a preferred embodiment, a real-time panoramic image stitching system based on multi-sensor fusion includes: The time sequence perception module is used to synchronously acquire image frame data through multiple image acquisition devices at different spatial positions, fuse the attitude data provided by the inertial measurement device with the position information provided by the positioning device, align the multi-source data based on a unified timestamp, and eliminate data segments with time delays exceeding a preset threshold to form a structured fused data sequence; The texture parsing module is used to extract high-density feature points from areas with grayscale value changes in the image frame, and construct the context structure of the feature points through the local texture direction histogram of the image block, identify image areas with repetitive texture features, and reduce the confidence level of the feature points in the matching process in this area; The dynamic segmentation module is used to extract a set of non-rigid displacement vectors based on sparse optical flow calculations between consecutive image frames, identify foreground dynamic occlusion areas and background static areas, and after completing the region segmentation, perform geometric transformation model fitting only on the feature points of the background static area to estimate the perspective transformation parameters of the current frame; The fusion solution module is used to construct a multi-frame image residual sequence within a time window. Based on the historical stitching path and the characteristic drift trend of the current frame image, the spatiotemporal residual values ​​of the three data sources of image, inertial, and positioning are calculated, and converted into participation proportions through a preset fusion weight distribution function to achieve adaptive solution of the stitching pose. The image fusion module is used to align the image perspective according to the stitching pose of the current frame, and perform smoothing of the image edge areas, blending of brightness overlapping areas, and color consistency correction. At the same time, the fusion stability is judged based on the stitching residual trend and image structure evaluation indicators. When the fusion status refresh is triggered, the initial fusion configuration of subsequent frames is updated.

[0013] The technical effects and advantages of the present invention are as follows: The present invention uses multiple spatially distinct image acquisition devices to synchronously acquire image frame data. This data is then combined with the attitude data provided by an inertial measurement unit (IMU) and the position information output by a positioning device. This method aligns the multi-source data using a unified timestamp mechanism, eliminating data segments with time delays exceeding a preset threshold to effectively construct a structured fused data sequence. This mechanism significantly improves the synchronization accuracy of multi-source data in the temporal dimension, avoiding spatial misalignment caused by acquisition lags or data drift. This provides a stable, continuous, and consistent data foundation for subsequent image registration and fusion operations, enhancing the overall reliability of panoramic stitching.

[0014] During the feature extraction phase, the present invention extracts high-density feature points from areas of grayscale variation within the image frame. Regional feature contexts are constructed based on the local texture orientation histograms of the image blocks. The spatial distribution patterns of the feature points are then combined to identify regions with repetitive textures and reduce the weight of these regional features in the matching process. This process helps mitigate mismatches caused by periodic or structurally repetitive patterns, enhancing matching discrimination without sacrificing the number of feature points. This improves the accuracy and stability of image pairing and provides higher-quality input features for accurate estimation of perspective transformation parameters.

[0015] After image alignment, the present invention effectively addresses the problems of sudden changes in boundaries, uneven brightness, and inconsistent tones that may occur after image stitching by performing smoothing of image edge areas, blending of areas with overlapping brightness, and color consistency adjustment. Furthermore, after generating a continuous panoramic image output, the system refreshes the fusion state in real time to address changes in feature drift trends and ensure that subsequent image frames maintain state coherence and adaptive processing strategies during the fusion process. This technical approach significantly enhances the natural expressiveness of the stitched image in terms of visual coherence, improving the display quality and stitching stability of the entire panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings; Figure 1 This is a schematic diagram of the real-time panoramic image stitching algorithm based on multi-sensor fusion in the present invention.

[0017] Figure 2 This is a schematic diagram of the real-time panoramic image stitching system based on multi-sensor fusion in the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] Reference Figure 1 - Figure 2 The following examples were obtained: Example 1: A real-time panoramic image stitching algorithm based on multi-sensor fusion, comprising the following steps: The first step is to synchronously acquire and align multi-source data: by setting up multiple image acquisition devices at different spatial positions, image frame data of the same scene can be acquired from different perspectives, which can be used to subsequently construct a panoramic image with a wide field of view. At the same time, in order to improve the accuracy of image stitching, it is also necessary to collect the attitude data (such as angular velocity and acceleration) output by the inertial measurement device and the geographic location information output by the positioning device in real time, so as to obtain the motion state of the equipment in the scene in time and space. Since the operating frequencies and sampling times of multi-source data acquisition devices are different, all data must be time-series aligned based on a unified timestamp, and data segments with time delays exceeding the set synchronization threshold must be eliminated to ensure time synchronization and spatial correspondence between various types of information. Finally, the aligned multi-source information is organized into a structured fusion data sequence to provide a unified input basis for subsequent fusion processing and stitching calculations.

[0020] The second step is image feature extraction and redundant texture reduction: In the acquired image frames, there are a large number of areas with significant changes in grayscale values. These areas often contain rich feature information such as edges and corners. In order to improve the accuracy of image registration, this step performs high-density feature point extraction on such areas to generate a set of key points for matching between images. Subsequently, based on the local texture direction distribution of the image block where each feature point is located, a direction histogram of the image block is constructed, and on this basis, the contextual structure information of the feature point is formed. In order to avoid feature matching confusion caused by repetitive structures (such as grids, curtain walls, and bricks), this step further combines the spatial arrangement pattern of feature points in the image to identify areas with periodic texture features, and introduces a confidence suppression mechanism for the feature points in such areas to reduce their participation weight in subsequent feature matching, thereby improving matching robustness and accuracy.

[0021] The third step: foreground dynamic stripping and background perspective estimation: Since there are often dynamic foreground objects such as pedestrians, vehicles, and robotic arms in real scenes, their movement may cause error interference to image stitching. Therefore, this step uses a sparse optical flow algorithm to track the feature points between image frames and analyze their motion trajectories on the image plane. By comparing the displacement direction and speed differences between adjacent feature points, a set of feature points with inconsistent motion can be extracted to form a non-rigid displacement vector set. Furthermore, a region expansion method is used to expand such feature points to form a foreground dynamic occlusion region, and separate it from the unmarked background static region. Subsequently, only the feature points in the background region are retained to estimate the perspective transformation parameters between the current image frame and the reference frame, which usually includes an affine or homography model, so that the pose estimation is not affected by dynamic interference, thereby improving the stability and accuracy of geometric alignment.

[0022] Step 4: Multi-source fusion optimization and adaptive stitching pose solution: To achieve accurate and dynamically adaptive image stitching, this step constructs a historical residual sequence of multi-frame image data within a time window, extracting the feature drift trend of the current image frame. Combined with the previous stitching trajectory, this step forms a priori estimates of the current motion state. Simultaneously, the spatiotemporal residuals of image features, inertial data, and positioning information are calculated for the current frame to measure the reliability of these three data types in the pose estimation. These residual values ​​are fed into a set of preset fusion weight mapping functions to generate a normalized confidence factor, automatically adjusting the contribution of each data source to the final pose estimate. This adaptive update mechanism for fusion parameters effectively prevents misleading stitching results caused by environmental interference (such as IMU drift, GPS loss, or image blur) from a particular data source, resulting in a more robust optimal stitching pose solution.

[0023] Step 5: Image fusion output and state refresh control: After obtaining the stitching pose of the current frame, this step applies it to the geometric transformation operation of the image frame to align the image perspective and then perform the image fusion operation. To ensure the visual continuity and overall quality of the stitching result, the system performs edge smoothing on the connecting transition areas at the edge of the image to alleviate the boundary mutation problem caused by the change in perspective; the brightness histogram equalization and gamma adjustment strategy are used for the brightness overlapping areas in multiple frames to achieve local brightness coordination; at the same time, the offset between the image color channels is normalized and corrected to ensure the naturalness of the final output image in terms of color consistency. Finally, the stability of the current fusion state is evaluated based on indicators such as the stitching residual trend and image structure integrity. If the fusion quality is detected to be degraded or the drift trend is enhanced, the fusion parameter refresh mechanism is triggered to prepare the updated weight initial value and feature configuration for the next frame stitching, so as to achieve the continuity and stability of the stitching process.

[0024] In multi-sensor fusion systems, different data acquisition devices can experience sampling frequency discrepancies and sampling delays. This is particularly true when an image acquisition device, inertial measurement unit, and positioning system are used together. The timestamps generated by these devices may not be strictly aligned. If used directly for fusion without processing, this can easily lead to spatial misalignment or temporal drift in the stitched image. To address this issue, the system implements a temporal residual determination method before multi-source data fusion. This method extracts the timestamp information from data from multiple devices within a given sampling period and calculates the time interval error between each data source. For example, assuming the image frame timestamp is T1, the IMU data timestamp is T2, and the GPS data timestamp is T3, the temporal residual at that sampling moment is defined as: ΔT = max{|T1-T2|,|T1-T3|,|T2-T3|}. The system sets a preset synchronization accuracy threshold, for example, 20 milliseconds. If ΔT exceeds this threshold, the multi-source data at that moment is deemed to have unacceptable time offset, and all data at that moment is discarded to prevent it from participating in subsequent registration and fusion calculations. To maintain data structure continuity, the system further sets a maximum expected inter-frame value (e.g., 50 milliseconds) to determine the interval between image frames. If the effective time difference between two consecutive image frames exceeds this expected value, it indicates that data is missing. To prevent gaps in the stitched trajectory, the system generates a reference pose using an interpolation algorithm. This reference pose can be obtained by linearly interpolating the transformation parameters of the two preceding and succeeding valid image frames, or by predicting it using inertial measurement data.

[0025] The reference pose generation process uses an interpolation algorithm, which can be implemented in the following two ways: Linear interpolation based on image frame transformation parameters: When the system detects that the interframe interval between two consecutive valid image frames exceeds the maximum expected interframe interval, it first extracts the pose transformation parameters of the two frames. These parameters are generally calculated by matching image feature points and can include translation and rotation angles. The system represents the pose parameters of the two frames as vector sets of spatial position and orientation, respectively, and performs element-by-element linear interpolation between them. Specifically, for the translation parameter, the difference between the two position vectors is taken and distributed proportionally to the time interval. For the rotation angle parameter, the angle is converted into three axial rotations using the Euler angle method, and then a weighted average is performed. Through this interpolation process, the system obtains an intermediate pose that is between the two valid frames in terms of temporal position and smoothly transitions in terms of spatial attitude. This pose is then inserted into the fused data sequence as the interpolation reference pose.

[0026] Prediction interpolation method combined with inertial measurement data: In some scenarios, if there is a long time interval between image frames, or the image quality cannot support stable feature matching, the system can also use inertial measurement data for pose interpolation prediction. This method is based on the continuous angular velocity and acceleration information provided by the inertial measurement device, and calculates the rotation change and position change of the device within a specified time period through integration. In specific implementation, the system accumulates and integrates the acceleration and angular velocity information after the time point of the previous frame's valid image to obtain the change in motion state over a period of time, and applies it to the known pose of the previous frame image to estimate the reference position and orientation of the predicted frame. The estimated result is the reference pose, which is used to fill in the gaps in the data structure.

[0027] Regardless of which approach is used, the generated reference pose must maintain geometric continuity with the preceding and following frames in terms of spatial structure, ensuring that the image stitching path does not experience abrupt changes due to data loss, thereby improving the robustness and temporal stability of the overall stitching system. Inserting the reference pose repairs the previously gapped structured fusion data sequence, resulting in smooth continuity in the temporal dimension, which helps improve image stitching stability and the continuous optimization capabilities of the subsequent adaptive fusion process.

[0028] To achieve high-precision perspective registration between images, it is necessary to extract a sufficient number of evenly distributed, highly distinguishable local feature points from the image frames as the basis for subsequent image matching and geometric transformation estimation. To this end, the system uses a high-density feature point extraction method that targets areas of image grayscale variation and extracts locations with significant grayscale gradient abrupt changes as candidate features.

[0029] Specifically, for each pixel in a frame of image, its first-order gradient response value in the horizontal and vertical directions is calculated respectively, that is, the grayscale difference between the left / right pixel and the upper / lower pixel respectively. These gradient response values ​​are used to indicate the severity of the grayscale change at the pixel. The gradient response value of the entire image is constructed as a gradient intensity map, in which each pixel value represents the edge response intensity of the point. In the gradient intensity map, the system selects the local maximum point as the initial feature point candidate set. The so-called local maximum means that the response value of the point is higher than all other pixels within a certain neighborhood range. For example, a 3×3 pixel window can be set, and the center point in the window is selected as the local extreme value.

[0030] Taking into account the possibility of excessive clustering of feature points (i.e., multiple feature points concentrated in a local area), the system further performs non-maximum suppression. When the pixel distance between two candidate points is less than the set minimum spacing (e.g., 5 pixels), the system retains the one with the higher response value, suppresses repeated responses, and improves the uniformity of spatial distribution. Ultimately, the system obtains a set of feature points with global distribution coverage, significant response values, and low repetition rate. This set is a high-density feature point set, which not only has sufficient local detail information but also supports subsequent feature context construction, texture recognition, and matching optimization. It is one of the most critical data structures in the entire image stitching chain.

[0031] After the system extracts a set of high-density feature points, in order to improve the semantic recognition ability of the feature points and the accuracy of subsequent feature matching, the system performs a context construction process on each high-density feature point. The context construction process means that the system not only analyzes the local grayscale characteristics of the feature point itself, but also considers the structure, texture and direction information of the area where it is located, so as to obtain more discriminative structural description features. With the current feature point as the center, the system cuts out a fixed-size image block area from the image. The size of this area is generally set to a square image area with a side length of eleven or thirteen pixels centered on the feature point. The size can be adjusted according to the image resolution and processing power. In this image block, the system calculates the gradient direction for each pixel, that is, the direction of the grayscale change in the neighborhood around the pixel is used to represent the edge direction of the pixel.

[0032] After counting the gradient directions of all pixels, the system constructs a directional histogram using multiple directional segments (for example, eight directions, each separated by 45 degrees) as coordinate axes. This histogram records the number of gradient pixels in each direction within the image block. To eliminate the influence of differences in pixel brightness or size between different image blocks, the system normalizes the histogram so that its value range falls between zero and one. This produces a structural description that is independent of absolute gradients and reflects only the directional distribution. After completing the contextual description of all high-density feature points, the system compares the directional histograms corresponding to these feature points pairwise to analyze their similarity. The similarity of directional histograms can be calculated by calculating the cross-similarity between two normalized histograms. Common calculation methods include cosine similarity or histogram cross ratio. When the directional histogram similarity between two or more feature points exceeds a preset similarity threshold (for example, 85%), and these feature points exhibit a periodic arrangement in spatial position within the image (such as repeated occurrence within the same line or grid), the system determines that these feature points belong to a repetitive texture region.

[0033] Repetitive texture regions refer to images containing numerous patterns with similar shapes, orientations, and structures. These are commonly found in scenes such as grids, window frames, curtain walls, and brick floors. These regions significantly increase feature point mismatches, impacting registration accuracy. To address this issue, the system introduces a suppression factor into the matching process for feature points identified as repetitive texture regions. The suppression factor is a scaling parameter that reduces the weight of these feature points in the feature matching phase. Its value is smaller than the standard matching weight, limiting the ability of these feature points to interfere with the feature matching results. For example, the system sets the standard matching confidence to one and sets a suppression value between 0.6 and 0.3 for feature points within repetitive regions, determined by the directional histogram similarity and repetition density of the region. By introducing the suppression factor, the system effectively weakens the dominant weight of feature points within repetitive regions during the matching process, thereby increasing the matching priority of features within non-repetitive regions and significantly improving the discrimination accuracy of feature matching. This, in turn, enhances the accuracy and stability of geometric transformation parameter estimation between images.

[0034] In image processing systems, feature point matching is a crucial process for determining image stitching accuracy. To minimize the impact of repetitive texture regions on feature matching accuracy, after identifying repetitive texture regions in an image, the system must further control the degree to which each feature point in that region participates in the matching process. This step achieves this goal by generating a set of numerical adjustment parameters, known as suppression factors.

[0035] The suppression factor is a coefficient used to adjust the matching score of a feature point. Its function is to reduce the weight of the feature point in the scoring algorithm, thereby reducing the influence of the point on the candidate matching results in the matching algorithm. Each feature point in the repetitive texture area is assigned a separate suppression factor.

[0036] The generation mechanism of the suppression factor is based on the calculation of the structural repetition between multiple feature points in the area, and the structural repetition refers to the degree of similarity between the contextual directional histograms corresponding to these feature points. The directional histogram, as a description method of the contextual structure of the feature point, has been disclosed in the previous article. The comparison result of multiple directional histograms will be input into an attenuation function as a similarity score. The attenuation function is defined in this technical solution as a type of monotonically decreasing function with the following properties: its input is a similarity score with a value range of zero to one; its output is a normalization coefficient, also between zero and one. The higher the input similarity, the lower the output normalization coefficient; conversely, when the similarity is low, the output coefficient maintains a higher value. The suppression factor is only used to adjust the matching score in the feature matching stage, and does not change the state of the feature point participating in motion estimation and mask generation.

[0037] In actual implementation, the normalization factor is multiplied by the original matching confidence of the feature point to form a new scoring factor, which serves as the effective weight of the feature point when participating in the matching calculation. For example, a feature point is assigned a standard matching confidence of one in the initial extraction stage. If its directional histogram similarity with adjacent points is 90%, the normalization factor output by the attenuation function may be 0.35. At this time, the matching score of the feature point will be compressed to 35% of the original confidence, significantly reducing its influence in the matching process. This method can be dynamically adjusted according to the density and structural complexity of the repeated texture area in the specific image. In particular, in the context of building facades, grid structures or repeated stripes, such feature points have a high probability of mismatching. Through the quantitative control mechanism of the suppression factor, the system's ability to distinguish the quality of feature point matching can be significantly improved.

[0038] When the similarity between the directional histograms approaches a predefined maximum similarity value (e.g., exceeding 95%), the system will reduce the corresponding suppression factor to a set minimum value, such as 0.2 or lower. This ensures that feature points in highly repetitive regions no longer dominate the matching results, effectively preventing spurious matching points from entering the registration parameter estimation model, ultimately improving the robustness of the geometric transformation estimation between images and the overall stitching accuracy.

[0039] Sparse optical flow calculation involves tracking the high-density feature points extracted in the second step between consecutive image frames and calculating the inter-frame displacement of each feature point on the image's two-dimensional coordinate plane. Sparse optical flow involves calculating the spatial positional trajectory of only a subset of pixels with significant texture or edge characteristics, namely the high-density feature points extracted previously, across consecutive frames. This avoids the computational burden of pixel-by-pixel optical flow estimation for the entire image. This displacement is established based on the assumption of constant image grayscale, assuming that the grayscale value of a point in the image remains constant between consecutive frames due to the short time interval between frames. Based on this assumption, the system uses the local photometric consistency criterion to minimize the inter-frame displacement vector for that point by finding the corresponding point within the feature point's neighborhood that minimizes the grayscale difference. Ultimately, the system obtains a set of motion vectors for each feature point from the current frame to the next, representing the direction and velocity of the feature point's movement on the image coordinate plane.

[0040] The process of extracting a set of non-rigid displacement vectors includes: comparing the displacement vector of each feature point with the displacement vectors of its surrounding adjacent feature points. If the direction difference is greater than the direction deviation angle threshold and the speed difference exceeds the motion consistency threshold, it is marked as a motion inconsistent point. "Non-rigid" in this implementation means that the movement direction and speed of the point in the area deviate significantly from its neighborhood, indicating that it is not a rigid area that moves with the background as a whole, but a possible foreground object with independent movement. The "direction deviation angle threshold" can be set between thirty and forty-five degrees; the "speed difference threshold" can be set to a movement amount between three pixels and five pixels in image resolution units. Once it is detected that the movement direction of a feature point deviates significantly from the average direction of the neighboring points, and the movement speed is also significantly different, the system will mark it as a motion inconsistent point and deem it to be a possible dynamic foreground component.

[0041] The set of all motion-inconsistent points is then processed through a preset region expansion algorithm to generate a foreground dynamic occlusion region mask. Region expansion can be based on connectivity judgment and edge closure strategies, where scattered dynamic points are aggregated to form closed regions. For example, pixel connectivity judgment is performed based on a four-neighborhood or eight-neighborhood approach, while expanding to adjacent weakly responsive edges to form occlusion regions. Ultimately, a mask region is formed, representing the image region that may be occluded by dynamic targets in the current frame. The system defines the remaining unmarked regions as background static regions, meaning that the motion direction and speed of the feature points in these regions are highly consistent, and the credibility is high.

[0042] Only feature points within the static background region are retained for subsequent view transformation parameter estimation to eliminate interference from dynamic foreground occlusions. View transformation parameter estimation involves performing feature point matching on the feature points retained within the static background region and establishing a set of candidate geometric transformation models based on their coordinate relationships in adjacent image frames. These models include affine and homography models. Affine models are suitable for scenes with small-scale camera motion or planar structures, while homography models are used to describe structural transformations in scenes with perspective changes. The system uses a least-squares error fitting method to solve the parameters of each of these models. This involves optimizing the mapping error between matched point pairs to determine the optimal transformation matrix. To ensure robustness of the estimated model, the system calculates the reprojection error for each candidate model. This is the distance error between the actual observed point and the reference frame after the points are mapped to their locations in the target frame using the transformation model. Finally, among all the scored models, the transformation model with the smallest reprojection error that is less than a set error tolerance threshold is selected as the final view transformation parameter solution for the current image frame and used for image alignment.

[0043] To further improve fusion accuracy and robustness to multi-source information misalignment during real-time image sequence stitching, the system implements a residual-driven fusion optimization mechanism in the fourth step. This mechanism first constructs a multi-frame image residual sequence within a time window. Specifically, starting with the current image frame, the system selects a finite-length reference time window consisting of multiple consecutive image frames preceding it. This window length can be set to commonly used values ​​such as three, five, or seven frames, depending on the frame rate, processing latency, or system task requirements.

[0044] In this time window, the system performs the following operations on the high-density feature points extracted in the second step in each frame: Based on the system's historical splicing path and the current frame's perspective transformation trend, it predicts the position where each feature point should appear in the current frame, and calculates the difference with the actual detected position of the point in the current image frame. The difference is expressed as the Euclidean distance on the two-dimensional coordinate plane of the image, indicating the geometric offset between the prediction and the actual position of each feature point. The offsets of all the above feature points in consecutive frames are recorded in chronological order, and the system forms a feature point residual sequence to describe the stability or drift degree of the local structure of the image in the time dimension. The system calculates the changing trend of the residual sequence in the time dimension as a feature drift trend vector, which can be expressed as the temporal change slope or fluctuation range of the average residual of each feature point, representing the current image structure stability and potential drift rate. In the current image frame, the system obtains pose prediction results from three types of data sources, including: image information: obtained by solving the feature point geometric matching and perspective transformation model in the third step; inertial measurement information: derived from the integral of angular velocity and acceleration output by the inertial measurement device; positioning information: the coordinate data provided by the positioning device is used to calculate the device position and orientation.

[0045] The system compares and analyzes the predicted pose of each data source with the feature drift trend vector, and calculates the image residual value between the image prediction and the drift trend, the posture residual value between the inertial prediction and the drift trend, and the positioning residual value between the positioning prediction and the drift trend. These three types of residual values ​​together constitute the multimodal residual vector under the current frame. The system inputs the three types of residual values ​​into a preset confidence distribution function, which is a type of mapping relationship used to convert the residual values ​​into corresponding information source participation weights. This function structure is an instance function of a class of monotonically decreasing functions with the following characteristics: when the residual value is small, the output confidence weight is large, indicating that the prediction result of the data source is more reliable; when the residual value becomes larger, the output confidence weight decreases, indicating that the contribution of the data source to the pose of the current frame needs to be weakened.

[0046] For example, when the image residual value is within two pixels, the fusion weight factor of the image information can approach a maximum value, such as 0.9; if the positioning residual reaches ten meters, the weight may drop below 0.3. This method can dynamically control the participation of multimodal data in different frames. Ultimately, the system assigns these three confidence weight factors to image information, inertial measurement information, and positioning information, respectively, and solves the spliced ​​pose solution in the current frame pose estimation using a weighted fusion method. That is, the predicted poses of the three types of information sources are fused by weighted average according to their respective confidence levels to obtain the optimal pose solution for the current frame. This mechanism effectively implements fusion control driven by data reliability, enabling the system to dynamically adjust the fusion strategy under abnormal conditions such as image blur, inertial drift, and positioning offset, thereby improving the overall splicing robustness and spatiotemporal consistency.

[0047] After the first four steps have completed perspective transformation estimation and image frame geometric alignment, the system enters the fifth step, which is to post-process the image frame data that has completed perspective alignment to improve the quality performance of the panoramic image in terms of visual continuity, lighting smoothness, and color consistency. The perspective-aligned image frame refers to the stitching pose (i.e., transformation matrix parameters) estimated in the fourth step. The current image frame is mapped to the target panoramic image reference coordinate system so that multiple image frames are overlapped in spatial position. On this basis, the system further enhances the visual quality of the image content. The processing flow is as follows: Image edge smoothing: During the image stitching process, there are often problems such as grayscale discontinuity and texture mutation at the boundary connections between image frames, which manifest as visible stitching lines or structural jumps. To this end, the system performs edge smoothing based on Gaussian weighted convolution in the image boundary area. The so-called Gaussian weighted convolution means: setting a sliding window around the stitching area, and performing weighted averaging of the pixels in the window according to the Gaussian distribution weights of the distance from the center point. After the edge pixels are smoothed by multiple convolutions, their grayscale values ​​are slowly transitioned to the adjacent image area, thereby reducing visual mutations. For example, at the splicing seam of two image frames, the system can set a five by five pixel window and perform two convolutions using a Gaussian kernel with a standard deviation of 1.5. The convolution result retains the edge texture while eliminating boundary jumps and improving the naturalness of the fusion.

[0048] Blending of overlapping brightness areas: Due to the different exposure conditions of each image acquisition device, even if the same scene is captured, there may be differences in brightness between images, especially when the stitching area overlaps. To solve this problem, the system performs distribution statistics on the pixel values ​​in the brightness overlapping area between image frames. Specifically, the system extracts the pixel grayscale distribution of the overlapping parts in the stitching area of ​​the two image frames, constructs histograms for each, and then performs a histogram equalization operation to stretch the grayscale distribution of each image frame to a uniform range. At the same time, combined with the dynamic gamma adjustment method, the pixel response curve is dynamically adjusted for the phenomenon that the overall image brightness is too bright or too dark, so as to achieve balanced and unified local brightness. For example, if the overlapping area of ​​the left image is brighter and the right image is darker, the system will perform a darkening operation with a gamma value less than one on the left image and a brightening operation with a gamma value greater than one on the right image, so that the median brightness of the two images tends to be consistent.

[0049] After balancing the brightness of the stitched image areas, the system performs color offset correction and mapping to further improve the overall color consistency of the multi-source images. The system performs color sampling on the overlapping areas between the image frames, dividing the stitched image area into several image blocks according to a set step size. Within each block, the pixel mean values ​​of the three color channels (red, green, and blue) are calculated. The difference between the mean values ​​of the three channels and the corresponding image blocks in the adjacent frames is calculated, defining this difference as the color offset. Based on this offset, the system generates a set of color offset correction parameters, which can include linear gain and offset factors for each color channel. For example, if the red channel mean value is 20 grayscale levels higher, the system can set the red channel offset correction parameter to minus 20, while retaining the original gain at 1, indicating a translational mapping compression of the red component. The system then calls a preset color mapping function, a channel-by-channel mapping model that takes the pixel values ​​of the original image in the three channels as input and outputs the pixel values ​​of the adjusted image. The mapping method is controlled by the color offset correction parameters. The function execution method includes the following operations: performing a linear transformation of "first subtracting the offset and then multiplying by the gain" on the red, green, and blue channels of each pixel; or using a lookup table to map the original pixel value to a standardized color distribution curve to improve color naturalness.

[0050] For example, if the average value of the blue channel of the original image is significantly low and unsaturated, the system can set the blue channel offset to plus twenty-five and the gain coefficient to one point one, thereby improving the overall blue level and enhancing the visual expression. The above color mapping operation can be performed at the image block level or uniformly across the entire image. The system can set the execution range based on the difference in color offset distribution in the stitching area. After the mapping is completed, the image frame maintains a unified trend in color distribution with the adjacent image frames, and there is no obvious color level mutation in the stitching area, thereby achieving the goal of adjusting the tonal consistency of the panoramic image.

[0051] Before performing color shift correction, the system first samples the overlapping area between the current frame and the reference frame to obtain the average difference of each color channel. The sampled red, green, and blue channel means are set as the mean difference between the current image frame and the reference image frame, respectively. This difference is used to calculate the color correction parameters for the current image frame.

[0052] The linear gain factor is a factor that linearly amplifies or compresses the pixel values ​​of a color channel in an image. It is often used to enhance or weaken the perceived brightness of that channel. Setting the gain factor greater than one brightens the corresponding channel; setting it less than one darkens the channel. It is often used to enhance low-contrast or dark channels.

[0053] During color adjustment, the offset factor is a constant term that uniformly increases or decreases the values ​​of all pixels in an image to shift the image channel mean to the target reference value. A positive offset increases the overall brightness of a channel, while a negative offset suppresses overly bright channels. It is often used to correct for differences in channel mean values ​​between images.

[0054] The present invention sets the linear gain coefficient and the offset coefficient according to the following strategy: Offset coefficient setting rules: For each color channel, set the offset coefficient equal to the difference in pixel mean between it and the reference image frame; if the difference is positive, the channel as a whole will be lowered by that value; if the difference is negative, the channel as a whole will be raised by that value; example: if the mean value of the red channel in the current frame is 160 and that of the reference frame is 140, then the offset coefficient is negative 20, indicating that the channel as a whole needs to be darkened by 20 gray levels.

[0055] Linear gain coefficient setting rules: When the channel contrast is obviously insufficient (that is, the standard deviation is lower than the set threshold, such as 10), the system sets its gain coefficient to a value greater than 1; when the channel is overexposed or saturated (that is, the pixel values ​​are concentrated in the high grayscale area), the system sets its gain to less than 1. For example: if the blue channel brightness is low and the standard deviation is 8, the system sets the gain to 1.25 to make the color performance more prominent.

[0056] The final mapping function of each color channel is as follows: Correction value = original value × linear gain coefficient + offset coefficient; after correction, pixel values ​​outside the range of zero to two hundred and fifty-five will be clipped to the valid range to ensure normal image display.

[0057] After completing the current frame's image output, the system comprehensively evaluates the current stitching status based on the stitching residual trend and image structural integrity indicators. The stitching residual trend refers to whether the feature point registration error between the current image frame and the reference frame is on an upward trend. Image structural integrity indicators include multiple preset local structural distribution indicators, such as edge coherence and image block texture consistency. If the system detects an increasing stitching error or a disruption of image structural continuity, such as increased feature drift or broken edges, it triggers a fusion state refresh process. This fusion state refresh proactively resets the initial values ​​of the next frame's fusion parameters, including but not limited to the feature point confidence distribution and fusion weight coefficients, so that the next frame can be fused from the updated state, avoiding error accumulation. Through this series of processing operations, the system ultimately generates a continuous panoramic image output that is not only structurally accurately aligned but also visually coherent in terms of brightness, color, and edges. This ensures the stability and controllability of the subsequent frame stitching process in terms of both visual quality and pose estimation.

[0058] Example 2: A real-time panoramic image stitching system based on multi-sensor fusion, comprising: The time sequence perception module is used to synchronously acquire image frame data through multiple image acquisition devices at different spatial positions, fuse the attitude data provided by the inertial measurement device with the position information provided by the positioning device, align the multi-source data based on a unified timestamp, and eliminate data segments with time delays exceeding a preset threshold to form a structured fused data sequence; The texture parsing module is used to extract high-density feature points from areas with grayscale value changes in the image frame, and construct the context structure of the feature points through the local texture direction histogram of the image block, identify image areas with repetitive texture features, and reduce the confidence level of the feature points in the matching process in this area; The dynamic segmentation module is used to extract a set of non-rigid displacement vectors based on sparse optical flow calculations between consecutive image frames, identify foreground dynamic occlusion areas and background static areas, and after completing the region segmentation, perform geometric transformation model fitting only on the feature points of the background static area to estimate the perspective transformation parameters of the current frame; The fusion solution module is used to construct a multi-frame image residual sequence within a time window. Based on the historical stitching path and the characteristic drift trend of the current frame image, the spatiotemporal residual values ​​of the three data sources of image, inertial, and positioning are calculated, and converted into participation proportions through a preset fusion weight distribution function to achieve adaptive solution of the stitching pose. The image fusion module is used to align the image perspective according to the stitching pose of the current frame, and perform smoothing of the image edge areas, blending of brightness overlapping areas, and color consistency correction. At the same time, it judges the fusion stability based on the stitching residual trend and image structure evaluation indicators, triggers the fusion status refresh, and updates the initial fusion configuration of subsequent frames.

[0059] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0060] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0061] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0062] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0063] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A real-time panoramic image stitching algorithm based on multi-sensor fusion, characterized by: The following steps are involved: The first step is to synchronously acquire image frame data through multiple image acquisition devices at different spatial positions, combine the posture data provided by the inertial measurement device with the position information output by the positioning device, align the multi-source data according to the timestamp, and eliminate data segments with time delays exceeding a preset threshold to form a structured fusion data sequence; The second step is to extract high-density feature points from the grayscale value change areas in the image frame, and use the local texture direction histogram of the image block to build the regional feature context. On this basis, each feature point is associated with the structural characteristics of the region where it is located, and the repetitive texture region is identified based on the spatial distribution pattern, reducing the participation weight of such regional features in the matching; The third step is to extract a set of non-rigid displacement vectors by calculating sparse optical flow between consecutive image frames, separate the foreground dynamic occlusion area and the background static area on the image plane, and estimate the perspective transformation parameters only for the static area to eliminate the influence of the foreground area on the registration process; The fourth step is to construct a multi-frame image residual sequence within the time window. Based on the historical fusion path and the feature drift trend of the current frame image, the spatiotemporal residual values ​​of the corresponding position information of different sensor sources are calculated. The fusion parameters are adaptively updated according to the confidence distribution function to obtain the optimal splicing pose solution. In the fifth step, according to the estimated stitching pose, the image frame data after perspective alignment is smoothed, the brightness overlapping area is blended, and the color consistency is adjusted to generate a continuous panoramic image output. The fusion status is refreshed in real time to ensure the stability of the subsequent frame stitching process.

2. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 1 is characterized in that: When fusing data collected by multiple image acquisition devices with corresponding inertial measurement devices and positioning devices, a time series residual discrimination method is used to filter out asynchronous data. Specifically, the time interval error between the timestamp sequences of each data source is calculated. When the error exceeds the set synchronization accuracy threshold, all data at that moment are removed from the structured fusion data sequence. To maintain data continuity, when the time difference between two consecutive valid image frames exceeds the maximum inter-frame expected value, the reference pose derived from the two frames of data is inserted into the data structure, and finally a structured fusion data sequence is obtained.

3. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 2 is characterized in that: The process of high-density feature point extraction includes: calculating the first-order gradient response values ​​in the horizontal and vertical directions for each pixel point in the image frame to construct a gradient intensity map; in the gradient intensity map, selecting the local maximum point as the initial feature point candidate set, and performing non-maximum suppression processing on the feature points in the candidate set whose adjacent distance is less than the set pixel spacing, retaining the ones with the largest response intensity as valid feature points; finally, a high-density feature point set that is evenly distributed in the grayscale value change area of ​​the image frame and has significant response intensity characteristics is obtained, which provides a basis for subsequent feature context construction and matching.

4. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 3 is characterized in that: The context construction process performed for each high-density feature point includes: extracting a fixed-size image block with the feature point as the center, calculating the pixel gradient directional distribution within the image block, and generating a normalized directional histogram as the structural description information of the area; comparing and analyzing the directional histograms corresponding to multiple feature points, when the similarity between the directional histograms is greater than the similarity threshold and their spatial positions are periodically arranged and distributed, they are identified as repetitive texture areas; setting an inhibition factor for all feature points participating in the matching process in the area to reduce the influence of repeated structural feature points in the feature matching process and improve the discrimination accuracy of feature matching.

5. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 4 is characterized in that: After completing the identification of repetitive texture areas, in order to control the participation intensity of feature points in the area in feature matching, a corresponding inhibition factor is generated as a score adjustment parameter; the value of the inhibition factor is obtained by inputting the structural repetition calculation result between the context direction histograms of the feature points in the area into the attenuation function. The attenuation function is a monotonically decreasing function, the input variable is the similarity score between the direction histograms, and the output result is the normalization factor value. The normalization factor is multiplied by the original feature point confidence to update the matching score, that is, to adjust the participation matching weight.

6. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 5, characterized in that: The sparse optical flow calculation is as follows: between consecutive image frames, the high-density feature points extracted in the second step are used as tracking targets, and the inter-frame displacement of each feature point on the two-dimensional coordinate plane of the image is calculated. The displacement is established based on the constant grayscale assumption of the image and is minimized using the local photometric consistency criterion to obtain the motion vector of the feature point between consecutive frames; The process of extracting the non-rigid displacement vector set includes: comparing the displacement vector of each feature point with the displacement vectors of its adjacent feature points. If the direction difference is greater than the direction deviation angle threshold and the speed difference exceeds the motion consistency threshold, it is marked as a motion inconsistent point; The set of all motion inconsistent points is processed through a preset region expansion algorithm to generate a foreground dynamic occlusion region mask. The remaining unmarked area is defined as the background static region. Only the feature points in the background static region are retained for the subsequent estimation of the view transformation parameters to eliminate the interference of the foreground dynamic occlusion region on the registration result. The perspective transformation parameter estimation is as follows: perform feature point matching on the feature points retained in the background static area, and establish a candidate set of geometric transformation models based on their coordinate relationship in adjacent image frames, including affine transformation model and homography transformation model; use the least squares error fitting method to solve the parameters of each model, and calculate the reprojection error for each model as the model adaptation score; in the model scoring, select the transformation model with the smallest reprojection error that is less than the set error tolerance threshold as the final perspective transformation parameter solution.

7. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 6, characterized in that: The fourth step constructs a multi-frame image residual sequence within the time window, including: starting from the current image frame, selecting several preceding image frames as reference time windows, and calculating the Euclidean distance between the predicted position and the actual position of the feature points extracted in the second step in each frame to form a feature point residual sequence; The changing trend of the sequence in the time dimension is extracted as the characteristic drift trend vector; Obtain the pose prediction results of image information, inertial measurement information, and positioning information in the current frame respectively, compare them with the feature drift trend, and calculate the spatiotemporal residual values ​​of the three data sources; The three types of residual values ​​are input into a preset confidence distribution function. The confidence distribution function has a fusion weight allocation mapping relationship, which is based on a class of functions in the monotonically decreasing function family. Its input is the residual value and its output is the normalized weight factor. The three weight factors are used as the participation proportions of the corresponding data sources to calculate the splicing pose solution of the current frame, thereby realizing adaptive fusion control based on residual driving.

8. The real-time panoramic image stitching algorithm based on multi-sensor fusion according to claim 7, characterized in that: The processing of the image frame data after the perspective alignment in the fifth step includes the following operations: performing edge smoothing based on Gaussian weighted convolution in the image boundary area to reduce the intensity mutation at the image splicing boundary; The pixel values ​​in the brightness overlap area between image frames are distributed statistically, and local brightness is harmonized through histogram equalization combined with dynamic gamma adjustment. The mean difference between the three-channel color data in the image blocks is sampled and calculated to generate color offset correction parameters, which are used to execute the preset color mapping function to achieve color consistency adjustment. After completing the output of the current frame image, a status assessment is performed based on the stitching residual trend and image structure integrity indicators. If there is a trend of decreased stability or increased feature drift, the fusion status refresh process is triggered and the fusion weight of the next frame is reset to ensure the stability and continuity of the subsequent image frame stitching process.

9. A panoramic image real-time stitching system based on multi-sensor fusion, based on the panoramic image real-time stitching algorithm based on multi-sensor fusion according to any one of claims 1 to 8, characterized in that: include: The time sequence perception module is used to synchronously acquire image frame data through multiple image acquisition devices at different spatial positions, fuse the attitude data provided by the inertial measurement device with the position information provided by the positioning device, align the multi-source data based on a unified timestamp, and eliminate data segments with time delays exceeding a preset threshold to form a structured fused data sequence; The texture parsing module is used to extract high-density feature points from areas with grayscale value changes in the image frame, and construct the context structure of the feature points through the local texture direction histogram of the image block, identify image areas with repetitive texture features, and reduce the confidence level of the feature points in the matching process in this area; The dynamic segmentation module is used to extract a set of non-rigid displacement vectors based on sparse optical flow calculations between consecutive image frames, identify foreground dynamic occlusion areas and background static areas, and after completing the region segmentation, perform geometric transformation model fitting only on the feature points of the background static area to estimate the perspective transformation parameters of the current frame; The fusion solution module is used to construct a multi-frame image residual sequence within a time window. Based on the historical stitching path and the characteristic drift trend of the current frame image, the spatiotemporal residual values ​​of the three data sources of image, inertial, and positioning are calculated, and converted into participation proportions through a preset fusion weight distribution function to achieve adaptive solution of the stitching pose. The image fusion module is used to align the image perspective according to the stitching pose of the current frame, and perform smoothing of the image edge areas, blending of brightness overlapping areas, and color consistency correction. At the same time, the fusion stability is judged based on the stitching residual trend and image structure evaluation indicators. When the fusion status refresh is triggered, the initial fusion configuration of subsequent frames is updated.

Images