Viewport-driven embedded panoramic selective stitching method and system

Abstract

This invention relates to the field of image stitching technology, and proposes an embedded panoramic selective stitching method and system based on viewport-driven technology. The method includes: performing feature matching on adjacent images in a panoramic image sequence to identify the position information of the overlapping area to be stitched; performing secondary screening based on geometric scale, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping parts; when the detection result indicates that processing is required, extracting the geometrically overlapping part of the viewport boundary rectangle and the preset overlapping area as the intersection area; offsetting the intersection area according to the viewport motion parameters to generate a pre-fused area; and obtaining multiple local image blocks from the panoramic image sequence; adaptively and seamlessly fusing the multiple local image blocks to generate stitched local image blocks for the pre-fused area, and writing them into the panoramic image at the position corresponding to the pre-fused area. This invention can improve the efficiency of embedded panoramic selective stitching.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, and in particular to an embedded panoramic selective stitching method and system based on viewport-driven technology. Background Technology

[0002] Existing panoramic image stitching typically uses a sequence of images captured by multiple cameras as input. It extracts and matches features from adjacent images or the global image, estimates geometric transformation relationships, determines overlapping areas, and generates a panoramic image using methods such as weighted fusion and multi-band fusion. This approach usually treats the entire panoramic image or all potentially overlapping areas as the processing target. The stitching range and fusion strategy are pre-set at the acquisition or processing end, and the user's current viewing area and motion trends are generally not involved in the stitching decision. Its reliance on conditions includes: continuous read / write capability of complete panoramic image data; processing platform with computing power and storage bandwidth matching the panoramic resolution; good consistency in lighting and color among the images; and scene content remaining relatively static or with gentle motion during the stitching process. Because the stitching processing range is bound to the entire panoramic image, and the stitching decision is independent of the user's current viewing area and motion trends, the system needs to perform the same feature matching, geometric calculation, and fusion operations on a large number of non-interested areas. This is the fundamental reason for the high system complexity and resource consumption.

[0003] In the aforementioned global processing mode, when dynamic changes occur in the application scenario, such as frequent viewport movement, rapid scanning, or reverse movement, the stitching process still needs to cover the entire panoramic image. The computational load and data read / write load continue to increase with the increase in panoramic resolution, further amplifying the processing latency, power consumption, and storage bandwidth pressure on the embedded platform. Simultaneously, the fusion region is usually determined according to fixed rules or full-image overlap relationships, making it difficult to adjust synchronously with the viewport movement trend. When the viewport changes rapidly, the fusion region may lag or misalign with the actual viewing range, causing image stuttering, local blurring, or stitching seams entering the visible area. Under complex conditions such as changes in lighting, motion occlusion, and poor texture, the feature matching stability in global stitching decreases. Problems such as ghosting, color deviation, and geometric misalignment accumulate and propagate throughout the entire panoramic image, thus affecting the real-time performance, visual consistency, and integrity of the panoramic display. Summary of the Invention

[0004] This invention provides an embedded panoramic selective stitching method and system based on viewport driving to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides an embedded panoramic selective stitching method based on viewport driving, comprising: Pt.1. Obtain the panoramic image sequence, viewport boundary rectangle and viewport motion parameters, perform feature matching on adjacent images in the panoramic image sequence, and identify the position information of the overlapping area to be stitched. Pt.2. Determine whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched. Perform secondary screening based on the geometric scale of the potential overlapping part, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping part to generate a detection result indicating the preset overlapping area. Pt.3 When the detection result indicates that processing is required, the geometric overlap between the viewport boundary rectangle and the preset overlapping area is extracted as the intersection area. The position of the intersection area is offset according to the viewport motion parameters to generate a pre-fusion area. Multi-path local image blocks corresponding to the pre-fusion area are obtained from the panoramic image sequence. Pt.4. Adaptively and seamlessly fuse the multi-path local image blocks to generate stitched local image blocks for the pre-fused region, and write them into the panoramic image at the positions corresponding to the pre-fused region.

[0006] To address the above problems, the present invention also provides an embedded panoramic selective stitching system based on viewport driving, the system comprising: The feature registration module is used to acquire the panoramic image sequence, viewport boundary rectangle and viewport motion parameters, perform feature matching on adjacent images in the panoramic image sequence, and identify the position information of the overlapping area to be stitched. The overlap filtering module is used to determine whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched. It performs secondary filtering based on the geometric scale of the potential overlapping part, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping part, and generates a detection result indicating the preset overlapping area. The region extraction module is used to extract the geometrically overlapping part of the viewport boundary rectangle and the preset overlapping area as the intersection region when the detection result indicates that processing is required, offset the position of the intersection region according to the viewport motion parameters, generate a pre-fusion region, and obtain multiple local image blocks corresponding to the pre-fusion region from the panoramic image sequence. The fusion backfill module is used to adaptively and seamlessly fuse the multi-path local image blocks, generate stitched local image blocks for the pre-fused region, and write them into the panoramic image at the positions corresponding to the pre-fused region.

[0007] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention introduces viewport boundary rectangles and viewport motion parameters as the basis for stitching decisions. It performs intersection judgment on the overlapping areas to be stitched obtained by feature matching, and performs secondary screening based on the geometric scale of the potential overlapping parts, the position of the viewport boundary rectangle in the panoramic image, and the texture richness. Only the geometrically overlapping parts of the preset overlapping areas and the viewport boundary are extracted as the intersection areas. Then, the intersection areas are offset according to the viewport motion parameters to generate pre-fusion areas. Only the multi-path local image blocks corresponding to the pre-fusion areas are adaptively seamlessly fused and backfilled. This makes the stitching processing range converge from the entire panoramic image to the viewport area of ​​interest and the range reached by its motion trend. This fundamentally overcomes the defects of high system complexity and large resource consumption caused by the independence of stitching decisions and viewport state in the global processing mode and the repeated calculation and reading of a large number of non-interested areas.

[0008] 2. This invention identifies the current viewport motion state by using instantaneous velocity vector and instantaneous acceleration vector. For steady-state uniform motion, it uses displacement step size time-domain mean smoothing to obtain the viewport boundary position of subsequent frames. For directional scanning motion, it uses median filtering and linear recursion to generate a pre-fusion region, so that the delineation of the pre-fusion region matches the current motion state, further improving the accuracy and adaptability of the fusion region prediction. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an embedded panoramic selective stitching method based on viewport driving, according to an embodiment of the present invention. Figure 2 A functional block diagram of an embedded panoramic selective stitching system based on viewport driving provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0010] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0011] This application provides a viewport-driven embedded panoramic selective stitching method. The execution entity of this viewport-driven embedded panoramic selective stitching method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the viewport-driven embedded panoramic selective stitching method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0012] Reference Figure 1 The diagram shown is a flowchart illustrating a viewport-driven embedded panoramic selective stitching method according to an embodiment of the present invention. In this embodiment, the viewport-driven embedded panoramic selective stitching method includes: Pt.1. Obtain the panoramic image sequence, viewport boundary rectangle and viewport motion parameters, perform feature matching on adjacent images in the panoramic image sequence, and identify the position information of the overlapping area to be stitched. In this embodiment of the invention, under dynamic lighting and viewport motion conditions, if descriptors are directly constructed using the original pixel grayscale and global matching is performed, when the brightness difference between the left and right images reaches 30% to 60%, the descriptor distance of the same physical point can increase by 1.5 to 3 times. The distance distributions of different physical points overlap, the nearest neighbor to second nearest neighbor distance ratio approaches 1, and the mismatch rate increases significantly. Therefore, in some embodiments, when acquiring panoramic image sequences, the frame rate of the panoramic image sequence is preferably 25 to 60 frames per second, more preferably 30 to 60 frames per second, and this range can be determined by the display refresh rate and embedded acquisition bandwidth. The viewport boundary rectangle is represented by center coordinates, width, and height. The width and height are preferably 10% to 50% of the panoramic image width and height, more preferably 20% to 40%, and this range can be determined according to the display window size and field of view. The viewport motion parameters include the displacement, instantaneous velocity vector, instantaneous acceleration vector, and angular velocity of the viewport center at adjacent sampling times. The sampling frequency is preferably 30 Hz to 120 Hz, more preferably 60 Hz to 90 Hz, and this range can be determined based on the motion sensor bandwidth and embedded computing power. The instantaneous velocity vector can be obtained by dividing the displacement vector of the viewport center at adjacent sampling times by the sampling time interval, which is preferably 1 / 120 to 1 / 30 of a second. The instantaneous acceleration vector can be obtained by dividing the difference in velocity vectors at adjacent sampling times by the sampling time interval. By obtaining the above parameters, a quantitative expression of the viewport state and motion trend is achieved, providing an input basis for subsequent selective stitching.

[0013] When performing feature matching on adjacent images, in some embodiments, scale-invariant feature transformation, accelerated robust feature extraction, or any one of directional FAST and rotation BRIEF can be used to extract feature points. The number of feature points per frame is preferably 200 to 2000, more preferably 500 to 1500, and this range can be determined based on texture richness and embedded computing power. A descriptor is constructed for each feature point, and the descriptor dimension is preferably 64 to 256 dimensions. During matching, the nearest neighbor distance and the second nearest neighbor distance are calculated, and the ratio of the nearest neighbor distance to the second nearest neighbor distance is calculated. When this ratio is less than or equal to the distance ratio threshold, the match is retained. The distance ratio threshold is preferably 0.6 to 0.9, more preferably 0.7 to 0.8, and this range can be determined based on the balance between false matching rate and matching recall rate. Through distance ratio screening, preliminary suppression of false matching is achieved.

[0014] In other embodiments, to reduce the impact of brightness differences between the left and right images on matching, horizontal and vertical gradient transformations can be performed on adjacent images respectively to obtain horizontal and vertical gradient maps. Gradient domain feature descriptors are then constructed based on the gradient magnitude and direction. The gradient magnitude can be obtained by taking the square root of the sum of the squares of the horizontal and vertical gradients, and the gradient direction can be obtained by the arctangent of the ratio of the vertical to the horizontal gradient. Constructing descriptors based on gradient magnitude and direction weakens brightness differences at the descriptor level. Gradient domain descriptor matching improves matching stability under varying illumination conditions.

[0015] After obtaining the initial matching point pairs, in some embodiments, a random sampling consensus algorithm can be used to estimate the homography matrix between adjacent images. The inlier determination criterion is that the reprojection error of the feature points after homography matrix transformation is less than or equal to a reprojection error threshold. The reprojection error threshold is preferably 0.5 pixels to 5 pixels, more preferably 1 pixel to 3 pixels, and this range can be determined according to the image resolution and noise level. Through inlier selection, robust estimation of geometric transformation relationships is achieved.

[0016] Based on the coordinate distribution of the matching interior point pairs in the first and second images, the location information of the overlapping region to be stitched can be determined. In some embodiments, the minimum and maximum values ​​of the horizontal and vertical coordinates of all interior points in adjacent images are taken, and the smallest bounding rectangle enclosed by the minimum, maximum, minimum, and maximum values ​​of the horizontal and vertical coordinates is taken as the overlapping region to be stitched, and the rectangle may be extended outward by 5 to 20 pixels. Alternatively, the convex hull of all interior points may be taken as the overlapping region to be stitched. The minimum width of the overlapping region to be stitched is preferably 5% to 30% of the image width, more preferably 10% to 20%, and this range can be determined according to the overlap width required for subsequent fusion. Through the above region identification, the accurate output of the location information of the overlapping region to be stitched is achieved, providing a basis for subsequent determination of whether there is an intersection between the viewport boundary rectangle and the region.

[0017] Pt.2. Determine whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched. Perform secondary screening based on the geometric scale of the potential overlapping part, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping part to generate a detection result indicating the preset overlapping area. In one feasible implementation of this invention, when determining whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched, the horizontal and vertical ranges of the viewport boundary rectangle in the image coordinate system, as well as the horizontal and vertical ranges of the overlapping area to be stitched, can be obtained respectively. If there are overlapping intervals in both the horizontal and vertical directions, it is determined that there is an intersection, and the area corresponding to the overlapping interval is taken as a potential overlapping part; if there are no overlapping intervals in either the horizontal or vertical direction, it is determined that there is no intersection, and no further filtering is performed. Alternatively, the viewport boundary rectangle and the overlapping area to be stitched can be converted into binary masks respectively, and a pixel-by-pixel logical AND operation can be performed on the two masks, with the set of pixels that have a valid result being taken as a potential overlapping part. Through the above intersection determination, the spatial overlap relationship between the viewport boundary rectangle and the overlapping area to be stitched is quickly determined, avoiding invalid subsequent processing for areas that do not have an intersection.

[0018] For potentially overlapping portions that intersect, further filtering is performed based on their geometric scale. In some embodiments, the geometric scale can be characterized by at least one of the width, height, or area of ​​the potentially overlapping portion. The width of the potentially overlapping portion is compared with a preset minimum stitching width, the height of the potentially overlapping portion is compared with a preset minimum stitching height, or the area of ​​the potentially overlapping portion is compared with a preset minimum stitching area. When the width is less than the preset minimum stitching width, the height is less than the preset minimum stitching height, or the area is less than the preset minimum stitching area, the potentially overlapping portion is marked as an invalid region. The preset minimum stitching width is preferably 5% to 20% of the width of the panoramic image, more preferably 8% to 15%; the preset minimum stitching height is preferably 5% to 20% of the height of the panoramic image, more preferably 8% to 15%; the preset minimum stitching area is preferably 0.5% to 5% of the total number of pixels in the panoramic image, more preferably 1% to 3%. The above thresholds can be determined jointly by the minimum number of overlapping pixels required by the fusion algorithm, the image resolution, and the computational load allowed by the embedded platform. By using geometric scale filtering, potential overlapping parts that are difficult to support stable fusion due to insufficient available pixels are eliminated, reducing the risk of visible seams in subsequent fusion.

[0019] For potential overlapping portions filtered by geometric scale, further filtering is performed based on the position of the viewport boundary rectangle in the panoramic image. In some embodiments, the offset between the center of the viewport boundary rectangle and the center of the panoramic image is calculated, and this offset is divided by the maximum distance from the center of the panoramic image to a corner point of the image to obtain a normalized distance. When the normalized distance is greater than a preset position threshold, it is determined that the viewport boundary rectangle is close to the edge of the panoramic image and far from the center, and the filtering priority of the potential overlapping portion is reduced. The preset position threshold is preferably 0.6 to 0.9, more preferably 0.7 to 0.85. This range can be determined based on the distribution of attention at the center of the field of view and the degree of edge distortion. Alternatively, the offsets between the center of the viewport boundary rectangle and the center of the panoramic image in the horizontal and vertical directions can be calculated separately. When the horizontal offset or the vertical offset exceeds the preset offset threshold in the corresponding direction, the filtering priority of the potential overlapping portion is reduced. Through position filtering, priority retention of areas near the center of attention of the viewport is achieved, so that stitching processing resources are preferentially allocated to areas that are more likely to enter the viewing range.

[0020] For potential overlapping regions filtered by geometric scale, further filtering is performed based on their texture richness. In some embodiments, horizontal and vertical gradient operators can be applied to the potential overlapping regions to obtain horizontal and vertical gradient maps, and gradient magnitude maps can be obtained accordingly. The mean and variance of the gradient magnitudes of all pixels within the potential overlapping region are statistically analyzed, and the mean and variance are weighted together to obtain the texture richness. The mean weight is preferably 0.6 to 0.8, the variance weight is preferably 0.2 to 0.4, and the sum of the two is one. This weight can be determined by experimental statistics of texture type and noise level. In other embodiments, the number of pixels with gradient magnitudes exceeding a preset gradient threshold can also be counted, and the ratio of this number of pixels to the total number of pixels in the potential overlapping region can be calculated, and this ratio can be used as the texture richness. The preset gradient threshold is preferably 10 to 50 gray levels per pixel, more preferably 15 to 30 gray levels per pixel. This ratio is compared with a sparse texture threshold, and when the ratio is less than the sparse texture threshold, the potential overlapping region is marked as a low-texture region. The sparse texture threshold is preferably between 0.05 and 0.2, and more preferably between 0.08 and 0.15. The gradient operator can be any one of the Sobel operator, Schkas operator, or Roberts operator. Through texture richness filtering, the identification and exclusion of potential overlapping parts with sparse features, poor matching stability, and a tendency to produce ghosting or geometric misalignment after fusion are achieved.

[0021] Based on the combined results of geometric scale screening, position screening, and texture richness screening, a detection result indicating a preset overlapping region is generated. In some embodiments, potential overlapping portions marked as invalid regions are removed; for potential overlapping portions not marked as invalid regions, if their screening priority is reduced or they are marked as low-texture regions, they are excluded or placed in a lower processing order; potential overlapping portions that simultaneously meet the geometric scale criteria, are not reduced in position priority, and have texture richness not lower than the sparse texture threshold are identified as preset overlapping regions, and a detection result indicating that the preset overlapping region needs to be processed is generated. Alternatively, geometric scale, position, and texture richness can be assigned scoring weights respectively, and a comprehensive score for each potential overlapping portion can be calculated. Potential overlapping portions with a comprehensive score higher than a preset total score threshold are identified as preset overlapping regions. The preset total score threshold is preferably between 0.6 and 0.8, and this range can be determined based on the balance between the false negative rate and the false positive rate. Through the above-mentioned secondary screening, the preset overlapping regions with sufficient geometric scale, within the viewport's range of interest, and rich texture are accurately located in the panoramic image. This avoids performing subsequent stitching and fusion operations on invalid or low-value regions, thereby reducing the computational and storage overhead of the embedded platform and providing a reliable basis for subsequent extraction of intersection regions and generation of pre-fused regions.

[0022] Pt.3 When the detection result indicates that processing is required, the geometric overlap between the viewport boundary rectangle and the preset overlapping area is extracted as the intersection area. The position of the intersection area is offset according to the viewport motion parameters to generate a pre-fusion area. Multi-path local image blocks corresponding to the pre-fusion area are obtained from the panoramic image sequence. In this embodiment of the invention, when the detection result indicates that processing is required, the geometrically overlapping portion of the viewport boundary rectangle and the preset overlapping area is extracted as the intersection region. In some embodiments, the horizontal and vertical ranges of the viewport boundary rectangle and the preset overlapping area can be obtained respectively, and the overlapping intervals in the horizontal and vertical directions are taken. The area enclosed by the overlapping intervals is the intersection region. Alternatively, the viewport boundary rectangle and the preset overlapping area can be converted into binary masks respectively, and a pixel-by-pixel logical AND operation can be performed on the two binary masks. The set of pixels with a valid result is taken as the intersection region. The outer rectangle of the intersection region can be extended outward by 0 to 20 pixels, more preferably 2 to 10 pixels. This range can be determined according to the subsequent fusion seam width and feature point distribution density. By extracting the geometrically overlapping portion as the intersection region, the subsequent processing range is strictly limited to the actual overlapping range of the viewport and the preset overlapping area, avoiding redundant operations on non-overlapping areas.

[0023] The intersection region is offset according to the viewport motion parameters to generate a pre-fusion region. Under dynamic conditions, the instantaneous speed of the viewport can reach 100 to 800 pixels per second. If only the intersection region of the current frame is used as the subsequent fusion region, the viewport may have moved out of the region when the next frame arrives, resulting in a lag in the fusion region. Therefore, in some embodiments, the offset direction and offset amount are determined according to the instantaneous velocity vector in the viewport motion parameters. The horizontal offset amount is equal to the product of the horizontal component of the instantaneous velocity vector and the prediction time interval, and the vertical offset amount is equal to the product of the vertical component of the instantaneous velocity vector and the prediction time interval. The prediction time interval is preferably 1 / 120 to 1 / 30 of a second, more preferably 1 / 60 to 1 / 30 of a second, and this range can be determined according to the viewport motion speed, embedded processing frame rate, and the number of predicted look-ahead frames. The intersection region is offset by the corresponding offset amount in the horizontal and vertical directions to obtain the offset intersection region. Then, the area covered by the intersection region and the offset intersection region is merged to form the pre-fusion region. Alternatively, the intersection region can be expanded outward by an offset, preferably by 5 to 30 pixels, more preferably by 10 to 20 pixels, and this range can be determined based on the velocity modulus and sampling interval. By offsetting the intersection region according to the viewport motion parameters, the pre-fusion region that may enter the viewing range in subsequent frames is pre-defined, making the pre-fusion region match the viewport motion trend and reducing the risk of lag in the fusion region or misalignment with the actual viewing range.

[0024] Multiple local image blocks corresponding to the pre-fusion region are obtained from the panoramic image sequence. In some embodiments, the panoramic image sequence consists of a fisheye image sequence containing fisheye images from different viewpoints. Isorectometric correction is performed on the fisheye images from different viewpoints to obtain corrected images. Based on the coordinate range of the pre-fusion region in the panoramic image, local image blocks corresponding to the pre-fusion region are extracted from each corrected image to obtain multiple local image blocks. The size of the local image block can be the same as the pre-fusion region, or it can extend 5 to 30 pixels beyond the boundary of the pre-fusion region, more preferably 10 to 20 pixels, which can be determined according to the overlap width required by the fusion algorithm. In other embodiments, the multiple local image blocks may include a left-path local image block and a right-path local image block, respectively from images of adjacent viewpoints in the panoramic image sequence. By obtaining multiple local image blocks corresponding to the pre-fusion region from the panoramic image sequence, only the local image data involved in the pre-fusion region is read and processed, avoiding repeated reading and writing of the entire panoramic image data, reducing the storage bandwidth and computational overhead of the embedded platform, and providing a data foundation for subsequent adaptive seamless fusion.

[0025] Pt.4. Adaptively and seamlessly fuse the multi-path local image blocks to generate stitched local image blocks for the pre-fused region, and write them into the panoramic image at the positions corresponding to the pre-fused region.

[0026] In this embodiment of the invention, the multi-path local image blocks are adaptively and seamlessly fused to generate stitched local image blocks for the pre-fused region. Under dynamic lighting and parallax conditions, the multi-path local image blocks may have brightness differences and geometric misalignments in the overlapping area. If a fixed-weight average fusion is used, when the color difference on both sides of the seam is large, the fixed weight cannot adaptively adjust with distance and color difference, resulting in visible seams or ghosting. Therefore, in some embodiments, the seam distance of the left and right local image blocks in the multi-path local image blocks is extracted separately. Each distance value in the left-path distance field represents the number of pixel intervals from the left-path pixel to the stitching seam, and each distance value in the right-path distance field represents the number of pixel intervals from the right-path pixel to the stitching seam. The seam position can be taken as the center line of the overlapping area of ​​the left and right local image blocks, or as the path with the minimum gradient energy in the overlapping area. The distance field can be calculated in the horizontal direction, vertical direction, or a combination of horizontal and vertical directions. The horizontal distance field is preferably determined by the difference between the pixel column number and the seam column number, the vertical distance field is preferably determined by the difference between the pixel row number and the seam row number, and the joint distance field is preferably determined by the square root of the sum of the squares of the horizontal and vertical distances. The distance value range is preferably 0 to 200 pixels, more preferably 0 to 100 pixels, and this range can be determined based on the width of the overlapping area and the width of the fusion transition band. By adaptively and seamlessly fusing multiple local image blocks, the fusion weights change smoothly with the distance from the pixel to the seam, reducing abrupt changes at the seam.

[0027] Further color difference detection is performed on the left and right local image blocks, comparing the left and right pixel values ​​pixel by pixel to generate a color difference map. In some embodiments, the color difference can be characterized by luminance difference, chrominance difference, or a combined luminance and chrominance difference. The luminance difference can be the absolute value of the difference between the luminance of the left and right pixels, the chrominance difference can be the absolute value of the difference between the chrominance of the left and right pixels, and the combined difference can be the weighted sum of the luminance and chrominance differences. The color difference threshold is preferably 5 to 30 gray levels, more preferably 10 to 20 gray levels, and this range can be determined according to the sensor noise level and color difference tolerance. In other embodiments, histogram distribution sampling can be performed on the overlapping region of the left and right local image blocks to obtain left and right histogram distributions. The cumulative distribution functions of the left and right histogram distributions are then matched to analyze the tone mapping deviation between the images. Gain coefficient compensation is then applied to the left and right local image blocks based on this tone mapping deviation, making the cumulative distribution functions of the images in the overlapping region more consistent. The gain coefficient is preferably between 0.8 and 1.2, more preferably between 0.9 and 1.1, and this range can be determined based on the differences in exposure and color response between the left and right channels. Through color difference detection and gain compensation, the tone deviation between the left and right local image blocks is reduced, providing a color consistency basis for subsequent weight construction.

[0028] A comprehensive weight map is constructed based on the left-path distance field, the right-path distance field, and the color difference map. In some embodiments, in the comprehensive weight map, regions with small color differences on both sides of the stitching seam are assigned weights that change smoothly with distance, while regions with large color differences are assigned weights close to binarization. Specifically, for regions with color differences less than the color difference threshold, the left-path weight can increase with the increase of the left-path distance, and the right-path weight can increase with the increase of the right-path distance, with the sum of the two weights being 1; for regions with color differences greater than the color difference threshold, the weight of the path with smaller color differences is set to close to 1, and the weight of the other path is set to close to 0, thereby avoiding the participation of pixels with large color differences in the mixing. The color difference threshold is preferably 10 to 40 gray levels, more preferably 15 to 30 gray levels, and this range can be determined based on the visibility of the fusion seam and color fidelity. Alternatively, a multi-band fusion method can be used, performing Laplacian pyramid decomposition on multiple local image blocks respectively, constructing frequency band weights based on the distance field and color difference map for each frequency band, and then reconstructing the image band by band. The number of pyramid layers is preferably 3 to 7, and more preferably 4 to 6, with this range determined based on image resolution and embedded computing power. A comprehensive weight map is used to perform pixel-by-pixel weighted mixing of the left and right local image blocks to generate stitched local image blocks. By constructing the comprehensive weight map, smooth transitions are achieved in areas with consistent color, while erroneous pixels are suppressed from participating in the fusion process in areas with significant color differences, thus balancing seam smoothness and color fidelity.

[0029] The stitched partial image blocks are written into the panoramic image at positions corresponding to the pre-fusion region. In some embodiments, during the startup phase, the required storage space for the panoramic image and the stitched partial image blocks is allocated as a contiguous storage space. When writing the stitched partial image blocks into the panoramic image at positions corresponding to the pre-fusion region, the positions corresponding to the pre-fusion region in the contiguous storage space are directly reused, and the storage address of the panoramic image remains unchanged before and after writing. The writing method can be line-by-line writing, block-by-block writing, or writing the entire block according to the bounding rectangle of the pre-fusion region. The writing boundary can extend beyond the pre-fusion region by 0 to 20 pixels, more preferably 2 to 10 pixels, and this range can be determined according to the width of the fusion transition band and the writing alignment accuracy. In other embodiments, after writing is completed, the write bounding box and write sequence number corresponding to each write operation can be recorded to generate a write history. Adjacency detection is performed on the write bounding boxes in the write history. When the write bounding boxes of two adjacent writes overlap, consistency verification is performed on the pixel values ​​of the overlapping area to obtain color difference and misalignment measurements. Local color migration is then performed based on the color difference measurements, and feature point reprojection is performed based on the misalignment measurements to generate a color-aligned panoramic image or a geometrically calibrated panoramic image. By writing the stitched local image blocks into the panoramic image at positions corresponding to the pre-fused region, only the local storage content corresponding to the pre-fused region is updated. This avoids repeated reading and writing of the entire panoramic image, reduces the storage bandwidth and write overhead of the embedded platform, and ensures the stability of the panoramic image storage address, facilitating continuous reading of the currently viewed image by the display device.

[0030] like Figure 2 The diagram shown is a functional block diagram of an embedded panoramic selective stitching system based on viewport driving provided in an embodiment of the present invention.

[0031] The viewport-driven embedded panoramic selective stitching system described in this invention can be installed in electronic devices. Depending on the functions implemented, the viewport-driven embedded panoramic selective stitching system may include a feature registration module, an overlap filtering module, a region extraction module, and a fusion backfilling module. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0032] In this embodiment, the functions of each module / unit are as follows: The feature registration module is used to acquire panoramic image sequences, viewport boundary rectangles and viewport motion parameters, perform feature matching on adjacent images in the panoramic image sequence, and identify the location information of the overlapping area to be stitched. The overlap filtering module is used to determine whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched, and to perform secondary filtering based on the geometric scale of the potential overlapping part, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping part, and to generate a detection result indicating the preset overlapping area. The region extraction module is used to extract the geometrically overlapping part of the viewport boundary rectangle and the preset overlapping area as the intersection area when the detection result indicates that processing is required, offset the position of the intersection area according to the viewport motion parameters, generate a pre-fusion area, and obtain multiple local image blocks corresponding to the pre-fusion area from the panoramic image sequence. The fusion backfill module is used to adaptively and seamlessly fuse the multi-path local image blocks, generate stitched local image blocks for the pre-fused region, and write them into the panoramic image at the positions corresponding to the pre-fused region.

[0033] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0034] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0035] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0036] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0037] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A viewport-driven based embedded panoramic selective stitching method, characterized in that, The method includes: Pt.

1. Obtain the panoramic image sequence, viewport boundary rectangle and viewport motion parameters, perform feature matching on adjacent images in the panoramic image sequence, and identify the position information of the overlapping area to be stitched. Pt.

2. Determine whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched. Perform secondary screening based on the geometric scale of the potential overlapping part, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping part to generate a detection result indicating the preset overlapping area. Pt.3 When the detection result indicates that processing is required, the geometric overlap between the viewport boundary rectangle and the preset overlapping area is extracted as the intersection area. The position of the intersection area is offset according to the viewport motion parameters to generate a pre-fusion area. Multi-path local image blocks corresponding to the pre-fusion area are obtained from the panoramic image sequence. Pt.

4. Adaptively and seamlessly fuse the multi-path local image blocks to generate stitched local image blocks for the pre-fused region, and write them into the panoramic image at the positions corresponding to the pre-fused region.

2. The embedded panoramic selective stitching method based on viewport driving as described in claim 1, characterized in that, The step of performing feature matching on adjacent images in the panoramic image sequence to identify the location information of the overlapping area to be stitched includes: Gradient transformation is performed on adjacent images in the panoramic image sequence to generate horizontal and vertical gradient maps of the adjacent images. Based on the horizontal gradient map and the vertical gradient map, gradient domain feature descriptors are constructed for the adjacent images, and the gradient domain feature descriptors eliminate the brightness differences between the adjacent images; The gradient domain feature descriptors of the adjacent images are matched to obtain gradient domain matching point pairs; Based on the positional distribution of the gradient domain matching point pairs in the adjacent images, the positional information of the overlapping region to be stitched is determined, and the positional information of the overlapping region to be stitched is updated as the content of the adjacent images changes.

3. The embedded panoramic selective stitching method based on viewport driving as described in claim 2, characterized in that, The viewport motion parameters include: In response to a user's dragging operation on the panoramic image, the dragging trajectory of the dragging operation on the panoramic image is obtained; The dragging direction and speed of the dragging operation are determined based on the dragging trajectory. The dragging direction and the dragging speed are combined to generate a viewport motion vector, which is used as a viewport motion parameter.

4. The embedded panoramic selective stitching method based on viewport driving as described in claim 1, characterized in that, The secondary screening based on the geometric scale of the potential overlapping portion, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping portion generates detection results indicating a preset overlapping area, including: The geometric scale of the potential overlapping part is compared with the coverage area of ​​the viewport boundary rectangle in the panoramic image. When the geometric scale is smaller than the coverage area, the potential overlapping part is marked as an invalid fusion area with no visual impact. The position of the viewport boundary rectangle in the panoramic image is compared with the position of the potential overlapping part. When the viewport boundary rectangle deviates from the potential overlapping part, the potential overlapping part is marked as an invalid fusion area with no visual impact. The texture richness of the potential overlapping part is compared with the texture richness of the area covered by the viewport boundary rectangle. When the texture richness of the potential overlapping part is lower than the texture richness of the area covered by the viewport boundary rectangle, the potential overlapping part is marked as an invalid fusion area with no visual impact. The potential overlapping portions that are not marked as invalid fusion regions are used as the preset overlapping regions, and detection results indicating the preset overlapping regions are generated.

5. The embedded panoramic selective stitching method based on viewport driving as described in claim 4, characterized in that, The texture richness of the potentially overlapping portions includes: A horizontal gradient operator and a vertical gradient operator are applied to the potential overlapping portion to generate a horizontal gradient map and a vertical gradient map of the potential overlapping portion. Based on the horizontal gradient map and the vertical gradient map, the gradient magnitude map of the potential overlapping portion is obtained; The gradient magnitude map is statistically analyzed to obtain its mean and variance. The mean and variance are combined to obtain the texture richness of the potential overlapping portion, which is then compared with the texture richness of the area covered by the viewport boundary rectangle.

6. The embedded panoramic selective stitching method based on viewport driving as described in claim 1, characterized in that, The step of offsetting the intersection region based on the viewport motion parameters to generate a pre-fused region includes: The viewport's direction and speed of movement on the panoramic image are determined based on the viewport motion parameters. According to the moving direction and the moving speed, the position of the intersection region in the panoramic image is shifted along the moving direction to obtain the shifted intersection region; The area covered by the intersection region and the shifted intersection region is merged to form the pre-fusion region, which is used to obtain the corresponding multi-path local image blocks from the panoramic image sequence.

7. The embedded panoramic selective stitching method based on viewport driving as described in claim 1, characterized in that, The step of obtaining multiple local image patches corresponding to the pre-fused region from the panoramic image sequence includes: The panoramic image sequence is composed of a fisheye image sequence, which contains fisheye images from different viewpoints; The fisheye images from different viewpoints are subjected to equal rectangle correction to obtain corrected images for each path; Based on the detection results, local image blocks corresponding to the pre-fusion region are extracted from each of the corrected images to obtain the multi-path local image blocks; The multiple local image blocks are independently subjected to adaptive seamless fusion to generate each stitched local image block; The stitched local image blocks are merged according to their positions in the panoramic image and written into the position in the panoramic image corresponding to the pre-fused region.

8. The embedded panoramic selective stitching method based on viewport driving as described in claim 1, characterized in that, The adaptive seamless fusion of the multiple local image blocks to generate stitched local image blocks for the pre-fused region includes: Bidirectional optical flow estimation is performed on the multi-path local image blocks to obtain the forward optical flow field and the reverse optical flow field of the multi-path local image blocks; Based on the amplitudes of the forward optical flow field and the reverse optical flow field, cross-direction modulation weights are generated. In the cross-direction modulation weights, the weights corresponding to the local image blocks in the multi-path local image blocks are obtained based on the optical flow amplitudes of other local image blocks. A smoothing switch is generated based on the color differences between the multiple local image blocks; When the smoothing switch indicates the same color, the multi-path local image blocks are weighted and summed according to the cross-direction modulation weight to generate a fused image block. When the smoothing switch indicates inconsistent colors, the weight of the local image block with the largest color difference among the multiple local image blocks is reset to zero, and the local image block with the smallest color difference is used as the fused image block. Local color correction is performed on the fused image blocks to make the brightness and color temperature of the multiple local image blocks more consistent, thus generating stitched local image blocks.

9. The embedded panoramic selective stitching method based on viewport driving as described in claim 8, characterized in that, After writing the image to the position corresponding to the pre-fused region, the method further includes: During the startup phase, the storage space required for the panoramic image and the stitched local image blocks is allocated as a contiguous storage space. When writing the stitched local image block into the panoramic image at the position corresponding to the pre-fused region, the position corresponding to the pre-fused region in the continuous storage space is directly reused, and the storage address of the panoramic image remains unchanged before and after writing. The panoramic image after writing the stitched local image blocks is used as the updated panoramic image; Based on the viewport boundary rectangle, the portion corresponding to the viewport boundary rectangle is cropped from the updated panoramic image and used as the current viewing range image, which is then output to the display device.

10. An embedded panoramic selective stitching system based on viewport-driven architecture, characterized in that, The system for implementing the viewport-driven embedded panoramic selective stitching method of claim 1 includes: The feature registration module is used to acquire the panoramic image sequence, viewport boundary rectangle and viewport motion parameters, perform feature matching on adjacent images in the panoramic image sequence, and identify the position information of the overlapping area to be stitched. The overlap filtering module is used to determine whether there is an intersection between the viewport boundary rectangle and the overlapping area to be stitched. It performs secondary filtering based on the geometric scale of the potential overlapping part, the position of the viewport boundary rectangle in the panoramic image, and the texture richness of the potential overlapping part, and generates a detection result indicating the preset overlapping area. The region extraction module is used to extract the geometrically overlapping part of the viewport boundary rectangle and the preset overlapping area as the intersection region when the detection result indicates that processing is required, offset the position of the intersection region according to the viewport motion parameters, generate a pre-fusion region, and obtain multiple local image blocks corresponding to the pre-fusion region from the panoramic image sequence. The fusion backfill module is used to adaptively and seamlessly fuse the multi-path local image blocks, generate stitched local image blocks for the pre-fused region, and write them into the panoramic image at the positions corresponding to the pre-fused region.

Images