Machine room scene-oriented panoramic image splicing method based on spherical projection
By employing a spherical projection algorithm and a linear interpolation weighting image stitching method, the problem of poor panoramic image stitching quality from a single camera in a base station equipment room was solved, achieving high-quality panoramic image stitching and improved user interaction experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-03-13
AI Technical Summary
Existing technologies for stitching panoramic images from a single camera in a base station equipment room suffer from image feature matching errors, resulting in poor stitching quality. This makes it particularly difficult to achieve high-quality panoramic image output in complex environments.
A panoramic image stitching method based on spherical projection is adopted. Image distortion is corrected by camera calibration data, image groups taken parallel to the ground are identified, the optimal pixel focal length and dynamic FOV are calculated, cylindrical projection estimation algorithm and spherical projection algorithm are used to generate ERP images, and images are superimposed by mask and linear interpolation weight to ensure stitching quality.
It improves the quality and robustness of image stitching, enabling high-quality panoramic image stitching in complex environments, and enhancing the user's immersive interactive experience and perspective navigation capabilities.
Smart Images
Figure CN121660877A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a panoramic image stitching method for computer room scenarios based on spherical projection. Background Technology
[0002] With the rapid development of information technology, modern base station equipment rooms have become core facilities for many enterprises and organizations. Due to the complex internal environment and numerous devices within base station equipment rooms, their frequent unattended locations in remote areas, and the prevalence of potential security risks and management challenges, these rooms face various issues. Given the significant resource and economic costs associated with redeploying multiple cameras or introducing high-precision cameras in base station equipment rooms, exploring solutions for achieving panoramic image acquisition with existing single-camera configurations is particularly important. This need has prompted us to conduct in-depth research on how to improve image acquisition and processing capabilities through technological innovation under limited hardware conditions to achieve panoramic monitoring effects.
[0003] By controlling the rotation of a single camera in the base station equipment room to capture multiple images from different angles and perspectives, panoramic stitching technology is used to stitch these images together into a more comprehensive image and a clearer monitoring view. This enables real-time monitoring and maintenance of the equipment room, improving its intelligence level and visualization capabilities. A key challenge in this process is achieving seamless stitching between multiple images.
[0004] Existing feature-based image stitching methods tend to employ increasingly complex geometric features to achieve better content alignment and shape preservation. However, they heavily rely on the geometric features of the image, especially in complex environments such as base station equipment rooms where feature mismatches may occur, leading to poor stitching quality and affecting the output quality of panoramic images. Summary of the Invention To address this issue, the present invention provides a panoramic image stitching method for computer room scenarios based on spherical projection, which can solve the problem of poor panoramic image output quality.
[0005] To achieve the above objectives, this invention provides a panoramic image stitching method for computer room scenarios based on spherical projection, the method comprising: Acquire several initial images, and perform geometric transformations on the initial images based on camera calibration data to obtain several corrected images; Identify images from a plurality of corrected images that are taken parallel to the ground, and determine an image group based on the identification results; The optimal pixel focal length is calculated based on the cylindrical projection estimation algorithm and the image group. The dynamic FOV is determined based on three sets of preset physical focal lengths, the optimal pixel focal length, and the FOV calculation formula. The corrected images are converted into ERP images using a spherical projection algorithm, and a mask is generated. The target image is obtained by progressively overlaying several ERP images using the mask and linear interpolation weights.
[0006] Furthermore, the step of performing geometric transformations on several of the initial images based on camera calibration data includes: Extract the intrinsic parameter matrix and distortion coefficients from the camera calibration data; Based on the intrinsic parameter matrix and the distortion coefficients, several initial images are corrected to obtain several corrected images.
[0007] Furthermore, the step of identifying the image among the plurality of corrected images that corresponds to the image taken parallel to the ground includes: Edge features are extracted from the corrected image, and significant horizontal structures in the edge features are detected based on the Hough transform. The main lines in the significant horizontal structure are extracted based on straight line fitting, and the actual tilt angle of the main lines is calculated. The actual tilt angle is compared with a preset angle threshold, and candidate images are determined based on the comparison results; The candidate images are simulated by cylindrical projection. The reprojection error of the projected candidate images in the vertical direction is calculated to verify whether they meet the geometric constraints of cylindrical projection. The image group is determined based on the verification results.
[0008] Furthermore, the step of calculating the optimal pixel focal length based on the cylindrical projection estimation algorithm and the image group includes: Determine the focal length range of candidate pixels; Calculate the SIFT features of several corrected images in the image group, calculate the image correlation based on the several SIFT features, and determine two related images based on the image correlation calculation results; Based on several candidate pixel focal lengths, the relevant images are cylindrically projected and the quality of the stitched region is calculated to select the optimal pixel focal length.
[0009] Furthermore, the step of calculating image correlation based on several of the SIFT features includes: Determine the first candidate image and the second candidate image; The SIFT algorithm is applied to the first candidate image and the second candidate image respectively to extract a number of first key points, a number of first descriptors, a number of second key points and a number of second descriptors; Calculate the Euclidean distance between any of the first descriptors and several of the second descriptors, and select the minimum value among the several Euclidean distances as the initial matching pair; The Euclidean distances corresponding to several initial matching pairs are compared with preset distances, and valid matching pairs are determined based on the comparison results. The number of valid matching pairs is counted, and the average value of the Euclidean distances corresponding to several valid matching pairs is calculated. A similarity score is determined based on the effective number and the average value, and related images are determined based on the similarity scores corresponding to several corrected images.
[0010] Furthermore, the step of performing cylindrical projection on the relevant image based on several candidate pixel focal lengths and calculating the quality of the stitched region includes: Based on the focal length of any candidate pixel, cylindrical projection is performed on the relevant image to obtain several projected images; Several projected images are stitched together, and the quality evaluation index of the stitched area is calculated. The candidate pixel focal length with the best quality evaluation index is selected as the optimal pixel focal length.
[0011] Furthermore, the step of converting the plurality of corrected images into ERP images using a spherical projection algorithm includes: The several corrected images are transformed from a two-dimensional planar coordinate system to a three-dimensional spherical coordinate system, wherein each pixel of the corrected image corresponds to a point on the sphere. Based on the three-dimensional spherical coordinate system, the corrected image is spherically projected to obtain the ERP image.
[0012] Furthermore, the steps for generating the mask include: Based on the camera's physical parameters and rotation angle, calculate the actual coverage area of a single ERP image in the spherical coordinate system; By using the spherical projection formula, the pixel coordinates corresponding to the corrected image are mapped to the ERP plane to determine the spherical latitude and longitude corresponding to each ERP pixel; For each ERP pixel in the ERP image, determine whether it is within the actual coverage area based on its corresponding spherical latitude and longitude, and determine the valid and invalid pixels based on the determination result, and determine the mask based on the valid and invalid pixels; Based on the mask, multiple images corresponding to it are superimposed. If a certain pixel is marked as valid in any mask, then the pixel in the final mask is marked as valid, and the rest are invalid. The calculation result is saved as a binary matrix.
[0013] Furthermore, the step of progressively overlaying several of the ERP images using the mask and linear interpolation weights includes: Determine the position of the effective pixels in each ERP image, and calculate the linear interpolation weight of each ERP image in the overlay region based on the mask matrix; For each pixel in the overlapping region, the linear interpolation weight of that pixel in all ERP images is calculated based on its position. The corresponding pixel values of several ERP images are then weighted and averaged to obtain the pixel values of the final stitched image. Repeat the above steps, iterating through all overlapping pixels, until all ERP images are overlaid to obtain a panoramic image.
[0014] Furthermore, the step of calculating the linear interpolation weights of each ERP image in the overlay region based on the mask matrix includes: Based on the weight setting rules and the mask matrix and its corresponding spatial location, calculate the single weight map corresponding to any of the ERP images; Superimpose all the single-weighted graphs and normalize the single-graph weights.
[0015] Compared with the prior art, the present invention has the following advantages: 1. The optimal pixel focal length of a set of images taken parallel to the ground is obtained through a cylindrical projection estimation algorithm. Since the images are taken with a zoom lens, the dynamic FOV is obtained by adjusting the pixel focal length, which ensures accurate spherical projection stitching and greatly improves the quality of image stitching.
[0016] 2. Utilize dynamic field of view (FOV) to perform spherical projection, thereby enabling the stitching of multiple images. Compared to other projection methods, spherical projection exhibits higher robustness in image stitching tasks, ensuring a high success rate under a wider range of shooting conditions and scene variations.
[0017] 3. By employing spherical projection to stitch images together and generate ERP format images, it can completely record panoramic field-of-view information. Combined with multi-view switching functionality, it can significantly enhance the user's immersive interactive experience, providing the possibility of highly flexible perspective navigation and detailed exploration capabilities.
[0018] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the panoramic image stitching method for computer room scenarios based on spherical projection provided in an embodiment of the present invention. Figure 2 This is a top view of the projection model in the panoramic image stitching method for computer room scenes based on spherical projection provided in an embodiment of the present invention. Figure 3This is a cylindrical projection structure diagram of two images in the panoramic image stitching method for computer room scenes based on spherical projection provided in an embodiment of the present invention. Figure 4 The spherical coordinate system is used in the panoramic image stitching method for computer room scenes based on spherical projection provided in the embodiments of the present invention. Figure 5 The z-axis coordinate representation of point P in the panoramic image stitching method for computer room scenes based on spherical projection provided in the embodiments of the present invention; Figure 6 The diagram shows the x and y coordinates of point P in the panoramic image stitching method for computer room scenarios based on spherical projection provided in this embodiment of the invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0021] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0022] It should be noted that in the description of this invention, the terms "upper," "lower," "left," "right," "inner," and "outer," etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is merely for ease of description and does not indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.
[0023] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0024] Please see Figure 1 As shown in the figure, this embodiment of the invention provides a panoramic image stitching method for computer room scenes based on spherical projection. The method includes: Step S100: Acquire several initial images, and perform geometric transformation on the several initial images based on camera calibration data to obtain several corrected images; Step S200: Identify the images taken parallel to the ground from among the several corrected images, and determine the image group based on the identification results; Step S300: Calculate the optimal pixel focal length based on the cylindrical projection estimation algorithm and the image group; Step S400: Determine the dynamic FOV based on three sets of preset physical focal lengths, the optimal pixel focal length, and the FOV calculation formula; Step S500: Convert the plurality of corrected images into ERP images using a spherical projection algorithm and generate a mask; Step S600: The target image is obtained by progressively superimposing several ERP images using the mask and linear interpolation weights.
[0025] In this embodiment, the initial images are multiple initial images captured by rotating a single camera.
[0026] Specifically, this invention performs geometric transformations on the initial image using camera calibration data to correct radial or tangential distortion caused by the lens, ensuring geometric accuracy for subsequent projection and stitching. The corrected image has a more realistic geometric structure, reducing stitching misalignment caused by lens distortion and providing high-quality input for subsequent steps. By grouping and focusing on images that meet projection conditions, the amount of invalid data processing is reduced, improving algorithm efficiency. The optimal pixel focal length is calculated using a cylindrical projection model to solve the projection inconsistency problem caused by focal length changes in zoom lenses. Focal length selection is optimized through evaluation metrics to ensure pixel alignment in overlapping areas, reducing stitching artifacts. Based on the physical focal length and the optimal... The system dynamically adjusts the horizontal and vertical field of view based on the optimal pixel focal length, resolving parallax issues when shooting objects at different distances in the equipment room. The dynamic FOV ensures that multi-angle images are proportionally consistent in the spherical projection, avoiding local stretching or compression caused by focal length differences. It converts the spherical projection to ERP format to fully record the field of view information, facilitating storage and interaction. The generated mask distinguishes between effective shooting areas and invalid blank areas, guiding subsequent fusion to process only effective pixels and avoid noise interference. Linear interpolation weights achieve gradual fusion in overlapping areas, eliminating stitching seams and improving visual continuity. The mask controls invalid areas from participating in fusion, and combined with normalization processing, it prevents uneven brightness or overexposure, ensuring the quality of the output image.
[0027] Specifically, the step of performing geometric transformations on several initial images based on camera calibration data includes: Extract the intrinsic parameter matrix and distortion coefficients from the camera calibration data; Based on the intrinsic parameter matrix and the distortion coefficients, several initial images are corrected to obtain several corrected images.
[0028] In this embodiment, the intrinsic parameter matrix includes focal length. , and principal point coordinates ; The distortion coefficient includes the radial distortion coefficient. , , and tangential distortion coefficient , .
[0029] In this embodiment, a distortion correction algorithm is applied to each initial image, mapping the initial image pixel (x,y) to the corrected coordinates (x′,y′) using the following formula:
[0030] In the formula, r is the distance from the initial image pixel to the image center, so as to correct the initial image according to the above algorithm.
[0031] Specifically, the step of identifying the image that was taken parallel to the ground among the plurality of corrected images includes: Edge features are extracted from the corrected image, and significant horizontal structures in the edge features are detected based on the Hough transform. The main lines in the significant horizontal structure are extracted based on straight line fitting, and the actual tilt angle of the main lines is calculated. The actual tilt angle is compared with a preset angle threshold, and candidate images are determined based on the comparison results; The candidate images are simulated by cylindrical projection. The reprojection error of the projected candidate images in the vertical direction is calculated to verify whether they meet the geometric constraints of cylindrical projection. The image group is determined based on the verification results.
[0032] In this embodiment, edge features in the corrected image are extracted using edge detection algorithms (such as Canny and Sobel).
[0033] In this embodiment, several edge features are detected by Hough transform or deep learning models (such as semantic segmentation based on convolutional neural networks) to identify significant horizontal structures parallel to the ground.
[0034] In this embodiment, the Hough Transform is used to detect straight lines in the image, obtaining the parameters (represented in polar coordinates) of each line: ρ (distance) is the perpendicular distance from the line to the origin, and θ is the perpendicular distance from the line to the origin. x (Angle) The angle between the line and the x-axis (horizontal axis), ranging from 0 to θ. x <π (0°~180°); θ x Convert to actual tilt angle, when 0 ≤ θx When the angle is less than π / 2, the line slopes upwards and to the right, with an actual angle of inclination of θ. 实际 =θ x When π / 2 ≤ θ x When π < π, the line slopes upwards and to the left, with an actual angle of inclination of θ. 实际 =θ x π, which is a negative angle; All detected angles are counted, and the angle with the highest frequency is selected as the actual tilt angle.
[0035] In this embodiment, the preset angle threshold can be set to ±2°. When the absolute value of the actual tilt angle is less than the absolute value of the preset angle threshold, the corrected image corresponding to its main lines is used as the candidate image.
[0036] In this embodiment, the candidate image is simulated by cylindrical projection. By calculating the reprojection error (such as feature point alignment or pixel continuity) of the projected image in the vertical direction, it is verified whether it meets the geometric constraints of cylindrical projection. If the vertical distortion is less than the threshold and the horizontal feature alignment is high, it is determined to be an image taken parallel to the ground.
[0037] Specifically, the steps for calculating the optimal pixel focal length based on the cylindrical projection estimation algorithm and the image group include: Determine the focal length range of candidate pixels; Calculate the SIFT features of several corrected images in the image group, calculate the image correlation based on the several SIFT features, and determine two related images based on the image correlation calculation results; Based on several candidate pixel focal lengths, the relevant images are cylindrically projected and the quality of the stitched region is calculated to select the optimal pixel focal length.
[0038] In this embodiment, based on practical experience, the focal length range of the candidate pixels is set to be between 1000 and 1800.
[0039] In this embodiment, the principle of cylindrical projection is to place the optical center of the camera at the center of the cylinder, and the pixel focal length of the image is the radius of the cylinder. Therefore, the image captured by the camera is tangent to the cylinder. A line connecting a point P on the image to the optical center and the cylindrical surface intersects at point P', which is the projection of point P onto the cylindrical surface. In the simplified model of cylindrical projection, the vertical direction (y′) usually remains unchanged; only the horizontal coordinate (x′) undergoes a projection transformation. Figure 2 This is a top view of the projected model, based on Figure 2 As can be seen, taking point A in the diagram as the origin, the horizontal coordinate of point P is calculated from the plane coordinate system to the cylindrical coordinate system P' based on the geometric correspondence, while the vertical coordinate remains unchanged. The transformation formula is as follows:
[0040] In the formula, x′ is the horizontal pixel coordinate in the plane coordinate system of the corrected image (with the left side of the image as the origin), w is the total width of the original image (in pixels), f is the focal length of the candidate pixel, that is, the radius of the cylinder in the cylindrical projection, θ is the longitude angle (in radians) of the line connecting point P and the optical center on the cylinder surface, representing the horizontal angle after projection, α is the total horizontal field of view (in radians) after the cylindrical projection, and w′ is the total width of the unfolded image after projection onto the cylindrical coordinate system (in pixels). y' represents the horizontal coordinates after projection onto the cylindrical coordinate system (with the center of the unfolded image as the origin), and y' represents the vertical pixel coordinates in the original image's planar coordinate system. These are the vertical coordinates projected onto the cylindrical coordinate system, and they remain consistent with the original vertical coordinates.
[0041] Since the pixel coordinates of an image are discrete, directly calculating the coordinates of the target image from the coordinates of the original image may result in holes in the pixel values of the target image. Therefore, it is necessary to use interpolation to map the coordinates of the target image to the coordinate system of the original image and estimate the pixel values corresponding to the coordinates of the target image.
[0042] In this embodiment, based on prior knowledge, if the image is rotated every 36 degrees, then the translation distance in the cylindrical coordinate system is... ,in , Figure 3 Given the cylindrical projection structure of two images, then as follows: Figure 3 As shown, the overlapping region is AB. The difference and similarity of the overlapping region are measured by calculating two evaluation indicators, SSD and SSIM. Finally, the average pixel focal length returned by the two indicators is selected as the optimal pixel focal length to ensure that the pixel difference of the overlapping region is minimized and the structural similarity is maximized.
[0043] In this embodiment, SSD and SSIM in the cylindrical projection estimation algorithm can be replaced with other evaluation metrics, such as PSNR or MSE.
[0044] In this embodiment, the SIFT features in the cylindrical projection estimation algorithm can be replaced with other feature extraction calculation methods, such as gradient sum, Sobel operator, Canny operator, etc.
[0045] Specifically, the steps for calculating image correlation based on several of the aforementioned SIFT features include: Determine the first candidate image and the second candidate image; The SIFT algorithm is applied to the first candidate image and the second candidate image respectively to extract a number of first key points, a number of first descriptors, a number of second key points and a number of second descriptors; Calculate the Euclidean distance between any of the first descriptors and several of the second descriptors, and select the minimum value among the several Euclidean distances as the initial matching pair; The Euclidean distances corresponding to several initial matching pairs are compared with preset distances, and valid matching pairs are determined based on the comparison results. The number of valid matching pairs is counted, and the average value of the Euclidean distances corresponding to several valid matching pairs is calculated. A similarity score is determined based on the effective number and the average value, and related images are determined based on the similarity scores corresponding to several corrected images.
[0046] In this embodiment, the similarity score = number of valid samples / average score.
[0047] In this embodiment, the correlation between two images can also be determined by considering them to be the best if the sum of the SIFT values of two adjacent images is the largest and the feature difference is the smallest.
[0048] Specifically, the steps of performing cylindrical projection on the relevant image based on several candidate pixel focal lengths and calculating the quality of the stitched region include: Based on the focal length of any candidate pixel, cylindrical projection is performed on the relevant image to obtain several projected images; Several projected images are stitched together, and the quality evaluation index of the stitched area is calculated. The candidate pixel focal length with the best quality evaluation index is selected as the optimal pixel focal length.
[0049] In this embodiment, the quality evaluation indicators include, but are not limited to, structural similarity (SSIM) and mean square error (MSE). The stitching quality is evaluated by calculating the SSIM and MSE of the stitched area. The closer the SSIM is to 1, the higher the structural similarity of the stitched area. The smaller the MSE, the smaller the pixel difference of the stitched area. The optimal candidate pixel focal length is selected by combining the two.
[0050] In this embodiment, due to the limitations of the camera's shooting angle, adjusting the physical focal length is necessary to capture clear images of scenes that are both far from and close to the lens. Therefore, changes in the physical focal length will affect the panoramic stitching effect of 30 images. Based on the three sets of physical focal lengths (short focal length, medium focal length, and long focal length) and the optimal pixel focal length obtained by the cylindrical projection estimation algorithm, the three sets of pixel focal lengths can be calculated proportionally using the FOV calculation formula in spherical projection. , The dynamic field of view (FOV) can be calculated. This dynamic FOV is then used in a spherical projection algorithm for panoramic stitching.
[0051] Specifically, the steps of converting the plurality of corrected images into ERP images using a spherical projection algorithm include: The several corrected images are transformed from a two-dimensional planar coordinate system to a three-dimensional spherical coordinate system, wherein each pixel of the corrected image corresponds to a point on the sphere. Based on the three-dimensional spherical coordinate system, the corrected image is spherically projected to obtain the ERP image.
[0052] In this embodiment, since 2D images need to be generated for storage and viewing in reality, the spherical projection algorithm uses equidistant cylindrical projection (ERP) for representation. The image is projected from the world coordinate system onto the ERP through the spherical projection, thereby generating multiple images with transformed perspectives in the spherical coordinate system.
[0053] The principle of ERP is based on the assumption that a sphere and a cylinder are tangent to the equator. The latitude and longitude lines on the sphere are projected onto the cylinder, and then unfolded into a plane along a generatrix of the cylinder.
[0054] The principle of spherical projection is that projecting an object onto a sphere involves three coordinate systems: the world coordinate system, the camera coordinate system (spherical coordinate system), and the image plane coordinate system (spherical image coordinates, i.e., ERP format). The spherical coordinate system is as follows: Figure 2 As shown, when transforming the world coordinate system into the camera coordinate system, the origin of the camera coordinate system is at the center of the sphere. At this point, the camera coordinates are 3-dimensional. To represent this, they need to be transformed into planar coordinates, so two variables are defined. , . Corresponding longitude angle, starting from The positive semi-axis of the axis, i.e., the position of the prime meridian, is positive when rotated counterclockwise, and its range is... . Corresponding dimensions, starting from the equator, with upward rotation being positive. . Figure 4 Using a spherical coordinate system, Figure 5 The graph shows the z-axis coordinates of point P. Figure 6 For a graph showing the x and y coordinates of point P, see [link / reference]. Figure 4 , Figure 5 and Figure 6 As shown, the coordinates of point P can be expressed as... Based on the shooting angle, the horizontal and vertical rotation matrices can be calculated, allowing for viewpoint adjustment of the coordinates. Then, each 3D point on the sphere is projected onto the image plane to obtain its coordinates, and an ERP image is generated based on this coordinate mapping.
[0055] Specifically, the steps for generating the mask include: Based on the camera's physical parameters and rotation angle, calculate the actual coverage area of a single ERP image in the spherical coordinate system; By using the spherical projection formula, the pixel coordinates corresponding to the corrected image are mapped to the ERP plane to determine the spherical latitude and longitude corresponding to each ERP pixel; For each ERP pixel in the ERP image, determine whether it is within the actual coverage area based on its corresponding spherical latitude and longitude, and determine the valid and invalid pixels based on the determination result, and determine the mask based on the valid and invalid pixels; Based on the mask, multiple images corresponding to it are superimposed. If a certain pixel is marked as valid in any mask, then the pixel in the final mask is marked as valid, and the rest are invalid. The calculation result is saved as a binary matrix.
[0056] In this embodiment, in order to prevent holes from being generated during later fusion, it is necessary to calculate a mask, which is a binary mask matrix used to filter and control the projection results, indicating which pixels are valid and which pixels are invalid. The generated mask guides linear fusion.
[0057] In this embodiment, the physical parameters of the camera include the horizontal field of view (FOV_W), the vertical field of view (FOV_H), and the rotation angle of 36°.
[0058] In this embodiment, for each pixel in the ERP image ( According to its latitude and longitude ( Determine if the image is within the original image's shooting range: ,in, , Determined by the camera's rotation angle and field of view; Valid pixels: If the above conditions are met, mark it as 1 (white); Invalid pixel: If the condition is not met, mark it as 0 (black).
[0059] In this embodiment, when multiple masks are superimposed, as long as a certain pixel is 1 in any mask, that pixel is marked as 1 in the final mask.
[0060] Specifically, the step of progressively overlaying several ERP images using the mask and linear interpolation weights includes: Determine the position of the effective pixels in each ERP image, and calculate the linear interpolation weight of each ERP image in the overlay region based on the mask matrix; For each pixel in the overlapping region, the linear interpolation weight of that pixel in all ERP images is calculated based on its position. The corresponding pixel values of several ERP images are then weighted and averaged to obtain the pixel values of the final stitched image. Repeat the above steps, iterating through all overlapping pixels, until all ERP images are overlaid to obtain a panoramic image.
[0061] In this embodiment, multiple generated ERP images are stitched together, and overlapping areas are smoothly fused. By calculating masks and linear interpolation weights, the images and masks are gradually superimposed to generate a single stitched image. Due to the influence of rotation angle in spherical projection, the captured image is located in the center of the entire area. Depending on the subsequent camera rotation angle, the image position will change. Since the ERP display may have left and right side regions segmented, similar to the Atlantic Ocean being located on the left and right sides of a world map, precise control of image fusion is required during linear fusion. The fusion stage is also divided into left and right side segmented fusion, and the overall fusion yields a complete stitched image. The fusion stage uses linear weighting, Gaussian blur, and smoothing to generate a high-quality reference image weight mask, guiding the transition and fusion process and ensuring natural connection in overlapping areas.
[0062] In this embodiment, the linear fusion method used in the image fusion stage can be replaced with other fusion methods, such as Laplacian pyramid fusion.
[0063] Specifically, the steps for calculating the linear interpolation weights of each ERP image in the overlay region based on the mask matrix include: Based on the weight setting rules and the mask matrix and its corresponding spatial location, calculate the single weight map corresponding to any of the ERP images; Superimpose all the single-weighted graphs and normalize the single-graph weights.
[0064] In this embodiment, the weight setting rules include center decay weight, which is a linear decay of weight from the center of the image to the edge, with the center weight being the highest (e.g., 1) and the edge weight being the lowest (e.g., 0), or boundary smoothing weight, which is to apply Gaussian blur or gradient weight to the edges of the effective area to avoid hard boundaries.
[0065] In this embodiment, the generation of a weight map for a single image includes: Assume the coordinates of the image center are ( , ) For each pixel (x) 像 ,y 像 ), calculate the Euclidean distance d1 from it to the center point, and normalize d1; Linear decay weights are generated based on Euclidean distance d1: W i =max(1-d1,0), (center is 1, edge is 0); When combined with a mask matrix, only the weights of the effective mask regions are retained, while the weights of the remaining regions are set to 0.
[0066] In this embodiment, the total weight is calculated for each pixel in the panoramic image. If the total weight is greater than 1, the weight of each image is reduced proportionally, where the ratio is the ratio of a single weight to the total weight. If the total weight is less than or equal to 1, the original weight is maintained.
[0067] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.
[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A panoramic image stitching method for computer room scenarios based on spherical projection, characterized in that, include: Acquire several initial images, and perform geometric transformations on the initial images based on camera calibration data to obtain several corrected images; Identify images from a plurality of corrected images that are taken parallel to the ground, and determine an image group based on the identification results; The optimal pixel focal length is calculated based on the cylindrical projection estimation algorithm and the image group. The dynamic FOV is determined based on three sets of preset physical focal lengths, the optimal pixel focal length, and the FOV calculation formula. The corrected images are converted into ERP images using a spherical projection algorithm, and a mask is generated. The target image is obtained by progressively overlaying several ERP images using the mask and linear interpolation weights.
2. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 1, characterized in that, The steps of performing geometric transformations on several initial images based on camera calibration data include: Extract the intrinsic parameter matrix and distortion coefficients from the camera calibration data; Based on the intrinsic parameter matrix and the distortion coefficients, several initial images are corrected to obtain several corrected images.
3. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 2, characterized in that, The step of identifying the image that was taken parallel to the ground from among the several corrected images includes: Edge features are extracted from the corrected image, and significant horizontal structures in the edge features are detected based on the Hough transform. The main lines in the significant horizontal structure are extracted based on straight line fitting, and the actual tilt angle of the main lines is calculated. The actual tilt angle is compared with a preset angle threshold, and candidate images are determined based on the comparison results; The candidate images are simulated by cylindrical projection. The reprojection error of the projected candidate images in the vertical direction is calculated to verify whether they meet the geometric constraints of cylindrical projection. The image group is determined based on the verification results.
4. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 3, characterized in that, The steps for calculating the optimal pixel focal length based on the cylindrical projection estimation algorithm and the image group include: Determine the focal length range of candidate pixels; Calculate the SIFT features of several corrected images in the image group, calculate the image correlation based on the several SIFT features, and determine two related images based on the image correlation calculation results; Based on several candidate pixel focal lengths, the relevant images are cylindrically projected and the quality of the stitched region is calculated to select the optimal pixel focal length.
5. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 4, characterized in that, The steps for calculating image correlation based on several of the aforementioned SIFT features include: Determine the first candidate image and the second candidate image; The SIFT algorithm is applied to the first candidate image and the second candidate image respectively to extract a number of first key points, a number of first descriptors, a number of second key points and a number of second descriptors; Calculate the Euclidean distance between any of the first descriptors and several of the second descriptors, and select the minimum value among the several Euclidean distances as the initial matching pair; The Euclidean distances corresponding to several initial matching pairs are compared with preset distances, and valid matching pairs are determined based on the comparison results. The number of valid matching pairs is counted, and the average value of the Euclidean distances corresponding to several valid matching pairs is calculated. A similarity score is determined based on the effective number and the average value, and related images are determined based on the similarity scores corresponding to several corrected images.
6. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 5, characterized in that, The steps of performing cylindrical projection on the relevant images based on several candidate pixel focal lengths and calculating the quality of the stitched region include: Based on the focal length of any candidate pixel, cylindrical projection is performed on the relevant image to obtain several projected images; Several projected images are stitched together, and the quality evaluation index of the stitched area is calculated. The candidate pixel focal length with the best quality evaluation index is selected as the optimal pixel focal length.
7. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 6, characterized in that, The steps of converting the several corrected images into ERP images using a spherical projection algorithm include: The several corrected images are transformed from a two-dimensional planar coordinate system to a three-dimensional spherical coordinate system, wherein each pixel of the corrected image corresponds to a point on the sphere. Based on the three-dimensional spherical coordinate system, the corrected image is spherically projected to obtain the ERP image.
8. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 7, characterized in that, The steps to generate a mask include: Based on the camera's physical parameters and rotation angle, calculate the actual coverage area of a single ERP image in the spherical coordinate system; By using the spherical projection formula, the pixel coordinates corresponding to the corrected image are mapped to the ERP plane to determine the spherical latitude and longitude corresponding to each ERP pixel; For each ERP pixel in the ERP image, determine whether it is within the actual coverage area based on its corresponding spherical latitude and longitude, and determine the valid and invalid pixels based on the determination result, and determine the mask based on the valid and invalid pixels; Based on the mask, multiple images corresponding to it are superimposed. If a certain pixel is marked as valid in any mask, then the pixel in the final mask is marked as valid, and the rest are invalid. The calculation result is saved as a binary matrix.
9. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 8, characterized in that, The step of progressively overlaying several ERP images using the mask and linear interpolation weights includes: Determine the position of the effective pixels in each ERP image, and calculate the linear interpolation weight of each ERP image in the overlay region based on the mask matrix; For each pixel in the overlapping region, the linear interpolation weight of that pixel in all ERP images is calculated based on its position. The corresponding pixel values of several ERP images are then weighted and averaged to obtain the pixel values of the final stitched image. Repeat the above steps, iterating through all overlapping pixels, until all ERP images are overlaid to obtain a panoramic image.
10. The panoramic image stitching method for computer room scenarios based on spherical projection according to claim 9, characterized in that, The steps for calculating the linear interpolation weights of each ERP image in the overlay region based on the mask matrix include: Based on the weight setting rules and the mask matrix and its corresponding spatial location, calculate the single weight map corresponding to any of the ERP images; Superimpose all the single-weighted graphs and normalize the single-graph weights.