A video fusion method and fusion system for fisheye cameras

Abstract

The present invention provides a video fusion method and system for fisheye cameras, belonging to the field of video fusion technology. The method comprises obtaining original videos from at least two fisheye cameras, converting the fisheye images into planar images based on a pre-established image mapping table, determining the overlapping areas of the planar images and arranging them in descending order of size, calculating the splicing relationship of the first overlapping area and performing splicing, quantifying the similarity of subsequent overlapping areas to obtain a similarity value, and determining whether the similarity value meets a preset threshold. If so, splicing is performed directly; otherwise, the splicing is performed after updating the splicing relationship. The present invention improves splicing accuracy through feature matching and transformation matrix calculation; optimizes the processing of overlapping areas through descending order arrangement and similarity evaluation, and enhances the robustness of the system; can achieve high-quality panoramic video output in real-time applications, reduces the risk of mismatching, and ensures the visual effect of the final spliced ​​image.

Description

Technical Field

[0001] The present invention relates to the technical field of video fusion, and in particular to a video fusion method and a fusion system for a fisheye camera. Background Art

[0002] With the continuous development of video surveillance technology, fisheye cameras, due to their unique advantage of capturing an ultra-wide field of view, have been widely used in monitoring systems, virtual reality, panoramic photography, and other fields. However, due to their unique spherical lens design, fisheye cameras often capture images with severe distortion. This distortion not only affects visual quality but also poses challenges for subsequent image processing and video fusion.

[0003] In traditional video processing, directly stitching together raw videos captured by fisheye cameras often results in unsatisfactory stitching due to differences in image distortion, resulting in misalignment, ghosting, or noticeable stitching artifacts. Furthermore, the complex and variable perspective relationships between different fisheye cameras make detecting and stitching overlapping areas even more difficult.

[0004] In the existing technology, although there are some distortion correction and image stitching methods for fisheye images, these methods are mostly aimed at static images or single-view videos and are difficult to adapt to multi-camera and dynamically changing surveillance scenarios.

[0005] Therefore, it is necessary to provide a video fusion method and fusion system for a fisheye camera to solve the above technical problems. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a video fusion method and fusion system for a fisheye camera. Through a pre-established image mapping table, the fisheye image in the original video shot by the fisheye camera is mapped into a distortion-free plane image, effectively eliminating the influence of distortion on subsequent processing. At the same time, the method determines the overlapping area based on the camera viewing angle relationship, and arranges the overlapping areas in descending order according to their sizes, giving priority to large overlapping areas to improve the accuracy and efficiency of stitching.

[0007] The present invention provides a video fusion method for a fisheye camera, the fusion method comprising the following steps:

[0008] S1: acquiring original videos shot from at least two fisheye cameras respectively, and mapping the fisheye images in all original videos into planar images based on a pre-established image mapping table;

[0009] S2: determining overlapping areas of planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arranging the overlapping areas in descending order according to the size of the overlapping areas;

[0010] S3: calculating the splicing relationship of the overlapping regions that are at the top after sorting in descending order, and splicing the overlapping regions that are at the top according to the splicing relationship of the overlapping regions that are at the top;

[0011] S4: For subsequent overlapping regions, quantify the similarity between the current overlapping region and the previous overlapping region to obtain a similarity value;

[0012] S5: Determine whether the similarity value meets a preset threshold.

[0013] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0014] If not, the splicing relationship of the previous overlapping area is updated, and the updated splicing relationship is used to splice the current overlapping area.

[0015] Preferably, step S1 includes the following steps:

[0016] S101: identifying a single-frame fisheye image for the original video captured by each fisheye camera;

[0017] S102: Loading a pre-built image mapping table, wherein the mapping table contains conversion parameters from the fisheye image to the planar image, and the conversion parameters include a distortion correction coefficient, a projection model, and a corresponding mapping matrix;

[0018] S103: traversing all fisheye images, and transforming all fisheye images using the transformation parameters, so as to map all fisheye images into distortion-free plane images.

[0019] Preferably, step S2 includes the following steps:

[0020] S201: Calculating the coverage of the shooting area of ​​each fisheye camera in three-dimensional space according to the position, orientation, and focal length of each fisheye camera;

[0021] S202: Determine the overlapping area of ​​the coverage on the converted plane image by using a spatial geometric analysis method;

[0022] S203: Calculate the number of pixels in each overlapping area, and sort all overlapping areas in descending order according to the number of pixels.

[0023] Preferably, step S3 includes the following steps:

[0024] S301: extracting feature points from two adjacent plane images in the overlapping area that is at the first position after being sorted in descending order, and matching the extracted feature points using a SIFT feature matching method to obtain multiple feature matching pairs;

[0025] S302: Calculating a transformation matrix between two adjacent planar images based on the multiple feature matching pairs, wherein the transformation matrix includes a translation transformation relationship, a rotation transformation relationship, and a scaling transformation relationship;

[0026] S303: aligning two adjacent planar images in the first overlapping area using the transformation matrix;

[0027] S304: Using an image fusion method to stitch the two adjacent planar images in the aligned overlapping area.

[0028] Preferably, step S4 includes:

[0029] For subsequent overlapping areas, the pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the previous overlapping area to obtain the similarity value at the pixel level.

[0030] Preferably, in step S5, the splicing relationship of the previous overlapping area is transformed, including:

[0031] S501: Identifying feature differences between a current overlapping region and a previous overlapping region, wherein the feature differences include translation offset, rotation angle, and scale change;

[0032] S502: Determine update parameters according to the feature differences;

[0033] S503: Using the update parameters, update the splicing relationship of the previous overlapping area.

[0034] The present invention also provides a fisheye camera video fusion system for executing the fisheye camera video fusion method, the fusion system comprising:

[0035] An image mapping module is used to obtain original videos shot from at least two fisheye cameras respectively, and map the fisheye images in all original videos into planar images based on a pre-established image mapping table;

[0036] an overlapping area sorting module, configured to determine overlapping areas of planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and to sort the overlapping areas in descending order according to the size of the overlapping areas;

[0037] A first splicing module is used to calculate the splicing relationship of the overlapping areas that are at the top after the descending order is sorted, and splice the overlapping areas that are at the top according to the splicing relationship of the overlapping areas that are at the top;

[0038] A similarity quantification module is used to quantify the similarity between the current overlapping area and the previous overlapping area for subsequent overlapping areas to obtain a similarity value;

[0039] The second splicing module is used to determine whether the similarity value meets a preset threshold.

[0040] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0041] If not, the splicing relationship of the previous overlapping area is updated, and the updated splicing relationship is used to splice the current overlapping area.

[0042] Compared with related technologies, the video fusion method and fusion system of a fisheye camera provided by the present invention have the following beneficial effects:

[0043] The present invention uses the SIFT feature matching method to extract and match feature points, and then calculates the transformation matrix to align images from different cameras, thereby significantly improving the quality of the stitched image.

[0044] At the same time, through the pre-built image mapping table and parameters such as distortion correction coefficients, the fisheye image can be quickly and effectively converted into a planar image, which simplifies the image processing process and reduces the computational complexity.

[0045] The robustness of the system is enhanced by dynamically adjusting the stitching relationship to adapt to different camera positions and orientation changes. It can maintain good stitching effects even when the camera moves or the environment changes. By arranging the overlapping areas in descending order of size and processing them step by step, the largest overlapping areas are prioritized to ensure the integrity of the main visual information. At the same time, the stitching of subsequent areas is further optimized through similarity evaluation, which improves the consistency and coherence of the overall image. It can provide high-quality panoramic video output in real-time applications, reduce the risk of mismatching, and ensure the visual effect of the final stitched image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flow chart of a video fusion method for a fisheye camera provided by the present invention;

[0047] Figure 2 This is a module structure diagram of a video fusion system for a fisheye camera provided by the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all of the structures. Furthermore, the embodiments of the present invention and the features of the embodiments may be combined with one another unless there is a conflict.

[0049] It should also be noted that, for ease of description, only the part relevant to the present invention, rather than all of the content, is shown in the accompanying drawings. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow charts describe each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0050] Example 1

[0051] The present invention provides a video fusion method of a fisheye camera, referring to Figure 1 As shown, the fusion method includes the following steps:

[0052] S1: Original videos captured by at least two fisheye cameras are obtained respectively, and fisheye images in all original videos are mapped into planar images based on a pre-established image mapping table.

[0053] In this embodiment, the original videos captured by at least two fisheye cameras are first obtained. In order to convert the fisheye image into a planar image, mapping is required based on a pre-established image mapping table. The image mapping table contains conversion parameters from the fisheye image to the planar image, including a distortion correction coefficient, a projection model, and a corresponding mapping matrix. Through these parameters, the image distortion caused by the fisheye lens can be effectively eliminated, and the fisheye image can be converted into a planar image close to the real scene.

[0054] S2: Determine overlapping areas of planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arrange them in descending order according to the size of the overlapping areas.

[0055] In this embodiment, based on the planar image converted in step S1, overlapping areas under different camera viewing angles are determined, and the overlapping areas are sorted according to their sizes for subsequent stitching operations.

[0056] S3: Calculate the splicing relationship of the overlapping region that is at the top after sorting in descending order, and splice the overlapping region that is at the top according to the splicing relationship of the overlapping region that is at the top.

[0057] S4: For subsequent overlapping regions, quantify the similarity between the current overlapping region and the previous overlapping region to obtain a similarity value.

[0058] S5: Determine whether the similarity value meets a preset threshold.

[0059] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0060] If not, the splicing relationship of the previous overlapping area is updated, and the updated splicing relationship is used to splice the current overlapping area.

[0061] Specifically, step S1 includes the following steps:

[0062] S101: For each original video captured by the fisheye camera, identify a single-frame fisheye image.

[0063] In this embodiment, for each original video captured by the fisheye camera, the video stream is read frame by frame, and frame by frame processing is performed to ensure that each frame of image can be correctly converted.

[0064] Each frame of image is preprocessed, including but not limited to grayscale conversion and denoising, to reduce the complexity of subsequent processing and improve image quality and processing efficiency.

[0065] The fisheye image in each frame is identified to prepare for subsequent image conversion. The identified fisheye image serves as the basis for subsequent processing, ensuring the accuracy of the processed object.

[0066] S102: Loading a pre-built image mapping table, wherein the mapping table contains conversion parameters from the fisheye image to the planar image, and the conversion parameters include a distortion correction coefficient, a projection model, and a corresponding mapping matrix.

[0067] In this embodiment, a pre-built image mapping table is loaded. The mapping table is generated during the fisheye camera calibration process. The conversion parameters in the mapping table include distortion correction coefficients, projection models and corresponding mapping matrices. These parameters ensure the accuracy of the conversion from fisheye images to planar images.

[0068] By using pre-built mapping tables, the image conversion process is simplified and the processing speed is improved. At the same time, the conversion parameters ensure the accuracy of distortion correction and projection transformation, thereby improving image quality.

[0069] S103: traversing all fisheye images, and transforming all fisheye images using the transformation parameters, so as to map all fisheye images into distortion-free plane images.

[0070] In this embodiment, all fisheye images identified in step S101 are traversed, and distortion correction is performed on each fisheye image using the conversion parameters in the mapping table loaded in step S102. The projection model and the corresponding mapping matrix are applied to convert the distortion-corrected fisheye image into a planar image.

[0071] Furthermore, the converted flat images may be further processed, including but not limited to cropping or resizing, to ensure that all images have consistent dimensions.

[0072] Distortion correction and projection transformation significantly improve the image quality and reduce the distortion at the image edges. The flat image is more suitable for subsequent image stitching operations, which helps to improve the accuracy and visual effect of the stitched image.

[0073] Specifically, step S2 includes the following steps:

[0074] S201: Calculate the coverage of the shooting area of ​​each fisheye camera in three-dimensional space according to the position, orientation and focal length of each fisheye camera.

[0075] In this embodiment, the position coordinates, orientation angle, and focal length of each fisheye camera are obtained. The horizontal and vertical field of view angles of each camera are calculated based on the focal length and sensor size. A world coordinate system is defined to facilitate the calculation of the 3D coverage of each camera. Then, the boundary points of the 3D coverage of each camera are calculated:

[0076] Calculate the coordinates of the boundary points of each camera in the world coordinate system based on the field of view angle and focal length;

[0077] Use the spherical coordinates to Cartesian coordinates conversion formula to convert the spherical coordinates of the boundary points to Cartesian coordinates.

[0078] S202: Determine the overlapping area of ​​the coverage on the converted plane image by using a spatial geometric analysis method.

[0079] In this embodiment, the three-dimensional spatial coverage calculated in step S201 is projected onto a two-dimensional plane, and a spatial geometric analysis method is used to determine the overlapping parts between the converted plane images under the perspectives of the various cameras.

[0080] For each camera, the 3D boundary points are projected onto a 2D plane using a perspective projection matrix.

[0081] Use polygon intersection algorithms from computer graphics to determine overlapping areas.

[0082] Image processing techniques, including but not limited to edge detection and shape matching methods, are used to further refine the boundaries of the overlapping area. Canny edge detection is applied to identify the edges of the overlapping area. Hough transform method is used to fit the edge contour and determine the most appropriate overlapping area.

[0083] S203: Calculate the number of pixels in each overlapping area, and sort all overlapping areas in descending order according to the number of pixels.

[0084] In this embodiment, the area of ​​each overlapping region determined in step S202, ie, the number of pixels, is calculated. Specifically, image processing software or a programming language (including Python's OpenCV library) is used to calculate the area of ​​the overlapping region.

[0085] Arrange all overlapping areas in descending order of area size. This helps prioritize larger overlapping areas during subsequent stitching. By processing larger overlapping areas first, a more stable "skeleton" can be established during the stitching process. Subsequent smaller overlapping areas can be more easily aligned with this skeleton. The larger overlapping area can be used as a benchmark, and the smaller overlapping areas can be adjusted based on this benchmark, thus simplifying the entire stitching process.

[0086] The overlapping regions are sorted by area using a sorting method. The sorted results are used to guide the stitching order in step S3, with priority given to overlapping regions with larger areas.

[0087] Specifically, step S3 includes the following steps:

[0088] S301: extracting feature points from two adjacent plane images in the overlapping area that is at the first position after being sorted in descending order, and matching the extracted feature points using a SIFT feature matching method to obtain a plurality of feature matching pairs.

[0089] In this embodiment, a Scale Invariant Feature Transform (SIFT) algorithm is used to extract feature points from two adjacent planar images in the overlapping region that is first in descending order. The SIFT algorithm detects key points in the image and generates a descriptor for each key point.

[0090] Use Difference of Gaussian (DoG) pyramid to detect key points and find local extreme points in the image.

[0091] Descriptors are extracted from the area around each key point and the descriptors are formed using the gradient direction and magnitude information.

[0092] Feature matching is performed using the feature descriptors in the SIFT algorithm to obtain multiple feature matching pairs.

[0093] Compare the SIFT descriptors of the two images and find the closest descriptor pair as a candidate matching pair.

[0094] A ratio test strategy is used to remove false matches, that is, if the ratio of the nearest neighbor distance to the next nearest neighbor distance is less than a certain threshold (usually 0.8), the match is considered valid.

[0095] S302: Based on the multiple feature matching pairs, calculate a transformation matrix between two adjacent planar images, wherein the transformation matrix includes a translation transformation relationship, a rotation transformation relationship, and a scaling transformation relationship.

[0096] In this embodiment, the RANSAC algorithm (Random Sample Consensus Algorithm) or other robust estimation methods are used to select the best matching pair from the feature matching pairs to eliminate possible erroneous matching.

[0097] Randomly select the minimum number of matching point pairs (usually 3 pairs for affine transformation and 4 pairs for perspective transformation) and calculate the corresponding transformation matrix.

[0098] Repeat the above process multiple times, selecting different matching point pairs each time, and counting how many matching point pairs under each transformation matrix conform to the transformation.

[0099] The transformation matrix with the most inliers is selected as the final transformation matrix.

[0100] Based on the selected best matching pair, the transformation matrix between two adjacent planar images is calculated, which contains translation, rotation and scaling information.

[0101] S303: aligning two adjacent planar images in the first overlapping area using the transformation matrix.

[0102] In this embodiment, the transformation matrix calculated in step S302 is used to map pixel positions in one planar image to corresponding positions in another image, thereby achieving image alignment.

[0103] Applies a transformation matrix to one image, adjusting its size, position, and orientation to align it with another image, and uses bilinear interpolation or nearest neighbor interpolation to account for any resulting shifts in pixel positions.

[0104] S304: Using an image fusion method to stitch the two adjacent planar images in the aligned overlapping area.

[0105] In this embodiment, the aligned images are fused to generate a complete stitched image, and the overlapping areas are fused at the pixel level using a pixel-level fusion method to avoid obvious boundaries. During the fusion process, gradient blending technology can also be used to reduce the visual differences at the stitching points to ensure a natural transition of the fused image.

[0106] Specifically, step S4 includes:

[0107] For subsequent overlapping areas, the pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the previous overlapping area to obtain the similarity value at the pixel level.

[0108] In this embodiment, for the overlapping area that has been spliced ​​in step S3, the next overlapping area is selected.

[0109] The pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the overlapping area that has been stitched. Pixel similarity can be achieved by calculating the color difference between the corresponding pixels in the two overlapping areas. Specifically, the mean square error (MSE) or structural similarity index (SSIM) is used to quantify the similarity between pixels.

[0110] Specifically, the corresponding pixels of the two overlapping areas are paired, and for each pixel pair, their color difference is calculated (using the distance in the RGB color space), and the color differences of all pixel pairs are averaged to obtain the mean square error (MSE) or the structural similarity (SSIM) method.

[0111] Specifically, in step S5, the splicing relationship of the previous overlapping area is transformed, including:

[0112] S501: Identify feature differences between a current overlapping region and a previous overlapping region, wherein the feature differences include translation offset, rotation angle, and scaling change.

[0113] In this embodiment, the feature differences between the image contents of the current overlapping region and the last overlapped region that has been stitched are identified, including:

[0114] Translation offset: The relative movement between two overlapping areas in the horizontal and vertical directions.

[0115] Rotation Angle: The relative rotation angle between the two overlapping areas.

[0116] Scaling change: The relative enlargement or reduction ratio between two overlapping areas.

[0117] Next, the SIFT feature matching method is used to find matching feature points between the two overlapping areas, and the translation offset, rotation angle and scale change are calculated based on the matching feature points.

[0118] S502: Determine update parameters according to the feature differences.

[0119] In this embodiment, according to the characteristic differences identified in S501, parameters for updating the splicing relationship are determined, wherein the update parameters include:

[0120] Translation Vector: A two-dimensional vector used to describe the translation offset.

[0121] Rotation Matrix: A two-dimensional rotation matrix used to describe the rotation angle.

[0122] Scaling factor: A proportional factor used to describe scale changes.

[0123] Calculating the updated parameters based on the identified feature differences includes the following steps:

[0124] a: Calculate the translation update amount:

[0125] Based on the set of matched feature points, the translation offset between the two overlapping areas is calculated, and the translation update amount is obtained by averaging the translation offsets of all matching point pairs.

[0126] b: Calculate the rotation update amount:

[0127] The matched feature points are used to estimate the rotation angle update between the two overlapping areas, and the least squares method is used to calculate the rotation update to ensure the accuracy of the rotation angle update.

[0128] c: Calculate the zoom update amount:

[0129] Based on the set of matched feature points, the scale change update between the two overlapping areas is calculated, and the scale update is obtained by averaging the scale changes of all matching point pairs.

[0130] In addition, the least squares method can be used to optimize the translation update amount, rotation update amount, and scale update amount to ensure the accuracy of the updated parameters.

[0131] S503: Using the update parameters, update the splicing relationship of the previous overlapping area.

[0132] In this embodiment, the splicing relationship of the previous overlapping area is updated according to the update parameters determined in S502, and the splicing relationship of the previous overlapping area is adjusted using the determined update parameters (translation vector, rotation matrix, scaling factor).

[0133] Exemplarily, the update can be achieved by modifying the transformation matrix, and the updated splicing relationship will be used to splice the current overlapping area.

[0134] The working principle of the video fusion method of a fisheye camera provided by the present invention is as follows:

[0135] First, raw video is captured from fisheye cameras at different locations. Each frame of the fisheye image is then accurately mapped into a distortion-free flat image. Next, by analyzing the spatial layout and shooting angle of the cameras, the overlapping areas between these flat images can be intelligently identified and sorted according to their importance (such as area size).

[0136] For overlapping areas, a feature matching method is used to calculate the transformation relationship between adjacent images, thereby achieving high-precision image alignment. Subsequently, an image fusion algorithm is used to ensure that these overlapping areas can transition naturally when stitching, with almost no visible seams.

[0137] For subsequent processing of overlapping regions, an adaptive similarity assessment mechanism is introduced. This mechanism dynamically determines whether the stitching strategy needs to be adjusted by comparing the pixel-level differences between the current overlapping region and the already stitched region. If the differences are small (i.e., the similarity is high), the previous stitching parameters are retained to improve processing efficiency. If the differences are significant, the stitching parameters are automatically adjusted based on the detected feature changes to ensure continuity and consistency throughout the video fusion process.

[0138] In summary, this innovative technology not only solves the image distortion and splicing problems in fisheye camera video fusion, but also significantly improves the quality and efficiency of video fusion through intelligent processing and adaptive adjustment strategies.

[0139] Example 2

[0140] The present invention also provides a fisheye camera video fusion system for executing the fisheye camera video fusion method, referring to Figure 2 As shown, the fusion system includes:

[0141] The image mapping module 100 is configured to obtain original videos shot from at least two fisheye cameras respectively, and map the fisheye images in all the original videos into planar images based on a pre-established image mapping table.

[0142] The overlapping area sorting module 200 is used to determine the overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and sort them in descending order according to the size of the overlapping area.

[0143] The first splicing module 300 is configured to calculate the splicing relationship of the first overlapping region after being sorted in descending order, and splice the first overlapping region according to the splicing relationship of the first overlapping region.

[0144] The similarity quantification module 400 is configured to quantify the similarity between the current overlapping region and the previous overlapping region for subsequent overlapping regions to obtain a similarity value.

[0145] The second splicing module 500 is used to determine whether the similarity value meets a preset threshold.

[0146] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0147] If not, the splicing relationship of the previous overlapping area is updated, and the updated splicing relationship is used to splice the current overlapping area.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0150] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. A video fusion method for a fisheye camera, characterized in that: The fusion method comprises the following steps: S1: Original videos captured by at least two fisheye cameras are obtained respectively, and fisheye images in all original videos are mapped into planar images based on a pre-established image mapping table. Step S1 includes the following steps: S101: identifying a single-frame fisheye image for the original video captured by each fisheye camera; S102: Loading a pre-built image mapping table, wherein the mapping table contains conversion parameters from the fisheye image to the planar image, and the conversion parameters include a distortion correction coefficient, a projection model, and a corresponding mapping matrix; S103: traversing all fisheye images, and transforming all fisheye images using the transformation parameters, so as to map all fisheye images into plane images without distortion; S2: determining overlapping areas of planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arranging the overlapping areas in descending order according to the size of the overlapping areas; S3: Calculate the splicing relationship of the overlapping areas that are at the top after sorting in descending order, and splice the overlapping areas that are at the top according to the splicing relationship of the overlapping areas that are at the top. Step S3 includes the following steps: S301: extracting feature points from two adjacent plane images in the overlapping area that is at the first position after being sorted in descending order, and matching the extracted feature points using a SIFT feature matching method to obtain multiple feature matching pairs; S302: Calculating a transformation matrix between two adjacent planar images based on the multiple feature matching pairs, wherein the transformation matrix includes a translation transformation relationship, a rotation transformation relationship, and a scaling transformation relationship; S303: aligning two adjacent planar images in the first overlapping area using the transformation matrix; S304: Using an image fusion method to stitch the two adjacent plane images in the aligned overlapping area; S4: For subsequent overlapping regions, quantify the similarity between the current overlapping region and the previous overlapping region to obtain a similarity value; S5: Determine whether the similarity value meets a preset threshold. If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area. If not, the splicing relationship of the previous overlapping area is updated, and the updated splicing relationship is used to splice the current overlapping area.

2. The video fusion method of a fisheye camera according to claim 1, characterized in that: Step S2 includes the following steps: S201: Calculating the coverage of the shooting area of ​​each fisheye camera in three-dimensional space according to the position, orientation, and focal length of each fisheye camera; S202: Determine the overlapping area of ​​the coverage on the converted plane image by using a spatial geometric analysis method; S203: Calculate the number of pixels in each overlapping area, and sort all overlapping areas in descending order according to the number of pixels.

3. The video fusion method of a fisheye camera according to claim 2, characterized in that: Step S4 includes: For subsequent overlapping areas, the pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the previous overlapping area to obtain the similarity value at the pixel level.

4. The video fusion method of a fisheye camera according to claim 3, characterized in that: In step S5, the splicing relationship of the previous overlapping area is transformed, including: S501: Identifying feature differences between a current overlapping region and a previous overlapping region, wherein the feature differences include translation offset, rotation angle, and scale change; S502: Determine update parameters according to the feature differences; S503: Using the update parameters, update the splicing relationship of the previous overlapping area.

5. A fisheye camera video fusion system, used to execute the fisheye camera video fusion method according to any one of claims 1 to 4, characterized in that: The fusion system includes: An image mapping module is used to obtain original videos shot from at least two fisheye cameras respectively, and map the fisheye images in all original videos into planar images based on a pre-established image mapping table; an overlapping area sorting module, configured to determine overlapping areas of planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and to sort the overlapping areas in descending order according to the size of the overlapping areas; A first splicing module is used to calculate the splicing relationship of the overlapping areas that are at the top after the descending order is sorted, and splice the overlapping areas that are at the top according to the splicing relationship of the overlapping areas that are at the top; A similarity quantification module is used to quantify the similarity between the current overlapping area and the previous overlapping area for subsequent overlapping areas to obtain a similarity value; The second splicing module is used to determine whether the similarity value meets a preset threshold. If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area. If not, the splicing relationship of the previous overlapping area is updated, and the updated splicing relationship is used to splice the current overlapping area.

Images