Video fusion method and fusion system of fisheye camera

Through the pre-established image mapping table and SIFT feature matching method, the problem of image distortion and stitching in fisheye camera videos is solved, and high-quality panoramic video output is achieved, adapting to multi-camera and dynamically changing surveillance scenarios.

CN119067842AActive Publication Date: 2024-12-03SICHUAN GUANGXIN TIANXIA MEDIA CO LTD

Patent Information

Application Number
CN202411093497.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The images captured by fisheye cameras are severely distorted due to the spherical lens design, which affects the video stitching effect. The existing technology is difficult to adapt to multi-camera and dynamically changing surveillance scenarios.

Method used

The fisheye image is mapped into a plane image without distortion through a pre-established image mapping table, and the overlapping area is determined based on the camera's viewing angle relationship. The transformation matrix is ​​calculated by using the SIFT feature matching method for splicing, and the splicing relationship is dynamically adjusted to adapt to the camera position and orientation change. .

Benefits of technology

It significantly improves the quality of stitching images, simplifies the image processing process, reduces the computational complexity, enhances the robustness of the system, and ensures high-quality output and visual effects of panoramic videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067842A_ABST
    Figure CN119067842A_ABST
Patent Text Reader

Abstract

The invention provides a video fusion method and fusion system for fisheye cameras, and belongs to the technical field of video fusion, and the method comprises the steps: respectively obtaining original videos from at least two fisheye cameras, and converting fisheye images into plane images based on a pre-established image mapping table; determining overlapping areas of the plane images and arranging the overlapping areas in a descending order according to sizes; calculating a splicing relation of the first overlapping area and performing splicing; performing similarity quantification on the subsequent overlapping regions to obtain a similarity value; whether the similarity value meets a preset threshold value or not is judged, if yes, direct splicing is conducted, and if not, splicing is conducted again after the splicing relation is updated. According to the invention, through feature matching and transformation matrix calculation, the splicing precision is improved; through descending order arrangement and similarity evaluation, the processing of overlapped regions is optimized, and the robustness of the system is enhanced; according to the invention, high-quality panoramic video output in real-time application can be realized, the mismatching risk is reduced, and the visual effect of the final spliced image is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video fusion, and in particular to a video fusion method and a fusion system of a fisheye camera. Background Art

[0002] With the continuous development of video surveillance technology, fisheye cameras have been widely used in monitoring systems, virtual reality, panoramic shooting and other fields due to their unique advantage of being able to capture ultra-wide fields of view. However, due to their special spherical lens design, the images captured by fisheye cameras often have severe distortion, which not only affects the visual effect, but also brings challenges to subsequent image processing and video fusion.

[0003] In the traditional video processing process, directly stitching the original videos shot by the fisheye camera often results in unsatisfactory stitching effects due to the distortion differences between the images, resulting in misalignment, ghosting, or obvious stitching marks. In addition, the perspective relationship between different fisheye cameras is complex and changeable, making the detection and stitching of overlapping areas more difficult.

[0004] In the prior art, although there are some distortion correction and image stitching methods for fisheye images, most of these methods are for static images or videos with a single perspective and are difficult to adapt to multi-camera and dynamically changing surveillance scenarios.

[0005] Therefore, it is necessary to provide a video fusion method and fusion system for a fisheye camera to solve the above technical problems. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a video fusion method and a fusion system for a fisheye camera. Through a pre-established image mapping table, the fisheye image in the original video taken by the fisheye camera is mapped into a distortion-free plane image, which effectively eliminates the influence of distortion on subsequent processing. At the same time, the method determines the overlapping area based on the camera viewing angle relationship, and arranges the overlapping areas in descending order according to their sizes, giving priority to large overlapping areas to improve the accuracy and efficiency of stitching.

[0007] The present invention provides a video fusion method of a fisheye camera, the fusion method comprising the following steps:

[0008] S1: acquiring original videos shot from at least two fisheye cameras respectively, and mapping the fisheye images in all the original videos into planar images based on a pre-established image mapping table;

[0009] S2: determining an overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arranging the overlapping areas in descending order according to the size of the overlapping areas;

[0010] S3: calculating the splicing relationship of the first overlapping region after the descending order, and splicing the first overlapping region according to the splicing relationship of the first overlapping region;

[0011] S4: for subsequent overlapping regions, quantify the similarity between the current overlapping region and the previous overlapping region to obtain a similarity value;

[0012] S5: Determine whether the similarity value meets a preset threshold.

[0013] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0014] If not, the splicing relationship of the previous overlapping area is updated, and the current overlapping area is spliced ​​using the updated splicing relationship.

[0015] Preferably, step S1 comprises the following steps:

[0016] S101: for each original video captured by the fisheye camera, identify a single-frame fisheye image;

[0017] S102: Loading a pre-built image mapping table, wherein the mapping table contains conversion parameters from a fisheye image to a plane image, and the conversion parameters include a distortion correction coefficient, a projection model, and a corresponding mapping matrix;

[0018] S103: traversing all fisheye images, and transforming all fisheye images using the transformation parameters, so as to map all fisheye images into distortion-free plane images.

[0019] Preferably, step S2 comprises the following steps:

[0020] S201: Calculating the coverage of the shooting area of ​​each fisheye camera in the three-dimensional space according to the position, orientation and focal length of each fisheye camera;

[0021] S202: Determine the overlapping area of ​​the coverage on the converted plane image by using a spatial geometric analysis method;

[0022] S203: Calculate the number of pixels in each overlapping area, and arrange all overlapping areas in descending order according to the number of pixels.

[0023] Preferably, step S3 comprises the following steps:

[0024] S301: extracting feature points from two adjacent plane images in the overlapping area that are at the first position after being arranged in descending order, and matching the extracted feature points using a SIFT feature matching method to obtain a plurality of feature matching pairs;

[0025] S302: Based on the multiple feature matching pairs, calculate a transformation matrix between two adjacent plane images, wherein the transformation matrix includes a translation transformation relationship, a rotation transformation relationship, and a scaling transformation relationship;

[0026] S303: aligning two adjacent plane images in the overlapping area at the first position using the transformation matrix;

[0027] S304: Using an image fusion method to stitch the two adjacent plane images in the aligned overlapping area.

[0028] Preferably, step S4 comprises:

[0029] For subsequent overlapping areas, a pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the previous overlapping area to obtain a similarity value at the pixel level.

[0030] Preferably, in step S5, the splicing relationship of the previous overlapping area is transformed, including:

[0031] S501: Identify feature differences between a current overlapping region and a previous overlapping region, wherein the feature differences include translation offset, rotation angle, and scale change;

[0032] S502: Determine update parameters according to the feature difference;

[0033] S503: Using the update parameters, update the splicing relationship of the previous overlapping area.

[0034] The present invention also provides a video fusion system of a fisheye camera, which is used to execute the video fusion method of a fisheye camera, and the fusion system comprises:

[0035] An image mapping module, used to obtain original videos shot from at least two fisheye cameras respectively, and map the fisheye images in all original videos into planar images based on a pre-established image mapping table;

[0036] An overlapping area sorting module, used to determine the overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arrange them in descending order according to the size of the overlapping area;

[0037] A first splicing module, used for calculating the splicing relationship of the first overlapping region after the descending order is arranged, and splicing the first overlapping region according to the splicing relationship of the first overlapping region;

[0038] A similarity quantification module is used to quantify the similarity between the current overlapping area and the previous overlapping area for subsequent overlapping areas to obtain a similarity value;

[0039] The second splicing module is used to determine whether the similarity value meets a preset threshold.

[0040] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0041] If not, the splicing relationship of the previous overlapping area is updated, and the current overlapping area is spliced ​​using the updated splicing relationship.

[0042] Compared with the related art, the video fusion method and fusion system of a fisheye camera provided by the present invention have the following beneficial effects:

[0043] The present invention uses the SIFT feature matching method to extract and match feature points, and then calculates a transformation matrix to align images from different cameras, thereby significantly improving the quality of the spliced ​​image.

[0044] At the same time, through the pre-constructed image mapping table and parameters such as distortion correction coefficients, the fisheye image can be quickly and effectively converted into a planar image, which simplifies the image processing process and reduces the computational complexity.

[0045] The robustness of the system is enhanced by dynamically adjusting the stitching relationship to adapt to different camera positions and orientation changes, and a good stitching effect can be maintained even when the camera moves or the environment changes. The overlapping areas are arranged in descending order of size and processed step by step, with the largest overlapping areas being prioritized to ensure the integrity of the main visual information. The stitching of subsequent areas is further optimized through similarity evaluation, which improves the consistency and coherence of the overall image. It can provide high-quality panoramic video output in real-time applications, reduce the risk of mismatching, and ensure the visual effect of the final stitched image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flow chart of a video fusion method of a fisheye camera provided by the present invention;

[0047] Figure 2 A module structure diagram of a video fusion system of a fisheye camera provided by the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0049] It should also be noted that, for ease of description, only the part relevant to the present invention but not all the content is shown in the accompanying drawings. It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as the processing or method described as a flow chart. Although the flow chart describes each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0050] Embodiment 1

[0051] The present invention provides a video fusion method of a fisheye camera, referring to Figure 1 As shown, the fusion method comprises the following steps:

[0052] S1: Original videos captured by at least two fisheye cameras are obtained respectively, and fisheye images in all original videos are mapped into planar images based on a pre-established image mapping table.

[0053] In this embodiment, the original videos shot by at least two fisheye cameras are first obtained respectively. In order to convert the fisheye image into a plane image, it is necessary to perform mapping based on a pre-established image mapping table. The image mapping table contains conversion parameters from the fisheye image to the plane image, including a distortion correction coefficient, a projection model and a corresponding mapping matrix. Through these parameters, the image distortion caused by the fisheye lens can be effectively eliminated, and the fisheye image can be converted into a plane image close to the real scene.

[0054] S2: Determine the overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arrange them in descending order according to the size of the overlapping area.

[0055] In this embodiment, based on the planar image converted in step S1, the overlapping areas under different camera viewing angles are determined, and the overlapping areas are sorted according to their sizes for subsequent stitching operations.

[0056] S3: Calculate the splicing relationship of the first overlapping region after the descending order, and splice the first overlapping region according to the splicing relationship of the first overlapping region.

[0057] S4: For subsequent overlapping regions, quantify the similarity between the current overlapping region and the previous overlapping region to obtain a similarity value.

[0058] S5: Determine whether the similarity value meets a preset threshold.

[0059] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0060] If not, the splicing relationship of the previous overlapping area is updated, and the current overlapping area is spliced ​​using the updated splicing relationship.

[0061] Specifically, step S1 includes the following steps:

[0062] S101: For each original video captured by the fisheye camera, a single-frame fisheye image is identified.

[0063] In this embodiment, for each original video captured by the fisheye camera, the video stream is read frame by frame, and each frame of the image is ensured to be correctly converted through frame-by-frame processing.

[0064] Each frame of the image is preprocessed, including but not limited to grayscale conversion and denoising, to reduce the complexity of subsequent processing and improve image quality and processing efficiency.

[0065] The fisheye image in each frame is identified to prepare for subsequent image conversion. The identified fisheye image serves as the basis for subsequent processing, ensuring the accuracy of the processed object.

[0066] S102: Loading a pre-built image mapping table, wherein the mapping table contains conversion parameters from a fisheye image to a plane image, and the conversion parameters include a distortion correction coefficient, a projection model, and a corresponding mapping matrix.

[0067] In this embodiment, a pre-built image mapping table is loaded. The mapping table is generated during the fisheye camera calibration process. The conversion parameters in the mapping table include distortion correction coefficients, projection models and corresponding mapping matrices. These parameters ensure the accuracy of the conversion from fisheye images to planar images.

[0068] By using pre-built mapping tables, the image conversion process is simplified and the processing speed is improved. At the same time, the conversion parameters ensure the accuracy of distortion correction and projection transformation, thereby improving image quality.

[0069] S103: traversing all fisheye images, and transforming all fisheye images using the transformation parameters, so as to map all fisheye images into distortion-free plane images.

[0070] In this embodiment, all fisheye images identified in step S101 are traversed, and distortion correction is performed on each fisheye image using the conversion parameters in the mapping table loaded in step S102. The projection model and the corresponding mapping matrix are applied to convert the fisheye image after distortion correction into a plane image.

[0071] Furthermore, the converted flat images may be further processed, including but not limited to cropping or resizing, to ensure that all images have a consistent size.

[0072] Distortion correction and projection transformation significantly improve the image quality and reduce the distortion at the image edges. The flat image is more suitable for subsequent image stitching operations, which helps to improve the accuracy and visual effect of the stitched image.

[0073] Specifically, step S2 includes the following steps:

[0074] S201: Calculate the coverage of the shooting area of ​​each fisheye camera in three-dimensional space according to the position, orientation and focal length of each fisheye camera.

[0075] In this embodiment, the position coordinates, orientation angle and focal length of each fisheye camera are obtained, the horizontal and vertical field of view angles of each camera are calculated according to the focal length and sensor size, a world coordinate system is defined to facilitate the calculation of the three-dimensional coverage of each camera, and then the three-dimensional coverage boundary points of each camera are calculated:

[0076] Calculate the coordinates of the boundary points of each camera in the world coordinate system based on the field of view angle and focal length;

[0077] Use the spherical coordinates to Cartesian coordinates conversion formula to convert the spherical coordinates of the boundary points to Cartesian coordinates.

[0078] S202: Determine the overlapping area of ​​the coverage on the converted plane image by using a spatial geometric analysis method.

[0079] In this embodiment, the three-dimensional space coverage calculated in step S201 is projected onto a two-dimensional plane, and the overlapping parts between the converted plane images under the viewing angles of the various cameras are determined using a spatial geometric analysis method.

[0080] For each camera, the 3D boundary points are projected onto a 2D plane using a perspective projection matrix.

[0081] Use polygon intersection algorithms from computer graphics to determine overlapping areas.

[0082] Image processing techniques, including but not limited to edge detection and shape matching methods, are used to further refine the boundaries of the overlapping area. Canny edge detection is applied to identify the edges of the overlapping area. Hough transform method is used to fit the edge contour and determine the most appropriate overlapping area.

[0083] S203: Calculate the number of pixels in each overlapping area, and arrange all overlapping areas in descending order according to the number of pixels.

[0084] In this embodiment, the area of ​​each overlapping region determined in step S202, ie, the number of pixels, is calculated. Specifically, image processing software or a programming language (including Python's OpenCV library) is used to calculate the area of ​​the overlapping region.

[0085] Arrange all overlapping areas in descending order by area size. This helps prioritize larger overlapping areas during subsequent stitching. By processing larger overlapping areas first, a more stable "skeleton" can be established during the stitching process. Subsequent small overlapping areas can be more easily aligned with this skeleton. Larger overlapping areas can be used as a benchmark, and smaller overlapping areas can be adjusted based on this benchmark, thus simplifying the entire stitching process.

[0086] The overlapping regions are sorted by area using a sorting method, and the sorted results are used to guide the stitching order in step S3, with the overlapping regions with larger areas being processed first.

[0087] Specifically, step S3 includes the following steps:

[0088] S301: extracting feature points from two adjacent plane images in the overlapping area that are at the first place after being arranged in descending order, and matching the extracted feature points using a SIFT feature matching method to obtain a plurality of feature matching pairs.

[0089] In this embodiment, a Scale Invariant Feature Transform (SIFT) algorithm is used to extract feature points from two adjacent plane images in the overlapping region that is at the first position after being arranged in descending order. The SIFT algorithm detects key points in the image and generates a descriptor for each key point.

[0090] Use Difference of Gaussian (DoG) pyramid to detect key points and find local extreme points in the image.

[0091] Descriptors are extracted from the area around each key point, and descriptors are formed using gradient direction and magnitude information.

[0092] Feature matching is performed using the feature descriptors in the SIFT algorithm to obtain multiple feature matching pairs.

[0093] Compare the SIFT descriptors of the two images and find the closest descriptor pair as a candidate matching pair.

[0094] A ratio test strategy is used to remove false matches, that is, if the ratio of the nearest neighbor distance to the next nearest neighbor distance is less than a certain threshold (usually 0.8), the match is considered valid.

[0095] S302: Based on the multiple feature matching pairs, a transformation matrix between two adjacent planar images is calculated, wherein the transformation matrix includes a translation transformation relationship, a rotation transformation relationship, and a scaling transformation relationship.

[0096] In this embodiment, the RANSAC algorithm (Random Sampling Consensus Algorithm) or other robust estimation methods are used to select the best matching pair from the feature matching pairs to exclude possible erroneous matching.

[0097] Randomly select the minimum number of matching point pairs (usually 3 pairs for affine transformation and 4 pairs for perspective transformation) and calculate the corresponding transformation matrix.

[0098] Repeat the above process multiple times, select different matching point pairs each time, and count how many matching point pairs under each transformation matrix conform to the transformation.

[0099] The transformation matrix with the most inliers is selected as the final transformation matrix.

[0100] Based on the selected best matching pair, the transformation matrix between two adjacent planar images is calculated, which contains translation, rotation and scaling information.

[0101] S303: using the transformation matrix to align two adjacent planar images in the first overlapping area.

[0102] In this embodiment, the transformation matrix calculated in step S302 is used to map pixel positions in one plane image to corresponding positions in another image, thereby achieving image alignment.

[0103] Applies a transformation matrix to one image to adjust its size, position, and orientation to align it with another image, using bilinear interpolation or nearest neighbor interpolation to handle the resulting shift in pixel position.

[0104] S304: Using an image fusion method to stitch the two adjacent plane images in the aligned overlapping area.

[0105] In this embodiment, the aligned images are fused to generate a complete stitched image, and the overlapping areas are fused at the pixel level using a pixel-level fusion method to avoid obvious boundaries. During the fusion process, gradient blending technology can also be used to reduce the visual differences at the stitching points to ensure a natural transition of the fused image.

[0106] Specifically, step S4 includes:

[0107] For subsequent overlapping areas, a pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the previous overlapping area to obtain a similarity value at the pixel level.

[0108] In this embodiment, for the overlapping area that has been spliced ​​in step S3, the next overlapping area is selected.

[0109] The pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the overlapping area that has been stitched. Pixel similarity can be achieved by calculating the color difference of corresponding pixels between the two overlapping areas. Specifically, the mean square error (MSE) or structural similarity index (SSIM) is used to quantify the similarity between pixels.

[0110] Specifically, the corresponding pixels of the two overlapping areas are paired, and for each pixel pair, their color difference is calculated (using the distance in the RGB color space), and the color differences of all pixel pairs are averaged to obtain the mean square error (MSE) or according to the structural similarity (SSIM) method.

[0111] Specifically, in step S5, the splicing relationship of the previous overlapping area is transformed, including:

[0112] S501: Identify feature differences between a current overlapping region and a previous overlapping region, wherein the feature differences include translation offset, rotation angle, and scaling change.

[0113] In this embodiment, for the image contents between the current overlapping area and the last overlapped area that has been stitched, identifying the feature differences between them includes:

[0114] Translation offset: The relative movement between two overlapping areas in the horizontal and vertical directions.

[0115] Rotation Angle: The relative rotation angle between the two overlapping areas.

[0116] Scaling change: The relative enlargement or reduction ratio between two overlapping areas.

[0117] Next, the SIFT feature matching method is used to find matching feature points between the two overlapping areas, and the translation offset, rotation angle and scale change are calculated based on the matching feature points.

[0118] S502: Determine update parameters according to the feature difference.

[0119] In this embodiment, according to the feature difference identified in S501, the parameters for updating the splicing relationship are determined, wherein the update parameters include:

[0120] Translation Vector: A two-dimensional vector used to describe the translation offset.

[0121] Rotation Matrix: A two-dimensional rotation matrix used to describe the rotation angle.

[0122] Scaling factor: A proportional factor used to describe scale changes.

[0123] Calculating update parameters based on the identified feature differences specifically includes the following steps:

[0124] a: Calculate the translation update amount:

[0125] Based on the set of matched feature points, the translation offset between the two overlapping areas is calculated, and the translation update amount is obtained by averaging the translation offsets of all matching point pairs.

[0126] b: Calculate the rotation update amount:

[0127] The matched feature points are used to estimate the rotation angle update between the two overlapping areas, and the least squares method is used to calculate the rotation update to ensure the accuracy of the rotation angle update.

[0128] c: Calculate the zoom update amount:

[0129] Based on the matched feature point set, the scale change update between the two overlapping areas is calculated, and the scale update is obtained by averaging the scale changes of all matching point pairs.

[0130] In addition, the least squares method can be used to optimize the translation update amount, rotation update amount, and scale update amount to ensure the accuracy of the update parameters.

[0131] S503: Using the update parameters, update the splicing relationship of the previous overlapping area.

[0132] In this embodiment, according to the update parameters determined in S502, the splicing relationship of the last overlapping area is updated, and the splicing relationship of the last overlapping area is adjusted using the determined update parameters (translation vector, rotation matrix, scaling factor).

[0133] Exemplarily, the update may be achieved by modifying the transformation matrix, and the updated stitching relationship will be used to stitch the current overlapping area.

[0134] The working principle of the video fusion method of a fisheye camera provided by the present invention is as follows:

[0135] First, the original video is captured from fisheye cameras at different positions, and then each frame of the fisheye image is accurately mapped into a distortion-free plane image. Next, by analyzing the spatial layout and shooting angle of the camera, the overlapping areas between these plane images can be intelligently identified and sorted according to their importance (such as area size).

[0136] For overlapping areas, feature matching methods are used to calculate the transformation relationship between adjacent images, thereby achieving high-precision image alignment. Subsequently, image fusion algorithms are used to ensure that these overlapping areas can transition naturally when stitching, with almost no visible seams.

[0137] For the subsequent overlapping area processing, an adaptive similarity evaluation mechanism is introduced. This mechanism dynamically determines whether the stitching strategy needs to be adjusted by comparing the pixel-level differences between the current overlapping area and the stitched area. If the difference is small, that is, the similarity is high, the previous stitching parameters are directly used to improve processing efficiency; if the difference is significant, the stitching parameters are automatically adjusted according to the detected feature changes to ensure the continuity and consistency of the entire video fusion process.

[0138] In summary, this innovative technology not only solves the image distortion and splicing problems in fisheye camera video fusion, but also significantly improves the quality and efficiency of video fusion through intelligent processing and adaptive adjustment strategies.

[0139] Embodiment 2

[0140] The present invention also provides a fisheye camera video fusion system for executing the fisheye camera video fusion method, referring to Figure 2 As shown, the fusion system comprises:

[0141] The image mapping module 100 is used to obtain original videos shot from at least two fisheye cameras respectively, and map the fisheye images in all the original videos into planar images based on a pre-established image mapping table.

[0142] The overlapping area sorting module 200 is used to determine the overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arrange them in descending order according to the size of the overlapping area.

[0143] The first splicing module 300 is used to calculate the splicing relationship of the first overlapping area after being arranged in descending order, and splice the first overlapping area according to the splicing relationship of the first overlapping area.

[0144] The similarity quantification module 400 is used to quantify the similarity between the current overlapping region and the previous overlapping region for subsequent overlapping regions to obtain a similarity value.

[0145] The second splicing module 500 is used to determine whether the similarity value meets a preset threshold.

[0146] If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area.

[0147] If not, the splicing relationship of the previous overlapping area is updated, and the current overlapping area is spliced ​​using the updated splicing relationship.

[0148] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0150] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

Claims

1. A video fusion method for a fisheye camera, characterized in that: The fusion method comprises the following steps: S1: acquiring original videos shot from at least two fisheye cameras respectively, and mapping the fisheye images in all the original videos into planar images based on a pre-established image mapping table; S2: determining an overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arranging the overlapping areas in descending order according to the size of the overlapping areas; S3: calculating the splicing relationship of the first overlapping region after the descending order, and splicing the first overlapping region according to the splicing relationship of the first overlapping region; S4: for subsequent overlapping regions, quantify the similarity between the current overlapping region and the previous overlapping region to obtain a similarity value; S5: Determine whether the similarity value meets a preset threshold. If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area. If not, the splicing relationship of the previous overlapping area is updated, and the current overlapping area is spliced ​​using the updated splicing relationship.

2. The video fusion method of a fisheye camera according to claim 1, characterized in that: Step S1 includes the following steps: S101: for each original video captured by the fisheye camera, identify a single-frame fisheye image; S102: Loading a pre-built image mapping table, wherein the mapping table contains conversion parameters from a fisheye image to a plane image, and the conversion parameters include a distortion correction coefficient, a projection model, and a corresponding mapping matrix; S103: traversing all fisheye images, and transforming all fisheye images using the transformation parameters, so as to map all fisheye images into distortion-free plane images.

3. The video fusion method of a fisheye camera according to claim 2, characterized in that: Step S2 includes the following steps: S201: Calculating the coverage of the shooting area of ​​each fisheye camera in the three-dimensional space according to the position, orientation and focal length of each fisheye camera; S202: Determine the overlapping area of ​​the coverage on the converted plane image by using a spatial geometric analysis method; S203: Calculate the number of pixels in each overlapping area, and arrange all overlapping areas in descending order according to the number of pixels.

4. The video fusion method of a fisheye camera according to claim 3, characterized in that: Step S3 includes the following steps: S301: extracting feature points from two adjacent plane images in the overlapping area that are at the first position after being arranged in descending order, and matching the extracted feature points using a SIFT feature matching method to obtain a plurality of feature matching pairs; S302: Based on the multiple feature matching pairs, calculate a transformation matrix between two adjacent plane images, wherein the transformation matrix includes a translation transformation relationship, a rotation transformation relationship, and a scaling transformation relationship; S303: aligning two adjacent plane images in the overlapping area at the first position using the transformation matrix; S304: Using an image fusion method to stitch the two adjacent plane images in the aligned overlapping area.

5. The video fusion method of a fisheye camera according to claim 4, characterized in that: Step S4 includes: For subsequent overlapping areas, a pixel similarity calculation method is used to evaluate the similarity between the current overlapping area and the previous overlapping area to obtain a similarity value at the pixel level.

6. The video fusion method of a fisheye camera according to claim 5, characterized in that: In step S5, the splicing relationship of the previous overlapping area is transformed, including: S501: Identify feature differences between a current overlapping region and a previous overlapping region, wherein the feature differences include translation offset, rotation angle, and scale change; S502: Determine update parameters according to the feature difference; S503: Using the update parameters, update the splicing relationship of the previous overlapping area.

7. A video fusion system of a fisheye camera, used to execute a video fusion method of a fisheye camera as claimed in any one of claims 1 to 6, characterized in that: The fusion system comprises: An image mapping module, used to obtain original videos shot from at least two fisheye cameras respectively, and map the fisheye images in all original videos into planar images based on a pre-established image mapping table; An overlapping area sorting module, used to determine the overlapping area of ​​the planar images in the original video based on the viewing angle relationship of the at least two fisheye cameras, and arrange them in descending order according to the size of the overlapping area; A first splicing module, used for calculating the splicing relationship of the first overlapping region after the descending order is arranged, and splicing the first overlapping region according to the splicing relationship of the first overlapping region; A similarity quantification module is used to quantify the similarity between the current overlapping area and the previous overlapping area for subsequent overlapping areas to obtain a similarity value; The second splicing module is used to determine whether the similarity value meets a preset threshold. If it is satisfied, the current overlapping area is spliced ​​using the splicing relationship of the previous overlapping area. If not, the splicing relationship of the previous overlapping area is updated, and the current overlapping area is spliced ​​using the updated splicing relationship.

Citation Information

Patent Citations

  • Method for splicing video in real time based on multiple cameras

    CN102006425A

  • Panoramic image stitching method and panoramic image stitching device

    CN106157241A

  • Image splicing method and device for fisheye lens

    CN110009567A

  • Panoramic video synthesis method and device and electronic equipment

    CN110519528A

  • Image splicing method and device, electronic equipment and storage medium

    CN113436068A

Cited By

  • Video image stitching method and device, terminal equipment and storage medium

    CN120219161A

  • A video image splicing method and device, a terminal device and a storage medium

    CN120219161B

  • Tunnel traffic flow detection method based on video cloud networking technology

    CN121483046A

  • Holographic projection method based on compound eye array camera

    CN121792713A