A multi-view overlapping video fusion splicing method and system

Abstract

The present invention discloses a multi-view overlapping video fusion and splicing method and system, which relates to the technical field of video fusion and splicing, and includes the following steps: performing basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before splicing; using a feature point detection algorithm to extract key feature points in the image; using feature descriptors to match the key feature points of the two images to find similar feature point pairs in adjacent images. The present invention uses intelligent perception of matching errors, combined with machine learning models and correction algorithms to perform real-time detection and optimization, ensure that the feature points meet geometric consistency, and use homography matrices to achieve precise alignment. Through unified processing of brightness and color and fusion of overlapping areas, the generated panoramic image not only has high-quality visual effects, but also improves the accuracy of target recognition and behavior analysis, and is particularly suitable for high-quality splicing demand scenarios such as security monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of video fusion and splicing, and in particular to a multi-view overlapping video fusion and splicing method and system. Background Art

[0002] Multi-view overlapping video fusion and splicing technology is very important in situations where multiple angles need to be displayed, especially in the field of security monitoring. Due to the limited viewing angle, a traditional single camera is difficult to cover every corner of the monitoring area, which may miss key details. However, by shooting synchronously from different angles with multiple cameras, the monitoring range can be effectively expanded. By fusing and splicing these images from different angles through image processing technology, and using algorithms such as feature point matching, image correction, and color smoothing, a seamless panoramic view can be generated, providing a more comprehensive and detailed picture. This technology not only improves the monitoring effect, but also helps to quickly locate and analyze abnormal events, providing important support for security work.

[0003] The prior art has the following deficiencies:

[0004] In the process of fusion stitching of multi-view overlapping videos, if there is a potential feature point matching error between two images in adjacent areas, existing technologies are usually unable to intelligently perceive and correct it in time. This error will directly lead to geometric misalignment between images, forming an obvious fault or misalignment at the stitching boundary. This phenomenon will not only seriously damage the overall coherence of the picture, but may also cause important target objects (such as people and vehicles) in the surveillance video to be segmented or deformed in the stitching area. This distortion effect will seriously affect the accurate understanding and analysis of the scene, and then lead to misjudgment or omission of key tasks such as target recognition and behavior tracking.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-view overlapping video fusion stitching method and system, which intelligently perceives potential matching errors, and combines machine learning models and correction algorithms to detect and optimize errors in real time, ensuring that matching points meet geometric consistency requirements, thereby eliminating faults or misalignments in the stitching process. In addition, when the match is correct, the homography matrix is ​​used to accurately align the images to ensure seamless stitching of the images in space. Through unified processing of image brightness and color and fusion of overlapping areas, the panoramic image finally generated not only maintains high-quality visual effects, but also effectively improves the accuracy of target recognition and behavior analysis. It is particularly suitable for scenes with high requirements on stitching quality, such as security monitoring, to solve the problems in the above-mentioned background technology.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a multi-view overlapping video fusion splicing method, comprising the following steps:

[0008] Perform basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before stitching;

[0009] Use feature point detection algorithm to extract key feature points in the image;

[0010] Use feature descriptors to match the key feature points of the two images and find similar feature point pairs in adjacent images;

[0011] When matching feature points, the matching information of the feature points is obtained, and features are extracted from the obtained matching information. After abnormal analysis of the extracted features, the analyzed features are input into a pre-trained machine learning model to intelligently perceive the potential matching errors when matching feature points.

[0012] When potential matching errors are identified in feature point matching, a correction algorithm is used to re-verify and optimize the detected matching error feature points to ensure that the matching pairs meet geometric consistency;

[0013] When no potential matching error is found in the feature point matching, the homography transformation matrix between the two images is calculated, and the homography matrix is ​​used to transform one image into the coordinate space of the other image so that the two images are aligned in space;

[0014] The homography transformed image is fused with the target image used as the main image in the stitching process in the overlapping area.

[0015] Preferably, the process of extracting key feature points from an image using a feature point detection algorithm comprises the following steps:

[0016] Convert the input image to grayscale to reduce computational complexity and ensure that the algorithm focuses on the structural features of the image rather than color information;

[0017] Use SIFT’s Difference of Gaussian algorithm to locate key points in grayscale images;

[0018] Extract local image information around the detected new key points and generate descriptors to represent the characteristics of each feature point;

[0019] The detected feature points are screened and optimized to ensure that the feature points are distinctive and stable.

[0020] Preferably, the specific steps of matching the feature points of two images using feature descriptors are as follows:

[0021] In each image, the corresponding feature descriptor is generated according to the detected feature points;

[0022] Use distance metric method to calculate the similarity between descriptors;

[0023] For each feature point descriptor, find the closest descriptor in the other image to form a feature point matching pair. In order to improve the accuracy, bidirectional nearest neighbor matching is used, that is, the matching point pairs are the nearest neighbors of each other in the two images.

[0024] Use the RANSAC algorithm to screen the matching point pairs, remove the mismatched point pairs that do not conform to the geometric transformation model, and retain the internal point set to improve the robustness and accuracy of the matching;

[0025] Based on the screened matching point pairs, the matching results are further optimized to ensure the accuracy of the positions of the matching points in the image, providing a reliable data basis for subsequent image registration and stitching.

[0026] Preferably, when matching feature points, feature extraction is performed on the acquired matching information, wherein the extracted features include the abnormal degree of change in the geometric relationship between the feature point matching pairs and the proportion of the descriptor distance distribution of the feature matching point pairs that deviates from the normal range. After abnormal analysis is performed on the abnormal degree of change in the geometric relationship between the feature point matching pairs and the proportion of the descriptor distance distribution of the feature matching point pairs that deviates from the normal range, a mismatch jump index and a matching distance anomaly index are generated respectively. The mismatch jump index is used to quantify the abnormal degree of change in the geometric relationship between the feature point matching pairs, and the matching distance anomaly index is used to quantify the proportion of the descriptor distance distribution of the feature matching point pairs that deviates from the normal range.

[0027] Preferably, after obtaining the mismatch jump index and the matching distance anomaly index obtained by performing an anomaly analysis on the extracted features, the mismatch jump index and the matching distance anomaly index are input into a pre-trained machine learning model to generate a matching error coefficient, and the matching error coefficient is used to intelligently perceive the potential matching error when matching feature points.

[0028] Preferably, the matching error coefficient generated when the potential matching error during feature point matching is predicted by the machine learning model is compared and analyzed with a preset matching error coefficient reference threshold, and the potential matching error during feature point matching is intelligently perceived. The specific steps are as follows:

[0029] If the matching error coefficient is greater than or equal to the matching error coefficient reference threshold, the matching situation of the feature point pair is identified as a matching error;

[0030] If the matching error coefficient is less than the matching error coefficient reference threshold, the matching situation of the feature point pair is identified as an efficient match.

[0031] Preferably, when matching feature points, the specific steps of generating a mismatch jump index after performing an abnormal analysis on the abnormal degree of geometric relationship change between feature point matching pairs are as follows:

[0032] For two adjacent feature point matching pairs and , define the geometric transformation matrix of each pair of adjacent matching points as and , in a two-dimensional image, the transformation matrix It is expressed as: , is the rotation angle between matching points, and are the translation amounts between matching point pairs respectively;

[0033] Then calculate the geometric relationship differences between adjacent matching pairs , ;

[0034] The degree of jump is quantified by calculating the difference in the geometric relationship between adjacent matching pairs, and the mismatch jump index is generated. The calculation expression of the mismatch jump index is: ,in, N is the total number of matching pairs, is the Euclidean norm of the geometric relationship difference, which indicates the degree of change of the geometric transformation between the two feature point matches: , is the Euclidean norm of the initial geometric relation, is a small positive number, is a weight parameter that controls the contribution of geometric relationship differences. , Indicates the mismatch transition index.

[0035] Preferably, when matching feature points, the specific steps of generating a matching distance anomaly index after performing an abnormal analysis on the proportion of descriptor distance distribution of feature matching point pairs that deviates from a normal range are as follows:

[0036] First, the matching distance outlier is calculated. Based on the descriptor distance of each feature matching point pair and the specific characteristics of the descriptor distance distribution, the descriptor distance of each pair of matching points is compared with the descriptor distances of other surrounding point pairs. The relative difference between the descriptor distance of each pair of matching points and other matching points is calculated based on the local similarity measurement method. The calculation expression is: , where Indicates k The feature point and u The relative difference measure of the distance between feature point descriptors, and They are k and u The descriptor distance of feature points;

[0037] Based on the above relative difference measurement, the deviation degree of all feature point pairs is summarized, and a weighting coefficient is used to give a higher weight to the matching pairs with serious deviations, and the matching distance anomaly index is calculated. The calculation expression is: , where Represents the final generated matching distance anomaly index, P is the set of all feature point matching pairs, represents any pair of matching points, is an adjustable parameter used to control the penalty for abnormal matching distance. is the weight of each matching pair.

[0038] Preferably, the homography transformation matrix between the two images is calculated, and the specific steps are:

[0039] Use the RANSAC algorithm to screen the matched feature point pairs and remove the mismatched point pairs to ensure the accuracy of the calculation;

[0040] Based on these selected feature point pairs, the homography transformation matrix is ​​calculated using the least squares method or other optimization methods;

[0041] The image to be transformed is perspective transformed using the homography transformation matrix and mapped to the coordinate space of the target image used as the main image in the stitching process, so that the two images are geometrically aligned.

[0042] A multi-view overlapping video fusion and splicing system, comprising an image preprocessing module, a feature point detection module, a feature point matching module, a matching information extraction module, an intelligent error perception module, a matching error correction module, an image transformation alignment module and an overlapping area fusion module;

[0043] The image preprocessing module performs basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before stitching;

[0044] Feature point detection module, which uses feature point detection algorithm to extract key feature points in the image;

[0045] The feature point matching module uses feature descriptors to match the key feature points of two images and find similar feature point pairs in adjacent images;

[0046] A matching information extraction module obtains matching information when feature points are matched, and performs feature extraction on the obtained information;

[0047] The intelligent error perception module performs abnormal analysis on the extracted features and inputs the analyzed features into a pre-trained machine learning model to intelligently perceive the potential matching errors when matching feature points.

[0048] The matching error correction module, when identifying potential matching errors in feature point matching, uses a correction algorithm to re-verify and optimize the matching error feature points detected to ensure that the matching pairs meet geometric consistency;

[0049] The image transformation alignment module calculates the homography transformation matrix between the two images when no feature point matching is identified and there is a potential matching error. The homography transformation matrix is ​​used to transform one image into the coordinate space of the other image so that the two images are aligned in space.

[0050] The overlapping region fusion module fuses the homography transformed image with the target image used as the main image in the stitching process in the overlapping region.

[0051] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0052] The present invention uses intelligent perception of potential matching errors, and combines machine learning models and correction algorithms to detect and optimize errors in real time, ensuring that the matching points meet the requirements of geometric consistency, thereby eliminating faults or dislocations in the stitching process. In addition, when the match is correct, the homography matrix is ​​used to accurately align the images to ensure seamless stitching of the images in space. Through unified processing of image brightness and color and fusion of overlapping areas, the panoramic image finally generated not only maintains high-quality visual effects, but also effectively improves the accuracy of target recognition and behavior analysis. It is particularly suitable for scenes with high requirements on stitching quality, such as security monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0054] Figure 1 The present invention is a method flow chart of a multi-view overlapping video fusion and splicing method.

[0055] Figure 2 The present invention is a module schematic diagram of a multi-view overlapping video fusion and splicing system. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0057] The present invention provides Figure 1 A multi-view overlapping video fusion and splicing method is shown, comprising the following steps:

[0058] Perform basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before stitching;

[0059] The images captured by each camera are processed in a basic way to ensure that the images have consistent brightness and color before stitching, thereby improving the naturalness and coherence of the stitching effect. Basic processing usually includes the following steps: brightness equalization, adjusting the image brightness distribution through histogram equalization or adaptive histogram equalization; color correction, using the white balance adjustment algorithm to correct the color deviation caused by different cameras due to differences in lighting or sensors; denoising, reducing the noise in the image through median filtering, bilateral filtering or other denoising algorithms; distortion correction, correcting the geometric distortion caused by lens distortion based on camera calibration parameters (such as intrinsic and extrinsic parameters). These steps together ensure that the images before stitching have good consistency, laying a solid foundation for subsequent feature point matching and image fusion.

[0060] Use feature point detection algorithms (such as SIFT, SURF, ORB, etc.) to extract key feature points in the image;

[0061] The process of extracting key feature points in an image using feature point detection algorithms (such as SIFT, SURF, ORB, etc.) usually includes the following steps:

[0062] Convert the input image to grayscale to reduce computational complexity and ensure that the algorithm focuses on the structural features of the image rather than color information.

[0063] Use specific algorithms (such as SIFT's Gaussian difference, SURF's Hessian matrix, and ORB's FAST corner point) to locate key points in the image. These key points are usually located at edges, corners, or obvious texture areas, and can represent important features of the image.

[0064] Extract local image information around the detected new key points and generate descriptors to represent the characteristics of each feature point. Different algorithms use different methods, such as SIFT uses gradient directional histograms, SURF uses Haar wavelet responses, and ORB uses binary feature descriptors.

[0065] The detected feature points are screened and optimized, such as eliminating feature points in edge areas or feature points with low repeatability, to ensure that the feature points have good distinguishability and stability.

[0066] Through the above steps, the algorithm can extract a set of key feature points and their descriptors from the image, providing a basis for subsequent image matching, transformation and splicing.

[0067] Use feature descriptors to match the key feature points of the two images and find similar feature point pairs in adjacent images;

[0068] The specific steps for matching the feature points of two images using feature descriptors are as follows:

[0069] In each image, the corresponding feature descriptors (such as gradient directional histograms for SIFT, Haar wavelet responses for SURF, and binary descriptors for ORB) are generated based on the detected feature points.

[0070] The similarity between descriptors is calculated using a distance metric. For example, for SIFT and SURF descriptors, Euclidean distance is often used; for ORB descriptors, Hamming distance is used.

[0071] For each feature point descriptor, find the closest descriptor in the other image to form a feature point matching pair. In order to improve the accuracy, bidirectional nearest neighbor matching can be used, that is, the matching point pairs are the nearest neighbors of each other in the two images.

[0072] The RANSAC (Random Sampling Consensus) algorithm is used to screen the matching point pairs, remove the mismatched point pairs that do not conform to the geometric transformation model, and retain the internal point set to improve the robustness and accuracy of the matching.

[0073] Based on the screened matching point pairs, the matching results are further optimized to ensure the accuracy of the positions of the matching points in the image, providing a reliable data basis for subsequent image registration and stitching.

[0074] Through the above steps, similar feature point pairs can be found in the two images, laying the foundation for subsequent image transformation and stitching.

[0075] When matching feature points, the matching information of the feature points is obtained, and features are extracted from the obtained matching information. After abnormal analysis of the extracted features, the analyzed features are input into a pre-trained machine learning model to intelligently perceive the potential matching errors when matching feature points.

[0076] When matching feature points, feature extraction is performed on the acquired matching information, wherein the extracted features include the abnormal degree of geometric relationship change between feature point matching pairs and the proportion of descriptor distance distribution of feature matching point pairs that deviates from the normal range. After abnormal analysis of the abnormal degree of geometric relationship change between feature point matching pairs and the proportion of descriptor distance distribution of feature matching point pairs that deviates from the normal range, a mismatch jump index and a matching distance anomaly index are generated respectively. The abnormal degree of geometric relationship change between feature point matching pairs is quantified by the mismatch jump index, and the proportion of descriptor distance distribution of feature matching point pairs that deviates from the normal range is quantified by the matching distance anomaly index.

[0077] After obtaining the mismatch jump index and matching distance anomaly index obtained by performing anomaly analysis on the extracted features, the mismatch jump index and matching distance anomaly index are input into a pre-trained machine learning model to generate a matching error coefficient. The matching error coefficient is used to intelligently perceive the potential matching error when matching feature points.

[0078] A pre-trained machine learning model is a model that uses a large amount of labeled training data for learning and optimization in the data-driven learning process. Through repeated training, the model can extract valuable patterns and features from the data and has certain prediction or classification capabilities. In the intelligent perception process of feature point matching, the role of the pre-trained machine learning model is to predict and determine whether there is a potential matching error based on the input information of feature point matching (such as the mismatch jump index and the matching distance anomaly index).

[0079] When creating a pre-trained machine learning model, you first need to collect a large amount of image data, especially various scenes containing feature point matching errors and normal matches. This data is manually annotated to mark which feature point matches are correct, which are incorrect, or under what circumstances feature point matching will fail. By training this data, the machine learning model can learn the characteristics of different types of matching errors. During the training process, the model continuously adjusts parameters through algorithms (such as neural networks, support vector machines, etc.) to optimize its performance on a given data set and reduce prediction errors. Ultimately, the model will have the ability to reason about unknown data, that is, to make accurate matching error judgments for the input mismatch jump index and matching distance anomaly index. At this point, the model no longer relies on simple rules, but intelligently perceives potential matching errors through patterns and relationships learned from the data.

[0080] The core function of the pre-trained machine learning model is to use the learned patterns and rules to intelligently predict and classify the newly input data (such as the mismatch jump index and the matching distance anomaly index). During the feature point matching process, by analyzing these index values, the model can determine whether there is an error in the current match, and then implement error correction. Specifically, the model can respond to different input data through the relationship established during the training process, generate a matching error coefficient, and judge whether the feature point matching is reliable based on this coefficient.

[0081] Once the information from the feature point matching process (such as the degree of abnormal geometric relationship changes and the degree of abnormal descriptor distance) is input into the pre-trained machine learning model, the model will determine whether the input data indicates potential errors in the match based on the patterns and data rules it has previously learned. The pre-trained machine learning model has mastered the rules for identifying matching errors by learning features from a large amount of annotated data, so that it can intelligently perceive new image matches, generate matching error coefficients, and further correct potential errors. This method not only improves the degree of automation of the stitching process, but also greatly improves the accuracy and efficiency of stitching, especially when dealing with complex dynamic scenes, it can significantly improve the quality of image stitching and the accuracy of monitoring analysis.

[0082] When matching feature points, the geometric relationship between the feature point matching pairs changes abnormally to a high degree, which usually indicates the existence of potential matching errors. Abnormal changes in geometric relationships mean that the differences in geometric transformations such as translation and rotation between adjacent feature point matching pairs are too large, which is usually caused by incorrect correspondences or mismatched points in the matching process. This abnormal change can cause geometric misalignment or distortion in the stitching process, affecting the accuracy and consistency of the final image. When the geometric relationship changes too much, it means that there are unreasonable jumps in the matching process between feature points, which may cause obvious boundary faults, image distortions, or misalignment of target objects when stitching images, thus affecting the overall stitching effect and subsequent analysis.

[0083] When matching feature points, the specific steps for generating the mismatch jump index after abnormal analysis of the abnormal degree of geometric relationship changes between feature point matching pairs are as follows:

[0084] For two adjacent feature point matching pairs and , define the geometric transformation matrix of each pair of adjacent matching points as and , where the geometric relationship can be described by transformations such as translation and rotation. In a two-dimensional image, the transformation matrix It is expressed as: , is the rotation angle between matching points, and are the translations between the matching point pairs, and then calculate the geometric relationship differences between adjacent matching pairs. , ;

[0085] The difference matrix represents the relative changes of the two matching pairs in the geometric space, including changes in rotation and translation. It reflects the change in the spatial position of adjacent feature point matching pairs. If the change is large, it may mean that there is an incorrect match.

[0086] The degree of jump is quantified by calculating the difference in the geometric relationship between adjacent matching pairs, and the mismatch jump index is generated. The calculation expression of the mismatch jump index is: , N is the total number of matching pairs, is the Euclidean norm of the geometric relationship difference, which indicates the degree of change of the geometric transformation between the two feature point matches: , is the Euclidean norm of the initial geometric relation, is a very small constant (such as 0.00001, to prevent division by zero errors), is a weight parameter that controls the contribution of geometric relationship differences and is usually taken as , in actual application, it is adjusted according to the data characteristics. Indicates the mismatch transition index.

[0087] It can be seen from the mismatch jump index that when matching feature points, the larger the performance value of the mismatch jump index generated after abnormal analysis of the abnormal degree of geometric relationship changes between feature point matching pairs, the more drastic the geometric relationship changes between adjacent matching pairs during the feature point matching process, which usually indicates that there are large errors or inconsistencies in the matching process. When the difference in geometric relationships (such as rotation and translation changes) exceeds the normal range, or there is a significant jump between adjacent matching pairs, it is usually a manifestation of potential matching errors. This abnormal change may cause obvious geometric dislocation, boundary discontinuity and other problems in the stitched image, thereby affecting the overall quality and accuracy of the image. Therefore, when the value of the mismatch jump index is high, it indicates that the hidden danger of matching error is large; conversely, when the mismatch jump index is low, it means that the geometric relationship changes in the matching process are relatively stable, the hidden danger of matching error is small, and the stability of the image stitching result is good.

[0088] When matching feature points, a high proportion of the descriptor distance distribution of the feature matching point pairs deviates from the normal range, which usually indicates that there is a hidden danger of potential matching errors in feature point matching. Descriptor distance is an indicator used to measure the similarity between two feature points, usually obtained by calculating the difference in the feature vectors of the two points in the local area. Under normal circumstances, the matched feature points should have a small descriptor distance, indicating that their positions and appearance characteristics in the image are similar. If the descriptor distance distribution deviates from the normal range, that is, there is a large distance difference, it means that the similarity of these matching pairs is poor, which may be due to image distortion, occlusion, noise, change in viewing angle and other factors. This deviation usually means that the inconsistency in the feature point matching process increases, which may cause geometric dislocation, boundary faults, and even deformation of the target object during stitching.

[0089] When matching feature points, the specific steps for generating the matching distance anomaly index after performing an abnormal analysis on the proportion of the descriptor distance distribution of the feature matching point pairs that deviates from the normal range are as follows:

[0090] First, the matching distance outlier is calculated. Based on the descriptor distance of each feature matching point pair and the specific characteristics of the descriptor distance distribution, the descriptor distance of each pair of matching points is compared with the descriptor distances of other surrounding point pairs to calculate the relative difference between the descriptor distance of each pair of matching points and other matching points based on the local similarity metric. The calculation expression is: , where Indicates k The feature point and u The relative difference measure of the distance between feature point descriptors, and They are k and u The descriptor distance of feature points;

[0091] This step quantifies the difference in the descriptor distance of each pair of feature points to measure their relative abnormality.

[0092] Based on the above relative difference metric, the deviation degree of all feature point pairs is summarized, and the weighting coefficient is used to give higher weights to the matching pairs with serious deviations. The matching distance anomaly index is calculated. The matching distance anomaly index not only reflects the severity of the deviation, but also can be combined with the global matching quality of the image for comprehensive evaluation. The calculation expression of the matching distance anomaly index is: , where Represents the final generated matching distance anomaly index, P is the set of all feature point matching pairs, represents any pair of matching points, It is an adjustable parameter used to control the penalty for abnormal matching distance. Its value is usually adjusted according to experimental data. is the weight of each matching pair, which is usually related to the importance of the feature point in the image or the reliability of the match;

[0093] High-quality matching pairs have larger weights, and low-quality matching pairs have smaller weights.

[0094] It can be seen from the matching distance anomaly index that when matching feature points, the larger the performance value of the matching distance anomaly index generated after the anomaly analysis of the proportion of the descriptor distance distribution of the feature matching point pair that deviates from the normal range, the greater the hidden danger of potential matching errors when matching feature points. The matching distance anomaly index quantifies the potential risk of matching errors by analyzing the degree to which the descriptor distance of the feature point matching pair deviates from the normal range. It combines the descriptor difference between the matching point pairs, the severity of the relative difference, and the importance of the matching pair. The higher the performance value, the greater the difference in descriptor distance, the lower the similarity of the matching pair, and the greater the potential risk of mismatching, which may cause geometric dislocation, image distortion, or target recognition errors during splicing. When the performance value of the matching distance anomaly index is small, it indicates that there are fewer anomalies in the feature point matching process, the descriptor distance of the matching pair is more consistent, and the hidden danger of mismatching is relatively small, so the splicing result is more stable and reliable. Therefore, the size of the matching distance anomaly index can be used as an important indicator to measure the matching quality. The larger the value, the more significant the potential matching error, and the smaller the value, the higher the matching accuracy.

[0095] The machine learning model is not specifically limited here, and can achieve the mismatch jump index and matching distance anomaly index Perform comprehensive analysis to generate matching error coefficients In order to realize the technical solution of the present invention, the present invention provides a specific implementation method;

[0096] Matching error coefficient The resulting calculation formula is: , where , Mismatch Transition Index and matching distance anomaly index The preset scaling factor of , Both are greater than 0.

[0097] According to the matching error coefficient, when the feature points are matched, the larger the performance value of the mismatch jump index generated after the abnormal analysis of the abnormal degree of change in the geometric relationship between the feature point matching pairs is, the larger the performance value of the matching distance abnormality index generated after the abnormal analysis of the proportion of the descriptor distance distribution of the feature matching point pairs that deviates from the normal range is, that is, the larger the performance value of the matching error coefficient generated when the potential matching error during feature point matching is predicted by the machine learning model, the greater the hidden danger of potential matching error during feature point matching, and vice versa.

[0098] The matching error coefficient generated when the machine learning model predicts the potential matching error during feature point matching is compared and analyzed with the preset matching error coefficient reference threshold, and the potential matching error during feature point matching is intelligently perceived. The specific steps are as follows:

[0099] If the matching error coefficient is greater than or equal to the matching error coefficient reference threshold, the matching situation of the feature point pair is identified as a matching error;

[0100] If the matching error coefficient is less than the matching error coefficient reference threshold, the matching situation of the feature point pair is identified as an efficient match.

[0101] Matching error refers to potential mismatches that are identified after analysis during feature point matching. This error usually indicates that two feature points are not corresponding points in the same object or scene.

[0102] Efficient matching means that in the feature point matching process, a pair of accurately matched feature points is successfully found, which are the real corresponding points in the same object or scene. Efficient matching not only means that the matched feature points are very close in spatial position, but also have high similarity in the descriptor space, which reflects the consistency of the corresponding geometric relationship and texture features of this pair of feature points in the image.

[0103] When potential matching errors are identified in feature point matching, a correction algorithm (such as the RANSAC algorithm) is used to re-verify and optimize the detected matching error feature points to ensure that the matching pairs meet geometric consistency;

[0104] The core idea of ​​the RANSAC algorithm is to estimate a geometric model (such as perspective transformation or homography matrix) by randomly selecting a small number of matching point pairs, and check whether other points conform to this model. If most points conform to the model, the model is considered correct, and those mismatched points that do not meet the geometric consistency requirements are eliminated. In this way, the system can eliminate incorrect matches when stitching images, ensure that the stitched images are geometrically consistent, avoid unnatural stitching effects such as misalignment and distortion, and thus ensure the accuracy and quality of image stitching.

[0105] When no potential matching error is found in the feature point matching, the homography transformation matrix between the two images is calculated, and the homography matrix is ​​used to transform one image into the coordinate space of the other image so that the two images are aligned in space;

[0106] The homography transformation matrix can describe the geometric relationship of images, such as translation, rotation, scale transformation, etc. The specific steps to calculate the homography transformation matrix between two images are as follows:

[0107] The RANSAC algorithm is used to screen the matching feature point pairs and remove the mismatched point pairs to ensure the accuracy of the calculation; based on these screened feature point pairs, the least squares method or other optimization methods are used to calculate the homography transformation matrix (i.e., a 3×3 matrix). Using this matrix, one of the images can be perspective transformed and mapped to the coordinate space of the other image, so that the two images are geometrically aligned, providing a basis for subsequent image stitching and fusion.

[0108] The image after homography transformation is fused with the target image used as the main image in the stitching process in the overlapping area;

[0109] Common fusion methods include direct splicing, linear gradient, Poisson fusion, multi-band fusion, etc. Choose a suitable fusion algorithm to ensure a smooth transition in the splicing area. If there is a significant difference in brightness or color, brightness or color smoothing may be required to make the transition more natural.

[0110] The present invention uses intelligent perception of potential matching errors, and combines machine learning models and correction algorithms to detect and optimize errors in real time, ensuring that the matching points meet the requirements of geometric consistency, thereby eliminating faults or dislocations in the stitching process. In addition, when the match is correct, the homography matrix is ​​used to accurately align the images to ensure seamless stitching of the images in space. Through unified processing of image brightness and color and fusion of overlapping areas, the panoramic image finally generated not only maintains high-quality visual effects, but also effectively improves the accuracy of target recognition and behavior analysis. It is particularly suitable for scenes with high requirements on stitching quality, such as security monitoring.

[0111] The present invention provides Figure 2 A multi-view overlapping video fusion and splicing system shown includes an image preprocessing module, a feature point detection module, a feature point matching module, a matching information extraction module, an intelligent error perception module, a matching error correction module, an image transformation alignment module and an overlapping area fusion module;

[0112] The image preprocessing module performs basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before stitching;

[0113] Feature point detection module, which uses feature point detection algorithm to extract key feature points in the image;

[0114] The feature point matching module uses feature descriptors to match the key feature points of two images and find similar feature point pairs in adjacent images;

[0115] A matching information extraction module obtains matching information when feature points are matched, and performs feature extraction on the obtained information;

[0116] The intelligent error perception module performs abnormal analysis on the extracted features and inputs the analyzed features into a pre-trained machine learning model to intelligently perceive the potential matching errors when matching feature points.

[0117] The matching error correction module, when identifying potential matching errors in feature point matching, uses a correction algorithm to re-verify and optimize the matching error feature points detected to ensure that the matching pairs meet geometric consistency;

[0118] The image transformation alignment module calculates the homography transformation matrix between the two images when no feature point matching is identified and there is a potential matching error. The homography transformation matrix is ​​used to transform one image into the coordinate space of the other image so that the two images are aligned in space.

[0119] The overlapping region fusion module fuses the homography transformed image with the target image used as the main image in the stitching process in the overlapping region;

[0120] A multi-perspective overlapping video fusion and stitching method provided in an embodiment of the present invention is implemented by the above-mentioned multi-perspective overlapping video fusion and stitching system. The specific method and process of a multi-perspective overlapping video fusion and stitching system are detailed in the embodiment of the above-mentioned multi-perspective overlapping video fusion and stitching method, which will not be repeated here.

[0121] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0123] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multi-view overlapping video fusion and splicing method, characterized in that: The following steps are involved: Perform basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before stitching; Use feature point detection algorithm to extract key feature points in the image; Use feature descriptors to match the key feature points of the two images and find similar feature point pairs in adjacent images; When matching feature points, the matching information of the feature points is obtained, and features are extracted from the obtained matching information. After abnormal analysis of the extracted features, the analyzed features are input into a pre-trained machine learning model to intelligently perceive the potential matching errors when matching feature points. When potential matching errors are identified in feature point matching, a correction algorithm is used to re-verify and optimize the detected matching error feature points to ensure that the matching pairs meet geometric consistency; When no potential matching error is found in the feature point matching, the homography transformation matrix between the two images is calculated, and the homography matrix is ​​used to transform one image into the coordinate space of the other image so that the two images are aligned in space; The image after homography transformation is fused with the target image used as the main image in the stitching process in the overlapping area; When matching feature points, feature extraction is performed on the acquired matching information, wherein the extracted features include the abnormal degree of geometric relationship change between feature point matching pairs and the proportion of descriptor distance distribution of feature matching point pairs that deviates from the normal range. After abnormal analysis is performed on the abnormal degree of geometric relationship change between feature point matching pairs and the proportion of descriptor distance distribution of feature matching point pairs that deviates from the normal range, a mismatch jump index and a matching distance abnormality index are generated respectively. The abnormal degree of geometric relationship change between feature point matching pairs is quantified by the mismatch jump index, and the proportion of descriptor distance distribution of feature matching point pairs that deviates from the normal range is quantified by the matching distance abnormality index. When matching feature points, the specific steps of generating the mismatch jump index after performing an abnormal analysis on the abnormal degree of geometric relationship changes between feature point matching pairs are as follows: For two adjacent feature point matching pairs and , define the geometric transformation matrix of each pair of adjacent matching points as and , in a two-dimensional image, the transformation matrix It is expressed as: , is the rotation angle between matching points, and are the translation amounts between matching point pairs respectively; Then calculate the geometric relationship differences between adjacent matching pairs , ; The degree of jump is quantified by calculating the difference in the geometric relationship between adjacent matching pairs, and the mismatch jump index is generated. The calculation expression of the mismatch jump index is: ,in, N is the total number of matching pairs, is the Euclidean norm of the geometric relationship difference, which indicates the degree of change of the geometric transformation between the two feature point matches: , is the Euclidean norm of the initial geometric relation, is a small positive number. is a weight parameter that controls the contribution of geometric relationship differences. , Indicates mismatch transition index; When matching feature points, the specific steps for generating the matching distance anomaly index after performing an abnormal analysis on the proportion of the descriptor distance distribution of the feature matching point pairs that deviates from the normal range are as follows: First, the matching distance outlier is calculated. Based on the descriptor distance of each feature matching point pair and the characteristics of the descriptor distance distribution, the descriptor distance of each pair of matching points is compared with the descriptor distances of other surrounding point pairs. The relative difference between the descriptor distance of each pair of matching points and other matching points is calculated based on the local similarity measurement method. The calculation expression is: , where Indicates k The feature point and u The relative difference measure of the feature point descriptor distance, and They are k and u The descriptor distance of feature points; Based on the relative difference metric, the deviation degree of all feature point pairs is summarized, and a weighting coefficient is used to give a higher weight to the matching pairs with serious deviations, and the matching distance anomaly index is calculated. The calculation expression is: , where Represents the final generated matching distance anomaly index, P is the set of all feature point matching pairs, represents any pair of matching points, is an adjustable parameter used to control the penalty for abnormal matching distance. is the weight of each matching pair.

2. The multi-view overlapping video fusion and splicing method according to claim 1, characterized in that: The process of extracting key feature points in an image using a feature point detection algorithm includes the following steps: Convert the input image to a grayscale image; Use SIFT’s Difference of Gaussian algorithm to locate key points in grayscale images; Extract local image information around the detected new key points and generate descriptors to represent the characteristics of each feature point; The detected feature points are screened and optimized to ensure that the feature points are distinctive and stable.

3. The multi-view overlapping video fusion and splicing method according to claim 1, characterized in that: The specific steps for matching the feature points of two images using feature descriptors are as follows: In each image, the corresponding feature descriptor is generated based on the detected feature points; Use distance metric method to calculate the similarity between descriptors; For each feature point descriptor, find the closest descriptor in the other image to form a feature point matching pair. In order to improve the accuracy, bidirectional nearest neighbor matching is used, that is, the matching point pairs are the nearest neighbors of each other in the two images. Use the RANSAC algorithm to screen the matching point pairs, remove the mismatched point pairs that do not conform to the geometric transformation model, and retain the internal point set to improve the robustness and accuracy of the matching; According to the screened matching point pairs, the matching results are optimized to ensure the accuracy of the matching points in the image, providing a reliable data basis for subsequent image registration and stitching.

4. The multi-view overlapping video fusion and splicing method according to claim 1, characterized in that: After obtaining the mismatch jump index and matching distance anomaly index obtained by abnormal analysis of the extracted features, the mismatch jump index and matching distance anomaly index are input into a pre-trained machine learning model to generate a matching error coefficient. The matching error coefficient is used to intelligently perceive the potential matching error when matching feature points.

5. The multi-view overlapping video fusion and splicing method according to claim 4, characterized in that: The matching error coefficient generated when the machine learning model predicts the potential matching error during feature point matching is compared and analyzed with the preset matching error coefficient reference threshold, and the potential matching error during feature point matching is intelligently perceived. The specific steps are as follows: If the matching error coefficient is greater than or equal to the matching error coefficient reference threshold, the matching situation of the feature point pair is identified as a matching error; If the matching error coefficient is less than the matching error coefficient reference threshold, the matching situation of the feature point pair is identified as an efficient match.

6. The multi-view overlapping video fusion and splicing method according to claim 1, characterized in that: Calculate the homography transformation matrix between two images. The specific steps are: Use the RANSAC algorithm to screen the matched feature point pairs and remove the mismatched point pairs to ensure the accuracy of the calculation; Based on these screened feature point pairs, the homography transformation matrix is ​​calculated using the least squares method; The image to be transformed is perspective transformed using the homography transformation matrix and mapped to the coordinate space of the target image used as the main image in the stitching process, so that the two images are geometrically aligned.

7. A multi-view overlapping video fusion and splicing system, used to implement the multi-view overlapping video fusion and splicing method described in any one of claims 1 to 6, characterized in that: It includes image preprocessing module, feature point detection module, feature point matching module, matching information extraction module, intelligent error perception module, matching error correction module, image transformation alignment module and overlapping area fusion module; The image preprocessing module performs basic processing on the images collected by each camera to ensure that the images have consistent brightness and color before stitching; Feature point detection module, which uses feature point detection algorithm to extract key feature points in the image; The feature point matching module uses feature descriptors to match the key feature points of two images and find similar feature point pairs in adjacent images; A matching information extraction module obtains matching information when feature points are matched, and performs feature extraction on the obtained information; The intelligent error perception module performs abnormal analysis on the extracted features and inputs the analyzed features into a pre-trained machine learning model to intelligently perceive the potential matching errors when matching feature points. The matching error correction module, when identifying potential matching errors in feature point matching, uses a correction algorithm to re-verify and optimize the matching error feature points detected to ensure that the matching pairs meet geometric consistency; The image transformation alignment module calculates the homography transformation matrix between the two images when no feature point matching is identified and there is a potential matching error. The homography transformation matrix is ​​used to transform one image into the coordinate space of the other image so that the two images are aligned in space. The overlapping region fusion module fuses the homography transformed image with the target image used as the main image in the stitching process in the overlapping region.

Images