AI video stitching fusion method based on BRISK and feature level fusion algorithm

Through the AI ​​video stitching method based on BRISK and feature-level fusion algorithm, the problems of inaccurate feature matching and insufficient image registration accuracy in complex image environments are solved, and efficient and accurate video stitching and fusion are achieved, which is suitable for a variety of application scenarios. Meet real-time needs.

CN119071566APending Publication Date: 2024-12-03宋健伟

Patent Information

Application Number
CN202411054412.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing video stitching technology is inaccurate feature matching, insufficient image registration accuracy, difficult processing of overlapping areas, high computational complexity, and poor real-time performance in complex image environments, especially when dealing with images with large viewing angle differences and perspective distortion. Efficient and accurate splicing.

Method used

Using AI video stitching method based on BRISK and feature-level fusion algorithms, feature extraction and matching are performed through end-to-end learning models of RTSP video stream reading, timestamp alignment, image preprocessing, SuperPoint and LightGlue algorithms, combined with weighted average feature fusion and self-attention mechanism, optimize feature point detection and descriptor generation, dynamically adjust matching thresholds and non-maximum suppression, and achieve efficient matching and stitching of images.

Benefits of technology

Improve the robustness of feature matching and the accuracy of image registration, optimize overlapping area processing, reduce computing complexity, enhance the versatility and adaptability of the technology, and achieve efficient, accurate and seamless video stitching fusion. Meet real-time processing needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071566A_ABST
    Figure CN119071566A_ABST
Patent Text Reader

Abstract

The AI video stitching fusion method based on the BRISK and the feature level fusion algorithm is used for realizing accurate capture and efficient matching of image features in a complex scene, and efficient and accurate feature extraction and matching are performed on video frames through a deep learning technology by taking an RTSP video stream as input; the whole processing flow comprises the key steps of video stream reading, image preprocessing, feature extraction, feature matching, result output and the like. The method aims to realize accurate capture and efficient matching of image features in a complex scene through an advanced deep learning algorithm, and provides powerful technical support for advanced applications such as image recognition and target tracking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an AI video splicing and fusion method based on BRISK and feature-level fusion algorithm. Background Art

[0002] In the development of video stitching and fusion technology, stitching methods based on feature matching have always been the focus of research. These methods extract key feature points of images, such as scale-invariant feature transform (SIFT) and speeded up robust features (SURF), and use the matching of these feature points to achieve stitching between different images. In order to improve the accuracy of feature point matching, researchers have developed a variety of algorithms, such as the random sampling consensus (RANSAC) algorithm for removing outliers, and the Harris corner detection algorithm for quickly locating corner features.

[0003] However, the feature matching-based method has limitations when facing complex image environments. Factors such as light changes, repeated textures, and blur in the image may lead to inaccurate feature point extraction and matching, which in turn affects the final stitching effect.

[0004] Another widely used method is image registration technology. This technology first requires geometric correction of the image, that is, registration, to ensure that the image is aligned in space. The registration process usually involves mathematical operations such as similarity transformation, affine transformation or perspective transformation. Through these transformations, the perspective and position differences of the image are corrected, laying the foundation for subsequent fusion or superposition. Although image registration technology can solve the perspective difference problem to a certain extent, it still faces challenges when processing images with large perspective differences and perspective distortion.

[0005] Panoramic stitching methods focus on seamlessly stitching multiple images into a panoramic image. This method usually includes multiple steps such as image registration, identification of overlapping areas, light adjustment and image fusion. Panoramic stitching technology has a wide range of applications in virtual reality, video surveillance and other fields, but how to efficiently and naturally process overlapping areas, and how to solve the problem of inconsistent color and brightness between different images, are still technical challenges that need to be overcome in this field.

[0006] A Chinese patent with application number CN202410155564.6 and name "A multi-camera image stitching method based on traffic roads" discloses a multi-camera image stitching method based on traffic roads. By collecting and acquiring traffic camera image data at intersections, a data set is made; the two images to be stitched are input into the trained SuperPoint model to obtain feature points, scores and descriptors, and then the output obtained by feature detection is used as the input of the LightGlue model to obtain the local feature point matching relationship of the two images, and RANSAC is used to calculate the homography matrix transformation of the two images and perform preliminary stitching. The logic of the APAP algorithm is used to cut the image into multiple small squares, and multiple local homography matrices and GSP global similarity are combined for image alignment. The Multi-Band Blending image fusion algorithm is used to eliminate the obvious edge traces in the overlapping area to obtain the final image stitching result, and the LightGlue model is used to match the local features in the image, so that the calculation amount adapts to the difficulty of each image pair, improves the accuracy of image transformation, and thus improves the quality of image stitching.

[0007] A Chinese patent with application number CN202410155564.6 and titled "A multi-camera image stitching method based on traffic roads" discloses a panoramic image stitching method and device based on an optimal mapping matrix. The method includes: collecting n video streams obtained by simultaneously shooting the airport scene area with n camera devices; each camera device shoots a part of the scene, and the scene shot by each camera device has an overlapping area with the scene shot by the adjacent camera device; n is a positive integer greater than 1; according to the preset distortion coefficients of each camera device, the image frames in the n video streams are distorted and corrected; the image frames at the same time after correction are processed using feature descriptors to obtain the optimal mapping matrix between the image frames; the image frames at that time are stitched according to the mapping matrix to obtain a panoramic image of the airport.

[0008] The limitations of existing technologies lie not only in the robustness of feature matching, but also in many other aspects. In complex image environments, feature matching-based methods are susceptible to factors such as lighting changes, repeated patterns, and blur, which interfere with the correct matching of feature points, thereby reducing the accuracy and reliability of stitching. In addition, the accuracy of image registration is also a key issue, especially when dealing with images with perspective distortion, large viewing angle differences, or distortion. Traditional registration techniques have difficulty in achieving accurate spatial alignment, resulting in misalignment and incoherence in the stitched image.

[0009] In the field of panoramic stitching, the handling of overlapping areas is particularly difficult. Direct stitching may result in obvious seams, affecting the visual effect. Although complex fusion technology can improve this problem, it often comes at the cost of increased computing resources and time. In addition, differences in shooting environments between different images, such as changes in light and color temperature, will introduce color differences and brightness inconsistencies during the stitching process, further affecting the coherence and overall look and feel of the stitched image.

[0010] With the development of deep learning technology, although deep learning-based methods have shown certain advantages in feature extraction and matching, their computational complexity is high and the hardware requirements are more stringent. When processing large-scale or high-resolution image data, the computational and storage costs of these methods increase significantly, limiting their use in resource-constrained application scenarios. At the same time, these deep learning-based methods may perform well in specific scenarios, but lack sufficient versatility and are difficult to adapt to changing application environments.

[0011] Real-time performance is also a challenge faced by existing technologies. In application scenarios that require fast response, such as video surveillance or autonomous driving, some image stitching algorithms have a long processing time and cannot meet the needs of real-time processing. Therefore, how to optimize the algorithm structure, improve algorithm efficiency, and reduce processing delays is the key to improving the practicality of image stitching technology.

[0012] In order to overcome the shortcomings of the prior art, the present invention aims to provide an improved video stitching and fusion method, which adopts advanced algorithms and optimization strategies to improve the robustness of feature matching, enhance the accuracy of image registration, optimize the processing of overlapping areas, solve the problem of color and brightness inconsistency, reduce computational complexity, enhance the versatility and adaptability of the technology, and improve the real-time performance of the algorithm, thereby achieving efficient, accurate and seamless video stitching and fusion in various application scenarios. Summary of the invention

[0013] The purpose of the present invention is to provide an AI video splicing and fusion method based on BRISK and feature-level fusion algorithm, which is used to accurately capture and efficiently match image features in complex scenes, and provide technical support for advanced applications including image recognition and target tracking, including the following steps:

[0014] S1.RTSP video stream reading

[0015] Use RTSP protocol to read video stream data in real time to ensure the real-time and stability of data;

[0016] S2. Timestamp alignment

[0017] S2.1. When reading the RTSP video stream, capture the timestamp information of each video frame;

[0018] S2.2. Timestamp calibration: By comparing the locations of feature points of the same scene in different video streams to capture information due to network delays and differences in video acquisition devices, these timestamps are calibrated to ensure that they reflect the actual synchronization time points;

[0019] S2.3. Frame rate synchronization: For the frame rate difference problem between different video streams, the frame rate synchronization between different video streams is achieved by dynamically adjusting the display time of the video frame or inserting additional frames;

[0020] S2.4. Buffer management: A buffer management strategy is adopted. The buffer temporarily stores a certain number of video frames. The system takes out the video frames from the buffer according to the timestamp information for processing to ensure the orderly output of the video frames to cope with network delay and jitter;

[0021] S3. Image preprocessing

[0022] After the video frame is read, the image is preprocessed; the preprocessing operation at least includes graying, noise reduction, and contrast enhancement processing of the image to eliminate noise and interference in the image and improve the accuracy of subsequent feature extraction and matching;

[0023] S4. Feature extraction

[0024] The SuperPoint algorithm is used to extract features from the preprocessed images in S3. The feature extraction process is learned directly from the original images through end-to-end learning:

[0025] S4.1. Feature point detection optimization based on weighted average feature fusion

[0026] Based on the SuperPoint algorithm, key point detection and descriptor generation are optimized, and a multi-scale feature fusion mechanism is introduced in the key point detector to capture multi-scale features in the image through feature maps of different scales. Feature point detection optimization is achieved by constructing a feature pyramid and performing weighted average feature fusion at different levels. The algorithm model is:

[0027] Its algorithm model is:

[0028]

[0029] Among them, F fused is the fused feature map, F i is the feature map of the i-th scale, W i is the corresponding weight, N is the number of scales;

[0030] S4.2. Optimization of descriptor generation based on self-attention mechanism

[0031] The attention mechanism is introduced into the descriptor generator, so that the model can focus on important feature areas and generate more representative descriptors. This is achieved by adding a self-attention module to the descriptor generation network. The algorithm model of the self-attention module is:

[0032]

[0033] Among them, Q, K, and V are query, key, and value matrices respectively. k is the dimension of the key, and the SoftMax function is used to calculate the attention weight;

[0034] S5. Feature matching

[0035] After feature extraction, the LightGlue algorithm is used to match feature points. LightGlue optimizes the matching strategy based on the adaptive matching threshold and learns the matching pattern directly from the image in an end-to-end learning manner, thus adapting to feature matching tasks in complex scenes.

[0036] Adaptively adjust the matching threshold according to the characteristics of the image; this is achieved by learning a threshold function related to the image characteristics, which outputs the matching threshold according to the parameter statistics of the image; the algorithm model is: τ=a·I brightness +b·I contrast +c; where T is the matching threshold, I brightness and I contrast are the brightness and contrast statistics of the image, respectively, and a, b, c are the learned parameters;

[0037] In order to remove redundant key points while retaining important feature points, improvements are made on the basis of the original NMS, at least adding local area judgment and adjusting the radius or threshold of NMS according to regional characteristics;

[0038] The algorithm model is:

[0039]

[0040] Among them, S(p) is the original score of point p, N(p) is the neighborhood set of point p, R(p) is the region set to which point p belongs, and T is the matching threshold.

[0041] Preferably, in S2, for different network environments and video acquisition devices, accurate timestamp alignment under various conditions is achieved through the timestamp alignment strategy; wherein, the method for frame rate difference detection and analysis is: in the video stream processing process, a high-precision timestamp is assigned to each video frame and passed through the FFmpeg video encoding / decoding library; the frame rate of each video stream is calculated in real time by counting the number of video frames received per unit time; a variance difference analysis module between each video stream is established, and the video stream that needs to be adjusted and the corresponding adjustment strategy are determined by receiving the frame rate data from each video stream and comparing the differences between them.

[0042] Preferably, in S2, for the frame rate difference problem existing in different video streams, the frame rate synchronization between different video streams is achieved by dynamically adjusting the display time of the video frames or inserting additional frames; the insertion of additional frames to achieve frame rate synchronization between different video streams is an intelligent frame rate adjustment strategy based on dynamic interpolation technology: according to the frame rate difference between the video streams, the number of frames to be inserted is dynamically calculated; by comparing the frame rate difference between the two video streams, when the frame rate of the target video stream is lower than that of the reference video stream, blank frames are supplemented in the target video stream or frames already existing in the original video stream are copied according to the frame rate difference.

[0043] Preferably, the position for inserting the additional frame is selected to insert the frame in an area where the feature point changes less and the stability is higher, so as to reduce the frame insertion problem caused by feature point matching errors.

[0044] Preferably, there is no b frame in the original video used for stitching.

[0045] Preferably, the video processing tool FFmpeg is used, and the -fflags nobuffer parameter is added in the command line to reduce the buffer.

[0046] Preferably, in S5, the complex scene at least includes weak texture areas, illumination changes, occlusions and partial visibility.

[0047] Preferably, the processing strategy for the weak texture area includes: based on SuperPoint, introducing or enhancing the feature description subnetwork to enable it to capture richer contextual information and local texture details, using a deeper convolutional network layer or introducing an attention mechanism to focus on important texture features; combining feature maps of different scales to capture texture information from coarse to fine, and efficiently processing weak texture areas.

[0048] Preferably, the strategy for processing illumination changes includes: during the training process, data augmentation by simulating various illumination conditions is performed so that the model learns to maintain feature consistency under illumination changes; during the feature extraction stage, features are normalized to reduce the impact of illumination changes on feature representation; and the robustness of the model to illumination changes is enhanced by selecting a triplet loss function that is insensitive to illumination changes.

[0049] Preferably, the occlusion and partial visibility processing strategy includes:

[0050] 1) During the matching process, obviously unmatched points are first discarded through a fast screening mechanism to reduce the impact of occlusion areas on the overall matching;

[0051] 2) Based on the preliminary matching, the matching points are further screened and verified through iterative optimization algorithms to improve the matching accuracy;

[0052] 3) Utilize the feature consistency of local regions to detect occluded regions, and try to compensate the feature representation of occluded regions through the features of surrounding unoccluded regions;

[0053] 4) During the matching process, the dustbin mechanism is dynamically adjusted to discard or temporarily ignore some difficult-to-match points to adapt to different degrees of occlusion.

[0054] The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm of the present invention has at least the following advantages:

[0055] 1. Efficiency of end-to-end learning

[0056] Through end-to-end learning models such as SuperPoint and LightGlue, this system can learn the feature extraction and matching process directly from the original RTSP video stream, without the tedious manual design of feature extractors and matching rules. This learning method improves the efficiency and accuracy of the entire system and reduces the possibility of human intervention and errors.

[0057] 2. Strong robustness and adaptability

[0058] The SuperPoint and LightGlue algorithms based on deep learning can learn the invariant features in the image, thereby providing robust and accurate feature point extraction and matching results in various complex scenarios (such as lighting changes, occlusion, perspective changes, etc.). This robustness and adaptability enable the system to be widely used in various practical scenarios, improving the practicality and versatility of the system.

[0059] 3. Ability to handle complex scenarios

[0060] The SuperPoint and LightGlue algorithms can effectively handle complex scenes and large-scale data, including large viewing angle differences, perspective distortion and other issues. This makes the system perform well in advanced applications such as panoramic image stitching and 3D reconstruction, and can provide users with high-quality image processing and feature matching results.

[0061] 4High real-time performance

[0062] By adopting efficient deep learning algorithms and optimized computing frameworks, this system can realize real-time processing of video streams. Even when processing high-definition video streams, the system can maintain a fast processing speed to meet the needs of real-time applications such as video surveillance and autonomous driving.

[0063] 5.5 Strong scalability

[0064] The system has good scalability and can be customized and optimized according to the needs of specific application scenarios. This enables the system to flexibly adapt to various new application scenarios and needs, maintaining the advancement and competitiveness of technology.

[0065] 5.6 Easy to integrate and deploy

[0066] The system is based on standardized RTSP video stream input, making it easy to integrate and deploy with other video processing systems and applications. This reduces the system's deployment and maintenance costs and improves the system's availability and ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of an AI video splicing and fusion method based on BRISK and feature-level fusion algorithm. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below through examples so that those skilled in the art can implement the invention with reference to the description.

[0069] It should be understood that the terms such as “having”, “including” and “comprising” used herein do not exclude the existence or addition of one or more other elements or combinations thereof.

[0070] An AI video stitching and fusion method based on BRISK and feature-level fusion algorithm is used to accurately capture and efficiently match image features in complex scenes, providing technical support for advanced applications including image recognition and target tracking, such as Figure 1 As shown, the following steps are included:

[0071] S1.RTSP video stream reading

[0072] Use RTSP protocol to read video stream data in real time to ensure the real-time and stability of data;

[0073] S2. Timestamp alignment

[0074] S2.1. When reading the RTSP video stream, capture the timestamp information of each video frame;

[0075] S2.2. Timestamp calibration: By comparing the locations of feature points of the same scene in different video streams to capture information due to network delays and differences in video acquisition devices, these timestamps are calibrated to ensure that they reflect the actual synchronization time points;

[0076] S2.3. Frame rate synchronization: For the frame rate difference problem between different video streams, the frame rate synchronization between different video streams is achieved by dynamically adjusting the display time of the video frame or inserting additional frames;

[0077] S2.4. Buffer management: A buffer management strategy is adopted. The buffer temporarily stores a certain number of video frames. The system takes out the video frames from the buffer according to the timestamp information for processing to ensure the orderly output of the video frames to cope with network delay and jitter;

[0078] S3. Image preprocessing

[0079] After the video frame is read, the image is preprocessed; the preprocessing operation at least includes graying, noise reduction, and contrast enhancement processing of the image to eliminate noise and interference in the image and improve the accuracy of subsequent feature extraction and matching;

[0080] S4. Feature extraction

[0081] The SuperPoint algorithm is used to extract features from the preprocessed images in S3. The feature extraction process is learned directly from the original images through end-to-end learning:

[0082] S4.1. Feature point detection optimization based on weighted average feature fusion

[0083] Based on the SuperPoint algorithm, key point detection and descriptor generation are optimized, and a multi-scale feature fusion mechanism is introduced in the key point detector to capture multi-scale features in the image through feature maps of different scales. Feature point detection optimization is achieved by constructing a feature pyramid and performing weighted average feature fusion at different levels. The algorithm model is:

[0084] Its algorithm model is:

[0085]

[0086] Among them, F fusedis the fused feature map, F i is the feature map of the i-th scale, W i is the corresponding weight, N is the number of scales;

[0087] S4.2. Optimization of descriptor generation based on self-attention mechanism

[0088] The attention mechanism is introduced into the descriptor generator, so that the model can focus on important feature areas and generate more representative descriptors. This is achieved by adding a self-attention module to the descriptor generation network. The algorithm model of the self-attention module is:

[0089]

[0090] Among them, Q, K, and V are query, key, and value matrices respectively. k is the dimension of the key, and the SoftMax function is used to calculate the attention weight;

[0091] S5. Feature matching

[0092] After feature extraction, the LightGlue algorithm is used to match feature points. LightGlue optimizes the matching strategy based on the adaptive matching threshold and learns the matching pattern directly from the image in an end-to-end learning manner, thus adapting to feature matching tasks in complex scenes.

[0093] Adaptively adjust the matching threshold according to the characteristics of the image (such as illumination, occlusion, etc.); this is achieved by learning a threshold function related to the image characteristics, which outputs the matching threshold according to the parameter statistics of the image (such as brightness, contrast, etc.); the algorithm model is: , τ = a·I brightness +b·I contrast +c; where τ is the matching threshold, I brightness and I contrast are the brightness and contrast statistics of the image, respectively, and a, b, c are the learned parameters;

[0094] In order to remove redundant key points while retaining important feature points, improvements are made on the basis of the original NMS, at least adding local area judgment and adjusting the radius or threshold of NMS according to regional characteristics;

[0095] The algorithm model is:

[0096]

[0097] Among them, S(p) is the original score of point p, N(p) is the neighborhood set of point p, R(p) is the region set to which point p belongs, T is the matching threshold, and τ is the formula τ = a·I brightness+b·I oontrast +c The matching threshold calculated, τ value is a dynamic variable, which is used in two main aspects: adaptive adjustment of matching threshold and improved non-maximum suppression (NMS).

[0098] In S2, for different network environments and video acquisition devices, accurate timestamp alignment under various conditions is achieved through the timestamp alignment strategy; wherein, the method for frame rate difference detection and analysis is as follows: during the video stream processing, a high-precision timestamp (such as millisecond level or higher) is assigned to each video frame and passed through the FFmpeg video encoding / decoding library; the frame rate of each video stream is calculated in real time by counting the number of video frames received within a unit time (such as one second); a variance difference analysis module is established between each video stream, and the video stream that needs to be adjusted and the corresponding adjustment strategy are determined by receiving the frame rate data from each video stream and comparing the differences between them.

[0099] In S2, for the frame rate difference problem existing in different video streams, the frame rate synchronization between different video streams is achieved by dynamically adjusting the display time of the video frames or inserting additional frames; the insertion of additional frames to achieve frame rate synchronization between different video streams is an intelligent frame rate adjustment strategy based on dynamic interpolation technology: according to the frame rate difference between the video streams, the number of frames to be inserted is dynamically calculated; by comparing the frame rate difference between the two video streams, when the frame rate of the target video stream is lower than that of the reference video stream, blank frames are supplemented in the target video stream according to the frame rate difference or frames already existing in the original video stream are copied; when copying frames already existing in the original video stream, frames near the position of the inserted frames can also be copied.

[0100] The location for inserting the additional frames is selected to insert frames in an area where the feature points change less and are more stable, such as inserting frames in an area where the scene is static or changes less and the movement is not violent, so as to reduce the frame insertion problem caused by feature point matching errors.

[0101] There is no b frame in the original video used for stitching.

[0102] Use the video processing tool FFmpeg and add the -fflags nobuffer parameter in the command line to reduce the cache.

[0103] In S5, the complex scene at least includes weak texture areas, illumination changes, occlusions, and partial visibility.

[0104] The processing strategy for weak texture areas includes: based on SuperPoint, introducing or enhancing the feature description subnetwork to enable it to capture richer contextual information and local texture details, using deeper convolutional network layers or introducing an attention mechanism to focus on important texture features; combining feature maps of different scales to capture texture information from coarse to fine and efficiently process weak texture areas.

[0105] The illumination change processing strategy includes: during the training process, by simulating various illumination conditions (such as brightness changes, shadows, different light sources, etc.) for data augmentation, the model learns to maintain feature consistency under illumination changes; in the feature extraction stage, the features are normalized to reduce the impact of illumination changes on feature representation; by selecting a triplet loss function that is insensitive to illumination changes, the robustness of the model to illumination changes is enhanced.

[0106] The occlusion and partial visibility processing strategies include:

[0107] 1) During the matching process, obviously unmatched points are first discarded through a fast screening mechanism (such as preliminary matching based on feature similarity) to reduce the impact of occluded areas on the overall matching;

[0108] 2) Based on the preliminary matching, further screen and verify the matching points through iterative optimization algorithms (such as RANSAC or its variants) to improve the matching accuracy;

[0109] 3) Utilize the feature consistency of local regions to detect occluded regions, and try to compensate the feature representation of occluded regions through the features of surrounding unoccluded regions;

[0110] 4) During the matching process, the dustbin mechanism is dynamically adjusted to discard or temporarily ignore some difficult-to-match points to adapt to different degrees of occlusion.

[0111] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the embodiments shown and described herein.

Claims

1. An AI video splicing and fusion method based on BRISK and feature-level fusion algorithm is used to accurately capture and efficiently match image features in complex scenes, and provide technical support for advanced applications including image recognition and target tracking, characterized in that: The steps include: S1.RTSP video stream reading Use RTSP protocol to read video stream data in real time to ensure the real-time and stability of data; S2. Timestamp alignment S2.

1. When reading the RTSP video stream, capture the timestamp information of each video frame; S2.

2. Timestamp calibration: By comparing the locations of feature points of the same scene in different video streams to capture information due to network delays and differences in video acquisition devices, these timestamps are calibrated to ensure that they reflect the actual synchronization time points; S2.

3. Frame rate synchronization: To solve the frame rate difference problem between different video streams, the frame rate synchronization between different video streams can be achieved by dynamically adjusting the display time of video frames or inserting additional frames. S2.

4. Buffer management: A buffer management strategy is adopted. The buffer temporarily stores a certain number of video frames. The system takes out the video frames from the buffer according to the timestamp information for processing to ensure the orderly output of the video frames to cope with network delay and jitter; S3. Image preprocessing After the video frame is read, the image is preprocessed; the preprocessing operation at least includes graying, noise reduction, and contrast enhancement processing of the image to eliminate noise and interference in the image and improve the accuracy of subsequent feature extraction and matching; S4. Feature extraction The SuperPoint algorithm is used to extract features from the preprocessed images in S3. The feature extraction process is learned directly from the original images through end-to-end learning: S4.

1. Feature point detection optimization based on weighted average feature fusion Based on the SuperPoint algorithm, the key point detection and descriptor generation are optimized, and a multi-scale feature fusion mechanism is introduced in the key point detector to capture the multi-scale features in the image through feature maps of different scales. Feature point detection optimization is achieved by constructing a feature pyramid and performing weighted average feature fusion at different levels; Its algorithm model is: Its algorithm model is: Among them, F fused is the fused feature map, F i is the feature map of the i-th scale, W i is the corresponding weight, N is the number of scales; S4.

2. Optimization of descriptor generation based on self-attention mechanism The attention mechanism is introduced into the descriptor generator, so that the model can focus on important feature areas and generate more representative descriptors. This is achieved by adding a self-attention module to the descriptor generation network. The algorithm model of the self-attention module is: Among them, Q, K, and V are query, key, and value matrices respectively. k is the dimension of the key, and the SoftMax function is used to calculate the attention weight; S5. Feature matching After feature extraction, the LightGlue algorithm is used to match feature points. LightGlue optimizes the matching strategy based on the adaptive matching threshold and learns the matching pattern directly from the image in an end-to-end learning manner, thus adapting to feature matching tasks in complex scenes. Adaptively adjust the matching threshold according to the characteristics of the image; this is achieved by learning a threshold function related to the image characteristics, which outputs the matching threshold according to the parameter statistics of the image; the algorithm model is: τ=a·I brightness +b·I contrast +c; where τ is the matching threshold, I brightness and I contrast are the brightness and contrast statistics of the image, respectively, and a, b, c are the learned parameters; In order to remove redundant key points while retaining important feature points, improvements are made on the basis of the original NMS, at least adding local area judgment and adjusting the radius or threshold of NMS according to regional characteristics; The algorithm model is: Among them, S(p) is the original score of point p, N(p) is the neighborhood set of point p, R(p) is the region set to which point p belongs, and τ is the matching threshold.

2. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 1, characterized in that: In S2, for different network environments and video acquisition devices, accurate timestamp alignment under various conditions is achieved through the timestamp alignment strategy; wherein, the method for frame rate difference detection and analysis is as follows: during the video stream processing, a high-precision timestamp is assigned to each video frame and passed through the FFmpeg video encoding / decoding library; the frame rate of each video stream is calculated in real time by counting the number of video frames received per unit time; a variance difference analysis module between each video stream is established, and the video stream that needs to be adjusted and the corresponding adjustment strategy are determined by receiving the frame rate data from each video stream and comparing the differences between them.

3. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 2 is characterized in that: In S2, for the frame rate difference problem between different video streams, the frame rate synchronization between different video streams is achieved by dynamically adjusting the display time of the video frames or inserting additional frames; inserting additional frames to achieve frame rate synchronization between different video streams is an intelligent frame rate adjustment strategy based on dynamic frame insertion technology: according to the frame rate difference between the video streams, the number of frames to be inserted is dynamically calculated; By comparing the frame rate difference between the two video streams, when the frame rate of the target video stream is lower than that of the reference video stream, blank frames are inserted into the target video stream or existing frames in the original video stream are copied according to the frame rate difference.

4. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 3 is characterized in that: The position for inserting the additional frame is selected to insert the frame in an area where the feature point changes less and the stability is higher, so as to reduce the frame insertion problem caused by feature point matching errors.

5. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 1, characterized in that: There is no b frame in the original video used for stitching.

6. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 1, characterized in that: Use the video processing tool FFmpeg and add the -fflagsnobuffer parameter in the command line to reduce the cache.

7. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 1, characterized in that: In S5, the complex scene at least includes weak texture areas, illumination changes, occlusions, and partial visibility.

8. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 7, characterized in that: The processing strategy for weak texture areas includes: based on SuperPoint, introducing or enhancing the feature description subnetwork to enable it to capture richer contextual information and local texture details, using deeper convolutional network layers or introducing an attention mechanism to focus on important texture features; combining feature maps of different scales to capture texture information from coarse to fine and efficiently process weak texture areas.

9. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 7, characterized in that: The illumination change processing strategy includes: in the training process, by simulating various illumination conditions for data augmentation, the model learns to maintain feature consistency under illumination changes; in the feature extraction stage, the features are normalized to reduce the impact of illumination changes on feature representation; by selecting a triplet loss function that is insensitive to illumination changes, the robustness of the model to illumination changes is enhanced.

10. The AI ​​video splicing and fusion method based on BRISK and feature-level fusion algorithm according to claim 7, characterized in that: The occlusion and partial visibility processing strategies include: 1) During the matching process, obviously unmatched points are first discarded through a fast screening mechanism to reduce the impact of occlusion areas on the overall matching; 2) Based on the preliminary matching, the matching points are further screened and verified through iterative optimization algorithms to improve the matching accuracy; 3) Utilize the feature consistency of local regions to detect occluded regions, and try to compensate the feature representation of occluded regions through the features of surrounding unoccluded regions; 4) During the matching process, the dustbin mechanism is dynamically adjusted to discard or temporarily ignore some difficult-to-match points to adapt to different degrees of occlusion.

Citation Information

Patent Citations

  • Multi-camera image splicing method based on traffic road

    CN117974437A

Cited By

  • AI endoscope visualization method and system based on focus tracking and storage medium

    CN120031915A

  • A method, system and storage medium for visualization of AI laparoscope based on focus tracking

    CN120031915B

  • Unmanned aerial vehicle inspection image registration and alignment method based on deep learning

    CN120563578A

  • A deep learning-based unmanned aerial vehicle inspection image registration alignment method

    CN120563578B