Camera offset detection and dynamic labeling method based on feature point matching

Abstract

The invention discloses a camera offset detection and dynamic labeling method based on feature point matching. The method comprises the following steps: constructing a target object feature point data model; acquiring a camera monitoring picture image in real time, performing feature point extraction on the picture and the video frame of the target object, and performing matching with the target object feature point data model by using a feature point matching algorithm to obtain a matched feature point pair; calculating the offset of the camera according to the matched feature point pairs, including the translation amount and the rotation amount; correcting the video frame according to the offset of the camera, and identifying and positioning a target object in the corrected video frame; and generating a corresponding label according to the identification and positioning result of the target object, and dynamically displaying the label in the video frame. The method can effectively solve the problems of poor real-time performance, low accuracy, poor robustness and the like in the prior art, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly relates to a camera offset detection and dynamic labeling method based on feature point matching. Background Art

[0002] With the rapid development of computer vision technology, object recognition and tracking technology has been widely applied in fields such as security monitoring, intelligent transportation, and human-computer interaction. Traditional object recognition and tracking methods usually rely on a fixed camera perspective. When the camera is offset, such as when the pan-tilt unit rotates or is manually moved, the position and angle of the target object in the image will change, resulting in target loss or incorrect recognition, the system being unable to continue tracking, and the label being unable to correctly label the target.

[0003] To solve the above problems, some camera offset detection methods based on feature point matching have been proposed in the prior art. These methods usually extract feature points in the image and use a feature point matching algorithm to calculate the offset of the camera. However, most of the existing methods only focus on the detection of camera offset and lack consideration of how to maintain the recognition stability of real-time recognition and marking of dynamic objects in a video under complex scenarios and changing perspectives, making it difficult to meet the requirements of practical applications.

[0004] Therefore, there is an urgent need to provide a camera offset detection and dynamic labeling method based on feature point matching to solve the above problems. Summary of the Invention

[0005] The present invention aims to provide a camera offset detection and dynamic labeling method based on feature point matching to solve the above problems existing in the prior art.

[0006] The object of the present invention is achieved by the following technical solutions:

[0007] A camera offset detection and dynamic labeling method based on feature point matching, the steps including:

[0008] S1. Construct a feature point data model of the target object;

[0009] S2. Real-time obtain the camera monitoring screen image, extract feature points from the pictures and video frames of the target object, and use a feature point matching algorithm to match with the feature point data model of the target object to obtain the matched feature point pairs;

[0010] S3. Calculate the offset of the camera according to the matched feature point pairs, including the translation amount and the rotation amount;

[0011] S4. Correct the video frames according to the offset of the camera, and identify and locate the target object in the corrected video;

[0012] S5. Generate corresponding labels according to the recognition and localization results of the target object, and display the labels in the video frame.

[0013] As a preferred technical solution of the present invention, the step of S1 for constructing the target object feature point data model includes:

[0014] S11. Collect the plan view and video stream of the target object, construct a data set and create a label file, and mark the targets in the images.

[0015] S12. Use a feature point extraction algorithm to extract the target object feature points and optimize the parameters; among them, the feature point extraction algorithms include SIFT, ORB, and SuperPoint.

[0016] S13. Generate binary descriptors and deep learning-based feature descriptors, and normalize the descriptors.

[0017] S14. Calibrate the spatial coordinates, including 2D calibration and 3D calibration.

[0018] S15. Store the model, including data structure, multi-scale modeling, and index optimization.

[0019] S16. Verify and optimize the model, use RRANSAC to remove false matches (iteration times ≥ 2000), and calculate the reprojection error.

[0020] Model optimization includes: increasing the number of key points or adjusting parameters, regularly re-collecting data to maintain the timeliness of the model.

[0021] As a preferred technical solution of the present invention, the S2 specifically includes the following steps:

[0022] S21. Obtain video stream data in real time and preprocess each frame of image in the video.

[0023] S22. Use a feature point extraction algorithm to locate the key points in the image and generate feature descriptors; the feature point extraction algorithms include SIFT, ORB, and SuperPoint.

[0024] S23. Load the target object feature point data model.

[0025] S24. Use a matching algorithm to compare the current frame feature points with the descriptors in the data model, and the matching algorithms include FLANN and brute-force matching.

[0026] S25. Perform matching verification through ratio testing and cross-validation, and retain the point pairs with the ratio of the best match to the second-best match distance less than the threshold.

[0027] S26. Eliminate false matches through a geometric transformation model;

[0028] S27. Output the successfully matched feature point pairs and store the matching results.

[0029] As a preferred technical solution of the present invention, the S3 specifically includes the following steps:

[0030] S31. According to the matched feature point pairs, select an appropriate geometric model. According to the scene type, a homography matrix is used for a planar scene, and a fundamental matrix or an essential matrix is used for a three-dimensional scene;

[0031] S32. Use a robust estimation algorithm to obtain the model parameters. Specifically, randomly select the smallest subset from the matching points, calculate the model parameter H based on the sampled subset, perform inlier screening, count the number of point pairs that conform to the current model, and finally retain the model with the most inliers;

[0032] S33. Decompose the model parameters to obtain the rotation matrix and the translation vector. According to the model type, perform parameter decomposition. If the target object is planar, the homography matrix H can be decomposed into the rotation matrix R and the translation vector t of the camera:

[0033] H = K·[R t]·K -1

[0034] Output multiple sets of possible solutions and select the correct solution through scene constraints;

[0035] The essential matrix E can be decomposed into the rotation matrix R and the translation vector t:

[0036] E = [t] × ·R

[0037] The scale of the translation vector t determines the actual displacement through a depth sensor (RGB-D camera);

[0038] S34. Optimize the results, use the Levenberg-Marquardt algorithm to optimize the rotation R and the translation t, and minimize the reprojection error;

[0039] S35. Output the offset, and output the relative motion parameters of the camera, including the rotation amount and the translation amount.

[0040] As a preferred technical solution of the present invention, the S4 is specifically to transform the current frame into the reference coordinate system according to the camera offset to eliminate the perspective change caused by the camera movement; in the corrected video frame, accurately identify the target object and locate its position, size, and direction; according to the positioning result of the target object, generate and update the label information in real time; including video frame set correction, target recognition and positioning after correction, dynamic label generation and tracking;

[0041] Among them, the video frame set correction includes determining the transformation model, image resampling and interpolation, and dynamic correction optimization; the target recognition and positioning after correction include feature point matching enhancement, target detection and positioning, and target position refinement; the dynamic label generation and tracking include label content generation and label dynamic tracking.

[0042] As a preferred technical solution of the present invention, S5 specifically includes the following steps:

[0043] S51. Obtain label information from the recognition result, including the target position, target category, and dynamic attributes;

[0044] S52. Define the label text;

[0045] S53. Overlay the label on the video frame;

[0046] S54. Real-time adjust the label position and content, including position update and content update;

[0047] S55. Process multi-target conflicts, use a multi-target tracker, and assign a unique ID to each target. The multi-target tracker can be SORT or DeepSORT.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention calculates the camera offset through feature point matching, can detect the camera offset in real time, and dynamically adjusts the recognition and positioning results of the target object. According to the recognition and positioning results of the target object, a label is attached to the target object. It can also perform dynamic recognition and real-time tagging during the rotation of the pan-tilt head, achieving precise detection of the camera offset, and real-time and accurate recognition and marking of specific objects in the video, maintaining high recognition ability even in the case of view changes caused by the rotation of the pan-tilt head. This application can effectively solve the problems of poor real-time performance, low accuracy, and poor robustness existing in the prior art, and has a wide range of application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is the flowchart of the present invention.

[0051] Figure 2 is the flowchart of step S1 of the present invention.

[0052] Figure 3 is the flowchart of step S2 of the present invention.

[0053] Figure 4 is the flowchart of step S3 of the present invention.

[0054] Figure 5 is the flowchart of step S5 of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0055] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0056] The specific implementation process of the present invention is as follows:

[0057] A camera offset detection and dynamic tagging method based on feature point matching, characterized in that the steps include:

[0058] S1. Construct a feature point data model of the target object, and the specific steps are as follows:

[0059] S11. Collect the plan view and video stream of the target object, construct a data set and create a tag file to mark the target in the image; the data collection is specifically to collect the target object data using a high-resolution camera, including multi-angle collection, multi-scale collection, multi-lighting scenarios, and 3D structure collection;

[0060] Multi-angle collection: Take 360° photos around the target object to cover different perspectives (at least 1 perspective every 30°);

[0061] Multi-scale collection: Take photos at different distances (0.5m - 3m) and scaling ratios to enhance scale invariance;

[0062] Multi-lighting scenarios: Include lighting scenarios such as normal light, low light, and strong backlight;

[0063] 3D structure collection: Use structured light / ToF camera to obtain depth information for three-dimensional objects.

[0064] S12. Use a feature point extraction algorithm to extract the feature points of the target object, optimize the parameters, and retain the top 20% of high-quality feature points through response value sorting; among them, the feature point extraction algorithms include SIFT, ORB, and SuperPoint.

[0065] S13. Generate binary descriptors (BRIEF and FREAK), feature descriptors based on deep learning (SuperPoint), and normalize the descriptors to improve the matching robustness.

[0066] S14. Calibrate the spatial coordinates, including 2D calibration and 3D calibration;

[0067] 2D calibration: Record the pixel coordinates of the feature points in the reference image;

[0068] 3D calibration includes: establishing an object coordinate system, calculating the internal and external parameters of the camera using a checkerboard calibration board, and solving the 3D coordinates of the feature points through the PnP algorithm.

[0069] S15. Store the model, including data structure, multi-scale modeling, and index optimization;

[0070] Data structure;

[0071] Multi-scale modeling: Construct a Gaussian pyramid (usually 3 - 5 layers) and store the features at each scale;

[0072] Index optimization: Use KD-Tree or FLANN to accelerate the matching and retrieval.

[0073] S16. Verify and optimize the model, use RANSAC to remove false matches (iteration times ≥ 2000), and calculate the reprojection error;

[0074] Model optimization includes increasing the number of key points or adjusting parameters, regularly re-acquiring data to maintain the timeliness of the model.

[0075] S2. Real-time obtain the images of the camera monitoring screen, extract the feature points of the pictures and video frames of the target object, and use the feature point matching algorithm to match with the feature point data model of the target object to obtain the matched feature point pairs. The specific steps are as follows:

[0076] S21. Real-time obtain the video stream data and preprocess each frame of the video; specifically, use multi-threading or asynchronous IO processing to avoid frame loss; adapt the resolution of the video stream (such as downsampling) to balance real-time performance and computer load;

[0077] Process each frame of the video, including denoising, light equalization, sharpening, etc. of the current frame to improve the robustness of subsequent feature point extraction;

[0078] Eliminate high-frequency noise by sampling Gaussian filtering;

[0079] Enhance the details of low-contrast regions through histogram equalization (CLAHE algorithm).

[0080] S22. Use the feature point extraction algorithm to locate the key points in the image and generate feature descriptors; the feature point extraction algorithms include SIFT, ORB, SuperPoint.

[0081] S23. Load the feature point data model of the target object.

[0082] S24. Use the matching algorithm to compare the feature points of the current frame with the descriptors in the data model; the matching algorithms include FLANN (for floating-point descriptors), brute-force matching (for binary descriptors).

[0083] S25. Conduct matching verification through ratio test and cross-validation to ensure the correctness of the matching, and retain the point pairs whose ratio of the best match to the second-best match distance is less than the threshold.

[0084] S26. Eliminate false matches through a geometric transformation model; the geometric transformation model includes a homography rectangle and a fundamental rectangle.

[0085] S27. Output the successfully matched feature point pairs and store the matching results; the stored matching result information includes the feature point coordinates of the current frame (i.e., image coordinates), the corresponding feature point coordinates in the data model (i.e., object coordinate system), and the matching confidence (such as distance score, proportion of inliers in RRANSAC).

[0086] S3. Calculate the offset of the camera, including translation and rotation, based on the matched feature point pairs. The specific steps are as follows:

[0087] S31. Select an appropriate geometric model based on the matched feature point pairs. According to the scene type, a homography matrix is used for a planar scene, and a fundamental matrix or an essential matrix is used for a three-dimensional scene.

[0088] The matched feature point pairs include:

[0089] Input the feature point coordinates of the current frame: pi = (xi, yi) (image coordinate system);

[0090] The corresponding feature point coordinates of the target object model: Pi = (Xi, Yi, Zi).

[0091] S32. Use a robust estimation algorithm to find the model parameters. Specifically, randomly select the smallest subset from the matched points (4 pairs are required for the homography matrix, and 5 pairs are required for the essential matrix), and calculate the model parameters based on the sampled subset; perform inlier screening and count the number of point pairs that conform to the current model (the error is less than the threshold, such as the reprojection error is less than 2 pixels); repeat the above steps, and finally retain the model with the most inliers.

[0092] S33. Decompose the model parameters to obtain the rotation matrix and translation vector. Decompose the parameters according to the model type. If the target object is planar, the homography matrix H can be decomposed into the rotation matrix R and translation vector t of the camera:

[0093] H = K · [R t] · K -1

[0094] Output multiple possible solutions, and select the correct solution through scene constraints, such as the target object being in front of the camera.

[0096] The essential matrix E can be decomposed into the rotation matrix R and translation vector t:

[0097] E = [t] × · R

[0098] The scale of the translation vector t determines the actual displacement through a depth sensor (RGB-D camera).

[0099] S34. Optimization result: Use the Levenberg - Marquardt algorithm to optimize the rotation R and translation t, and minimize the reprojection error;

[0100] S35. Output offset: Output the relative motion parameters of the camera, including the rotation amount and translation amount.

[0101] S4. According to the camera offset, transform the current frame into the reference coordinate system to eliminate the perspective change caused by the camera movement; in the corrected video frame, accurately identify the target object and locate its position, size, and direction; according to the positioning result of the target object, generate and update the label information in real - time; the specific steps are as follows:

[0102] S41. Video frame set correction includes:

[0103] Determine the transformation model: The transformation model includes homography correction (for planar scenes) and projection correction (for 3D scenes);

[0104] Homography correction: Use the homography matrix H for perspective transformation;

[0105] Corrected image = warpPerspective(current frame, H, output size);

[0106] Applicable when the target object is on a plane (such as a billboard, ground sign).

[0107] Projection correction: Project the current frame into the reference coordinate system through the camera pose (R, t) and the camera internal parameter K:

[0108] Corrected image = reprojection(current frame, K, R, t);

[0109] Applicable when the target object is a 3D structure (such as a vehicle, a human body);

[0110] If the reference coordinate system is the initial frame, it is necessary to calculate the relative pose Rrel = Rcurrent * R - 1initial.

[0111] Image resampling and interpolation: The interpolation methods include bilinear interpolation and Lanczos interpolation; process the edges, fill the black areas or expand the image boundaries.

[0112] Dynamic correction optimization: Dynamic correction optimization includes incremental update and local correction;

[0113] Incremental update: In a continuous video stream, accumulate the pose changes to avoid the cumulative error caused by global transformation frame by frame;

[0114] Local correction: Only transform the target region (ROI) to reduce the computational amount.

[0115] S42. Corrected target recognition and positioning, including:

[0116] Feature point matching enhancement: Based on model-based matching, match the corrected frame with the pre-stored feature point model (SIFT / ORB descriptor) of the target object, including using the FLANN or brute-force matching algorithm and further screening the matching point pairs through RANSAC.

[0117] Target detection and positioning: Use a lightweight target detection model (such as YOLO Nano, MobileNet-SSD) to locate the target in real time;

[0118] Target position refinement: Perform sub-pixel level optimization on the detected bounding box or feature points (such as the `cornerSubPix` function in OpenCV); Combine the known size of the target object (such as the length and width of a vehicle) to verify the rationality of the bounding box through perspective projection.

[0119] S43. Dynamic label generation and tracking, including:

[0120] Label content generation: Include static information and dynamic information. Static information includes the target object name and ID;

[0121] Dynamic information includes position and motion state. The position is the bounding box coordinates, and the motion state includes speed (multi-frame tracking) and direction (through optical flow or pose estimation).

[0122] Label dynamic tracking:

[0123] Use Kalman filtering or particle filtering to predict the target's next frame position and reduce jitter;

[0124] Perform moving average filtering on the label position and size to avoid jitter caused by detection jitter.

[0125] S5. Generate corresponding labels according to the recognition and positioning results of the target object, and display the labels in the video frame. The specific steps are as follows:

[0126] S51. Obtain label information from the recognition results, including target position, target category, and dynamic attributes; where the target position includes bounding box coordinates or center coordinates; the target category includes object name and confidence score; dynamic attributes include motion direction (velocity vector), displacement identifier (ID), and size (length, width, and height);

[0127] S52. Define the label text;

[0128] S53. Overlay the label on the video frame;

[0129] S54. Real-time adjust the label position and content, including:

[0130] Position update: Predict the position of the next frame through optical flow or Kalman filtering to reduce label jitter; if the camera is offset, the label coordinates need to be inversely transformed back to the original frame coordinate system;

[0131] Content update: Update the confidence level according to the detection results of the new frame; if the target disappears or reappears, update the visible and hidden state of the label.

[0132] S55. Handle multi-target conflicts, use a multi-target tracker to assign a unique ID to each target. The multi-target tracker can be SORT, DeepSORT; if multiple label areas overlap, merge the display or adjust the position; determine the label display hierarchy according to the confidence level or risk level.

[0133] This application calculates the camera offset through feature point matching, can detect the camera offset in real time, and dynamically adjusts the recognition and positioning results of the target object. According to the recognition and positioning results of the target object, labels are attached to the target object. It can also perform dynamic recognition and real-time labeling when the pan-tilt rotates, achieving accurate detection of the camera offset, as well as real-time and accurate recognition and marking of specific objects in the video, and maintaining high recognition ability even when the perspective changes due to the rotation of the pan-tilt. This application can effectively solve the problems of poor real-time performance, low accuracy, and poor robustness existing in the prior art, and has a wide range of application prospects.

[0134] The above embodiments only represent the implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A camera offset detection and dynamic labeling method based on feature point matching, characterized in that the steps include: S1. Constructing a target object feature point data model; S2, acquiring the camera monitoring screen image in real time, extracting feature points from the pictures and video frames of the target object, and matching them with the target object feature point data model using a feature point matching algorithm to obtain matching feature point pairs; S3, calculating the camera offset, including translation and rotation, based on the matched feature point pairs; S4, correcting the video frame according to the offset of the camera, and identifying and locating the target object in the corrected video; S5. Generate corresponding labels based on the recognition and positioning results of the target object, and display the labels in the video frame.

2. According to the method of camera offset detection and dynamic labeling based on feature point matching in claim 1, it is characterized in that: The step of constructing the target object feature point data model in S1 includes: S11, collect the plan view and video stream of the target object, build a data set and create a label file, and mark the target in the image; S12, extracting feature points of the target object using a feature point extraction algorithm and optimizing parameters; wherein the feature point extraction algorithm includes SIFT, ORB, and SuperPoint; S13, generating a binary descriptor and a feature descriptor based on deep learning, and normalizing the descriptor; S14, calibrating the spatial coordinates, including 2D calibration and 3D calibration; S15. Storing the model, including data structure, multi-scale modeling, and index optimization; S16. Verify and optimize the model, use RRANSAC to remove mismatches (iterations ≥ 2000), and calculate the reprojection error; Model optimization includes: increasing the number of key points or adjusting parameters, and regularly recollecting data to maintain the timeliness of the model.

3. The camera offset detection and dynamic labeling method based on feature point matching according to claim 1 is characterized in that: The S2 specifically includes the following steps: S21, acquiring video stream data in real time, and preprocessing each frame of the video; S22, using a feature point extraction algorithm to locate key points in the image and generate a feature descriptor; the feature point extraction algorithm includes SIFT, ORB, and SuperPoint; S23, loading the target object feature point data model; S24, using a matching algorithm to compare the feature points of the current frame with the descriptors in the data model, the matching algorithms including FLANN and brute force matching; S25, performing matching verification through ratio test and cross validation, and retaining point pairs whose distance ratio between the best match and the second best match is less than a threshold; S26, eliminating mismatches through a geometric transformation model; S27. Output the successfully matched feature point pairs and store the matching results.

4. The camera offset detection and dynamic labeling method based on feature point matching according to claim 1, characterized in that: The S3 specifically includes the following steps: S31, selecting an appropriate geometric model according to the matched feature point pairs, and selecting according to the scene type, using a homography matrix for a planar scene and a basic matrix or an essential matrix for a three-dimensional scene; S32, using a robust estimation algorithm to find model parameters, specifically, randomly selecting a minimum subset from the matching points, and calculating the model parameter H based on the sampled subset; performing inlier screening, counting the number of point pairs that meet the current model, and finally retaining the model with the most inliers; S33, decompose the model parameters to obtain the rotation matrix and translation vector, and perform parameter decomposition according to the model type. If the target object is a plane, the homography matrix H can be decomposed into the rotation matrix R and translation vector t of the camera: H=K·[R t]·K -1 Output multiple sets of possible solutions and select the correct solution based on scenario constraints; The essential matrix E can be decomposed into the rotation matrix R and the translation vector t: E=[t] × ·R The scale of the translation vector t is determined by the depth sensor (RGB-D camera) to determine the actual displacement; S34, optimization results, using the Levenberg-Marquardt algorithm to optimize the rotation R and translation t to minimize the reprojection error; S35, outputting the offset, outputting the relative motion parameters of the camera, including the rotation amount and the translation amount.

5. The camera offset detection and dynamic labeling method based on feature point matching according to claim 1, characterized in that: The S4 specifically transforms the current frame into a reference coordinate system according to the camera offset to eliminate the change in viewing angle caused by the camera movement; accurately identifies the target object in the corrected video frame and locates its position, size and direction; generates and updates the label information in real time according to the positioning result of the target object; Including video frame set correction, corrected target recognition and positioning, dynamic label generation and tracking; Among them, video frame set correction includes determining the transformation model, image resampling and interpolation, and dynamic correction optimization; corrected target recognition and positioning include feature point matching enhancement, target detection and positioning, and target position refinement; dynamic label generation and tracking include label content generation and label dynamic tracking.

6. The camera offset detection and dynamic labeling method based on feature point matching according to claim 1, characterized in that: The S5 specifically includes the following steps: S51, obtaining label information from the recognition result, including target location, target category, and dynamic attributes; S52, define label text; S53, superimposing the label onto the video frame; S54, adjusting the label position and content in real time, including updating the position and content; S55. Process multi-target conflicts by using a multi-target tracker to assign a unique ID to each target. The multi-target tracker may be SORT or DeepSORT.

Images