Target tracking method and system based on inertial sensor and visual camera
By combining the information of inertial sensors and vision cameras, multi-sensor fusion technology is used to achieve accurate positioning and tracking of the targets, solving the problem of unstable positioning in complex environments in the existing technology, and significantly improving the tracking accuracy and robustness.
Patent Information
- Application Number
- CN202411680046.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-14
AI Technical Summary
Existing pedestrian navigation and positioning technology is difficult to provide continuous and stable tracking services in large or complex environments, and the accuracy is unstable due to factors such as lighting conditions and occlusion.
The target tracking method based on inertial sensors and vision cameras is adopted, combined with inertial navigation, visual detection and multi-sensor fusion technology, data is collected through the mobile phone's built-in accelerometer and gyroscope, filtering and denoising processing, calculating the target position, and capturing images through the visual camera, feature extraction and matching to achieve accurate positioning and tracking of the target.
It significantly improves the accuracy and robustness of target tracking, can maintain high tracking accuracy in complex environments, reduces the problem of error accumulation of single sensors, and enhances the stability and adaptability of the system.
Smart Images
Figure CN119625342A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to pedestrian navigation and positioning, and in particular to a target tracking method and system based on inertial sensors and visual cameras. Background Art
[0002] Among the existing positioning technologies, the target positioning system based on Bluetooth sensors uses parameters such as the strength attenuation and arrival time difference of Bluetooth signals to calculate the position of the target object by receiving and processing Bluetooth signals. The core is to collect and process the information of Bluetooth signals and use the characteristics of wireless signal transmission for positioning. The transmission distance of Bluetooth signals is relatively short and is easily affected by walls and objects, which limits the effective coverage of the target tracking system, making it impossible for the system to provide continuous and stable tracking services in some large or complex environments. The accuracy is still affected by many factors, such as signal strength, propagation environment, etc., which affects the stability of the positioning results.
[0003] Pure camera-based target tracking systems are based on image processing and target detection algorithms. They use cameras to capture images of target objects, and then use image processing algorithms to extract features of target objects, such as color, shape, texture, etc. Next, target detection algorithms are used to search and match these features in consecutive video frames, thereby achieving real-time tracking and positioning of target objects. Pure camera-based target tracking systems rely heavily on image information, so lighting conditions have a great impact on the performance of the system. In the case of insufficient or excessive lighting, the contrast and clarity of the image may be reduced, resulting in a decrease in the accuracy of target detection and tracking; when the target object is occluded by other objects, the tracking ability of the system will be seriously affected. Occlusion may cause the loss of some or all of the feature information of the target object, making it difficult for the system to continue to accurately track the target; if the target object moves too fast, or the relative motion between the camera and the target object causes image blur, the tracking performance of the system will also decrease. Summary of the invention
[0004] The embodiment of the present application provides a target tracking method and system based on an inertial sensor and a visual camera to solve the problems existing in the related art. The technical solution is as follows: A target tracking method based on an inertial sensor and a visual camera includes inertial navigation, visual detection and multi-sensor fusion. The steps of the inertial navigation are as follows:
[0005] S11: Data collection: The built-in IMU sensors such as accelerometers and gyroscopes in the mobile phone collect the acceleration and angular velocity data of the target in three directions in real time, providing basic data for subsequent processing;
[0006] S12: Data processing: filtering and denoising the collected data;
[0007] S13: Position estimation: By integrating the acceleration data twice, the displacement of the target in three directions is obtained, thereby estimating the position of the target. At the same time, the attitude and heading of the target are estimated using the gyroscope data.
[0008]
[0009] in, is the position vector representing latitude, longitude and altitude, D -1 is the velocity vector, is the velocity vector in the navigation system, is the attitude transfer matrix between coordinate systems, is the angular velocity of the Earth's rotation in a certain coordinate system, is the angular velocity of rotation between coordinate systems, M is the mass of the earth;
[0010] S14: Error correction: Use the PDR algorithm to assist in correcting the positioning technology to improve positioning accuracy.
[0011] Furthermore, the PDR algorithm includes:
[0012] S21: Gait detection module: intelligently judge whether the target has taken a step according to the change characteristics of the acceleration data. If no gait change is detected, re-read the sensor data; if a gait change is detected, proceed to the next step of processing;
[0013] S22: Step length estimation module: Use mathematical models and algorithms to accurately calculate the step length L of each step. An estimation model in a nonlinear model is mainly used for step length estimation. The calculation formula is as follows:
[0014]
[0015] Among them, a is the acceleration z-axis data value at the peak and trough, and S is the step coefficient, which can be set within a certain range.
[0016] S23: Heading estimation module: uses acceleration, angular velocity and magnetic intensity sensor data, combined with attitude solution algorithm, to accurately calculate the heading angle of the target;
[0017] S24: Position update module: Based on the step length and heading angle, the position information of the target is updated in real time using the position update algorithm.
[0018] Furthermore, S21: the gait detection module adopts the peak-zero-crossing-valley detection method to extract the acceleration Z-axis data output by the mobile phone when the pedestrian is walking or obtains the modulus of the acceleration three-axis output to perform peak and trough detection.
[0019] Further, S23: heading estimation module: adopts Mahony-based complementary filtering algorithm to perform attitude solution, the model is as follows: according to the quaternion rotation matrix
[0020]
[0021] Proportional-integral operation on the error
[0022]
[0023] Using the first-order Runge-Kutta method to solve the quaternion
[0024]
[0025] Convert quaternion to Euler angles
[0026]
[0027] Among them, v bx 、v by 、v bz is the gravitational acceleration on three axes, q0, q1, q2, q3 are the four elements that make up the quaternion, q0 is the scalar part, q1, q2, q3 are the vector part, g bx , g by , g bz is the angular velocity on the three axes, K p is the control proportional gain of the current error response speed in the PI controller, e bx 、e by 、e bz is the cross product error on the three axes, eInt bx 、eInt by 、eInt bz is the error integral on three axes, roll, pitch, yaw are the roll angle, pitch angle and heading angle;
[0028] Furthermore, the change of the position over time in the position update module in S24 can be represented by a set of differential equations:
[0029]
[0030] in, λ, h are latitude, longitude and altitude, R M is the meridian curvature radius, v N 、v E 、v D is the velocity in a certain coordinate system.
[0031] Further, the steps of visual inspection are as follows:
[0032] S31: Image acquisition: Use the high-resolution visual camera on the vehicle to capture image data in the surrounding environment in real time;
[0033] S32: Image processing: preprocessing the collected image data, including denoising and contrast enhancement, to improve image quality;
[0034] S33: Feature extraction: Using deep learning algorithms, key features that are helpful for target detection, such as edges, colors, textures, etc., are extracted from the processed images;
[0035] S34: Feature analysis / model matching: Based on the extracted feature information, the deep learning model is used to identify, classify and detect the target object, and the target is accurately positioned through model matching;
[0036] S35: Result output and tracking: Output the detection results and combine image processing and feature matching technology to achieve continuous tracking of the target.
[0037] Furthermore, the steps of multi-sensor fusion are as follows:
[0038] S41: Sensor calibration and synchronization: After the camera's timestamp is shifted by td, the camera and IMU are synchronized;
[0039] S42: Data preprocessing: filtering and denoising the mobile phone inertial navigation data, and enhancing and denoising the images captured by the vehicle camera to improve the image quality and the accuracy of feature extraction;
[0040] S43: Data initialization: The mobile phone successfully connects to the system device via Bluetooth and starts up with the system. At this time, the target position information detected by the camera sensor is given to the mobile phone inertial navigation sensor to initialize it as the data information of the first frame or the first few frames;
[0041] S44: System initialization and scene monitoring: After the program is started, the system is initialized first, including configuration parameters, loading models, etc.
[0042] S45: target tracking strategy selection, including the following steps:
[0043] S4501: Target detection and ID acquisition: The system first performs target detection, identifies each target in the scene, assigns a unique ID to it, and records the location information of each target;
[0044] S4502: Target tracking and overlap judgment.
[0045] Further, S4502: target tracking and overlap judgment include the following situations:
[0046] 1. When the target exists and the following ID remains unchanged:
[0047] 1.1 Independent state: If there is no overlap between the following target box and other detection boxes (i.e., IOU = 0), then based on the logic of the detection and tracking algorithm, using the current position of the following target as the navigation point, perform the target following operation;
[0048] 1.2 Overlap state: If there is an overlap between the following target box and other detection boxes (i.e., IOU > 0), then perform the following processing logic:
[0049] a. Warning prompt: Use the IOU value calculated during the target detection process as an indicator to measure the overlap degree between two targets; when 0 < IOU < the preset threshold, it means that the following target and other targets are occluded but can be detected normally; when IOU > the preset threshold, it means that the occlusion is serious and the detection is unstable; at this time, the system gives a prompt through the warning light;
[0050] b. False positive determination: Combine the position information of the camera end and the mobile phone end, and use the false positive target fusion determination algorithm logic to perform false positive determination on the following target to reduce false tracking; among them, the false positive target fusion determination algorithm mainly compares the movement direction and angle difference during the movement of the two, and combines the cosine distance and the Mahalanobis distance, and introduces the corresponding weight coefficient for correlation matching, so as to update the IMU position information;
[0051] 2. When the target exists but the following ID changes:
[0052] If the target is not lost but the following ID changes, then based on the false negative target fusion determination algorithm logic, perform false negative determination on all targets; this determination process also needs to combine the position information of the camera end and the mobile phone end to ensure the accuracy and stability of tracking; among them, the main difference between the false negative target fusion determination algorithm and the false positive target fusion determination algorithm is that it uses the information of several frames before the change of the target ID as the measurement standard, and the position information before disappearance as the start and end points for subsequent correlation matching;
[0053] 3. Target loss determination and re-initialization:
[0054] 3.1 Loss judgment: If within a certain life cycle, the survival time of the tracker that does not match the box in the target detection exceeds the preset threshold, it is determined that the following target is lost;
[0055] 3.2 Re-initialization: Once it is determined that the following target is lost, perform data initialization operations, including resetting the initial position information of the inertial sensor and the visual camera sensor, and re-performing target detection and tracking.
[0056] The present invention also provides a target tracking system based on an inertial sensor and a visual camera, including:
[0057] Vision cameras, inertial measurement units, processors, and memory;
[0058] Wherein, the visual camera and the inertial measurement unit are both connected to the processor, and the memory stores executable instructions of the processor;
[0059] The processor preprocesses the collected sensor information, wherein the sensor information includes: image information of a visual camera and IMU data of an inertial measurement unit;
[0060] Get the external parameters of the system to which the visual camera and inertial measurement unit belong;
[0061] The external parameters of the system are the relative position and posture between the inertial measurement unit and the visual camera.
[0062] The advantages or beneficial effects of the above technical solution include:
[0063] (1) Significantly improves the accuracy of target tracking: By combining the information of the mobile phone inertial sensor and the visual camera, the present invention can fully utilize the advantages of both to achieve more accurate tracking of the target object. The mobile phone inertial sensor can provide real-time motion state information of the object, such as acceleration and angular velocity, while the visual camera can capture the image information of the target object, including position, shape and size. The fusion of the two information can significantly improve the accuracy of target tracking, especially in complex environments, such as lighting changes, occlusion, etc., and still maintain a high tracking accuracy.
[0064] (2) Enhanced robustness of target tracking: By fusing information from multiple sensors, the error accumulation problem that may occur in a single sensor during long-term tracking is effectively reduced. At the same time, the image information from the visual camera can provide additional visual clues to help the system better cope with emergencies and complex scenarios, such as sudden acceleration, deceleration, or steering of the target object. This fusion of multi-source information makes the target tracking system more robust and can maintain stable tracking effects in various complex environments.
[0065] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.
[0067] Figure 1 A schematic diagram of the inertial navigation positioning process according to the present application;
[0068] Figure 2 A schematic diagram of time synchronization between a camera and an IMU according to the present application;
[0069] Figure 3 This is a schematic diagram of the multi-sensor fusion process for this application. DETAILED DESCRIPTION
[0070] In the following, only some exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be illustrative and non-restrictive in nature.
[0071] A target tracking method based on inertial sensors and visual cameras, including inertial navigation, visual detection and multi-sensor fusion, such as Figure 1 As shown, the steps of inertial navigation are as follows:
[0072] S11: Data collection: The built-in accelerometer, gyroscope and other IMU sensors of the mobile phone collect the acceleration and angular velocity data of the target in three directions in real time, providing basic data for subsequent processing; the basic modeling formula is as follows:
[0073]
[0074] in, Represents the final output acceleration measurement data, Indicates the current true value of acceleration, g b represents the gravitational acceleration in a certain coordinate system, b a Indicates the offset of the accelerometer, n a represents the random measurement noise of the accelerometer; Represents the final output angular velocity measurement value, Indicates the true value of the current angular velocity, b g Indicates the offset of the gyroscope, n g represents the random measurement noise of the gyroscope.
[0075] S12: Data processing: Filtering and denoising the collected data; this is crucial to reducing sensor noise and errors. The basic processing methods are as follows:
[0076] Median filter:
[0077] y[n]=median(x[nk],…,x[n],…x[n+k])
[0078] Where x is the original signal, y is the filtered signal, n is the current position, and k is the window size.
[0079] Moving average filtering:
[0080]
[0081] Where x is the input signal, y is the output signal, and M is the number of points in the average.
[0082] S13: Position estimation: By integrating the acceleration data twice, the displacement of the target in three directions is obtained, thereby estimating the position of the target. At the same time, the attitude and heading of the target are estimated using the gyroscope data.
[0083]
[0084] in, is the position vector representing latitude, longitude and altitude, D -1 is the velocity vector, is the velocity vector in the navigation system, is the attitude transfer matrix between coordinate systems, is the angular velocity of the Earth's rotation in a certain coordinate system, is the angular velocity of rotation between coordinate systems, M is the mass of the earth;
[0085] S14: Error correction: Use the PDR algorithm to assist in correcting the positioning technology to improve positioning accuracy.
[0086] Furthermore, the PDR algorithm includes:
[0087] S21: Gait detection module: intelligently judge whether the target has taken a step according to the change characteristics of the acceleration data. If no gait change is detected, re-read the sensor data; if a gait change is detected, proceed to the next step of processing;
[0088] S22: Step length estimation module: Use mathematical models and algorithms to accurately calculate the step length L of each step. An estimation model in a nonlinear model is mainly used for step length estimation. The calculation formula is as follows:
[0089]
[0090] Among them, a is the acceleration z-axis data value at the peak and trough, and S is the step coefficient, which can be set within a certain range.
[0091] S23: Heading estimation module: uses acceleration, angular velocity and magnetic intensity sensor data, combined with attitude solution algorithm, to accurately calculate the heading angle of the target;
[0092] S24: Position update module: Based on the step length and heading angle, the position information of the target is updated in real time using the position update algorithm.
[0093] Furthermore, S21: the gait detection module adopts the peak-zero-crossing-valley detection method to extract the acceleration Z-axis data output by the mobile phone when the pedestrian is walking or obtains the modulus of the acceleration three-axis output to perform peak and trough detection.
[0094] Further, S23: heading estimation module: adopts Mahony-based complementary filtering algorithm to perform attitude solution, the model is as follows: according to the quaternion rotation matrix
[0095]
[0096] Proportional-integral operation on the error
[0097]
[0098] Using the first-order Runge-Kutta method to solve the quaternion
[0099]
[0100] Convert quaternion to Euler angles
[0101]
[0102] Among them, v bx 、v by 、v bz is the gravitational acceleration on three axes, q0, q1, q2, q3 are the four elements that make up the quaternion, q0 is the scalar part, q1, q2, q3 are the vector part, g bx , g by , g bz is the angular velocity on the three axes, K p is the control proportional gain of the current error response speed in the PI controller, e bx 、e by 、e bz is the cross product error on the three axes, eInt bx 、eInt by 、eInt bz is the error integral on three axes, roll, pitch, yaw are the roll angle, pitch angle and heading angle;
[0103] Furthermore, the change of the position over time in the position update module in S24 can be represented by a set of differential equations:
[0104]
[0105] in, λ, h are latitude, longitude and altitude, R M is the meridian curvature radius, v N 、v E 、v D is the velocity in a certain coordinate system.
[0106] Further, the steps of visual inspection are as follows:
[0107] S31: Image acquisition: Use the high-resolution visual camera on the vehicle to capture image data in the surrounding environment in real time;
[0108] S32: Image processing: preprocessing the collected image data, including denoising and contrast enhancement, to improve image quality;
[0109] S33: Feature extraction: Using deep learning algorithms, key features that are helpful for target detection, such as edges, colors, textures, etc., are extracted from the processed images;
[0110] S34: Feature analysis / model matching: Based on the extracted feature information, the deep learning model is used to identify, classify and detect the target object, and the target is accurately positioned through model matching;
[0111] S35: Result output and tracking: Output the detection results and combine image processing and feature matching technology to achieve continuous tracking of the target.
[0112] Specifically in terms of detection and following algorithms, this application uses the yolov5 target detection algorithm and the bytetrack target tracking algorithm on the RK3588 high-performance computing platform to achieve real-time detection and precise tracking of targets. The yolov5 algorithm, with its high efficiency and accuracy, can quickly detect targets in the video; while the bytetrack algorithm uses advanced tracking strategies to achieve continuous tracking of the target's motion trajectory. By combining the two, an efficient and accurate target detection and tracking system is constructed. The system can detect targets in the video in real time and track their motion trajectories, providing strong technical support for application scenarios such as autonomous driving.
[0113] Furthermore, if Figure 3 As shown; the steps of multi-sensor fusion are as follows:
[0114] S41: Sensor Calibration and Synchronization: After shifting the timestamp of the camera by td, synchronization between the camera and the IMU is achieved, as Figure 2 shown;
[0115] S42: Data Preprocessing: Filter and denoise the inertial data of the mobile phone, and enhance and denoise the images captured by the vehicle-mounted camera to improve the image quality and the accuracy of feature extraction;
[0116] S43: Data Initialization: The mobile phone successfully connects to the system device via Bluetooth and starts following the system. At this time, the target position information detected by the camera sensor is assigned to the inertial sensor of the mobile phone to initialize it as the data information of the first frame or the first few frames;
[0117] S44: System Initialization and Scene Monitoring: After the program starts, system initialization is first performed, including configuring parameters, loading models, etc.
[0118] S45: Target Tracking Strategy Selection, including the following steps:
[0119] S4501: Target Detection and ID Acquisition: The system first performs target detection, identifies each target in the scene, assigns a unique ID to it, and at the same time, records the position information of each target;
[0120] S4502: Target Tracking and Overlap Judgment.
[0121] Further, S4502: Target Tracking and Overlap Judgment includes the following situations:
[0122] 1. When the target exists and the following ID remains unchanged:
[0123] 1.1 Independent State: If there is no overlap between the following target box and other detection boxes (i.e., IOU = 0), then based on the logic of the detection and tracking algorithm, the current following target position is used as the navigation point to perform the target following operation;
[0124] 1.2 Overlap State: If there is an overlap between the following target box and other detection boxes (i.e., IOU>0), then the following processing logic is executed:
[0125] a. Early Warning Prompt: The IOU value calculated during target detection is used as an indicator to measure the overlap degree of two targets; when 0 < IOU < the preset threshold, it means that the following target and other targets are occluded but can be detected normally; when IOU > the preset threshold, it means that the occlusion is severe and the detection is unstable; at this time, the system gives a prompt through the warning light;
[0126] b. False positive determination: Combined with the position information of the camera and the mobile phone, the false positive target fusion determination algorithm logic is used to make false positive determinations on the follow-up target to reduce mistracking; the false positive target fusion determination algorithm mainly compares the movement direction and angle difference of the two during the movement process, and combines the cosine distance and Mahalanobis distance to introduce the corresponding weight coefficient for correlation matching, thereby updating the IMU position information;
[0127] 2. When the target exists but the tracking ID changes:
[0128] If the target is not lost, but the tracking ID changes, all targets are judged as false negative based on the false negative target fusion judgment algorithm logic; this judgment process also needs to combine the location information of the camera and the mobile phone to ensure the accuracy and stability of tracking; the main difference between the false negative target fusion judgment algorithm and the false positive target fusion judgment algorithm is that the information of the few frames before the target ID changes is used as the measurement standard, and the location information before disappearance is used as the start and end points for subsequent association matching;
[0129] 3. Target loss determination and reinitialization:
[0130] 3.1 Loss judgment: If the survival time of the tracker that is not matched to the frame in the target detection exceeds the preset threshold within a certain life cycle, it is judged that the tracking target is lost;
[0131] 3.2 Reinitialization: Once it is determined that the tracking target is lost, the data initialization operation is performed, including resetting the initial position information of the inertial sensor and visual camera sensor, and re-detecting and tracking the target.
[0132] The present invention also provides a target tracking system based on an inertial sensor and a visual camera, comprising:
[0133] Vision cameras, inertial measurement units, processors, and memory;
[0134] Wherein, the visual camera and the inertial measurement unit are both connected to the processor, and the memory stores executable instructions of the processor;
[0135] The processor preprocesses the collected sensor information, wherein the sensor information includes: image information of a visual camera and IMU data of an inertial measurement unit;
[0136] Get the external parameters of the system to which the visual camera and inertial measurement unit belong;
[0137] The external parameters of the system are the relative position and posture between the inertial measurement unit and the visual camera.
[0138] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. For example, the mobile phone inertial sensor can use other types of wearable inertial sensors, such as sensors on devices such as smart watches and smart bracelets. These devices also have functions such as accelerometers and gyroscopes, and can measure the motion state of objects in real time; it is also possible to consider introducing a depth camera, which can directly obtain the depth information of the scene, that is, the distance between the object and the camera. This information is very useful for target tracking because it can provide more accurate object position and shape information. Therefore, it is possible to consider using a depth camera in combination with a mobile phone inertial sensor to achieve more accurate target tracking; without departing from the scope and spirit of the embodiments described, many modifications and changes are obvious to ordinary technicians in the field of technology. The choice of terms used in this article is intended to best explain the principles, practical applications or technical improvements in the market of each embodiment, or to enable other ordinary technicians in the field of technology to understand the embodiments disclosed herein. Therefore, the scope of protection of this application shall be based on the scope of protection of the claims.
Claims
1. A target tracking method based on inertial sensor and visual camera, characterized in that: Including inertial navigation, visual detection and multi-sensor fusion, the steps of inertial navigation are as follows: S11: Data collection: The built-in IMU sensors such as accelerometers and gyroscopes in the mobile phone collect the acceleration and angular velocity data of the target in three directions in real time, providing basic data for subsequent processing; S12: Data processing: filtering and denoising the collected data; S13: Position estimation: By integrating the acceleration data twice, the displacement of the target in three directions is obtained, thereby estimating the position of the target. At the same time, the attitude and heading of the target are estimated using the gyroscope data. in, is the position vector representing latitude, longitude and altitude, D -1 is the velocity vector, is the velocity vector in the navigation system, is the attitude transfer matrix between coordinate systems, is the angular velocity of the Earth's rotation in a certain coordinate system, is the angular velocity of rotation between coordinate systems, M is the mass of the earth; S14: Error correction: Use the PDR algorithm to assist in correcting the positioning technology to improve positioning accuracy.
2. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: The PDR algorithm includes: S21: Gait detection module: intelligently judge whether the target has taken a step according to the change characteristics of the acceleration data. If no gait change is detected, re-read the sensor data; if a gait change is detected, proceed to the next step of processing; S22: Step length estimation module: Use mathematical models and algorithms to accurately calculate the step length L of each step. An estimation model in a nonlinear model is mainly used for step length estimation. The calculation formula is as follows: Among them, a is the acceleration z-axis data value at the peak and trough, and S is the step coefficient, which can be set within a certain range; S23: Heading estimation module: uses acceleration, angular velocity and magnetic intensity sensor data, combined with attitude solution algorithm, to accurately calculate the heading angle of the target; S24: Position update module: Based on the step length and heading angle, the position information of the target is updated in real time using the position update algorithm.
3. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: S21: The gait detection module adopts the peak-zero-crossing-valley detection method to extract the acceleration Z-axis data output by the mobile phone when the pedestrian is walking or obtains the modulus of the acceleration triaxial output to perform peak and trough detection.
4. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: The S23: heading estimation module: adopts the Mahony-based complementary filtering algorithm to perform attitude calculation, and the model is as follows: According to the quaternion rotation matrix Proportional-integral operation on the error Using the first-order Runge-Kutta method to solve the quaternion Convert quaternion to Euler angles Among them, v bx 、v by 、v bz is the gravitational acceleration on three axes, q0, q1, q2, q3 are the four elements that make up the quaternion, q0 is the scalar part, q1, q2, q3 are the vector part, g bx , g by , g bz is the angular velocity on the three axes, K p is the control proportional gain of the current error response speed in the PI controller, e bx 、e by 、e bz is the cross product error on the three axes, eInt bx 、eInt by 、eInt bz is the error integral on three axes, roll, pitch, and yaw are the roll angle, pitch angle, and heading angle.
5. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: The change of position over time in the position update module in S24 can be expressed by a set of differential equations: in, λ, h are latitude, longitude and altitude, R M is the meridian curvature radius, v N 、v E 、v D is the velocity in a certain coordinate system.
6. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: The steps of the visual inspection are as follows: S31: Image acquisition: Use the high-resolution visual camera on the vehicle to capture image data in the surrounding environment in real time; S32: Image processing: preprocessing the collected image data, including denoising and contrast enhancement, to improve image quality; S33: Feature extraction: Using deep learning algorithms, key features that are helpful for object detection are extracted from the processed images; S34: Feature analysis / model matching: Based on the extracted feature information, the deep learning model is used to identify, classify and detect the target object, and the target is accurately positioned through model matching; S35: Result output and tracking: Output the detection results and combine image processing and feature matching technology to achieve continuous tracking of the target.
7. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: The steps of multi-sensor fusion are as follows: S41: Sensor calibration and synchronization: After the camera's timestamp is shifted by td, the camera and IMU are synchronized; S42: Data preprocessing: Filter and denoise the inertial data of the mobile phone, and perform enhancement, denoising, etc. on the images captured by the vehicle-mounted camera to improve the image quality and the accuracy of feature extraction; S43: Data initialization: The mobile phone successfully connects to the system device via Bluetooth and starts following the system. At this time, the target position information detected by the camera sensor is assigned to the inertial sensor of the mobile phone to initialize it as the data information of the first frame or the first few frames; S44: System initialization and scene monitoring: After the program starts, system initialization is performed first; S45: Target tracking strategy selection, including the following steps: S4501: Target detection and ID acquisition: The system first performs target detection, identifies each target in the scene, assigns a unique ID to it, and at the same time records the position information of each target; S4502: Target tracking and overlap judgment.
8. The target tracking method based on inertial sensor and visual camera according to claim 1, characterized in that: The S4502: Target tracking and overlap judgment includes the following situations:
1. When the target exists and the following ID does not change: 1.1 Independent state: If there is no overlap between the following target box and other detection boxes (i.e., IOU = 0), then based on the detection and tracking algorithm logic, use the current following target position as the navigation point to perform the target following operation; 1.2 Overlap state: If there is an overlap between the following target box and other detection boxes (i.e., IOU>0), then perform the following processing logic: a. Warning prompt: According to the IOU value calculated during the target detection process, which is used as an index to measure the overlap degree of the two targets; when 0 < IOU < the preset threshold, it means that the following target and other targets have an occlusion phenomenon but can be detected normally; when IOU> the preset threshold, it means that the occlusion is serious and the detection is unstable; at this time, the system gives a prompt through the warning light; b. False positive determination: Combining the position information of the camera end and the mobile phone end, use the false positive target fusion determination algorithm logic to perform false positive determination on the following target to reduce false tracking; among them, the false positive target fusion determination algorithm mainly compares the motion direction and angle difference during the motion of the two, and combines the cosine distance and the Mahalanobis distance, and introduces the corresponding weight coefficient to perform correlation matching, so as to update the IMU position information; 2. When the target exists but the following ID changes: If the target is not lost but the following ID changes, then based on the false negative target fusion determination algorithm logic, perform false negative determination on all targets; this determination process also needs to combine the position information of the camera end and the mobile phone end to ensure the accuracy and stability of the tracking; among them, the main difference between the false negative target fusion determination algorithm and the false positive target fusion determination algorithm is that it uses the information of several frames before the change of the target ID as the measurement standard, and the position information before disappearance as the start and end points for subsequent correlation matching; 3. Target loss determination and re-initialization: 3.1 Loss judgment: If within a certain life cycle, the survival time of the tracker that does not match the box in the target detection exceeds the preset threshold, it is determined that the following target is lost; 3.2 Reinitialization: Once it is determined that the tracking target is lost, the data initialization operation is performed, including resetting the initial position information of the inertial sensor and visual camera sensor, and re-detecting and tracking the target.
9. A target tracking system based on inertial sensors and visual cameras, characterized in that: include: Vision cameras, inertial measurement units, processors, and memory; Wherein, the visual camera and the inertial measurement unit are both connected to the processor, and the memory stores executable instructions of the processor; The processor preprocesses the collected sensor information, wherein the sensor information includes: image information of a visual camera and IMU data of an inertial measurement unit; Get the external parameters of the system to which the visual camera and inertial measurement unit belong; The external parameters of the system are the relative position and posture between the inertial measurement unit and the visual camera.