Implementation method and system of a high-precision moving object tracking AI camera

Abstract

The present invention belongs to the technical field of artificial intelligence, and relates to an implementation method and system for a high-precision moving object tracking AI camera, including: obtaining the three-dimensional coordinates, visual image sequence, and motion state data of a target object; constructing a neural network dynamic weight model to adjust the data priorities of a millimeter-wave radar, a camera, and an inertial sensor; performing weighted calculation on the calibrated three-axis coordinates, and generating a continuous tracking path in combination with a velocity vector; constructing a motion pattern prediction model for predicting the future motion trajectory of the target object; screening an effective candidate path set; outputting a credibility score of the predicted path; if the credibility score does not reach the threshold, replacing and correcting; selecting the effective prediction path with the smallest deviation as the main solution, and retaining the sub-optimal prediction path as the backup solution. The present invention solves the problem that the traditional method lacks an effective correction process strategy when the tracking error is large.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and relates to an implementation method and system for a high-precision moving object tracking AI camera. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, moving object tracking technology is widely used in multiple fields such as security monitoring, intelligent transportation, sports training, and human-computer interaction. Traditional moving object tracking technologies face many challenges. Due to the complex and changeable environment, factors such as lighting conditions, occlusion, and motion blur will seriously affect the tracking accuracy and robustness. The data fusion problem of different sensors is also a key factor restricting the tracking performance.

[0003] When dealing with these problems, traditional methods often adopt a sensor data fusion strategy with fixed weights, and cannot dynamically adjust the priorities of each sensor according to real-time environmental changes, resulting in relatively large tracking errors. When traditional moving object tracking algorithms predict the future motion trajectory of the target object, they often lack sufficient context information and environmental perception capabilities, and it is difficult to accurately predict the object motion trajectory in complex scenarios.

[0004] Based on the above problems, traditional methods lack an effective correction process strategy when the tracking error is relatively large. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides an implementation method and system for a high-precision moving object tracking AI camera.

[0006] In the first aspect, the present invention provides an implementation method for a high-precision moving object tracking AI camera, adopting the following technical solutions:

[0007] An implementation method for a high-precision moving object tracking AI camera includes the following steps:

[0008] S1. Obtain the three-dimensional coordinates, visual image sequence, and motion state data of the target object, perform cross-modal spatio-temporal alignment calibration, and eliminate the differences in sensor coordinate systems;

[0009] S2. Construct a neural network dynamic weight model to adjust the data priorities of millimeter-wave radar, camera, and inertial sensors in real time;

[0010] S3. Perform weighted calculation on the calibrated three-axis coordinates, and generate a continuous tracking path in combination with the velocity vector;

[0011] S4. Construct a motion mode prediction model to obtain a motion mode prediction model for predicting the future motion trajectory of the target object;

[0012] S5. Predict and generate three candidate paths, and screen an effective set of candidate paths;

[0013] S6. Calculate the average deviation coefficient of the candidate paths, and combine the deviation threshold and the proportion of effective paths to output the confidence score of the predicted path;

[0014] S7. If the confidence score does not reach the threshold, correct it through iterative model training, weight reallocation, or sensor replacement;

[0015] S8. Select the effective predicted path with the smallest deviation in the three-axis coordinates as the main solution, and retain the sub-optimal predicted path as the backup solution.

[0016] In a further solution of the present invention, step S1 includes the following steps:

[0017] S11. Obtain the three-dimensional position coordinates and velocity information of the target object, capture the visual image sequence of the target object, and record the self-motion state data of the target object;

[0018] Specifically, utilize the principle that the millimeter-wave radar emits continuously varying frequency waves and receives the signals reflected by the object to obtain the three-dimensional coordinates and radial velocity of the target object; utilize the high-speed camera to capture the visual image sequence of the target object, and utilize the anti-shake inertial sensor to record the self-motion state data of the target object;

[0019] Specifically, when the anti-shake inertial sensor is installed on the target object, record the true lateral acceleration and displacement increment of the target object;

[0020] S12. Perform spatial alignment and calibration on the radar point cloud data, the pixel data of the visual image, and the displacement increment measured by the anti-shake inertial sensor;

[0021] Specifically, through a unified time signal, the time difference of the data of the millimeter-wave radar, the high-speed camera, and the anti-shake inertial sensor does not exceed 1 millisecond to ensure that the millimeter-wave radar, the high-speed camera, and the anti-shake inertial sensor collect data simultaneously;

[0022] Based on the three-dimensional coordinates of the radar point cloud, the two-dimensional pixel coordinates of the visual image, and the displacement increment coordinates measured by the anti-shake inertial sensor, establish a cross-modal association by extracting common feature points to try to solve the problem of the spatial pose difference between the radar, camera coordinate systems, and displacement increment coordinate systems; combine the key points of the target object, and convert the mapping relationship of the radar point cloud coordinates (X, Y, Z), the camera image coordinate system (U, V), and the displacement increment coordinate system (A, B) into a homogeneous transformation matrix, and optimize and eliminate the matching error through the least squares method.

[0023] In a further solution of the present invention, step S2 includes the following steps:

[0024] Integrate the light sensor and the meteorological sensing unit to obtain the current light intensity and visibility; calculate the moving speed of the target object based on the speed information output by the millimeter-wave radar and the self-motion state data of the anti-shake inertial sensor; then establish a dynamic weight allocation model to calculate the dynamic weight coefficients of the measurement data of the millimeter-wave radar, the measurement data of the high-speed camera, and the measurement data of the anti-shake inertial sensor. The dynamic weight allocation adjusts the priorities of different sensors in real time.

[0025] In a further solution of the present invention, step S3 includes the following steps:

[0026] S31. Extract the radar point cloud coordinates, the pixel coordinates of the image, and the displacement increment coordinates from the spatio-temporal alignment data, and perform three-axis coordinate weighted calculation according to the dynamic weight parameters measured in step S2.

[0027] Specifically, the displacement increment coordinates measured by the anti-shake inertial sensor ( ), the radar point cloud coordinates ( ), and the pixel coordinates of the image ( ); combined with the dynamic weight parameters: the weight coefficient of the anti-shake inertial sensor is , the weight coefficient of the millimeter-wave radar is , and the weight coefficient of the high-speed camera is , satisfying ;

[0028] Perform weighted calculations on the X, Y, and Z axes respectively, satisfying the following formula:

[0029]

[0030]

[0031]

[0032] S32. Convert the millimeter-wave radar into a velocity vector using the space coordinate system, and combine the weighted three-axis coordinate data and the velocity vector measured by the millimeter-wave radar ( ) to generate a tracking path including three-dimensional coordinates and a velocity vector.

[0033] In a further solution of the present invention, step S4 includes the following steps:

[0034] Based on the generated tracking path, intercept 100 consecutive frames of historical motion data as the basis for model training. Each frame of data includes the three-dimensional coordinates of the target object ( ), the velocity vector ( ), and the acceleration ( ), ensuring that the 100 frames of data cover at least a 10-second motion time window to capture the complete motion law of the target.

[0035] Combine the three-dimensional coordinates ( ), velocity vectors ( ), and accelerations ( ) in the historical valid motion data of a single frame into a nine-dimensional feature vector to form a historical training set of time series; train and generate a motion pattern prediction model using an LSTM network. Use the nine-dimensional feature vector of the first 70 frames as input and the three-dimensional coordinates of the next 30 frames as output. Train the double-layer LSTM network using the historical training set to obtain a motion pattern prediction model for predicting the future motion trajectory of the target object;

[0036] Collect the three-dimensional coordinates ( ), velocity vectors ( ), and accelerations ( ) in the current motion data of the target object and combine them into a nine-dimensional feature vector; input the nine-dimensional feature vector into the motion pattern prediction model to obtain the future motion trajectory of the target object.

[0037] A further solution of the present invention, step S5, includes the following steps:

[0038] Predict the future motion trajectory of the target object according to the motion pattern prediction model. Input the motion data of the target object in the most recent 3 seconds into the motion pattern prediction model to obtain the future 0.5-second motion trajectory of the target object; generate three candidate path plans by combining the future 0.5-second motion trajectory of the target object and the real-time environmental data;

[0039] Among them, the three candidate path plans correspond to three strategies of inertial continuation prediction, neighboring feature matching prediction, and scene map pre-judgment prediction;

[0040] Obtain the actual detection positions of the millimeter-wave radar and the vision sensor, and compare the predicted coordinates of the three candidate path plans with the actual detection coordinates point by point;

[0041] The X / Y plane deviation is in units of pixels of the image. Calculate the total X / Y plane deviation between the candidate path and the actual detection result, which satisfies the following formula,

[0042]

[0043] Among them, represents the total X / Y plane deviation between the candidate path and the actual detection result, represents the difference between the predicted coordinate and the actual detection coordinate of the candidate path in the X-axis direction, represents the difference between the predicted coordinate and the actual detection coordinate of the candidate path in the Y-axis direction;

[0044] The X / Y plane deviation is in pixels of the image, and the maximum allowable deviation threshold of the total X / Y plane deviation is 5 pixels; the Z-axis deviation is in meters, and the maximum allowable deviation threshold of the Z-axis deviation is 0.3 meters; determine whether the candidate path meets the threshold condition, mark the valid candidate paths that do not exceed the threshold condition, and construct a set of valid candidate path schemes.

[0045] A further solution of the present invention, step S6, includes the following steps:

[0046] Combined with the set of valid candidate path schemes, calculate the average deviation coefficient of the candidate predicted path scheme, which satisfies the following formula,

[0047]

[0048] where, represents the average deviation coefficient, represents the number of valid candidate paths, represents the th difference between the predicted coordinate and the actual detected coordinate of the candidate path in the X-axis direction, represents the th difference between the predicted coordinate and the actual detected coordinate of the path in the Y-axis direction, represents the th difference between the predicted coordinate and the actual detected coordinate of the path in the Z-axis, represents the th total deviation between the predicted coordinate and the actual detected coordinate of the path, represents the variance of the actual detected value;

[0049] Combined with the average deviation coefficient of the candidate predicted path scheme, calculate the credibility score of the candidate predicted path scheme, which satisfies the following formula,

[0050]

[0051] where, represents the credibility score, represents the number of valid candidate paths, represents the total number of candidate paths, represents the average deviation coefficient, represents the allowable deviation threshold preset by the system, which is usually the result obtained by multiple calculations in the laboratory.

[0052] A further solution of the present invention, step S7, includes the following steps:

[0053] Set the credibility score threshold of the candidate predicted path scheme Compare the credibility score of the candidate predicted path scheme with the credibility score threshold The size between;

[0054] If the confidence score , determine that the predicted trajectory is reliable;

[0055] If the confidence score , determine that the predicted trajectory is unreliable, execute the correction request, select a suitable compensation scheme, including replacing or upgrading a new sensor, reallocating dynamic weights, and iterative prediction;

[0056] Among them, for iterative prediction, re - collect a new historical training set, train and generate a corrected motion pattern prediction model;

[0057] Re - allocate dynamic weights, re - allocate the dynamic weight coefficients of the millimeter - wave radar, high - speed camera, and anti - shake inertial sensor, and generate a corrected predicted trajectory;

[0058] Replace or upgrade a new sensor, detect abnormal sensor data, replace or upgrade a new sensor, and regenerate a corrected predicted trajectory based on the new combination;

[0059] For the re - corrected predicted path, perform a confidence score again , if the confidence score , determine that the predicted path is reliable; if the confidence score , determine that the predicted trajectory is unreliable and execute the second correction request; if the correction fails three times in a row, with a maximum of 3 iterations triggered for a single correction, perform the operation of re - initializing the process.

[0060] A further solution of the present invention, step S8, includes the following steps:

[0061] Calculate the error between each solution in the set of valid candidate path solutions and the actual position, select the solution with the smallest error as the main path, and retain the sub - optimal solution as the backup path; calculate the absolute value of the three - axis coordinate deviation between each valid candidate path solution and the actual detection position,

[0062]

[0063] Among them, is the absolute value of the three - axis coordinate deviation, is the difference between the candidate path predicted coordinate and the actual detection coordinate in the X - axis direction, is the difference between the path predicted coordinate and the actual detection coordinate in the Y - axis direction, is the difference between the path predicted coordinate and the actual detection coordinate in the Z - axis;

[0064] Preset the error value of the maximum deviation threshold , compare the error value The size between the maximum deviation threshold ;

[0065] If When the error is the smallest, select the scheme with the smallest error as the main path and retain the sub-optimal scheme as the backup path; when the target object is in a complex motion state, retain two backup paths;

[0066] If When, perform the operation of re-initializing the process.

[0067] In a second aspect, the present invention provides an implementation system of a high-precision moving object tracking AI camera, adopting the following technical solutions:

[0068] A data acquisition module, which is used to obtain the three-dimensional coordinates, visual image sequence, and motion state data of the target object, and perform cross-modal spatio-temporal alignment and calibration;

[0069] A dynamic weight model construction module, which is used to construct a neural network dynamic weight model and adjust the data priorities of millimeter-wave radar, camera, and inertial sensor in real time;

[0070] A weighted calculation module, which is used to perform weighted calculation on the calibrated three-axis coordinates and generate a continuous tracking path in combination with the velocity vector;

[0071] A motion mode prediction model construction module, which is used to obtain a motion mode prediction model for predicting the future motion trajectory of the target object;

[0072] A candidate path generation and screening module, which is used to predict and generate three candidate paths and screen an effective candidate path set;

[0073] A credibility score calculation module, which is used to calculate the average deviation coefficient of the candidate path, and output the credibility score of the predicted path in combination with the deviation threshold and the proportion of the effective path;

[0074] A process correction module, if the credibility score does not reach the threshold, correct it by iterative model training, weight reallocation, or sensor replacement;

[0075] An effective prediction path selection module, which is used to select the effective prediction path with the smallest deviation of the three-axis coordinates as the main scheme and retain the sub-optimal prediction path as the backup scheme.

[0076] In summary, the present invention includes the following beneficial technical effects:

[0077] 1. Through the cross-modal data spatio-temporal alignment of millimeter-wave radar, high-speed camera, and anti-shake inertial sensor, combined with the adaptive weight distribution model of illumination, meteorology, and motion speed, the sensor data with the highest reliability is preferentially selected in rainy, foggy, low-light, or high-speed scenarios;

[0078] 2. The motion pattern prediction model constructed based on the LSTM network learns the target motion law through historical nine-dimensional feature vectors, and generates multi-strategy candidate paths by combining inertial continuation, adjacent feature matching, and scene map semantic prediction. Through the dynamic deviation threshold screening and credibility scoring mechanism, the system can automatically optimize the prediction path that best fits the actual trajectory;

[0079] 3. When the credibility score fails to meet the standard, the system triggers multiple real-time correction strategies, including iterative adjustment of the dynamic weights of sensors, incremental training and optimization of the LSTM model, and quick replacement in case of sensor failures; using the three-time iterative fault tolerance and initialization mechanism, it effectively avoids cumulative errors and ensures continuous tracking under extreme working conditions such as high-speed mutation or sensor anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. The drawings are used to provide a further understanding of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0081] Figure 1 The flowchart of the implementation method of a high-precision moving object tracking AI camera is disclosed.

[0082] Figure 2 The structural schematic diagram of the implementation system of a high-precision moving object tracking AI camera is disclosed. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0083] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0084] The following combines the attached Figure 1 - Figure 2 Make a preferred and detailed description of the present invention.

[0085] Referring to the attached Figure 1 , the present invention proposes an implementation method of a high-precision moving object tracking AI camera, including the following steps:

[0086] S1. Obtain the three-dimensional coordinates, visual image sequence, and motion state data of the target object, perform cross-modal spatio-temporal alignment and calibration, and eliminate the difference in the sensor coordinate system;

[0087] S2. Build a neural network dynamic weight model to adjust the data priorities of millimeter-wave radar, camera, and inertial sensor in real time;

[0088] S3. Perform weighted calculation on the calibrated three-axis coordinates and generate a continuous tracking path by combining with the velocity vector;

[0089] S4. Build a motion mode prediction model to obtain a motion mode prediction model for predicting the future motion trajectory of the target object;

[0090] S5. Anticipate and generate three candidate paths, and screen an effective set of candidate paths;

[0091] S6. Calculate the average deviation coefficient of the candidate paths, and combine with the deviation threshold and the proportion of effective paths to output the confidence score of the predicted path;

[0092] S7. If the confidence score does not reach the threshold, correct it through iterative model training, weight reallocation, or sensor replacement;

[0093] S8. Select the effective predicted path with the smallest three-axis coordinate deviation as the main solution, and retain the sub-optimal predicted path as the backup solution.

[0094] In one embodiment of the present invention, step S1 includes the following steps:

[0095] S11. Obtain the three-dimensional position coordinates and velocity information of the target object, capture the visual image sequence of the target object, and record the self-motion state data of the target object;

[0096] Specifically, use the principle that the millimeter-wave radar emits continuously changing frequency waves and receives the signals reflected by the object to obtain the three-dimensional coordinates and radial velocity of the target object; use a high-speed camera to capture the visual image sequence of the target object, and at the same time, the high-speed camera turns on the HDR mode to avoid overexposure or underexposure of the picture caused by strong light or shadow;

[0097] Use an anti-shake inertial sensor to record the self-motion state data of the target object;

[0098] Specifically, when the anti-shake inertial sensor is installed on the target object, record the real lateral acceleration and displacement increment of the target object.

[0099] S12. Perform spatial alignment calibration on the radar point cloud data, pixel data of the visual image, and displacement increment measured by the anti-shake inertial sensor;

[0100] Specifically, through a unified time signal, the time difference of the data of the millimeter-wave radar, high-speed camera, and anti-shake inertial sensor does not exceed 1 millisecond, ensuring that the millimeter-wave radar, high-speed camera, and anti-shake inertial sensor collect data simultaneously;

[0101] Based on the three-dimensional coordinates of radar point clouds, the two-dimensional pixel coordinates of visual images, and the displacement increment coordinates measured by a shake-proof inertial sensor, cross-modal associations are established by extracting common feature points to try to solve the problem of spatial pose differences among the radar, camera coordinate system, and displacement increment coordinate system; combined with the key points of the target object, the mapping relationships of the radar point cloud coordinates (X, Y, Z), the camera image coordinate system (U, V), and the displacement increment coordinate system (A, B) are converted into homogeneous transformation matrices, and the matching errors are eliminated by optimization using the least squares method; the radar point cloud coordinates, camera image coordinate system, and displacement increment coordinate system after spatial alignment and calibration are packaged into a unified format.

[0102] In one embodiment of the present invention, step S2 includes the following steps:

[0103] Integrate an illumination sensor and a meteorological sensing unit (e.g., a visibility sensor) to obtain the current illumination intensity and visibility; calculate the motion speed of the target object based on the speed information output by the millimeter-wave radar and the self-motion state data of the shake-proof inertial sensor; then establish a dynamic weight allocation model to calculate the dynamic weight coefficients of the measurement data of the millimeter-wave radar, the measurement data of the high-speed camera, and the measurement data of the shake-proof inertial sensor. The dynamic weight allocation significantly improves the performance of the multi-sensor system by adjusting the priorities of different sensors in real time.

[0104] In one embodiment of the present invention, establishing a dynamic weight allocation model includes the following steps:

[0105] Collect the historical illumination intensity, historical visibility, and historical motion speed of the target object around. Experts allocate and label the dynamic weight coefficients of the measurement data of the millimeter-wave radar, the measurement data of the high-speed camera, and the measurement data of the shake-proof inertial sensor based on the historical illumination intensity, historical visibility, and historical motion speed.

[0106] Collect the historical illumination intensity, historical visibility, and historical motion speed of the target object around to construct a historical data set; construct a dynamic weight allocation model based on a three-layer neural network model (input layer, hidden layer, and output layer). Using the illumination intensity, visibility, and motion speed of the target object around as inputs and the dynamic weight coefficients of the millimeter-wave radar, high-speed camera, and shake-proof inertial sensor as outputs, train the neural network model using the historical data set to obtain the dynamic weight allocation model;

[0107] Collect the current illumination intensity, current visibility, and current motion speed of the target object around to obtain an input data set;

[0108] Input the input data set into the dynamic weight allocation model to obtain the dynamic weight coefficient allocation values of the millimeter-wave radar, high-speed camera, and shake-proof inertial sensor.

[0109] Integrate the dynamic weight coefficient assignment values of each sensor (millimeter-wave radar, high-speed camera, anti-shake inertial sensor), and convert the normalized weight values output by the dynamic weight assignment model into the weight percentages of each sensor, which has a significant improvement compared to the fixed weight assignment scheme.

[0110] Exemplarily, for the environmental parameter associated optical sensor, when the light intensity > 1000 lux, the measurement data of the high-confidence millimeter-wave radar; for the environmental parameter associated meteorological sensing unit, when the visibility in rainy or foggy weather < 50 meters, the measurement data of the priority millimeter-wave radar;

[0111] When the moving speed of the target object > 30 m / s, rely on the measurement data of the anti-shake inertial sensor to compensate for the image blur or radar lag error caused by high-speed movement;

[0112] Among them, when the conditions of rain / fog and high-speed movement coexist, the priority of the environmental state is higher than that of the movement state (for example, when it is raining and the target speed > 30 m / s at the same time, the dynamic weight coefficient of the measurement data of the millimeter-wave radar is still assigned 80%);

[0113] Adopt the exponential decay method to avoid weight mutation (for example, when the light suddenly drops, the dynamic weight coefficient of the measurement data of the high-speed camera gradually drops from 70% to 30% within 10 frames), which can reduce the tracking error of the high-precision moving object tracking AI camera.

[0114] 1. Input of the dynamic weight assignment model: light intensity = 1500 lux, visibility in rainy or foggy weather = 2000 meters, moving speed of the target object = 35 m / s;

[0115] Output of the dynamic weight assignment model: weight of the measurement data of the millimeter-wave radar = 70% (high light ensures clear image), weight of the measurement data of the high-speed camera = 25%, weight of the measurement data of the anti-shake inertial sensor = 5%;

[0116] The high-speed camera dominates the positioning, and the millimeter-wave radar assists in correcting the movement offset.

[0117] 2. Input of the dynamic weight assignment model: light intensity = 300 lux, visibility in rainy or foggy weather = 30 meters, moving speed of the target object = 15 m / s;

[0118] Output of the dynamic weight assignment model: weight of the measurement data of the millimeter-wave radar = 80% (heavy fog weakens the reliability of vision), weight of the measurement data of the high-speed camera = 15%, weight of the measurement data of the anti-shake inertial sensor = 5%;

[0119] The millimeter-wave radar dominates the positioning, and the anti-shake inertial sensor eliminates the airflow disturbance error.

[0120] 3. Input of the dynamic weight distribution model: The movement speed of the target object = 0 m / s (suddenly accelerates to 40 m / s after standing still);

[0121] Output of the dynamic weight distribution model: The weight of the measurement data of the millimeter-wave radar = 15% (fog weakens the reliability of vision), the weight of the measurement data of the high-speed camera = 35%, and the weight of the measurement data of the anti-shake inertial sensor = 50%;

[0122] The anti-shake inertial sensor takes over emergently to avoid the failure of visual tracking caused by high-speed mutations.

[0123] In one embodiment of the present invention, step S3 includes the following steps:

[0124] S31. Extract the radar point cloud coordinates, pixel coordinates of the image, displacement increment coordinates from the spatio-temporal alignment data in step S1, and execute three-axis coordinate weighted calculation according to the dynamic weight parameters measured in step S2;

[0125] Specifically, the displacement increment coordinates measured by the anti-shake inertial sensor ( ), radar point cloud coordinates ( ), pixel coordinates of the image ( ); combined with the dynamic weight parameters: the weight coefficient of the anti-shake inertial sensor is , the weight coefficient of the millimeter-wave radar is , the weight coefficient of the high-speed camera is , satisfying ;

[0126] Execute weighted calculation for the X, Y, and Z axes respectively, satisfying the following formula:

[0127]

[0128]

[0129]

[0130] Exemplarily, in the rain and fog weather scenario ( = 80%, = 15%, = 5%);

[0131] Perform weighted calculation on the predetermined radar detection coordinates (10.235 m, 5.678 m, 2.314 m), visual detection coordinates (10.240 m, 5.665 m, 2.320 m), and displacement increment coordinates (10.238 m, 5.670 m, 2.315 m);

[0132] It is known that the final coordinates are (10.236m, 5.675m, 2.316m).

[0133] S32. Convert the millimeter-wave radar into a velocity vector using a spatial coordinate system, combine the weighted three-axis coordinate data with the velocity vector measured by the millimeter-wave radar ( ), generate a tracking path including three-dimensional coordinates and a velocity vector; update the weighted result at a frequency of 200HZ to form a frame-by-frame continuous path data stream; encapsulate the three-dimensional coordinates and the velocity vector into a JSON structure, including the weight parameters and timestamps of the sensor.

[0134] In one embodiment of the present invention, step S4 includes the following steps:

[0135] Based on the tracking path generated in step S3, intercept 100 consecutive frames of historical motion data as the basis for model training. Each frame of data includes the three-dimensional coordinates of the target object ( ), velocity vector ( ), and acceleration ( ), ensuring that the 100 frames of data cover at least a 10-second motion time window to capture the complete motion law of the target;

[0136] Combine the three-dimensional coordinates ( ), velocity vector ( ), and acceleration ( in the single-frame historical valid motion data into a nine-dimensional feature vector to form a historical training set of time series; use an LSTM network to train and generate a motion pattern prediction model. Use the nine-dimensional feature vector of the previous 70 frames as the input and the three-dimensional coordinates of the next 30 frames as the output. Use the historical training set to train the double-layer LSTM network to obtain a motion pattern prediction model for predicting the future motion trajectory of the target object.

[0137] Collect the three-dimensional coordinates ( ), velocity vector ( ), and acceleration ( in the current motion data of the target object and combine them into a nine-dimensional feature vector; input the nine-dimensional feature vector into the motion pattern prediction model to obtain the future motion trajectory of the target object, supporting the motion pattern learning of different types of target objects.

[0138] In one embodiment of the present invention, step S5 includes the following steps:

[0139] According to the motion pattern prediction model, predict the future motion trajectory of the target object. Input the recent 3-second motion data (60-frame high-speed camera data) of the target object into the motion pattern prediction model to obtain the future 0.5-second motion trajectory of the target object; combine the future 0.5-second motion trajectory of the target object with the real-time environment data to generate three candidate path plans;

[0140] Among them, the three candidate path solutions correspond to three strategies of inertial continuation prediction, adjacent feature matching prediction, and scene map pre-judgment prediction;

[0141] The inertial continuation prediction path calculates the future motion trajectory of the target object based on the motion inertia (including speed and acceleration) of the target object in the past 3 seconds. The target object maintains its current motion state unchanged, and the effective path extends along the straight path direction; relying on physical motion inertia, it is applicable to scenarios without environmental interference;

[0142] The adjacent feature matching prediction path searches for other target objects with similar features (shape, speed, size) in the adjacent area when the target object is partially occluded or there is environmental interference, and selects the motion trajectory of the most similar target object with the highest matching degree as the candidate prediction path; relying on feature matching, it is applicable to scenarios of target occlusion or group movement;

[0143] The scene map pre-judgment prediction path combines a high-precision scene map (such as lane lines, intersections, obstacle distribution) to predict the reasonable path of the target object. The path planning generates candidate routes that meet the scene constraints, and dynamically adjusts the trajectory curvature to ensure compliance with the maximum steering angle of the target object; relying on scene semantic understanding, it is applicable to structured road or fixed route scenarios.

[0144] The actual detection positions of the millimeter-wave radar and the vision sensor will be obtained, and the predicted coordinates of the three candidate path solutions will be compared with the actual detection coordinates point by point;

[0145] The X / Y plane deviation is in units of pixels of the image. Calculate the total X / Y plane deviation between the candidate path and the actual detection result, which satisfies the following formula,

[0146]

[0147] Among them, represents the total X / Y plane deviation between the candidate path and the actual detection result, represents the difference between the predicted coordinates and the actual detection coordinates of the candidate path in the X-axis direction, represents the difference between the predicted coordinates and the actual detection coordinates of the candidate path in the Y-axis direction.

[0148] The X / Y plane deviation is in units of pixels of the image, and the maximum allowable deviation threshold of the total X / Y plane deviation is 5 pixels; the Z-axis (depth direction) deviation is in units of meters, and the maximum allowable deviation threshold of the Z-axis deviation is 0.3 meters; judge whether the candidate path meets the threshold conditions, mark the valid candidate paths that do not exceed the threshold conditions, and construct a set of effective candidate path solutions.

[0149] In one embodiment of the present invention, step S6 includes the following steps:

[0150] Combine with the set of effective candidate path schemes, calculate the average deviation coefficient of the candidate prediction path scheme, which satisfies the following formula:

[0151]

[0152] Wherein: represents the average deviation coefficient; represents the number of effective candidate paths; represents the difference between the predicted coordinates and the actual detected coordinates of the th candidate path in the X-axis direction; represents the difference between the predicted coordinates and the actual detected coordinates of the th path in the Y-axis direction; represents the difference between the predicted coordinates and the actual detected coordinates of the th path in the Z-axis; represents the total deviation between the predicted coordinates and the actual detected coordinates of the th path; represents the variance of the actual detected value;

[0153] Combine with the average deviation coefficient of the candidate prediction path scheme, calculate the credibility score of the candidate prediction path scheme, which satisfies the following formula:

[0154]

[0155] Wherein: represents the credibility score; represents the number of effective candidate paths; represents the total number of candidate paths; represents the average deviation coefficient; represents the allowable deviation threshold preset by the system, which is usually the result obtained by multiple calculations in the laboratory.

[0156] Exemplarily, set three paths: represents an effective path with a total deviation of 0.07m; represents an invalid path with a total deviation of 0.4m; represents an effective path with a total deviation of 0.15m; the number of effective paths = 2, the total number of paths = 3, calculate the average deviation coefficient of the candidate prediction path scheme = 0.11;

[0157] Set the allowable deviation threshold preset by the system to = 0.3m, calculate the credibility score of the candidate prediction path scheme = 76.7%.

[0158] In one embodiment of the present invention, step S7 includes the following steps:

[0159] Set the confidence score threshold for the candidate prediction path scheme Compare the confidence scores of the candidate prediction path schemes with the confidence score threshold to determine their magnitudes;

[0160] If the confidence score , determine that the prediction trajectory is reliable;

[0161] If the confidence score , determine that the prediction trajectory is unreliable, execute the correction request, and select a suitable compensation scheme, including replacing or upgrading to a new sensor, reallocating dynamic weights, and iterative prediction;

[0162] Among them, for iterative prediction, re - collect a new historical training set, train and generate a corrected motion pattern prediction model;

[0163] Re - allocate dynamic weights, re - allocate the dynamic weight coefficients of the millimeter - wave radar, high - speed camera, and anti - shake inertial sensor, and generate a corrected prediction trajectory;

[0164] Replace or upgrade to a new sensor. When sensor data anomalies are detected, replace or upgrade to a new sensor, and regenerate a corrected prediction trajectory based on the new combination;

[0165] For the re - corrected prediction path, perform a confidence score again , if the confidence score , determine that the prediction path is reliable; if the confidence score , determine that the prediction trajectory is unreliable and execute the second correction request; if the correction fails three times in a row, with a maximum of 3 iterations triggered for a single correction, perform the operation of re - initializing the process.

[0166] In one embodiment of the present invention, step S8 includes the following steps:

[0167] Calculate the error between each scheme in the set of valid candidate path schemes and the actual position, select the scheme with the smallest error as the main path, and retain the sub - optimal scheme as the backup path; calculate the absolute value of the three - axis coordinate deviation between each valid candidate path scheme and the actual detection position,

[0168]

[0169] wherein, is the absolute value of the three - axis coordinate deviation, is the difference between the candidate path prediction coordinate and the actual detection coordinate in the X - axis direction, is the difference between the path prediction coordinate and the actual detection coordinate in the Y - axis direction, It is the difference between the path prediction coordinate and the actual detection coordinate on the Z-axis;

[0170] Predetermined error value The maximum deviation threshold , compare the error value with the maximum deviation threshold to determine the magnitude relationship;

[0171] If is the case, select the solution with the smallest error as the main path, and retain the sub-optimal solution as the backup path; when the target object is in a complex motion state, retain two backup paths;

[0172] If is the case, perform the operation of re-initializing the process.

[0173] Refer to the appendix Figure 2 , the present invention also proposes an implementation system of a high-precision moving object tracking AI camera, including the following modules:

[0174] Data acquisition module, used to obtain the three-dimensional coordinates, visual image sequence, and motion state data of the target object, and perform cross-modal spatio-temporal alignment and calibration;

[0175] Dynamic weight model construction module, used to construct a neural network dynamic weight model and adjust the data priorities of millimeter-wave radar, camera, and inertial sensor in real time;

[0176] Weighted calculation module, used to perform weighted calculation on the calibrated three-axis coordinates and generate a continuous tracking path in combination with the velocity vector;

[0177] Motion mode prediction model construction module, used to obtain a motion mode prediction model for predicting the future motion trajectory of the target object;

[0178] Candidate path generation and screening module, used to predict and generate three candidate paths and screen an effective set of candidate paths;

[0179] Reliability score calculation module, used to calculate the average deviation coefficient of the candidate path, and output the reliability score of the predicted path in combination with the deviation threshold and the proportion of effective paths;

[0180] Process correction module, if the reliability score does not reach the threshold, correct it through iterative model training, weight reallocation, or sensor replacement;

[0181] Effective predicted path selection module, used to select the effective predicted path with the smallest deviation of the three-axis coordinates as the main solution, and retain the sub-optimal predicted path as the backup solution.

[0182] Each of the described modules can be implemented in whole or in part by software, hardware, or a combination thereof, supporting a hardware form embedded in or independent of a processor in a computer device, and also supporting a software form stored in a memory in the computer device to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0183] It should be noted that the user information (including but not limited to user device information and personal information, etc.) and data (including but not limited to data for analysis, stored data, and displayed data, etc.) involved in the present invention are all information and data that have been authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data require relevant legal standards.

[0184] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An implementation method of a high-precision moving object tracking AI camera, characterized in that, The following steps are involved: S1, obtain the three-dimensional coordinates, visual image sequence, and motion state data of the target object, perform cross-modal spatiotemporal alignment calibration, and eliminate the difference in sensor coordinate systems. The sensors include millimeter wave radar, high-speed camera, and anti-shake inertial sensor; S2. Build a neural network dynamic weight model to adjust the data priority of millimeter wave radar, high-speed camera and anti-shake inertial sensor in real time; S3, performing weighted calculation on the calibrated three-axis coordinates and generating a continuous tracking path in combination with the velocity vector; S4, constructing a motion pattern prediction model to obtain a motion pattern prediction model for predicting the future motion trajectory of the target object; S5, predict and generate three candidate paths, and screen a valid set of candidate paths; The three candidate paths correspond to the three strategies of inertial continuation prediction, neighboring feature matching prediction, and scene map pre-judgment prediction; Inertial continuation prediction: the future motion trajectory of the target object is estimated based on the target object's motion inertia in the past 3 seconds. It is suitable for scenarios without environmental interference. Neighboring feature matching prediction: when the target object is partially blocked or interfered by the environment, search for other target objects with similar features in the neighboring area, and select the motion trajectory of the similar target object with the highest matching degree as the candidate prediction path. This is suitable for target occlusion or group motion scenes. Scene map prediction: Combined with high-precision scene maps, the reasonable path of the target object is predicted. Path planning generates candidate routes that meet the scene constraints and dynamically adjusts the trajectory curvature to ensure the maximum steering angle that meets the target object. It is suitable for structured roads or fixed route scenarios. The actual detection positions of the millimeter-wave radar and high-speed camera will be obtained, and the predicted coordinates of the three candidate path solutions will be compared with the actual detection coordinates point by point; The X / Y plane deviation is calculated in pixels of the image. The total X / Y plane deviation between the candidate path and the actual detection result satisfies the following formula: Among them, represents the total X / Y plane deviation between the candidate path and the actual detection result, represents the difference between the predicted coordinates of the candidate path and the actual detection coordinates in the X-axis direction, represents the difference between the predicted coordinates of the candidate path and the actual detection coordinates in the Y-axis direction; The X / Y plane deviation is in pixels of the image, and the maximum allowable deviation threshold of the total X / Y plane deviation is 5 pixels; The Z-axis deviation is in meters, and the maximum allowable deviation threshold of the Z-axis deviation is 0.3 meters; determine whether the candidate path meets the threshold condition, mark the valid candidate path that does not exceed the threshold condition, and build a valid candidate path solution set; S6. Calculate the average deviation coefficient of the candidate paths, combine the deviation threshold and the effective path ratio, and output the credibility score of the predicted path; S7. If the credibility score does not reach the threshold, correction is performed through iterative model training, weight redistribution, or sensor replacement; S8. Select the effective predicted path with the smallest three-axis coordinate deviation as the main solution, and retain the suboptimal predicted path as a backup solution.

2. The implementation method of a high-precision moving object tracking AI camera according to claim 1, characterized in that, Step S1 includes the following steps: S11, obtaining the three-dimensional position coordinates and speed information of the target object, capturing the visual image sequence of the target object, and recording the self-motion state data of the target object; Specifically, the millimeter-wave radar emits continuously changing frequency waves and receives the signal reflected by the object to obtain the three-dimensional coordinates and radial velocity of the target object; a high-speed camera is used to capture the visual image sequence of the target object, and an anti-shake inertial sensor is used to record the target object's own motion state data; When the anti-shake inertial sensor is installed on the target object, it records the true lateral acceleration and displacement increment of the target object; S12. Perform spatial alignment calibration for the radar point cloud data, pixel data of the visual image, and displacement increment measured by the anti-shake inertial sensor; Specifically, through a unified time signal, the time difference between the data of the millimeter-wave radar, high-speed camera, and anti-shake inertial sensor does not exceed 1 millisecond, ensuring that the millimeter-wave radar, high-speed camera, and anti-shake inertial sensor collect data simultaneously; Based on the three-dimensional coordinates of the radar point cloud, the camera image coordinate system of the visual image, and the displacement increment coordinate measured by the anti-shake inertial sensor, establish a cross-modal association by extracting common feature points; combine the key points of the target object, and convert the mapping relationship between the radar point cloud coordinates (X, Y, Z), camera image coordinate system (U, V), and displacement increment coordinate system (A, B) into a homogeneous transformation matrix, and optimize and eliminate the matching error by the least squares method.

3. The implementation method of a high-precision moving object tracking AI camera according to claim 2, characterized in that, Step S2 includes the following steps: Integrate the light sensor and meteorological sensing unit to obtain the current light intensity and visibility; calculate the moving speed of the target object based on the speed information output by the millimeter-wave radar and the self-motion state data of the anti-shake inertial sensor; then establish a dynamic weight allocation model to calculate the dynamic weight coefficients of the measurement data of the millimeter-wave radar, the measurement data of the high-speed camera, and the measurement data of the anti-shake inertial sensor, and the dynamic weight allocation adjusts the priorities of different sensors in real time.

4. The implementation method of a high-precision moving object tracking AI camera according to claim 3, characterized in that, Step S3 includes the following steps: S31. Extract the radar point cloud coordinates, pixel coordinates of the image, and displacement increment coordinates from the spatio-temporal alignment data, and perform weighted calculation of the three-axis coordinates according to the dynamic weight parameters measured in step S2; Specifically, the displacement increment coordinates measured by the anti-shake inertial sensor ( ), the radar point cloud coordinates ( ), and the pixel coordinates of the image ( ); combined with dynamic weight parameters: the weight coefficient of the anti-shake inertial sensor is , the weight coefficient of the millimeter-wave radar is , and the weight coefficient of the high-speed camera is , satisfying ; Perform weighted calculation on the X, Y, and Z axes respectively, satisfying the following formula: S32. The millimeter-wave radar generates a velocity vector using a spatial coordinate system ( ), combines it with the weighted three-axis coordinate data, and generates a tracking path including three-dimensional coordinates and a velocity vector.

5. The implementation method of a high-precision moving object tracking AI camera according to claim 4, characterized in that, Step S4 includes the following steps: Based on the generated tracking path, 100 consecutive frames of historical motion data are intercepted as the basis for model training. Each frame of data contains the three-dimensional coordinates of the target object ( ), velocity vector ( ), and acceleration ( ), ensuring that the 100-frame data covers a motion time window of at least 10 seconds to capture the complete motion pattern of the target; Combine the three-dimensional coordinates ( ), velocity vectors ( ), and accelerations ( ) in the historical valid motion data of a single frame into a nine-dimensional feature vector to form a historical training set of time series; use an LSTM network to train and generate a motion pattern prediction model, use the nine-dimensional feature vector of the previous 70 frames as input, and the three-dimensional coordinates of the next 30 frames as output, and use the historical training set to train the motion pattern prediction model to obtain a motion pattern prediction model for predicting the future motion trajectory of the target object; Collect the three-dimensional coordinates ( ), velocity vector ( ), and acceleration ( ) in the current motion data of the target object to form a nine-dimensional feature vector; input the nine-dimensional feature vector into the motion pattern prediction model to obtain the future motion trajectory of the target object.

6. The implementation method of a high-precision moving object tracking AI camera according to claim 5, characterized in that, Step S6 includes the following steps: Combine the effective candidate path plan set, calculate the average deviation coefficient of the candidate prediction path plan, satisfying the following formula, Among them, represents the average deviation coefficient, represents the number of effective candidate paths, represents the difference between the predicted coordinate and the actual detected coordinate of the th candidate path in the X-axis direction, represents the th path difference between the predicted coordinate and the actual detected coordinate in the Y-axis direction, represents the th path difference between the predicted coordinate and the actual detected coordinate in the Z-axis direction, represents the total deviation between the predicted coordinate and the actual detected coordinate of the th path; represents the variance of the actual detected value; Combine the average deviation coefficient of the candidate prediction path plan, calculate the credibility score of the candidate prediction path plan, satisfying the following formula, Among them, represents the confidence score, represents the number of valid candidate paths, represents the total number of candidate paths, represents the average deviation coefficient, represents the allowable deviation threshold preset by the system.

7. The implementation method of a high-precision moving object tracking AI camera according to claim 6, characterized in that, Step S7 includes the following steps: Set the confidence score threshold for the candidate prediction path plan , compare the confidence scores of the candidate prediction path plans with the confidence score threshold to determine their magnitudes; If the confidence score , determine that the predicted trajectory is reliable; If it is a credibility score , determine that the predicted trajectory is unreliable, execute the correction request, and select a compensation plan, including replacing or upgrading new sensors, reallocating dynamic weights, and iterative prediction; Among them, iterative prediction refers to re-collecting a new historical training set, training and generating a corrected motion mode prediction model; Re-distributing the dynamic weights refers to re-distributing the dynamic weight coefficients of the millimeter-wave radar, high-speed camera, and anti-shake inertial sensor to generate a corrected prediction trajectory; Replacing or upgrading a new sensor refers to detecting abnormal sensor data, replacing or upgrading a new sensor, and re-generating a corrected prediction trajectory based on the new combination; The re - corrected prediction path refers to re - performing the credibility scoring , if the credibility score , it is determined that the prediction path is reliable; if the credibility score , it is determined that the prediction trajectory is unreliable and the second correction request is executed; if the correction fails three times in a row, each single correction can trigger up to 3 iterations at most, and the operation of re - initializing the process is executed.

8. The implementation method of a high-precision moving object tracking AI camera according to claim 5, characterized in that, Step S8 includes the following steps: Calculate the error between each plan in the effective candidate path plan set and the actual position, select the plan with the smallest error as the main path, and retain the sub-optimal plan as the backup path; calculate the absolute value of the three-axis coordinate deviation between each effective candidate path plan and the actual detection position, Among them, is the absolute value of the three-axis coordinate deviation, is the difference between the candidate path prediction coordinate and the actual detection coordinate in the X-axis direction, is the difference between the path prediction coordinate and the actual detection coordinate in the Y-axis direction, is the difference between the path prediction coordinate and the actual detection coordinate in the Z-axis; Predetermined error value The maximum deviation threshold , compare the error value with the maximum deviation threshold to determine their magnitudes; If When, select the solution with the smallest error as the main path and retain the sub-optimal solution as the alternate path; when the target object is in a complex motion state, retain two alternate paths; If perform the operation of re-initialization process 9. An implementation system of a high-precision moving object tracking AI camera, which is used to implement the implementation method of a high-precision moving object tracking AI camera described in claim 1, and is characterized in that, Includes the following modules: The data acquisition module is used to obtain the three-dimensional coordinates, visual image sequence, and motion state data of the target object, perform cross-modal spatio-temporal alignment and calibration, and eliminate the differences in sensor coordinate systems. The sensors include millimeter-wave radar, high-speed cameras, and anti-shake inertial sensors; The dynamic weight model construction module is used to construct a neural network dynamic weight model and adjust the data priorities of millimeter-wave radar, high-speed cameras, and anti-shake inertial sensors in real time; The weighted calculation module is used to perform weighted calculations on the calibrated three-axis coordinates and generate a continuous tracking path in combination with the velocity vector; The motion mode prediction model construction module is used to obtain a motion mode prediction model for predicting the future motion trajectory of the target object; The candidate path generation and screening module is used to predict and generate three candidate paths and screen an effective set of candidate paths; The credibility score calculation module is used to calculate the average deviation coefficient of the candidate paths, and combine the deviation threshold and the proportion of effective paths to output the credibility score of the predicted path; The process correction module, if the credibility score does not reach the threshold, corrects it by iterative model training, weight reallocation, or sensor replacement; The effective prediction path selection module is used to select the effective prediction path with the smallest deviation of the three-axis coordinates as the main solution and retain the sub-optimal prediction path as the backup solution.

Images