Target tracking method and device for heavy-load equipment, equipment and medium

Abstract

The invention provides a target tracking method and device for heavy-load equipment, equipment and a medium. The target tracking method comprises the following steps: acquiring surrounding environment data of the heavy-load equipment; tracking and identifying each target object in the surrounding environment data, and detecting a tracking and identifying result: when it is detected that the target object is shielded, performing prediction processing according to state data of the target object before the target object is in a shielded state to obtain prediction state data of the target object in the shielded state; and when it is detected that a new target object appears in the surrounding environment data, tracking calibration is performed on the new target object according to the state data of the new target object and a historical tracking identification result. Through the target tracking method and device for the heavy-load equipment, the equipment and the medium provided by the invention, a new target object can be effectively matched and associated with a target object in a previous shielding state.

Description

Technical Field

[0001] The present invention relates to the industrial field, and in particular to a target tracking method, device, equipment and medium for heavy-load equipment. Background Art

[0002] In the working area of ​​large open-air heavy-load equipment, due to its large size, frequent obstructions from the surrounding environment and numerous blind spots during movement, moving target objects such as pedestrians or vehicles around it are often obscured by obstacles and cannot be directly observed, forming monitoring blind spots.

[0003] When a target object is obscured, the heavy-duty equipment not only interrupts the continuous calibration of its motion state, but also, after the obstruction ends, it becomes difficult to effectively match and associate the newly appearing target object with the previously obscured target object. This makes it impossible to accurately restore the target object's identity and motion trajectory, significantly increasing the risk of misjudgment and safety warning failure. Therefore, there is room for improvement. Summary of the Invention

[0004] The present invention provides a target tracking method, device, equipment and medium for heavy-duty equipment to solve the technical problem that a newly appeared target object cannot be effectively matched and associated with a target object that was previously in an occluded state.

[0005] The present invention provides a target tracking method for heavy-load equipment, comprising:

[0006] Collect environmental data around heavy-load equipment;

[0007] Track and identify each target object in the surrounding environment data, and detect the tracking and identification results:

[0008] When it is detected that the target object is blocked, prediction processing is performed based on the state data of the target object before the blocking state to obtain the predicted state data of the target object in the blocking state;

[0009] When a new target object is detected in the surrounding environment data, the new target object is tracked and calibrated according to its state data and historical tracking and recognition results.

[0010] In one embodiment of the present invention, the step of performing prediction processing based on the state data of the target object before being in the occlusion state to obtain the predicted state data of the target object in the occlusion state includes:

[0011] Based on a preset motion model, predicted state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the motion model represents the functional relationship between the predicted state data and the state data.

[0012] In one embodiment of the present invention, the motion model includes a uniform velocity sub-model, a uniform acceleration / deceleration sub-model, and a turning sub-model. The step of calculating predicted state data of the target object in the occlusion state based on the state data of the target object before the occlusion state based on the preset motion model; and characterizing the functional relationship between the predicted state data and the state data using the motion model includes:

[0013] Based on the uniform velocity sub-model, the predicted uniform velocity state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the uniform velocity sub-model represents the functional relationship between the predicted uniform velocity state data and the state data when the target object is in uniform motion;

[0014] Based on the uniform acceleration / deceleration sub-model, the predicted uniform acceleration / deceleration state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the uniform acceleration / deceleration sub-model represents the functional relationship between the predicted uniform acceleration / deceleration state data and the state data when the target object is in uniform acceleration / deceleration motion;

[0015] Based on the turning sub-model, predicted turning state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the turning sub-model represents the functional relationship between the predicted turning state data and the state data when the target object is in the turning motion;

[0016] The predicted state data is calculated based on the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data.

[0017] In one embodiment of the present invention, the step of calculating the predicted state data based on the predicted constant speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data includes:

[0018] Calculating the uniform speed credible data of the uniform speed sub-model, the uniform acceleration and deceleration credible data of the uniform acceleration and deceleration sub-model, and the turning credible data of the turning sub-model based on the state data of the target object before being in the occlusion state;

[0019] The predicted state data is calculated based on the predicted uniform speed state data, the uniform speed credible data, the predicted uniform acceleration / deceleration state data, the uniform acceleration / deceleration credible data, the predicted turning state data, and the turning credible data.

[0020] In one embodiment of the present invention, after the step of calculating the predicted state data based on the predicted constant speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data, the method further includes:

[0021] Calculating an error covariance matrix of the target object before the occlusion state according to the state data of the target object before the occlusion state;

[0022] Calculating the uniform speed error covariance matrix of the uniform speed sub-model, the uniform acceleration and deceleration error covariance matrix of the uniform acceleration and deceleration sub-model, and the turning error covariance matrix of the turning sub-model according to the error covariance matrix;

[0023] Calculating a fusion error covariance matrix based on the uniform speed credible data, the uniform acceleration / deceleration credible data, the turning credible data, the uniform speed error covariance matrix, the uniform acceleration / deceleration error covariance matrix, the turning error covariance matrix, the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, the predicted turning state data, and the predicted state data;

[0024] Calculating uncertainty data according to the fusion error covariance matrix;

[0025] Within a preset time period, the uncertainty data is judged against the corresponding threshold:

[0026] When the uncertainty data is less than a corresponding threshold, performing prediction processing on the corresponding target object in the occluded state;

[0027] Otherwise, no prediction processing is performed on the corresponding target object in the occluded state.

[0028] In one embodiment of the present invention, when a new target object is detected in the surrounding environment data, the step of tracking and calibrating the new target object according to its state data and historical tracking and recognition results includes:

[0029] When a new target object is detected in the surrounding environment data, obtaining state data of the new target object;

[0030] Matching the state data of the new target object with the predicted state data of each target object in the occluded state in the historical tracking and recognition results to obtain corresponding feature matching data and position matching data;

[0031] Calculating association probability data based on the feature matching data and the corresponding position matching data;

[0032] According to the comparison result of the association probability data and the preset matching threshold, the new target object is tracked and calibrated.

[0033] In one embodiment of the present invention, the step of tracking and calibrating the new target object based on the comparison result of the association probability data with a preset matching threshold includes:

[0034] Determine whether the association probability data matches a preset matching threshold:

[0035] When the association probability data is greater than a preset matching threshold, the new target object is associated with the corresponding target object in the occluded state in the historical tracking and recognition results, and the new target object is tracked and calibrated;

[0036] Otherwise, tracking and calibration are performed on the new target object.

[0037] The present invention also provides a target tracking device for heavy-load equipment, comprising:

[0038] Data acquisition module, used to collect environmental data around heavy-duty equipment;

[0039] The target tracking module is used to track and identify each target object in the surrounding environment data and detect the tracking and identification results:

[0040] A state prediction module is used to perform prediction processing based on the state data of the target object before the occlusion state when detecting that the target object is occluded, so as to obtain the predicted state data of the target object in the occlusion state;

[0041] The target matching module is used to track and calibrate the new target object based on its status data and historical tracking and recognition results when a new target object is detected in the surrounding environment data.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the target tracking method for heavy-loaded equipment are implemented.

[0043] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the target tracking method for heavy-load equipment are implemented.

[0044] The beneficial effects of the present invention are as follows: when the target object disappears due to occlusion, the state data before the occlusion is actively used to predict the motion trajectory; when the occlusion is released, by comparing the new target object with the historical tracking and recognition results, the new target object can be effectively matched and associated with the target object that was previously in the occluded state, so as to accurately restore the identity of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description in

[0046] In the drawings:

[0047] Figure 1 A flow chart of a target tracking method of a heavy load device according to an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a target tracking device of a heavy load device according to an embodiment of the present application;

[0049] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present application.

[0050] Reference signs are as follows: 100, data acquisition module; 200, target tracking module; 300, state prediction module; 400, target matching module; 10, electronic device; 11, memory; 12, processor. DETAILED DESCRIPTION

[0051] The above embodiments of the present application are merely intended for describing and illustrating the present application. The skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present specification. The present application can be implemented or applied in other different embodiments, and the details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] It should be noted that the diagrams provided in the following embodiments merely schematically illustrate the basic concept of the present application, and the drawings in the embodiments only show the components related to the present application rather than the number, shape and size of the components in actual implementation. The shape, number and ratio of the components in actual implementation can be arbitrarily changed, and the layout pattern of the components can be more complex.

[0053] In the following description, a large number of details are discussed to provide a more thorough explanation of embodiments consistent with the present application, however, it is apparent to one skilled in the art that the embodiments of the present application can be implemented without these specific details, and in other embodiments, the well-known structures and devices are shown in the form of block diagrams rather than in the form of details to avoid making the embodiments of the present application difficult to understand.

[0054] Reference will now be made to Figure 1The present invention discloses a target tracking method for heavy-duty equipment. The target tracking method can track a target object in surrounding environment information. When a target object is obscured, the target object's state data can be predicted. When a new target object appears, the newly appeared target object can be matched and associated with the previously obscured target object. The target tracking method may include the following steps: Step S10: Collecting environmental data surrounding the heavy-duty equipment.

[0055] In some embodiments, multiple types of sensors (including cameras, lidars, millimeter-wave radars, etc.) installed on heavy-load equipment can collect surrounding environment data. The surrounding environment data may include image data, point cloud data, etc. In the process of collecting surrounding environment data, the image data and point cloud data are aligned through high-precision clock synchronization and hardware timestamps to achieve temporal consistency of the image data and point cloud data.

[0056] In some embodiments, the target tracking method may further include the following steps: Step S20 , tracking and identifying each target object in the surrounding environment data, and detecting the tracking and identification results.

[0057] In some embodiments, a target tracking algorithm can be used to identify target objects in the surrounding environment data using image data. Specifically, the target tracking algorithm can include using a preset rule-based algorithm (e.g., screening based on geometric shape or size thresholds) or a traditional computer vision target tracking algorithm (e.g., feature extraction and classification based methods) to track specific categories of target objects, such as people, vehicles, and other equipment.

[0058] In some embodiments, a unique identifier can be assigned to the target object identified by the image data to continuously track and distinguish different target objects in subsequent frames, and the status data of each identified target object can also be extracted. Multi-dimensional information may include feature data, status data, etc. Feature data refers to visual features that describe the attributes of the target object (for example, color and texture information based on the appearance of the target), geometric features (for example, size, outer contour, shape information based on the point cloud), etc. The status data refers to the preliminary velocity data, acceleration data, angular velocity data, and position data formed based on the change information of its position in the current frame image data and the position in the previous frame image data. Position data refers to the position information of the target object in the current three-dimensional environment model coordinate system, and the position data can be obtained through point cloud data.

[0059] In some embodiments, for target objects that have been assigned displacement identifiers and are being tracked (including targets identified and continuously tracked in the previous frame or earlier), a filtering algorithm such as a Kalman filter can be applied to update their status. The Kalman filter algorithm can use the target object's position information and motion data (velocity data, acceleration data, angular velocity data) from the previous frame and the newly measured position information in the current frame to perform fusion and prediction.

[0060] In some embodiments, the target tracking method may further include the following steps: Step S30: When occlusion of the target object is detected, predictive processing is performed based on the state data of the target object before the occlusion to obtain predicted state data of the target object within the occlusion state. The predicted state data of the target object within the occlusion state may be calculated based on the state data of the target object before the occlusion state based on a preset motion model; the motion model represents a functional relationship between the predicted state data and the state data. The motion model includes a uniform speed sub-model, a uniform acceleration / deceleration sub-model, and a turning sub-model.

[0061] In some embodiments, when determining whether a target object is obscured, for each target object being tracked, the image data of the target object obtained after tracking and updating the current frame can be compared and analyzed with the image data of the target object recorded in the previous frame. Based on the degree of difference between the current frame and the previous frame image data (for example: changes in feature similarity, the degree of decrease in feature point matching rate, fluctuations in key appearance attributes, tracked bounding box size, changes in point cloud point count, etc.), an occlusion probability is calculated to characterize the degree to which the target object may be obscured. The occlusion probability is used to quantify the possibility of a negative change in the visibility of the target object from the previous frame to the current frame. The calculation formula for the occlusion probability can be expressed as: N obs / N exp ; Among them, N obs Represents the image data of the current frame; N exp Represents the image data of the previous frame.

[0062] In some embodiments, the state of each target object is then determined based on the occlusion probability and a preset threshold range. Specifically, the calculated occlusion probability of each target object can be compared with a preset threshold range. Based on the comparison result, the specific state of the target object in the current frame can be determined. The specific size of the preset threshold range is not limited and can be, for example, 0.9 to 1.1.

[0063] In some embodiments, when the occlusion probability is less than the lower limit of the preset threshold range, it is determined that the corresponding target object is in an occluded state. Specifically, when the calculated occlusion probability for a certain target object is less than the lower limit of the preset threshold range, it indicates that the image data of the target object has significantly decayed (e.g., unable to match or a large number of matching features lost) compared with the previous frame, and it can be determined that the target object is currently in an occluded state, i.e., the target object may become invisible or partially invisible in the current frame due to obstacles or other reasons.

[0064] In some embodiments, when the occlusion probability is within the preset threshold range, it is determined that the corresponding target object is in an unoccluded state. Specifically, when the calculated occlusion probability value for a certain target object is within the preset threshold range (i.e., not less than the lower limit and not greater than the upper limit), it indicates that the image data of the target object has maintained good continuity or the change is within an acceptable normal range compared with the previous frame, and it can be determined that the target object is currently in an unoccluded state.

[0065] In some embodiments, when the occlusion probability is greater than the upper limit of the preset threshold range, it is determined that the corresponding target object is in a de-occluded state, or is a new target object. Specifically, when the calculated occlusion probability for a certain target object is greater than the upper limit of the preset threshold range, it can represent the appearance of a new target that has not been previously identified and tracked, or a target object that was previously in an occluded state has become visible again and its feature data is successfully identified and extracted in the current frame. Therefore, in this situation, it can be determined that the target object is currently in a de-occluded state. This state indicates that a target with high confidence visibility in image data is identified and captured in the current frame, but this target object can be completely new or can be a target object that has recovered visibility from previous occlusion.

[0066] In some embodiments, for each target object in an occluded state, the state data recorded in the previous frame (i.e., the last valid observation frame before occlusion occurs) can be predicted by a motion model. The prediction can be based on the historical motion state of the target object before occlusion (i.e., the state data of the previous frame) and combined with the time extrapolation of the target object.

[0067] In some embodiments, through the above prediction processing, the motion trend and inertia law (such as motion direction retention, continuity of speed change) exhibited in the previous frame can output the predicted state data of the target object in an occluded state in the current frame. The predicted state data represents the state data (e.g., predicted velocity vector, predicted acceleration vector, predicted orientation angle, predicted position data, etc.) that the target object should have in the current frame under the assumption that its motion law has not changed during the occlusion period.

[0068] In some embodiments, step S30 may include the following steps: step S31, based on the uniform velocity sub-model, calculating the predicted uniform velocity state data of the target object in the occlusion state according to the state data of the target object before the occlusion state; the uniform velocity sub-model represents the functional relationship between the predicted uniform velocity state data and the state data when the target object is in uniform motion;

[0069] Step S32: Calculate predicted uniform acceleration / deceleration state data of the target object in the occlusion state based on the state data of the target object before the occlusion state based on the uniform acceleration / deceleration sub-model; the uniform acceleration / deceleration sub-model represents the functional relationship between the predicted uniform acceleration / deceleration state data and the state data when the target object is in uniform acceleration / deceleration motion;

[0070] Step S33: Based on the turning sub-model, the predicted turning state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the turning sub-model represents the functional relationship between the predicted turning state data and the state data when the target object is in a turning motion.

[0071] In some embodiments, for a motion model, a calculation formula for the predicted state data of a target object in an occluded state can be expressed as: Where F represents the state transfer matrix; It represents the status data of the k-1 time node; k represents the k time node.

[0072] In some embodiments, the state transfer matrix and process noise covariance matrix are different for different sub-models (the uniform velocity sub-model, the uniform acceleration / deceleration sub-model, and the turning sub-model). After obtaining the state data of the target object before the occlusion state, the state data can be independently and concurrently input into the three pre-set physical motion models, and calculations are performed based on the inherent characteristics of each model.

[0073] In some embodiments, the state data can be first input into a uniform velocity sub-model, which characterizes the motion of a target object in a state of uniform linear motion. Its core assumption is that the target object's velocity vector (including magnitude and direction) remains constant. Based on the velocity-maintaining characteristics defined by the uniform velocity sub-model and the input state data (particularly velocity data), the corresponding uniform velocity state data (e.g., a description of future position change trends under constant velocity) is calculated and output.

[0074] In some embodiments, the state data can be input into a uniform acceleration / deceleration sub-model. This uniform acceleration / deceleration sub-model characterizes the motion of a target object in a uniform acceleration / deceleration linear motion state. Its core assumption is that the acceleration vector of the target object remains constant (the velocity magnitude is allowed to change, but the direction remains linear). Based on the acceleration retention characteristics defined by this sub-model and the input state data (particularly velocity and acceleration data), the corresponding uniform acceleration / deceleration state data (e.g., velocity change trend and position prediction under constant acceleration) is calculated and output.

[0075] In some embodiments, the state data can be input into a turning sub-model. The turning sub-model characterizes the motion law of the target object under a state of changing the direction of motion (such as turning). Its core assumption involves that the target object has a non-zero angular velocity or heading angle change rate (for example, performing circular motion with a constant turning angular velocity or curvature radius). According to the direction change characteristics defined by the sub-model (such as angular velocity or turning rate), and the input state data (especially speed magnitude, heading angle, historical angular velocity data), the corresponding turning state data (for example, track change trend and position prediction under turning motion) is calculated and output.

[0076] In some embodiments, calculations using the uniform velocity sub-model, the uniform acceleration / deceleration sub-model, and the turning sub-model yield three independent sets of results: uniform velocity state data based on a constant velocity assumption, uniform acceleration / deceleration state data based on a constant acceleration (straight line) assumption, and turning state data based on a direction change (turning) assumption. Together, these data describe the state evolution of the target object under different possible motion patterns during occlusion.

[0077] In some embodiments, step S30 may further include the following steps: step S34, calculating predicted state data based on the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data.

[0078] In some embodiments, step S34 may include the following steps: step S341, based on the state data of the target object before being in the occlusion state, calculate the uniform speed credible data of the uniform speed sub-model, the uniform acceleration and deceleration credible data of the uniform acceleration and deceleration sub-model, and the turning credible data of the turning sub-model.

[0079] In some embodiments, the posterior probability of each model can be calculated by Bayes' theorem, and the calculation formula can be expressed as: P(M i Z)<<P(Z|M i )P(M i ); where M i Represented as different sub-models (uniform speed sub-model, uniform acceleration and deceleration sub-model, and turning sub-model); P(Z|M i) represents the probability of predicting the state data under a sub-model; P(M i ) represents the probability of each sub-model being used in the absence of predicted state data; P(M i |Z) represents the probability of each sub-model being adopted based on the state data before occlusion, that is, the credible data (including credible data of uniform speed, credible data of uniform acceleration and deceleration, and credible data of turning); ∝ represents the product of the probability of the credible data and the predicted state data appearing and the probability of each sub-model being used.

[0080] In some embodiments, step S34 may further include the following steps: step S342, calculating predicted state data based on predicted uniform speed state data, uniform speed credible data, predicted uniform acceleration / deceleration state data, uniform acceleration / deceleration credible data, predicted turning state data, and turning credible data.

[0081] In some embodiments, after obtaining the predicted uniform speed state data, the uniform speed credible data, the predicted uniform acceleration / deceleration state data, the uniform acceleration / deceleration credible data, the predicted turning state data, and the turning credible data, a calculation or integration process can be performed. This calculation or integration process can combine these three different motion hypotheses to generate a more robust and comprehensive predicted state data that can cover a variety of possible motion behaviors. This predicted state data is no longer the output of a single sub-model, but rather the result of integrating multiple potential motion patterns for subsequent motion trajectory prediction.

[0082] In some embodiments, the calculation formula for predicting state data can be expressed as: in, It is represented by predicted uniform speed state data, predicted uniform acceleration / deceleration state data, and predicted turning state data; i represents 1 to 3.

[0083] In some embodiments, after obtaining the predicted state data, the position data of the target object before the occlusion state can be used as the starting point or key reference point of the predicted trajectory. The fused predicted state data is used as the main factor driving the shape and trend of the trajectory (for example, the data contains key information such as the rate of change of velocity direction, acceleration trend, etc.). Based on the above combination, a suitable mathematical method can be used to recursively and extrapolate the motion state of future time steps. Ultimately, through the extrapolation process, a continuous predicted motion trajectory representing the possible motion path of the target object during the occlusion period (the current frame and several subsequent frames) is fitted. The trajectory contains a series of predicted position points arranged in chronological order (including predicted velocity data, acceleration data, angular velocity data, position data, etc.), which clearly depicts the most likely movement route or movement range of the target object when it cannot be directly observed.

[0084] In some embodiments, step S34 may further include the following steps: Step S343 , calculating an error covariance matrix of the target object before being in the occlusion state according to the state data of the target object before being in the occlusion state.

[0085] In some embodiments, the error covariance matrix P of the target object before the occlusion state is 0|k-1 It can be calculated through the state data of the target object before the occlusion state and the Kalman filter update equation.

[0086] In some embodiments, step S34 may further include the following steps: step S344, based on the error covariance matrix, calculating the uniform speed error covariance matrix of the uniform speed sub-model, the uniform acceleration and deceleration error covariance matrix of the uniform acceleration and deceleration sub-model, and the turning error covariance matrix of the turning sub-model.

[0087] In some embodiments, the calculation formula of the error covariance matrix can be expressed as: k|k-1 =F×P k-1|k-1 ×F T +Q; where P k|k-1 Expressed as the error covariance matrix of k time nodes, that is, the uncertainty of position, velocity and acceleration estimation; P k-1|k-1 is represented as the error covariance matrix at the k-1 time node; Q is represented as the process noise covariance matrix. Since the process noise covariance matrix Q and state transition matrix F are different for each sub-model, the corresponding uniform speed error covariance matrix, uniform acceleration and deceleration error covariance matrix, and turning error covariance matrix can be calculated.

[0088] In some embodiments, step S34 may also include the following steps: step S345, calculating the fusion error covariance matrix based on the uniform speed trusted data, uniform acceleration and deceleration trusted data, turning trusted data, uniform speed error covariance matrix, uniform acceleration and deceleration error covariance matrix, turning error covariance matrix, predicted uniform speed state data, predicted uniform acceleration and deceleration state data, predicted turning state data, and predicted state data.

[0089] In some embodiments, the calculation formula of the fusion error covariance matrix can be expressed as: in, It is expressed as the uniform speed error covariance matrix, the uniform acceleration and deceleration error covariance matrix, and the turning error covariance matrix.

[0090] In some embodiments, step S34 may further include the following steps: step S346, calculating uncertainty data according to the fusion error covariance matrix.

[0091] In some embodiments, the calculation formula of uncertainty data can be expressed as: Where diag represents extracting diagonal elements from a matrix.

[0092] In some embodiments, step S34 may also include the following steps: step S347, within a preset time period, determine the uncertainty data and the corresponding threshold: when the uncertainty data is less than the corresponding threshold, perform prediction processing on the corresponding target object in the occlusion state; otherwise, do not perform prediction processing on the corresponding target object in the occlusion state.

[0093] In some embodiments, in the process of continuously updating the predicted state data of the target object in the occluded state, for the target object that is currently in the occluded state, it is necessary to record the uncertainty data of each target object in the occluded state. At the same time, it is necessary to record the length of time the target object is in the occluded state. Within the preset time length, if the uncertainty data of multiple time nodes appear continuously and is greater than the corresponding threshold, it can be considered that the target object has been in a long-term occlusion state that exceeds the reasonable prediction range. The main reasons may include: the target object has completely left the scene; the occlusion is too severe and persistent, and the prediction result is highly unreliable; or the target object may have been eliminated by dynamic changes in the scene. At this point, the predicted state data update process for the specific target object can be stopped. The size of the preset time length can be unlimited, for example, it can be 20s, 30s, 40s, etc.

[0094] In some embodiments, the target tracking method may further include the following steps: Step S40: when a new target object is detected in the surrounding environment data, the new target object is tracked and calibrated according to its state data and historical tracking and recognition results.

[0095] In some embodiments, step S40 may include the following steps: Step S41 , when a new target object is detected in the surrounding environment data, obtaining status data of the new target object.

[0096] In some embodiments, when a previously occluded target object is subsequently tracked again by the sensor, it is necessary to determine whether the newly tracked target object is the same as a previously occluded target object. If so, the two are re-associated.

[0097] In some embodiments, step S40 may further include the following steps: Step S42, matching the state data of the new target object with the predicted state data of each target object in the occluded state in the historical tracking and recognition results to obtain corresponding feature matching data and position matching data.

[0098] In some embodiments, feature data of a new target object (i.e., a newly tracked target) can be extracted. The feature data generally includes but is not limited to: visual features (such as color, texture, shape), point cloud geometric features (such as size, outline), and state data (speed data, acceleration data, angular velocity data, position data). At the same time, all target objects in the historical tracking and recognition results that are in an occluded state (i.e., target objects that were previously occluded and are still being predicted and tracked) can be traversed to obtain their image data before being occluded. Subsequently, the feature data of the new target object can be matched with the feature data of each target object in an occluded state. The matching calculation can evaluate the degree of similarity between the feature data, and by calculating and outputting the corresponding feature matching data, the feature matching data quantifies the similarity between the two target objects in the feature space.

[0099] In some embodiments, the calculation formula for feature matching data may be: Among them, d i It is expressed as the difference between the feature vector of the feature data of the new target object and the feature vector of the feature data of the i-th target object in the occluded state in the feature space; σ1 is expressed as the standard deviation of feature matching; S feat,i It is represented as the feature matching data between the new target object and the i-th target object in the occluded state.

[0100] In some embodiments, after obtaining the position data of a new target object, the predicted position data in the predicted state data maintained and continuously updated by each target object in an occluded state can be obtained, by matching and calculating the position data of the new target object with the predicted position data in each predicted state data, and outputting corresponding position matching data, which quantifies the degree of consistency between the position data of the new target object and the predicted position data of the target object in the occluded state.

[0101] In some embodiments, the calculation formula for position matching data may be: Where Δx i It is represented as the error between the position data of the new target object and the predicted position data of the target object in the occluded state; σ2 is the standard deviation of the position measurement uncertainty; S pos,i It is represented as the position matching data between the new target object and the i-th target object in the occluded state.

[0102] In some embodiments, step S40 may further include the following steps: Step S43, calculating association probability data based on the feature matching data and the corresponding position matching data.

[0103] In some embodiments, a set of corresponding feature matching data and position matching data is generated for each new target object and each previously occluded target object. By combining these data, corresponding association probability data is calculated. This association probability data reflects the likelihood that the new target object is the real object corresponding to a previously occluded target object. Essentially, this data is a probabilistic judgment of target identity under the dual constraints of features and space.

[0104] In some embodiments, the calculation formula for the association probability data may be: Among them, P assoc,i It can be expressed as the association probability data between the new target object and the i-th target object in the occlusion state; Z is represented as a normalization constant to ensure that all association probability data are combined to 1; j is represented as the j-th target object in the occlusion state.

[0105] In some embodiments, step S40 may further include the following steps: step S44, tracking and calibrating the new target object according to the comparison result of the association probability data and the preset matching threshold.

[0106] In some embodiments, step S44 may include the following steps: judging the association probability data and a preset matching threshold: when the association probability data is greater than the preset matching threshold, associating the new target object with the corresponding target object in the occluded state in the historical tracking and recognition results, and tracking and calibrating the new target object; otherwise, tracking and calibrating the new target object.

[0107] In some embodiments, the association probability data is compared with a preset matching threshold, which is a configurable confidence threshold representing the minimum confidence level for accepting that a new target object is the same target object as an occluded target object.

[0108] In some embodiments, if the association probability data for a particular pair is determined to be greater than a preset matching threshold, the new target object can be confirmed to be the same target object as the particular occluded target object. A target association operation can then be performed, treating the two targets as preceding and following parts of the same trajectory chain, and assigning a corresponding unique identifier to the new target object.

[0109] In some embodiments, the target tracking method may include the following steps: continuously monitoring the spatial relationship between the target object in an obstructed state, the target object in an unobstructed state and the heavy-loaded equipment, and the monitoring content includes the minimum or real-time distance between the target object and the heavy-loaded equipment, and whether there is an intersection or collision risk point between the motion trajectory of the target object and the current or future planned driving path of the heavy-loaded equipment (that is, the motion paths of the two will overlap in time and space).

[0110] In some embodiments, when analysis determines that any target object meets any of the following risk conditions: its (predicted or actual) motion trajectory may substantially conflict with the driving path of the heavy-loaded equipment, and the distance between it and the heavy-loaded equipment is less than the set safety boundary distance (this distance is the safety threshold preset by the system, representing the minimum allowable safety distance); the heavy-loaded equipment can immediately generate and send a safety warning signal, which is intended to alert the operator that there is a collision or approach danger and that avoidance measures need to be taken immediately.

[0111] In some embodiments, after sending a safety warning signal, it is necessary to continuously and dynamically evaluate the distance change trend and relative speed between the target object and the heavy-loaded equipment. If it is determined that the approach trend of the target object relative to the heavy-loaded equipment persists, and it is predicted that the distance will shrink sharply to the critical risk range or smaller in a short period of time (that is, the risk of collision increases significantly), the preset emergency response linkage device of the heavy-loaded equipment can be further triggered. The execution level of this linkage device is higher than the warning signal, and it can directly drive the equipment to perform protective actions, such as: sending an automatic emergency braking command to the equipment control system to force the equipment speed to be reduced; sending an emergency stop command to the equipment control system to force the heavy-loaded equipment to stop operating immediately; triggering an audible, visual / tactile risk avoidance alarm (enhanced warning level). By actively and forcibly intervening in the operating status of the heavy-loaded equipment, the occurrence of potential collision accidents can be prevented to the greatest extent.

[0112] In some embodiments, when the target object in an obstructed state is unblocked, the re-acquired precise location data clearly shows that it is not in the driving path of the heavy-duty equipment, and the distance from the heavy-duty equipment continues to be greater than the safety boundary distance, and the target object has clearly left the monitored danger zone. At this point, it can be confirmed that the collision risk has been substantially eliminated, and the previously issued warning signal and any safety status restrictions that may have been triggered (such as releasing the brake / shutdown state lock) are automatically released. The system operating state returns to normal monitoring mode and continues to execute the target object detection, tracking, and safety assessment process.

[0113] It can be seen that in the above scheme, when the target object disappears due to occlusion, the state data before the occlusion is actively used to predict the motion trajectory. When the occlusion is lifted, by comparing the new target object with the historical tracking and recognition results, the new target object can be effectively matched and associated with the target object that was previously in the occluded state, so as to accurately restore the identity of the target object.

[0114] See also Figure 2 The present invention also discloses a target tracking device for heavy-duty equipment. The target tracking method described above can be applied to the target tracking device. The target tracking device can include: a data acquisition module 100, a target tracking module 200, a state prediction module 300, and a target matching module 400.

[0115] In some embodiments, the data collection module 100 may be used to collect environmental data surrounding heavy-duty equipment.

[0116] In some embodiments, the target tracking module 200 may be used to track and identify each target object in the surrounding environment data, and detect the tracking and identification results.

[0117] In some embodiments, the state prediction module 300 may be configured to perform prediction processing based on the state data of the target object before the occlusion state when detecting that the target object is occluded, to obtain predicted state data of the target object in the occlusion state.

[0118] In some embodiments, the target matching module 400 may be configured to, when a new target object is detected in the surrounding environment data, track and calibrate the new target object based on its state data and historical tracking and recognition results.

[0119] For the specific definition of the target tracking device, please refer to the definition of the target tracking method above, which will not be repeated here. The various modules in the above-mentioned target tracking device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the memory in the electronic device in the form of hardware, or can be stored in the memory in the electronic device in the form of software, so that the memory can call and execute the operations corresponding to the above modules.

[0120] See also Figure 3 In one embodiment, the electronic device 10 may include a memory 11, a processor 12, and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 12, such as a program for a target tracking method.

[0121] In one embodiment, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 10, such as a mobile hard disk of the electronic device 10. In other embodiments, the memory 11 can also be an external storage device of the electronic device 10, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 10. Furthermore, the memory 11 can also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 11 can be used not only to store application software and various types of data installed in the electronic device 10, such as target tracking code, but also to temporarily store data that has been output or is to be output.

[0122] In one embodiment, the processor 12 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 12 is the control core (Control Unit) of the electronic device 10, connecting the various components of the entire electronic device 10 using various interfaces and circuits. It executes or runs programs or modules (such as a target tracking method program) stored in the memory 11 and calls data stored in the memory 11 to perform various functions of the electronic device 10 and process data.

[0123] In one embodiment, the processor 12 executes the operating system and various installed applications of the electronic device 10. The processor 12 executes the applications to implement the steps in the above-mentioned target tracking method.

[0124] In one embodiment, the computer program may be divided into one or more modules, one or more of which are stored in the memory 11 and executed by the processor 12 to complete the present application. One or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 10. For example, the computer program may be divided into a data acquisition module 100, a target tracking module 200, a state prediction module 300, a target matching module 400, etc.

[0125] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A target tracking method for heavy-load equipment, characterized in that: include: Collect environmental data around heavy-load equipment; Track and identify each target object in the surrounding environment data, and detect the tracking and identification results: When it is detected that the target object is blocked, prediction processing is performed based on the state data of the target object before the blocking state to obtain the predicted state data of the target object in the blocking state; When a new target object is detected in the surrounding environment data, the new target object is tracked and calibrated according to its state data and historical tracking and recognition results.

2. The target tracking method for heavy-load equipment according to claim 1, characterized in that: The step of performing prediction processing based on the state data of the target object before being in the occlusion state to obtain the predicted state data of the target object in the occlusion state includes: Based on a preset motion model, predicted state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the motion model represents the functional relationship between the predicted state data and the state data.

3. The target tracking method for heavy-load equipment according to claim 2, characterized in that: The motion model includes a uniform speed sub-model, a uniform acceleration / deceleration sub-model, and a turning sub-model. The motion model is based on a preset state data of the target object before being in an occlusion state to calculate the predicted state data of the target object in the occlusion state. The step of characterizing the functional relationship between the predicted state data and the state data using the motion model comprises: Based on the uniform velocity sub-model, the predicted uniform velocity state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the uniform velocity sub-model represents the functional relationship between the predicted uniform velocity state data and the state data when the target object is in uniform motion; Based on the uniform acceleration / deceleration sub-model, the predicted uniform acceleration / deceleration state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the uniform acceleration / deceleration sub-model represents the functional relationship between the predicted uniform acceleration / deceleration state data and the state data when the target object is in uniform acceleration / deceleration motion; Based on the turning sub-model, predicted turning state data of the target object in the occlusion state is calculated according to the state data of the target object before the occlusion state; the turning sub-model represents the functional relationship between the predicted turning state data and the state data when the target object is in the turning motion; The predicted state data is calculated based on the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data.

4. The target tracking method for heavy-load equipment according to claim 3, characterized in that: The step of calculating the predicted state data based on the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data comprises: Calculating the uniform speed credible data of the uniform speed sub-model, the uniform acceleration and deceleration credible data of the uniform acceleration and deceleration sub-model, and the turning credible data of the turning sub-model based on the state data of the target object before being in the occlusion state; The predicted state data is calculated based on the predicted uniform speed state data, the uniform speed credible data, the predicted uniform acceleration / deceleration state data, the uniform acceleration / deceleration credible data, the predicted turning state data, and the turning credible data.

5. The target tracking method for heavy-load equipment according to claim 4, characterized in that: After the step of calculating the predicted state data based on the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, and the predicted turning state data, the method further includes: Calculating an error covariance matrix of the target object before the occlusion state according to the state data of the target object before the occlusion state; Calculating the uniform speed error covariance matrix of the uniform speed sub-model, the uniform acceleration and deceleration error covariance matrix of the uniform acceleration and deceleration sub-model, and the turning error covariance matrix of the turning sub-model according to the error covariance matrix; Calculating a fusion error covariance matrix based on the uniform speed credible data, the uniform acceleration / deceleration credible data, the turning credible data, the uniform speed error covariance matrix, the uniform acceleration / deceleration error covariance matrix, the turning error covariance matrix, the predicted uniform speed state data, the predicted uniform acceleration / deceleration state data, the predicted turning state data, and the predicted state data; Calculating uncertainty data according to the fusion error covariance matrix; Within a preset time period, the uncertainty data is judged against the corresponding threshold: When the uncertainty data is less than a corresponding threshold, performing prediction processing on the corresponding target object in the occluded state; Otherwise, no prediction processing is performed on the corresponding target object in the occluded state.

6. The target tracking method for heavy-load equipment according to claim 1, characterized in that: The step of tracking and calibrating the new target object according to its state data and historical tracking and recognition results when a new target object is detected in the surrounding environment data includes: When a new target object is detected in the surrounding environment data, obtaining state data of the new target object; Matching the state data of the new target object with the predicted state data of each target object in the occluded state in the historical tracking and recognition results to obtain corresponding feature matching data and position matching data; Calculating association probability data based on the feature matching data and the corresponding position matching data; According to the comparison result of the association probability data and the preset matching threshold, the new target object is tracked and calibrated.

7. The target tracking method for heavy-load equipment according to claim 6, characterized in that: The step of tracking and calibrating the new target object based on the comparison result of the association probability data and the preset matching threshold comprises: Determine whether the association probability data matches a preset matching threshold: When the association probability data is greater than a preset matching threshold, the new target object is associated with the corresponding target object in the occluded state in the historical tracking and recognition results, and the new target object is tracked and calibrated; Otherwise, tracking and calibration are performed on the new target object.

8. A target tracking device for heavy-load equipment, characterized in that: include: Data acquisition module, used to collect environmental data around heavy-duty equipment; The target tracking module is used to track and identify each target object in the surrounding environment data and detect the tracking and identification results: A state prediction module is used to perform prediction processing based on the state data of the target object before the occlusion state when detecting that the target object is occluded, so as to obtain the predicted state data of the target object in the occlusion state; The target matching module is used to track and calibrate the new target object based on its status data and historical tracking and recognition results when a new target object is detected in the surrounding environment data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the target tracking method for heavy-load equipment according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the target tracking method for heavy-load equipment according to any one of claims 1 to 7 are implemented.

Images