A vehicle surrounding object information processing method, device and vehicle

Abstract

The embodiment of the application provides a kind of vehicle surrounding environment object's information processing method, device and vehicle, wherein the method comprises: based on at least one first environment object in history time located around vehicle history moving track, the predicted moving state of first environment object at current time is predicted;In response to identifying at least one second environment object located around vehicle, based on predicted moving state and the feature information of second environment object at current time, determine association information;With association information, determine the association result between first environment object and second environment object;Based on association result and the feature information of second environment object, update the history moving track of first environment object.The application solves the technical problem that the information processing effect of vehicle surrounding environment object is poor.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and more specifically, to a method, apparatus, and vehicle for processing information about objects in the vehicle's surrounding environment. Background Technology

[0002] Currently, tracking objects in the vehicle's surrounding environment is fundamental for vehicle perception, decision-making, and planning. Related environmental object tracking methods associate objects with their appearance or motion. However, these methods are prone to failure under conditions of occlusion or changes in lighting, leading to errors in object association. These shortcomings make environmental object tracking unstable, directly impacting the accuracy of vehicle environmental perception and threatening driving safety. Therefore, the technical problem of poor information processing of the vehicle's surrounding environment remains.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and vehicle for processing information about objects in the surrounding environment of a vehicle, in order to at least solve the technical problem of poor information processing effect for objects in the surrounding environment of a vehicle.

[0005] According to one aspect of the embodiments of this application, an information processing method for environmental objects surrounding a vehicle is provided. The method may include: predicting the predicted movement state of at least one first environmental object located around the vehicle at a current time based on its historical movement trajectory over a historical period; in response to the identification of at least one second environmental object located around the vehicle, determining association information based on the predicted movement state and feature information of the second environmental object at the current time, wherein the association information represents the degree of association between the predicted movement state of the first environmental object and the actual movement state of the second environmental object; determining the association result between the first environmental object and the second environmental object using the association information; and updating the historical movement trajectory of the first environmental object based on the association result and the feature information of the second environmental object.

[0006] Optionally, based on the predicted movement state and the feature information of the second environmental object at the current moment, the association information is determined, including: based on the predicted movement state and the feature information of the second environmental object, determining a first sub-distance and a second sub-distance of the second environmental object, wherein the influence of the first sub-distance on the matching accuracy of the historical movement trajectory and the second environmental object is greater than the influence of the second sub-distance on the matching accuracy of the historical movement trajectory and the second environmental object; and determining the association information based on the first sub-distance and the second sub-distance.

[0007] Optionally, the first sub-distance includes a movement distance, which represents the correlation between the predicted movement characteristics of the first environmental object at the current moment and the actual movement characteristics of the second environmental object. The predicted movement state includes a predicted bounding box of the first environmental object at the current moment predicted using historical movement trajectories. The feature information includes the actual detection bounding box of the second environmental object. Based on the predicted movement state and the feature information, determining the first sub-distance of the second environmental object includes: determining the intersection-union ratio (IU) between the predicted bounding box and the actual detection bounding box, wherein the IU represents the degree of overlap between the predicted bounding box and the actual detection bounding box; and determining the movement distance as the reciprocal of the IU. And / or, the first sub-distance includes an appearance distance, which represents the correlation between the predicted appearance characteristics of the first environmental object at the current moment and the actual appearance characteristics of the second environmental object. The predicted movement state includes the appearance feature vector of the first environmental object at any historical moment in history. The feature information includes the appearance feature vector of the second environmental object. Based on the predicted movement state and the feature information, determining the first sub-distance of the environmental object includes: determining the similarity between the appearance feature vector of the first environmental object and the appearance feature vector of the second environmental object; and determining the appearance distance as the difference between a preset coefficient and the similarity.

[0008] Optionally, the second sub-distance includes height distance and confidence distance. The height distance is used to represent the correlation between the predicted height of the first environmental object at the current moment and the actual height of the second environmental object. The confidence distance is used to represent the reliability of detecting the second environmental object at the current moment. Determining the second sub-distance of the second environmental object based on the predicted movement state and feature information includes: determining the height distance based on the predicted movement state and feature information; and determining the confidence distance based on the feature information of the second environmental object.

[0009] Optionally, the predicted movement state includes the height value of a first environmental object at any historical moment in history, and the feature information includes the height value of a second environmental object. Based on the predicted movement state and the feature information, determining the height distance includes: normalizing the height values ​​of the first and second environmental objects respectively to obtain normalized height values ​​of the first and second environmental objects; determining the difference between the normalized height values ​​of the first and second environmental objects; and determining the height distance based on the difference.

[0010] Optionally, the feature information includes the detection confidence at the current moment. Based on the feature information of the second environmental object, the confidence distance is determined, including: determining the difference between the preset coefficient and the detection confidence as the confidence distance.

[0011] Optionally, determining association information based on the first sub-distance and the second sub-distance includes: assigning weight coefficients to the first sub-distance and the second sub-distance respectively, wherein the weight coefficient assigned to the first sub-distance is greater than the weight coefficient assigned to the second sub-distance; and fusing the first sub-distance assigned to the weight coefficient and the second sub-distance assigned to the weight coefficient to obtain association information.

[0012] Optionally, the association result between the first environment object and the second environment object is determined by using the association information, including: invoking a global matching strategy, using the association information to associate the first environment object and the second environment object, and obtaining the association result, wherein the global matching strategy is used to represent the rule that minimizes the total cost of associating the first environment object and the second environment object from the association information.

[0013] Optionally, a global matching strategy is invoked to associate the first environment object and the second environment object using the association information, resulting in an association result. This includes: constructing a bipartite graph using a first set of nodes with the historical movement trajectory of the first environment object as nodes and a second set of nodes with the second environment object as nodes, wherein the edge weights between nodes in the first set of nodes and nodes in the second set of nodes in the bipartite graph are used to represent the association information; determining the edges between nodes in the first set of nodes and nodes in the second set of nodes that satisfy the objective of minimizing edge weights in the bipartite graph; and determining the association result based on the edges.

[0014] Optionally, updating the historical movement trajectory of the first environmental object based on the association results and the feature information of the second environmental object includes: classifying the historical movement trajectory of the first environmental object and the second environmental object respectively according to the association results to obtain the classification results of the historical movement trajectory and the classification results of the second environmental object; and updating the historical movement trajectory based on the classification results of the historical movement trajectory, the classification results of the second environmental object, and the feature information.

[0015] Optionally, based on the association results, the historical movement trajectory of the first environmental object and the second environmental object are classified respectively to obtain the classification results of the historical movement trajectory and the classification results of the second environmental object, including: based on the association results, determining that the classification result of the historical movement trajectory includes at least one of the following: a historical movement trajectory that has been matched at the current time, or a historical movement trajectory that has not been matched at the current time; based on the association results, determining that the classification result of the second environmental object includes at least one of the following: an unmatched second environmental object, or a matched second environmental object.

[0016] Optionally, based on the classification results of historical movement trajectories, the classification results of the second environmental object, and feature information, the historical movement trajectories are updated, including at least one of the following: resetting the lifecycle state of matched historical movement trajectories; determining the lifecycle state of unmatched historical movement trajectories based on the movement continuity of unmatched historical movement trajectories; updating the lifecycle counter of unmatched historical movement trajectories in response to the lifecycle state of unmatched historical movement trajectories meeting the maintenance conditions; deleting unmatched historical movement trajectories in response to the lifecycle state of unmatched historical movement trajectories not meeting the maintenance conditions; determining the detection confidence and continuous occurrence state of unmatched second environmental objects from the feature information; and creating a historical movement trajectory corresponding to the unmatched second environmental object and updating the lifecycle counter of the created historical movement trajectory in response to the moment when the detection confidence is greater than or equal to the confidence threshold and the continuous occurrence state indicates that the unmatched second environmental object has continuously appeared a first target number.

[0017] Optionally, the method further includes: deleting the historical movement trajectory in response to at least one of the following conditions: the lifecycle counter of the historical movement trajectory is reduced to a counting threshold; the moment when the historical movement trajectory continuously appears for a second target number does not match the second environmental object, and the error of determining the predicted movement state using the historical movement trajectory is greater than an error threshold.

[0018] Optionally, based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period, the predicted movement state of the first environmental object at the current moment is predicted, including: generating a preliminary predicted movement state of the first environmental object at the current moment through a linear dynamics prediction model based on the movement state sequence in the historical movement trajectory; and extracting temporal features from the movement state sequence through a nonlinear temporal modeling network to generate a movement state correction vector of the first environmental object; and fusing the movement state correction vector with the preliminary predicted movement state to obtain the predicted movement state.

[0019] Optionally, the nonlinear temporal modeling network is a gated recurrent unit network based on an attention mechanism. The nonlinear temporal modeling network extracts temporal features from the movement state sequence to generate a movement state correction vector for the first environmental object. This includes: inputting a preset number of historical movement state sequences from the historical movement trajectory into the gated recurrent unit network; using the gated recurrent unit to encode long-term temporal features of the movement state sequences to generate a hidden state sequence containing long-term temporal movement dependencies; assigning weights to the hidden state sequences using an attention mechanism; and fusing the weighted hidden state sequences to obtain attention-weighted movement features; and determining the attention-weighted movement features as the movement state correction vector.

[0020] Optionally, the motion state correction vector is fused with the preliminary predicted motion state to obtain the predicted motion state, including: superimposing the motion state correction vector onto the state vector of the preliminary predicted motion state to obtain the predicted motion state, wherein the state vector of the preliminary predicted motion state includes at least one of the following: the center coordinates of the first environment object at the current time, the movement speed of the first environment object at the current time, and the size parameter of the first environment object at the current time.

[0021] Optionally, the method further includes: in response to the identification of at least one second environmental object located around the vehicle at the current moment, acquiring image data and point cloud data containing the second environmental object at the current moment; and extracting feature information of the second environmental object at the current moment from the image data and point cloud data.

[0022] Optionally, acquiring image data and point cloud data containing a second environmental object at the current moment includes: at the current moment, using an image acquisition device deployed on the vehicle to acquire initial image data of the environment surrounding the vehicle, and using a radar device deployed on the vehicle to acquire initial point cloud data of the environment surrounding the vehicle, wherein the image content of the initial image data includes at least one second environmental object, and the point cloud content of the initial point cloud data includes at least one second environmental object; performing spatiotemporal calibration processing on the initial image data and the initial point cloud data to obtain calibrated initial image data and calibrated initial point cloud data; and performing denoising processing on the calibrated initial image data and the calibrated initial point cloud data respectively to obtain image data and point cloud data.

[0023] Optionally, the feature information includes the actual detection bounding box of the second environmental object, the actual appearance features of the second environmental object, the detection confidence of the second environmental object, and the actual height of the second environmental object. The feature information is extracted from the image data and the point cloud data, including: extracting the actual detection bounding box, the actual appearance features, and the detection confidence from the image data; and extracting the actual height from the point cloud data.

[0024] According to another aspect of the embodiments of this application, an information processing apparatus for objects in the environment surrounding a vehicle is also provided. The apparatus may include: a prediction module, configured to predict the predicted movement state of the first environmental object at the current moment based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period; a first determination module, configured to determine association information based on the predicted movement state and feature information of the second environmental object at the current moment in response to the identification of at least one second environmental object located around the vehicle, wherein the association information is used to represent the degree of association between the predicted movement state of the first environmental object and the actual movement state of the second environmental object; a second determination module, configured to determine the association result between the first environmental object and the second environmental object using the association information; and an update module, configured to update the historical movement trajectory of the first environmental object based on the association result and the feature information of the second environmental object.

[0025] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of any one of the various embodiments of this application when it runs.

[0026] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of any one of the various embodiments of this application when it runs.

[0027] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the method of any one of the various embodiments of this application.

[0028] In this embodiment, the predicted movement state of at least one first environmental object located around the vehicle within a historical time period can be predicted at the current moment. Upon detection of at least one second environmental object around the vehicle, association information can be determined based on the predicted movement state and the feature information of the second environmental object at the current moment. The association information can be used to determine the association result between the first and second environmental objects. The historical movement trajectory of the first environmental object can be updated based on the association result and the feature information of the second environmental object. In other words, in this embodiment, the predicted movement state of the first environmental object around the vehicle and the feature information of the second environmental object at the current moment are fused to determine the association information between the predicted movement state of the first environmental object around the vehicle and the second environmental object at the current moment. The association information is then used to determine the association result between the first and second environmental objects. This method solves the problem in related technologies where environmental objects are associated only through simple data, leading to incorrect associations in scenarios such as occlusion and changes in lighting. By using feature information to associate environmental objects, the problem of incorrect associations in occlusion and lighting scenarios is eliminated, thereby improving the technical effect of information processing of environmental objects around the vehicle and solving the technical problem of poor information processing of environmental objects around the vehicle. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 This is a schematic diagram of an application scenario for information processing of objects surrounding a vehicle, according to an embodiment of this application.

[0031] Figure 2 This is a flowchart of an information processing method for vehicle surrounding environment objects according to an embodiment of this application;

[0032] Figure 3 This is a flowchart of a DynaSort-based multi-target tracking method for autonomous driving according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of an information processing device for objects surrounding a vehicle, according to an embodiment of this application.

[0034] Figure 5 This is a schematic diagram of a vehicle according to an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0038] Figure 1 This is a schematic diagram illustrating an application scenario of information processing of objects surrounding a vehicle according to an embodiment of this application, such as... Figure 1 As shown, the scenario described above may include a sensing device 10, a network 20, and a vehicle 30. The sensing device 10 is used to acquire sensing information about the presence of perceptible environmental objects around the vehicle and inputs the sensing information of the detected environmental objects. The sensing device may be a high-definition camera, LiDAR, or other similar device on the vehicle. The sensing information can be sent to the vehicle 30 via the network 20. At this point, the vehicle 30 needs to execute steps S101 to S104 to complete the information processing of the environmental objects around the vehicle.

[0039] The following steps can be performed by vehicle 30: Step S101, based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period, predict the predicted movement state of the first environmental object at the current moment; Step S102, in response to the identification of at least one second environmental object located around the vehicle, determine the association information based on the predicted movement state and the feature information of the second environmental object at the current moment; Step S103, using the association information, determine the association result between the first environmental object and the second environmental object; Step S104, based on the association result and the feature information of the second environmental object, update the historical movement trajectory of the first environmental object.

[0040] In this embodiment, through steps S101 to S104, the predicted movement state of the first environmental object around the vehicle and the feature information of the second environmental object at the current moment are fused to determine the association information between the predicted movement state of the first environmental object and the second environmental object at the current moment. Using this association information, the association result between the first and second environmental objects is determined. This method solves the problem in related technologies where environmental objects are associated only through simple data, leading to incorrect associations in scenarios such as occlusion and changes in lighting. By using feature information to associate environmental objects, the problem of incorrect associations in occlusion and lighting scenarios is eliminated, thereby improving the information processing effect of the environmental objects around the vehicle and solving the technical problem of poor information processing effect.

[0041] According to an embodiment of this application, an embodiment of a method for processing information about objects in the surrounding environment of a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] This embodiment provides a method for processing information about objects in the environment surrounding a vehicle. Figure 2 This is a flowchart of an information processing method for vehicle surrounding environment objects according to an embodiment of this application, such as... Figure 2 As shown, the method may include the following steps.

[0043] Step S201: Based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period, predict the predicted movement state of the first environmental object at the current moment.

[0044] In the technical solution provided by step S201 in the embodiments of this application, the aforementioned historical time can be a time window or time period before the current moment, such as 1 to 5 seconds past the current moment. If the current moment is the current frame, then the aforementioned historical time can be several frames prior to the current frame.

[0045] Optionally, at least one first environmental object surrounding the vehicle can be an environmental object with stored historical data, such as other vehicles, pedestrians, or obstacles around the vehicle. This environmental object can also be referred to as a target. The historical data can include the tracking identifier (ID) of the first environmental object, its historical movement trajectory, and feature information of the first environmental object within a historical time period. The feature information of the first environmental object can be a set of features acquired by the sensing device at a historical moment, representing multidimensional information about the first environmental object, such as a feature set consisting of appearance, movement, height, and confidence level.

[0046] Optionally, the aforementioned historical movement trajectory can be a set of state sequences (also known as historical movement state sequences) recording the first environmental object surrounding the vehicle over a historical period. The states in the state sequence set can include feature information of the first environmental object surrounding the vehicle at any historical moment within the historical time period, such as the center coordinates, velocity vector, and category label of the first environmental object at that historical moment.

[0047] Optionally, the predicted movement state of the first environmental object at the current moment can be the estimated state of the first environmental object around the vehicle at the current moment, calculated by a prediction model based on the historical movement trajectory of the first environmental object.

[0048] In this embodiment, if a historical movement trajectory of at least one first environmental object located around the vehicle is detected within a historical time period, the predicted movement state of the first environmental object at the current moment is predicted based on the historical movement trajectory.

[0049] Optionally, during the detection of the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period, the historical movement trajectory of at least one first environmental object can be retrieved from the vehicle's historical data storage unit. Specifically, a historical movement state sequence of at least one first environmental object over a past time period (e.g., the past N frames) can be extracted, and this historical movement state sequence constitutes the historical movement trajectory of the first environmental object.

[0050] Optionally, in the process of predicting the predicted movement state of the first environmental object at the current moment based on its historical movement trajectory, the predicted historical movement trajectory of the first environmental object around the vehicle can be input into the prediction model to obtain the predicted movement state of the first environmental object at the current moment. For example, the prediction model can be a Kalman Filter and Gated Recurrent Unit Attention Model (KF-GRU) model.

[0051] In step S202, in response to the identification of at least one second environmental object located around the vehicle, the association information is determined based on the predicted movement state and the feature information of the second environmental object at the current moment.

[0052] In the technical solution provided by step S202 of this application embodiment, the second environmental object can be an environmental object detected by a sensing device (e.g., camera, lidar) in the vehicle at the current moment, and this second environmental object has not yet been matched with a first environmental object with an existing historical movement trajectory. The feature information of the second environmental object can be a set of features acquired by the sensing device at the current moment that can represent multi-dimensional information about the second environmental object, such as a feature set composed of appearance, movement, height, and confidence level.

[0053] Optionally, the aforementioned correlation information can be a numerical value or index used to quantify the degree of matching or similarity between the predicted movement state of the first environmental object and the actual movement state of the second environmental object. The actual movement state of the second environmental object is the feature information of the second environmental object at the current moment, obtained through the sensing device.

[0054] In this embodiment, after detecting at least one second environmental object located around the vehicle, association information is determined based on the predicted movement state and the feature information of the second environmental object at the current moment.

[0055] Optionally, during the detection of at least one second environmental object located around the vehicle, at the current moment, sensing data of the environment around the vehicle can be collected by a sensing device, and at least one second environmental object can be identified and separated from the sensing data, and feature information corresponding to at least one second environmental object can be extracted.

[0056] Optionally, in determining the association information based on the predicted movement state and the feature information of the second environmental object at the current moment, the predicted movement state of the first environmental object at the current moment can be predicted using the historical movement trajectory of the first environmental object. This prediction is then combined with the actual movement state of the second environmental object collected and extracted by sensing devices at the current moment. A multi-dimensional distance metric is calculated between the predicted movement state of the first environmental object and the actual movement state of the second environmental object at the current moment, and these are then weighted and fused to obtain a comprehensive association index representing the degree of matching between the historical movement trajectory of the first environmental object and the second environmental object. This process quantifies the similarity between the historical movement trajectory and the current environmental object by fusing multiple feature information.

[0057] In this embodiment, the method determines the association information based on the predicted movement state and the feature information of the second environmental object at the current moment, enabling efficient and accurate matching of historical movement trajectories and environmental objects at the current moment. Specifically, this process combines the predicted movement state derived from the historical movement trajectory with the actual feature information of the environmental object at the current moment. It not only utilizes simple appearance or motion state feature information, but also combines multiple fused feature information including appearance, motion state, height, and confidence level. This effectively overcomes the problem of single-frame detection loss or false detection caused by temporary target occlusion, rapid movement, or sensor noise. Even if there is a detection deviation in a single frame, the tracking loss rate can be reduced through multiple feature information, thereby achieving the technical effect of improving the information processing effect of the vehicle's surrounding environment and solving the technical problem of poor information processing effect of the vehicle's surrounding environment.

[0058] Step S203: Using the association information, determine the association result between the first environment object and the second environment object.

[0059] In the technical solution provided by step S203 of the embodiments of this application, the association result can be the matching relationship between the historical movement trajectory of the first environmental object and the second environmental object at the current moment. For example, the association result can include matched pairs (i.e., a historical movement trajectory of a first environmental object is successfully matched with a second environmental object at the current moment), unmatched pairs (i.e., a historical movement trajectory of a first environmental object is not matched with any second environmental object at the current moment), and unmatched detections (i.e., a second environmental object is not matched by the historical movement trajectory of any first environmental object at the current moment).

[0060] In this embodiment, the association result between the first environmental object and the second environmental object at the current moment is determined by using the historical trajectory of the first environmental object and the association information of the second environmental object obtained at the current moment.

[0061] Optionally, in determining the association result between the first and second environmental objects by utilizing the obtained historical trajectory of the first environmental object and the association information of the second environmental object at the current moment, a correspondence can be established between the set of historical movement trajectories of historical environmental objects and the set of currently detected environmental objects based on the obtained historical trajectory of the first environmental object and the association information of the second environmental object at the current moment. Specifically, a matching strategy can be used to determine the matching relationship between the historical movement trajectories of historical environmental objects and the currently detected environmental objects, thereby obtaining the association result. The aforementioned matching strategy can be a global matching strategy, which selects the matching scheme that minimizes the total cost (or total distance) from all possible matching combinations to establish a one-to-one correspondence between the set of historical movement trajectories of historical environmental objects and the set of currently detected environmental objects.

[0062] In this embodiment, the above method uses association information to determine the association result between the first environmental object and the second environmental object, which can establish an accurate correspondence between the historical movement trajectory and the real-time environmental object. This effectively solves the problem of confusion of tracking identifiers (IDs) under occlusion and lighting scenes, avoids erroneous associations caused by local matching, and thus achieves the technical effect of improving the accuracy of information processing of the vehicle's surrounding environment objects, solving the technical problem of poor information processing effect of the vehicle's surrounding environment objects.

[0063] Step S204: Based on the association results and the feature information of the second environmental object, update the historical movement trajectory of the first environmental object.

[0064] In the technical solution provided by step S204 of the embodiments of this application, the process of updating the historical movement trajectory of the first environment object by using the association result and the feature information of the second environment object can refresh the status parameters of the historical movement trajectory of the first environment object or perform lifecycle management by adopting different strategies based on the association result between the first environment object and the second environment object at the current moment, thereby performing operations such as creating a new trajectory, maintaining a valid trajectory, or deleting an invalid trajectory.

[0065] For example, when the association result shows that the first environment object and the second environment object are successfully matched, the historical movement trajectory of the first environment object is updated using the feature information of the second environment object. At the same time, the lifecycle counter of the historical movement trajectory is reset. When the association result shows that the first environment object is not matched with any second environment object, the continuity of the motion state of the historical movement trajectory is checked. If the prediction error is within the threshold (e.g., ≤2 pixels) and the motion state is continuous, the lifecycle counter is decremented by 1 (e.g., from T=5 to T=4) to deal with possible temporary occlusion. If the lifecycle counter drops to 0, or there is no match for multiple consecutive frames (e.g., 3 frames) and the prediction error exceeds the threshold, the historical movement trajectory is determined to be invalid, and the historical movement trajectory of the first environment object is deleted from the historical movement trajectory set.

[0066] In the embodiments of this application, the above method updates the historical movement trajectory of the first environmental object based on the association result and the feature information of the second environmental object. This enables real-time state correction of effective targets to ensure tracking accuracy, while performing dynamic lifecycle management of unmatched targets. This effectively avoids the residue and accidental deletion of invalid historical movement trajectories, thereby achieving the technical effect of improving the stability and accuracy of information processing of the vehicle's surrounding environment and solving the technical problem of poor information processing effect of the vehicle's surrounding environment.

[0067] In steps S201 to S204 of this embodiment, the predicted movement state of at least one first environmental object located around the vehicle within a historical time period can be predicted at the current moment. Upon identifying at least one second environmental object around the vehicle, association information can be determined based on the predicted movement state and the feature information of the second environmental object at the current moment. The association information can be used to determine the association result between the first and second environmental objects. The historical movement trajectory of the first environmental object can be updated based on the association result and the feature information of the second environmental object. In other words, in this embodiment, the predicted movement state of the first environmental object around the vehicle and the feature information of the second environmental object at the current moment are fused to determine the association information between the predicted movement state of the first environmental object around the vehicle and the second environmental object at the current moment. The association information is then used to determine the association result between the first and second environmental objects. This method solves the problem in related technologies where environmental objects are associated only through simple data, leading to incorrect associations in scenarios such as occlusion and changes in lighting. By using feature information to associate environmental objects, the problems of occlusion and incorrect association of environmental objects under different lighting conditions are eliminated, thereby improving the information processing effect of environmental objects around the vehicle and solving the technical problem of poor information processing effect of environmental objects around the vehicle.

[0068] The embodiments of this application will be described in detail below with reference to the steps described above.

[0069] As an optional implementation, the association information is determined based on the predicted movement state and the feature information of the second environmental object at the current moment, including: determining a first sub-distance and a second sub-distance of the second environmental object based on the predicted movement state and the feature information of the second environmental object, wherein the influence of the first sub-distance on the matching accuracy of the historical movement trajectory and the second environmental object is greater than the influence of the second sub-distance on the matching accuracy of the historical movement trajectory and the second environmental object; and determining the association information based on the first sub-distance and the second sub-distance.

[0070] In this embodiment, if the predicted movement state of the first environmental object and the feature information of the second environmental object at the current moment are obtained, the first sub-distance and the second sub-distance of the second environmental object are calculated based on the predicted movement state and the feature information, and then the association information used for the final matching judgment is determined based on the calculated first sub-distance and the second sub-distance.

[0071] Optionally, sub-distance refers to a numerical indicator used to quantify the degree of difference between the historical movement trajectory (first environmental object) and the currently detected target (second environmental object). Based on the different weights of its influence on matching accuracy, it is divided into first sub-distance and second sub-distance. Among them, the first sub-distance represents the key feature difference with a greater influence, and the second sub-distance represents the auxiliary feature difference with a smaller influence.

[0072] Optionally, the process of obtaining the predicted movement state of the first environmental object and the feature information of the second environmental object at the current moment can be performed by performing state prediction processing on the historical movement trajectory of the first environmental object and feature extraction processing on the real-time perception data of the second environmental object. Specifically, on the one hand, based on the set of state sequences within the historical time period in the historical movement trajectory of the first environmental object, the predicted movement state of the first environmental object at the current moment is calculated using a preset state prediction model; on the other hand, by sensing the scene data around the vehicle at the current moment in real time through sensing devices, and after denoising, calibration and target detection processing, the appearance, motion, height and confidence of the second environmental object are extracted to form the feature information of the second environmental object at the current moment.

[0073] Optionally, the process of calculating the first sub-distance and the second sub-distance of the second environmental object based on the predicted movement state and feature information can be as follows: extract the historical predicted features of the first environmental object and the real-time observed features of the second environmental object, quantify the difference between the two in the key motion / appearance dimension through a metric function to obtain the first sub-distance, and quantify the difference between the two in the auxiliary dimension to obtain the second sub-distance.

[0074] Optionally, on the one hand, a first sub-distance representing high-impact matching accuracy is calculated based on the degree of overlap between the predicted motion state and the actual detection box or the similarity of appearance feature vectors; on the other hand, a second sub-distance representing low-impact matching accuracy is calculated based on the normalized difference between the predicted height and the actual height or the detection confidence.

[0075] Optionally, the process of determining the association information used for the final matching decision based on the calculated first sub-distance and second sub-distance can be achieved by assigning differentiated weight coefficients to the first sub-distance and second sub-distance, which represent the differences in matching across different dimensions. The weight coefficient assigned to the first sub-distance is higher than that assigned to the second sub-distance, so as to reflect the dominant role of the first sub-distance in influencing matching accuracy. Subsequently, the weighted first sub-distance and the weighted second sub-distance are fused together to obtain association information that comprehensively represents the degree of association between the predicted movement state of the first environmental object and the actual movement state of the second environmental object.

[0076] In the embodiments of this application, the above method can effectively balance the contribution weight of core features (such as motion and appearance) and auxiliary features (such as height and confidence) in the association process by distinguishing the different degrees of influence of the first sub-distance and the second sub-distance on the matching accuracy. This avoids association errors caused by the failure of a single clue in complex scenarios (such as occlusion and changes in lighting). At the same time, it uses other clues to supplement key information to improve the robustness of association, thereby improving the tracking accuracy and stability of objects in the vehicle's surrounding environment in dynamic scenarios.

[0077] As an optional implementation, the first sub-distance includes a movement distance, which represents the correlation between the predicted movement characteristics of the first environmental object at the current moment and the actual movement characteristics of the second environmental object. The predicted movement state includes a predicted bounding box of the first environmental object at the current moment predicted using historical movement trajectories. The feature information includes the actual detection bounding box of the second environmental object. Based on the predicted movement state and the feature information, determining the first sub-distance of the second environmental object includes: determining the intersection-union ratio (IU) between the predicted bounding box and the actual detection bounding box, wherein the IU represents the degree of overlap between the predicted bounding box and the actual detection bounding box; and determining the movement distance as the reciprocal of the IU. And / or, the first sub-distance includes an appearance distance, which represents the correlation between the predicted appearance characteristics of the first environmental object at the current moment and the actual appearance characteristics of the second environmental object. The predicted movement state includes the appearance feature vector of the first environmental object at any historical moment in history. The feature information includes the appearance feature vector of the second environmental object. Based on the predicted movement state and the feature information, determining the first sub-distance of the environmental object includes: determining the similarity between the appearance feature vector of the first environmental object and the appearance feature vector of the second environmental object; and determining the appearance distance as the difference between a preset coefficient and the similarity.

[0078] In this embodiment, if the predicted bounding box of the first environmental object is obtained at the current time, the intersection-union ratio (IUU) between the predicted bounding box and the actual detection bounding box of the second environmental object is determined, and the reciprocal of the IUU is determined as the moving distance. And / or, if the appearance feature vectors of the first environmental object and the second environmental object are obtained, the similarity between the two appearance feature vectors is determined, and the difference between the preset coefficient and the similarity is determined as the appearance distance.

[0079] Optionally, the intersection-union ratio (CIU) can be the ratio of the area of ​​the overlapping region between the predicted bounding box and the actual detected bounding box to the area of ​​their union. Specifically, the CIU can be obtained by calculating the intersection area of ​​the predicted bounding box and the actual detected bounding box and dividing it by the area of ​​their union. A higher CIU indicates a greater degree of overlap between the predicted bounding box and the actual detected bounding box.

[0080] Optionally, the similarity can be the cosine of the angle between the appearance feature vectors of the first and second environment objects. Specifically, the similarity can be obtained by calculating the dot product of the appearance feature vectors of the first and second environment objects and then dividing by the product of the magnitudes of these two vectors. The closer the value is to 1, the more similar the two appearance feature vectors are.

[0081] Optionally, the preset coefficient can be 1. This preset coefficient is used to convert similarity or detection confidence into a distance metric, so that the higher the similarity or detection confidence, the smaller the corresponding distance value, thereby reflecting a higher matching priority in association matching.

[0082] Optionally, the process of obtaining the predicted bounding box of the first environmental object at the current moment can be achieved by using the movement state sequence in the historical movement trajectory and a linear dynamics prediction model to calculate the theoretical spatial position and size of the first environmental object at the current moment. Specifically, the movement state sequence of the first environmental object over a past time period can be extracted, input into a Kalman filter module, and the center coordinates, movement speed, and size parameters of the first environmental object at the current moment can be predicted using the state transition equation, thereby generating the predicted bounding box of the first environmental object at the current moment.

[0083] Optionally, determining the intersection-union ratio (IUR) between the predicted bounding box and the actual detection bounding box of the second environmental object, and using the reciprocal of the IUR as the moving distance, can be achieved by quantifying the spatial overlap between the predicted bounding box and the actual detection bounding box using a geometric metric, and converting the spatial overlap into a distance index. Specifically, the intersection area of ​​the predicted bounding box and the actual detection bounding box can be calculated, as can their union area. The IUR is obtained by dividing the intersection area by the union area. Then, the IUR is subtracted using a preset coefficient (e.g., 1), and the difference is determined as the moving distance. A smaller moving distance indicates a higher degree of overlap between the predicted bounding box and the actual detection bounding box.

[0084] Optionally, the process of obtaining the appearance feature vectors of the first and second environmental objects can involve capturing visual images of the environmental objects using an image acquisition device and extracting high-order semantic features from the visual images using a deep learning model. Specifically, at the current moment, initial image data containing the first and second environmental objects can be acquired using an image acquisition device deployed on the vehicle. After denoising the initial image data, the image is input into a target detection algorithm to extract the appearance feature vectors of the first environmental object at a first historical moment and the appearance feature vectors of the second environmental object at the current moment from the feature extraction layer.

[0085] Optionally, the process of determining the similarity between the two appearance feature vectors and defining the difference between a preset coefficient and the similarity as the appearance distance can be achieved by quantifying the similarity between the two appearance feature vectors using vector space geometric metrics and converting the similarity into a distance index. Specifically, the appearance feature vectors of the first and second environmental objects can be normalized, and the dot product between these two normalized vectors can be calculated. This dot product is then used as the similarity. Subsequently, the similarity is subtracted from the preset coefficient (e.g., 1), and the resulting difference is defined as the appearance distance. A smaller appearance distance indicates greater similarity between the two appearance feature vectors.

[0086] In this embodiment, the above method, by calculating the movement distance representing the degree of overlap of spatial motion and the appearance distance representing the visual semantic similarity, can comprehensively quantify the degree of matching between the first environmental object and the second environmental object in terms of movement history trajectory and appearance features. This overcomes the limitations of a single feature in scenarios of target deformation, occlusion, or lighting changes, thereby improving the robustness of association matching and providing reliable data support for accurately determining the association results between the first environmental object and the second environmental object in the future.

[0087] As an optional implementation, the second sub-distance includes an altitude distance and a confidence distance. The altitude distance represents the correlation between the predicted altitude of the first environmental object at the current moment and the actual altitude of the second environmental object. The confidence distance represents the reliability of detecting the second environmental object at the current moment. Determining the second sub-distance of the second environmental object based on the predicted movement state and feature information includes: determining the altitude distance based on the predicted movement state and feature information; and determining the confidence distance based on the feature information of the second environmental object.

[0088] In this embodiment, if the predicted movement state and feature information are obtained, the height distance of the second environmental object is determined based on the predicted movement state and feature information; if the feature information of the second environmental object is obtained, the confidence distance of the second environmental object is determined based on the feature information of the second environmental object; wherein, the second sub-distance includes the height distance and the confidence distance.

[0089] Optionally, the height distance can be a distance metric for the degree of correlation between the predicted height of a first environmental object at the current moment and the actual height of a second environmental object, reflecting the difference in matching in the vertical spatial dimension.

[0090] Optionally, the confidence distance can be a distance metric for the reliability of detecting a second environmental object at the current moment. This distance reflects the confidence level of the detection algorithm in the existence and location accuracy of the current target.

[0091] Optionally, if the predicted movement state and feature information are obtained, the process of determining the height distance of the second environmental object based on the predicted movement state and feature information can be achieved by using the historical height information of the first environmental object contained in the predicted movement state and the current height information of the second environmental object contained in the feature information to calculate and quantify the difference between the two in the height dimension, thereby obtaining the height distance.

[0092] Optionally, the process of obtaining the detection confidence of the second environmental object at the current moment can be achieved by outputting a probability score of the existence of the target through an object detection algorithm. Specifically, preprocessed image data can be input into an object detection model. This model, while outputting the detection bounding box of the second environmental object, will also output a probability value representing the presence of a real target within that bounding box. This probability value is then determined as the detection confidence of the second environmental object at the current moment.

[0093] Optionally, if the feature information of the second environment object is obtained, the process of determining the confidence distance of the second environment object based on the feature information can be to use the detection confidence of the second environment object contained in the feature information to convert the detection confidence into a distance metric. Generally, the higher the detection confidence, the more reliable the detection result, and the smaller the corresponding confidence distance, thereby reflecting the impact of the reliability of the detection result on the association matching.

[0094] In this embodiment, the above method introduces lidar height information and detection confidence as weak cues to construct height distance and confidence distance, which can make up for the limitations of motion information and appearance features in occluded or feature-blurred scenarios. It enhances the robustness of association matching by utilizing physical constraints of vertical space dimension and detection reliability indicators, thereby effectively reducing the probability of mismatch and improving the stability and accuracy of multi-target tracking in complex autonomous driving scenarios.

[0095] As an optional implementation, the predicted movement state includes the height value of a first environmental object at any historical moment in history, and the feature information includes the height value of a second environmental object. Based on the predicted movement state and the feature information, determining the height distance includes: normalizing the height values ​​of the first and second environmental objects respectively to obtain normalized height values ​​of the first and second environmental objects; determining the difference between the normalized height values ​​of the first and second environmental objects; and determining the height distance based on the difference.

[0096] In this embodiment, if the height value of the first environmental object at any historical moment in the historical time and the height value of the second environmental object at the current moment are obtained, then these two height values ​​are normalized respectively to obtain the normalized height value of the first environmental object and the normalized height value of the second environmental object. If the normalized height value of the first environmental object and the normalized height value of the second environmental object are obtained, then the difference between them is determined, and the height distance is determined based on the difference.

[0097] Optionally, normalization can map the original height value to a preset standard range (e.g., [0,1] or [-1,1]). Here, a linear scaling formula can be used to convert the historical height value of the first environmental object and the current height value of the second environmental object into dimensionless relative values, thereby eliminating the influence of numerical magnitude caused by differences in sensor range or target physical size.

[0098] Optionally, the difference between the height value of the first environment object after normalization and the height value of the second environment object after normalization can be calculated as the absolute value of the difference between these two normalized values. Specifically, subtracting the height value of the second environment object from the height value of the first environment object after normalization and taking the absolute value of the result yields the degree of deviation between the two in the standardized height space.

[0099] Optionally, the height distance can be determined based on this difference, which can be directly used as the height distance. Since normalization ensures the values ​​are of the same order of magnitude, this difference directly reflects the vertical matching difference between the predicted and actual heights; the smaller the difference, the higher the degree of height correlation.

[0100] In this embodiment, the above method eliminates the dimensional differences of height data through normalization and determines the height distance using the difference in normalized height values. This can accurately quantify the matching degree between the first environmental object and the second environmental object in the vertical space dimension, and supplement the association strategy as a weak cue, thereby improving the tracking stability in scenarios with occlusion or failure of appearance features.

[0101] As an optional implementation, the feature information includes the detection confidence at the current moment. Based on the feature information of the second environmental object, the confidence distance is determined, including: determining the difference between the preset coefficient and the detection confidence as the confidence distance.

[0102] In this embodiment, if the detection confidence level of the second environmental object at the current moment is obtained, the difference between the preset coefficient and the detection confidence level is determined as the confidence distance.

[0103] Optionally, the detection confidence score can be a numerical value output by the target detection algorithm representing the reliability of the existence of the second environmental object at the current moment. This can be achieved by acquiring image data containing the second environmental object using an image acquisition device deployed on the vehicle, inputting the image data into the target detection algorithm, and obtaining the detection confidence score of the second environmental object from the algorithm's output layer. This value is typically between 0 and 1, with a higher value indicating more reliable detection.

[0104] Optionally, the preset coefficient can be 1. This preset coefficient is used to convert the detection confidence into a distance metric, so that the higher the detection confidence, the smaller the corresponding distance value, thereby reflecting a higher matching priority in association matching and ensuring that the detection results with high confidence have a greater influence in the association process.

[0105] Optionally, the process of determining the confidence distance as the difference between the preset coefficient and the detection confidence level can be achieved by quantifying the contribution of the detection confidence level to the association matching through arithmetic subtraction. Specifically, the detection confidence level can be subtracted from the preset coefficient (e.g., 1), and the calculated difference can be determined as the confidence distance. A smaller confidence distance indicates a higher detection confidence level, meaning a higher reliability of the detection result.

[0106] In the embodiments of this application, the above method determines the confidence distance by calculating the difference between the preset coefficient and the detection confidence. This can quantify the reliability of the detection result into a distance index, which can be used as a weak clue to supplement the association strategy. In scenarios where the target appearance features are fuzzy or the motion features are limited, the reliability of the detection can be used to help improve the accuracy of association matching and enhance the robustness of the multi-target tracking system.

[0107] As an optional implementation, determining association information based on a first sub-distance and a second sub-distance includes: assigning weight coefficients to the first sub-distance and the second sub-distance respectively, wherein the weight coefficient assigned to the first sub-distance is greater than the weight coefficient assigned to the second sub-distance; and fusing the first sub-distance assigned to the weight coefficient and the second sub-distance assigned to the weight coefficient to obtain association information.

[0108] In this embodiment, if the first sub-distance and the second sub-distance are obtained, weight coefficients are assigned to the first sub-distance and the second sub-distance respectively, wherein the weight coefficient corresponding to the first sub-distance is greater than the weight coefficient corresponding to the second sub-distance. Then, the weighted first sub-distance and the weighted second sub-distance are fused to obtain the final association information.

[0109] Optionally, the first sub-distance can be the movement distance and / or the appearance distance, used to characterize the degree of matching of strong cues. The movement distance characterizes the spatial overlap between the predicted location and the actual detection location, while the appearance distance characterizes the semantic similarity between historical appearance features and current appearance features.

[0110] Optionally, the second sub-distance can be the height distance and / or the confidence distance, used to characterize the degree of matching of weak cues. The height distance characterizes the matching difference in the vertical spatial dimension, while the confidence distance characterizes the reliability of the current detection result.

[0111] Optionally, the weighting coefficients can be numerical parameters used to adjust the proportion of contribution of different distance metrics to the final association result. Since the first sub-distance (e.g., movement, appearance) typically reflects the identity of the target more accurately than the second sub-distance (e.g., height, confidence), the weighting coefficient for the first sub-distance is set relatively large (e.g., 0.7 or 0.8), while the weighting coefficient for the second sub-distance is set relatively small (e.g., 0.3 or 0.2), and the sum of all weighting coefficients is 1.

[0112] Optionally, fusing the first sub-distance assigned to the weight coefficient and the second sub-distance assigned to the weight coefficient can be a weighted summation operation. Specifically, the first sub-distance can be multiplied by its corresponding weight coefficient to obtain a weighted first sub-distance, and the second sub-distance can be multiplied by its corresponding weight coefficient to obtain a weighted second sub-distance. Then, the weighted first sub-distance and the weighted second sub-distance are added together to obtain the association information (i.e., the total distance or total cost).

[0113] Optionally, the weighting coefficients can be assigned based on preset empirical values ​​or adaptively adjusted based on the scene. Specifically, in scenarios with significant occlusion, such as urban roads, the weight of confidence distance can be appropriately increased to leverage detection reliability for association assistance. In scenarios with intense movement, such as highways, the weight of movement distance can be increased to emphasize motion continuity.

[0114] In the embodiments of this application, the above method, by assigning different weight coefficients to the first sub-distance and the second sub-distance and fusing them, can reasonably balance the contributions of strong cues (motion, appearance) and weak cues (height, confidence) to the association results. This ensures the accuracy dominated by the main features in normal scenarios, while providing supplementary support by using weak cues when strong cues fail or become ambiguous. This improves the robustness and accuracy of multi-target tracking in complex autonomous driving scenarios.

[0115] As an optional implementation manner, determining the association result between the first environmental object and the second environmental object by using association information includes: invoking a global matching strategy, and associating the first environmental object and the second environmental object by using the association information to obtain the association result, wherein the global matching strategy is used to represent the rule that minimizes the total cost of associating the first environmental object and the second environmental object from the association information.

[0116] In this embodiment, if the association information is acquired, the global matching strategy is invoked to perform association matching on the first environmental object and the second environmental object based on the association information, so as to obtain a final association result. The global matching strategy aims to find a matching solution that minimizes the sum of association information (i.e., the total cost) between each matched pair.

[0117] Optionally, the association information may be a matching cost or a distance metric value between the first environmental object and the second environmental object. The association information may be the sum of the weighted first sub-distance and the weighted second sub-distance, and a smaller value indicates a higher possibility that the two objects are matched and a lower matching cost.

[0118] Optionally, the global matching strategy may be a matching algorithm based on optimization theory, such as the Hungarian Algorithm (abbreviated as HA). The strategy solves the linear assignment problem to find a set of unique matching pairs that minimize the total cost among all possible matching combinations, thereby avoiding tracking errors caused by local optimal matching.

[0119] Optionally, the association result may be a one-to-one mapping relationship between the first environmental object and the second environmental object. The association result includes which historical movement trajectories are successfully matched with which detection targets, which historical movement trajectories are not matched (may be lost), and which detection targets are new targets (not matched with any historical movement trajectories).

[0120] Optionally, the minimum total cost may be the minimization of the sum of the association information values of each matching pair. If there are N first environmental objects and M second environmental objects, the global matching strategy constructs a cost matrix, and selects N elements (if N<=M) or M elements (if M<N) therefrom, so that the sum of the selected elements is the smallest, and each object participates in at most one matching.

[0121] Optionally, the process of invoking the global matching strategy may be inputting the association information into an optimization solver for calculation. The first environmental objects may be used as nodes of one part of a bipartite graph, the second environmental objects may be used as nodes of the other part, the association information may be used as the weight of the connecting edge between the nodes, and a graph theory algorithm is used to find the minimum weight matching.

[0122] In this embodiment, the above method calls a global matching strategy and uses the associated information to perform a global optimal match on the first and second environment objects. This can overcome the problem that local greedy matching is prone to getting stuck in local optimal solutions. Especially in scenarios with dense targets, severe occlusion, or a large number of moving targets, the method ensures the consistency and accuracy of multi-target tracking by minimizing the total cost, and effectively reduces tracking identifier (ID) switching and tracking loss.

[0123] As an optional implementation, a global matching strategy is invoked, and association information is used to associate the first environment object and the second environment object to obtain the association result, including: constructing a bipartite graph with a first node set as nodes based on the historical movement trajectory of the first environment object and a second node set as nodes based on the second environment object, wherein the edge weights between nodes in the first node set and nodes in the second node set in the bipartite graph are used to represent the association information; determining the edges between nodes in the first node set and nodes in the second node set that satisfy the objective of minimizing the edge weights in the bipartite graph; and determining the association result based on the edges.

[0124] In this embodiment, if the historical movement trajectory of the first environmental object and the second environmental object are obtained, a bipartite graph containing nodes of these two types of objects is constructed, and the weights of the edges between nodes are set to represent the association information. Then, the bipartite graph is solved to find the matching edge set with the minimum sum of edge weights. Finally, the final association result is determined based on these selected edges.

[0125] Optionally, the first node set can be a set consisting of tracked historical movement trajectories. Each node in the first node set represents a historical movement trajectory of a first environment object. For example, there are N valid tracked historical movement trajectories existing at the current moment. Each historical movement trajectory includes a tracking identifier (ID), a predicted state, and historical features.

[0126] Optionally, the second node set can be a set consisting of currently detected targets. Each node in the second node set represents a second environment object detected at the current moment, such as M target bounding boxes detected in the current frame image or point cloud, and the corresponding feature information of the target bounding boxes.

[0127] Optionally, a bipartite graph can be a special graph structure where the set of vertices can be divided into two disjoint subsets, and each edge in the graph connects two vertices belonging to these two subsets respectively. Here, the first set of nodes and the second set of nodes serve as the two parts of the bipartite graph, and edges exist only between the first set of nodes and the second set of nodes; there are no edges within the same set.

[0128] Optionally, the edge weight can be a numerical value that characterizes the degree of association or matching cost between the first environment object and the second environment object. The edge weight can be a distance value calculated based on the association information (such as the sum of weighted movement distance, appearance distance, height distance and confidence distance). The smaller the value, the better the match, or the larger the value, the better the match (depending on the specific algorithm implementation, usually the minimum weight match corresponds to the minimum distance).

[0129] Optionally, satisfying the objective of minimizing the edge weights in the bipartite graph can be achieved by finding a matching scheme that minimizes the sum of the weights of all selected edges. This corresponds to the linear assignment problem or the minimum weight perfect matching problem (which typically introduces virtual nodes or allows partial matching when unmatched cases exist), aiming to globally optimally assign the correspondence between the first environment object and the second environment object.

[0130] Optionally, determining the edges between nodes in the first set of nodes and nodes in the second set of nodes can be achieved by solving for the minimum weight matching using graph theory algorithms. Specifically, combined optimization algorithms such as the Hungarian Algorithm (HA) or the Kuhn-Munkres Algorithm (KM) can be used to compute the set of edges that minimizes the total edge weight in polynomial time.

[0131] Optionally, determining the association result based on the edge can be done by considering the first environment object node and the second environment object node connected by the selected edge as a matching pair. Specifically, if the edge between node A (the first environment object) and node B (the second environment object) is selected, it is determined that the historical movement trajectory corresponding to node A and the detection target corresponding to node B are successfully matched, thereby establishing a unique one-to-one mapping relationship.

[0132] In this embodiment, the method models the multi-target tracking association problem as a bipartite graph minimum weight matching problem, enabling the efficient and accurate solution of the globally optimal matching scheme using mature graph theory algorithms. Compared to local optimization methods such as greedy matching, this method effectively avoids erroneous associations caused by local optimal choices, especially in complex scenarios with dense targets, mutual occlusion, or intersecting historical movement trajectories, thus improving the accuracy and stability of multi-target tracking.

[0133] As an optional implementation, updating the historical movement trajectory of the first environmental object based on the association results and the feature information of the second environmental object includes: classifying the historical movement trajectory of the first environmental object and the second environmental object respectively according to the association results to obtain the classification results of the historical movement trajectory and the classification results of the second environmental object; and updating the historical movement trajectory based on the classification results of the historical movement trajectory, the classification results of the second environmental object, and the feature information.

[0134] In this embodiment, if an association result is obtained, the historical movement trajectory of the first environmental object is divided into two categories: matched or unmatched, and the second environmental object is also divided into two categories: matched or unmatched. Subsequently, based on the classified categories and the currently detected feature information, corresponding historical movement trajectory update operations are performed, such as resetting the counter, maintaining the state, creating a new historical movement trajectory, or deleting an invalid historical movement trajectory.

[0135] Optionally, the classification result can be a category label for the object based on the association matching status. Specifically, for the first environment object, the classification result includes "matched" (i.e., the current frame is successfully associated with the detection target) and "not matched" (i.e., the current frame is not associated with any detection target); for the second environment object, the classification result includes "matched" and "not matched".

[0136] Optionally, the matched historical movement trajectory can be a historical movement trajectory that has successfully established a unique correspondence with a second environmental object at the current moment. Specifically, when the association result shows that the association cost between a certain historical movement trajectory and a certain second environmental object is minimal and meets the matching threshold, the historical movement trajectory is marked as "matched", indicating that the target has been effectively observed in the current frame.

[0137] Optionally, the unmatched historical movement trajectory can be a historical movement trajectory that has not established a corresponding relationship with any second environmental object at the current moment. Specifically, when the association results show that a certain historical movement trajectory fails to match any second environmental object, or is removed due to excessive matching cost, the historical movement trajectory is marked as "unmatched," which may mean that the target is occluded, disappeared, or missed by the detection algorithm.

[0138] Optionally, the matched second environmental object can be a second environmental object that has successfully established a unique correspondence with the historical movement trajectory of the first environmental object at the current moment. Specifically, when the association result shows that a second environmental object has successfully paired with a historical movement trajectory, the second environmental object is marked as "matched," indicating that the detection result has been included in the tracking framework.

[0139] Optionally, the unmatched second environmental object can be a second environmental object that has not established a corresponding relationship with any first environmental object's historical movement trajectory at the current moment. Specifically, when the association results show that a second environmental object fails to match any existing historical movement trajectory, that second environmental object is marked as "unmatched" and is typically considered a potential new target or a false detection target.

[0140] Optionally, updating the historical movement trajectories can involve executing specific state maintenance logic based on the classification results. The update operation includes: for matched historical movement trajectories, resetting the historical movement trajectory lifecycle counter and integrating it with current observation data; for unmatched but continuously moving historical movement trajectories, decrementing the historical movement trajectory lifecycle counter; and for unmatched second environmental objects that meet specific conditions (such as high confidence or continuous occurrence), creating new historical movement trajectories.

[0141] In this embodiment, the method described above effectively manages the lifecycle of targets by performing fine-grained classification of historical movement trajectories and detected targets based on association results and adopting differentiated update strategies for different categories of objects. This method not only prevents tracking identifier (ID) switching by maintaining the lifecycle of unmatched historical movement trajectories when targets are occluded, but also promptly creates or restores historical movement trajectories when targets reappear or new targets appear, thereby improving the continuity and robustness of multi-target tracking systems in complex dynamic scenarios.

[0142] As an optional implementation, based on the association results, the historical movement trajectory of the first environmental object and the second environmental object are classified respectively to obtain the classification results of the historical movement trajectory and the classification results of the second environmental object, including: based on the association results, determining that the classification result of the historical movement trajectory includes at least one of the following: a historical movement trajectory that has been matched at the current time, or a historical movement trajectory that has not been matched at the current time; based on the association results, determining that the classification result of the second environmental object includes at least one of the following: an unmatched second environmental object, or a matched second environmental object.

[0143] In this embodiment, if an association result is obtained, the historical movement trajectory of the first environmental object is traversed, and the historical movement trajectory is divided into matched or unmatched categories based on whether an association is established with the second environmental object at the current moment; at the same time, the second environmental object is traversed, and the second environmental object is divided into matched or unmatched categories based on whether an association is established with the historical movement trajectory of the first environmental object at the current moment.

[0144] Optionally, the process of determining the classification results described above can be an operation of label assignment based on an association matching matrix. An association matrix can be constructed, where rows represent historical movement trajectories, columns represent second environment objects, and matrix elements are association costs. For the minimum cost element selected in the matrix, the row (historical movement trajectory) and column (second environment object) corresponding to the minimum cost element are marked as "matched," unselected rows are marked as "unmatched historical movement trajectories," and unselected columns are marked as "unmatched second environment objects."

[0145] In this embodiment, the method described above clearly classifies historical movement trajectories and secondary environmental objects based on the association results, enabling a clear distinction between tracked targets, lost targets, and newly created targets. This classification mechanism provides clear data support for subsequent differentiated lifecycle management (such as resetting counters, decrementing counters, or creating new historical movement trajectories) for different categories of objects, thereby effectively addressing common issues in autonomous driving scenarios such as target occlusion, disappearance, and the appearance of new targets, thus improving the stability and accuracy of the multi-target tracking system.

[0146] As an optional implementation, the historical movement trajectory is updated based on the classification results of the historical movement trajectory, the classification results of the second environmental object, and feature information, including at least one of the following: resetting the lifecycle state of the matched historical movement trajectory; determining the lifecycle state of the unmatched historical movement trajectory based on the movement continuity of the unmatched historical movement trajectory; updating the lifecycle counter of the unmatched historical movement trajectory in response to the lifecycle state of the unmatched historical movement trajectory meeting the maintenance condition; deleting the unmatched historical movement trajectory in response to the lifecycle state of the unmatched historical movement trajectory not meeting the maintenance condition; determining the detection confidence and continuous occurrence state of the unmatched second environmental object from the feature information; and creating a historical movement trajectory corresponding to the unmatched second environmental object and updating the lifecycle counter of the created historical movement trajectory in response to the moment when the detection confidence is greater than or equal to the confidence threshold and the continuous occurrence state indicates that the unmatched second environmental object has continuously appeared for a first target number.

[0147] In this embodiment, if classification results and feature information are obtained, a status refresh operation is performed on the matched objects; for unmatched historical movement trajectories, it is checked whether the historical movement trajectory meets the maintenance conditions, and if so, a maintenance operation is performed; for unmatched detection objects, it is checked whether the detection confidence of the detection object exceeds the confidence threshold and continues to appear, and if so, it is further checked whether the current number of targets is within the allowable range, and if it is within the range, a new historical movement trajectory is created, thereby completing the update of all historical movement trajectories.

[0148] Optionally, updating the matched historical movement trajectories at the current moment can be achieved by resetting the lifecycle counter and fusing the current observation data. Specifically, for the matched historical movement trajectories, the lifecycle counter is reset to its maximum value (e.g., T=5), indicating that the target is active. Simultaneously, using the detection boxes and velocity information in the current feature information, the position and velocity estimates of the historical movement trajectories are updated through Kalman filtering, and the appearance feature library is updated.

[0149] Optionally, the maintenance condition can be that the prediction error of the unmatched historical movement trajectory is less than a preset threshold and the motion state is continuous. The maintenance condition is used to determine whether the unmatched historical movement trajectory still represents a real target (e.g., lost due to temporary occlusion). If the prediction error (e.g., center point offset) is ≤2 pixels and the velocity direction does not change abruptly, the maintenance condition is satisfied, and the historical movement trajectory is allowed to continue to exist.

[0150] Optionally, updating the unmatched historical movement trajectories at the current moment can be achieved by adjusting the lifecycle counter based on maintenance conditions. Specifically, if the unmatched historical movement trajectory meets the maintenance conditions, the lifecycle counter is decremented by 1, allowing it to continue existing for a certain period to cope with temporary occlusion; if the maintenance conditions are not met, it may be directly marked as disappeared.

[0151] Optionally, the detection confidence score can be the probability value output by the target detection algorithm that the target actually exists. The detection confidence score quantifies the reliability of the detection result; a higher value indicates that the detection box is more likely to be a real target, and it typically ranges from 0 to 1.

[0152] Optionally, the confidence threshold can be a preset probability lower limit for screening valid new targets. The confidence threshold is used to prevent false detections of noisy historical movement trajectories. For example, if the confidence threshold is set to 0.7, the detection result is considered reliable only when the detection confidence is ≥0.7, and then the creation of a new historical movement trajectory is considered.

[0153] Optionally, the number of targets can be the total number of valid historical motion trajectories currently being maintained in the system. The number of targets reflects the current load on the tracking system and is used to limit the upper limit of targets tracked simultaneously, preventing the exhaustion of computing resources or degradation of tracking performance due to an excessive number of targets.

[0154] Optionally, creating a new historical movement trajectory can initialize a new historical movement trajectory record containing current detection information. This involves assigning a new unique tracking identifier (ID) to the unmatched second environment object, initializing a lifecycle counter, and storing the second environment object's current detection bounding box, appearance features, detection confidence, and height information as the initial state in the historical movement trajectory database. This creation operation is only performed when the detection confidence is greater than or equal to a confidence threshold, stable detection occurs across multiple consecutive frames, and the current number of targets is less than a preset maximum tracking limit, to avoid invalid historical movement trajectories consuming computing resources or overloading the system.

[0155] Optionally, the process of determining whether the number of targets is less than the preset maximum tracking limit can be performed by reading the historical movement trajectory count value in the system's global variables. Specifically, the system maintains a global counter, incrementing the count by 1 each time a new historical movement trajectory is created or a historical movement trajectory is successfully maintained, and decrementing the count by 1 each time a historical movement trajectory is deleted. Before performing a creation operation, the count value is compared with a preset threshold (e.g., 100). If the count value is less than the threshold, creation is allowed; otherwise, the unmatched target is discarded to ensure system stability.

[0156] Optionally, the preset maximum tracking limit can be the maximum number of concurrent tracking determined based on the hardware computing power. If the hardware computing power is limited, the upper limit of the target number may be set lower (e.g., 50), while if the computing power is sufficient, the upper limit may be set higher (e.g., 200) to adapt to different levels of autonomous vehicles.

[0157] In this embodiment, the method distinguishes between matched and unmatched historical movement trajectories and unmatched detected objects, and performs update, maintenance, and creation operations respectively, achieving adaptive dynamic lifecycle management. This mechanism can prevent tracking identifier (ID) switching by maintaining unmatched historical movement trajectories when the target is occluded, avoid false detections that generate invalid historical movement trajectories through strict confidence filtering, and quickly respond to the entry of new targets, thus improving the stability and robustness of multi-target tracking in complex autonomous driving scenarios.

[0158] As an optional implementation, the method further includes: deleting the historical movement trajectory in response to at least one of the following conditions: the lifecycle counter of the historical movement trajectory is reduced to a counting threshold; the time when the historical movement trajectory continuously appears for a second target number does not match the second environmental object, and the error in determining the predicted movement state using the historical movement trajectory is greater than an error threshold.

[0159] In this embodiment, if an unmatched historical movement trajectory is obtained at the current moment, the prediction error of the historical movement trajectory at the current moment is first calculated. Then, the prediction error is compared with a counting threshold. If the prediction error exceeds the counting threshold, it indicates that the target's motion state is abnormal or has deviated, and the historical movement trajectory is directly deleted to release resources. If the prediction error is within the counting threshold range, it indicates that the target may be under normal occlusion or detection fluctuations, and the historical movement trajectory is maintained by decrementing the lifecycle counter, thereby balancing the continuity and accuracy of tracking.

[0160] Optionally, the prediction error can be a measure of the deviation between the predicted position of the historical trajectory at the current moment and the actual observed position at the current moment (or the effective position in the previous frame). Specifically, the prediction error can be quantified by calculating the Euclidean distance between the center coordinates of the predicted bounding box and the center coordinates of the actual detected bounding box, or by using the trace of the prediction covariance matrix in Kalman filtering, reflecting the degree of agreement between the target's motion state and the prediction model.

[0161] Optionally, the counting threshold can be an upper limit of error used to determine whether unmatched historical movement trajectories should be deleted due to prediction failure. The counting threshold sets a maximum permissible spatial deviation range, such as 2 pixels or 0.5 meters. When the prediction error exceeds this value, it is considered that the target's motion pattern has abruptly changed or the prediction has completely failed, making it unsuitable for maintenance through simple lifecycle reduction, and should be decisively deleted to avoid erroneous association.

[0162] Optionally, determining whether the prediction error is greater than the counting threshold can be achieved by comparing the calculated prediction error value with a preset counting threshold value. Specifically, if the prediction error is 1.8 pixels and the counting threshold is 2 pixels, then 1.8 < 2, and is determined to be no greater than; if the prediction error is 2.5 pixels, then 2.5 > 2, and is determined to be greater than.

[0163] Optionally, deleting the unmatched historical movement trajectory at the current moment can be done by removing the historical movement trajectory record from the system's historical movement trajectory library and releasing the memory resources occupied by the historical movement trajectory record. The deletion operation includes marking the tracking identifier (ID) of the historical movement trajectory as invalid, removing the historical movement trajectory from the list of active historical movement trajectories, and optionally archiving the historical movement trajectory to a historical log to prevent invalid historical movement trajectories from occupying computing resources for a long time or affecting subsequent target association matching.

[0164] Optionally, decrementing the lifecycle counter of the unmatched historical movement trajectory at the current moment can be done by decrementing an integer counter maintained within the historical movement trajectory by 1. The initial value of the lifecycle counter may be 5. Each time a frame of unmatched prediction errors occurs within a counting threshold, the counter is decremented by 1. When the counter reaches 0, it is further determined whether to delete the historical movement trajectory, thereby achieving more refined control over the maintenance of the historical movement trajectory.

[0165] Optionally, if the prediction error is less than or equal to the counting threshold, the logic for maintaining the existence of the historical movement trajectory can be to assume that the target is in temporary occlusion or detection noise. This logic is based on the common characteristics of autonomous driving scenarios, namely that it is normal for the target to be temporarily occluded. As long as the predicted position does not deviate drastically (i.e., the prediction error ≤ the counting threshold), it is assumed that the target is still nearby, and a certain "tolerance time" is given to wait for the next detection and matching.

[0166] Optionally, the prediction error may further include velocity prediction error. In addition to position deviation, the prediction error may also incorporate the difference between the predicted velocity and the detected velocity. If the velocity difference exceeds a preset range, it can be considered a prediction failure, triggering a deletion operation. Alternatively, when calculating the overall error, the position error and velocity error can be weighted and summed before being compared with a counting threshold.

[0167] Optionally, the counting threshold can be dynamically adjusted according to different autonomous driving scenarios. In scenarios such as urban roads where target movement is complex and frequent starts and stops are common, the counting threshold can be set to a larger value (e.g., 3 pixels) to tolerate more motion uncertainty; while in scenarios such as highways where target movement is relatively linear, the counting threshold can be set to a smaller value (e.g., 1 pixel) to improve sensitivity to prediction deviations, thereby more quickly eliminating erroneous historical movement trajectories.

[0168] In this embodiment, the method introduces a counting threshold as a key indicator for determining the retention or removal of unmatched historical movement trajectories, enabling intelligent differentiation between "temporary disconnection due to occlusion" and "true disconnection due to prediction failure or target disappearance." When the prediction error is within the counting threshold, the historical movement trajectory is maintained by decrementing the counter, effectively addressing the common target occlusion problem in autonomous driving scenarios and reducing tracking identifier (ID) switching. When the prediction error exceeds the counting threshold, the historical movement trajectory is promptly deleted, avoiding interference from invalid historical movement trajectories on subsequent tracking tasks and improving the overall tracking accuracy and resource utilization efficiency of the system.

[0169] As an optional implementation, based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period, the predicted movement state of the first environmental object at the current moment is predicted, including: generating a preliminary predicted movement state of the first environmental object at the current moment through a linear dynamics prediction model based on the movement state sequence in the historical movement trajectory; and extracting temporal features from the movement state sequence through a nonlinear temporal modeling network to generate a movement state correction vector of the first environmental object; and fusing the movement state correction vector with the preliminary predicted movement state to obtain the predicted movement state.

[0170] In this embodiment, if the historical movement trajectory of the first environmental object is obtained, a preliminary linear motion prediction is first performed using a linear dynamics prediction model to obtain a preliminary predicted movement state. Subsequently, historical appearance features are extracted and combined with the preliminary predicted state to construct a temporal feature sequence. The temporal feature sequence is then input into a nonlinear temporal modeling network to capture nonlinear motion patterns. Finally, the motion correction vector output by the network is fused with the preliminary prediction result to obtain a more accurate predicted movement state.

[0171] Optionally, the linear dynamics prediction model can be a linear state estimation model based on the laws of physical motion. This linear dynamics prediction model can be a Kalman filter (KF) or an extended Kalman filter. The model assumes that the target's motion in a short time follows a uniform or uniformly accelerated linear motion law, and uses state transition equations and measurement update equations to linearly predict and correct the target state.

[0172] Optionally, the preliminary predicted movement state can be the future state of the target, output by a linear dynamics prediction model and based solely on linear motion assumptions. The preliminary predicted movement state includes the predicted center coordinates, predicted velocity, predicted length and width, and predicted covariance matrix, reflecting the target's theoretical historical movement trajectory under the assumption of no complex maneuvers.

[0173] Optionally, historical appearance features can be target appearance representation data recorded in historical movement trajectories. Specifically, historical appearance features can be high-dimensional feature vectors obtained by processing image regions of the target captured by an image acquisition device at historical moments using a feature extraction network (such as ResNet), used to characterize the target's visual appearance attributes.

[0174] Optionally, the temporal feature sequence can be a sequence of data formed by concatenating or fusing historical appearance features with the preliminary predicted movement state in the time dimension. Specifically, the temporal feature sequence can be formed by concatenating the historical appearance feature vectors of the past N frames with the corresponding preliminary predicted movement state parameters to create a multi-dimensional time series tensor, which is then input into the temporal modeling network.

[0175] Optionally, the nonlinear temporal modeling network can be a deep learning network capable of capturing long-term dependencies and nonlinear relationships in time-series data. Specifically, the nonlinear temporal modeling network can be a Gated Recurrent Unit (GRU), a Long Short-Term Memory (LSTM) network, or a GRU network with an attention mechanism. It can adaptively focus on historical keyframes and learn the nonlinear motion patterns of the target in complex scenes.

[0176] Optionally, the motion correction vector can be a feature vector output by a nonlinear temporal modeling network used to correct linear prediction bias. Here, the motion correction vector is a vector with the same or related dimension as the state vector, containing prediction error information caused by nonlinear motion, used to compensate for deficiencies in the initial prediction of the moving state.

[0177] Optionally, fusing the motion correction vector with the preliminary predicted movement state can be achieved by performing vector addition or weighted fusion. The fusion process can involve directly superimposing the motion correction vector onto the preliminary predicted movement state, or by using a learnable linear layer to perform a weighted sum of the two, thereby obtaining the final predicted target state, i.e., the predicted movement state.

[0178] Optionally, the predicted motion state can be information that more accurately reflects the target's current and future motion state after nonlinear correction. The predicted motion state includes corrected parameters such as center coordinates, velocity, length, and width. Compared to the initial predicted motion state, it is better suited to complex nonlinear motion scenarios such as pedestrians crossing or vehicles changing lanes.

[0179] In this embodiment, the method combines the linear prediction capability of a linear dynamics prediction model with the nonlinear feature extraction capability of a nonlinear temporal modeling network to achieve accurate prediction of the target's motion state. The linear dynamics prediction model provides a stable basic prediction, while the nonlinear temporal modeling network captures long-term temporal dependencies and outputs correction vectors for complex motions. This fusion mechanism effectively overcomes the problem of large prediction bias in traditional single linear models when the target undergoes nonlinear motion, improving the prediction accuracy and robustness of multi-target tracking in complex autonomous driving scenarios.

[0180] As an optional implementation, the nonlinear temporal modeling network is a gated recurrent unit network based on an attention mechanism. The nonlinear temporal modeling network extracts temporal features from the movement state sequence to generate a movement state correction vector for the first environment object. This includes: inputting a preset number of historical movement state sequences from the historical movement trajectory into the gated recurrent unit network; using the gated recurrent unit to encode long-term temporal features of the movement state sequences to generate a hidden state sequence containing long-term temporal movement dependencies; assigning weights to the hidden state sequences using an attention mechanism; and fusing the weighted hidden state sequences to obtain attention-weighted movement features; and determining the attention-weighted movement features as the movement state correction vector.

[0181] In this embodiment, if a preset number of historical motion state sequences are obtained from the historical motion trajectory, the motion state sequences are input into a gated recurrent unit network for long-term feature encoding to generate a hidden state sequence. Subsequently, the hidden state sequences are weighted and fused using an attention mechanism to extract key motion features. Finally, the attention-weighted motion features are directly used as motion state correction vectors for subsequent fusion with the preliminary predicted motion state.

[0182] Optionally, the preset number can be the length of the historical frames used to construct the temporal input sequence. The preset number is a fixed positive integer, such as 5, 10, or 20, which determines the time span of the historical data used in nonlinear temporal modeling, i.e., how many frames of data the model looks at in the past to predict the current state.

[0183] Optionally, the motion state sequence of a preset number of historical moments in the historical motion trajectory can be a set of motion state data extracted from the target over the past N consecutive frames. Where the preset number is N, the sequence includes [ ,..., , There are N state data points at a time, where S represents the movement state at a certain time (such as center coordinates, velocity, length and width), forming a fixed-length input tensor.

[0184] Optionally, the gated recurrent unit network can be a recurrent neural network architecture capable of capturing long-term dependencies in sequential data. The gated recurrent unit (GRU) controls the flow of information by updating and resetting gates, effectively solving the gradient vanishing problem in traditional recurrent neural networks (RNNs), and is suitable for processing time-dependent sequential data such as the historical movement trajectories of vehicles and pedestrians.

[0185] Optionally, long-term temporal feature encoding can progressively pass and fuse time step information in the input sequence to capture motion patterns spanning multiple time steps. Specifically, a gated recurrent unit network processes each frame of data in a preset number of motion state sequences layer by layer, combining the hidden state of the previous moment with the input of the current moment, thereby encoding the long-term motion pattern of the target in the deep hidden state of the network.

[0186] Optionally, the hidden state sequence can be a set of hidden state vectors output by the gated recurrent unit network after processing the data at each time step in the sequence. The hidden state sequence has a length equal to a preset number and contains multiple hidden state vectors of the same length as the input sequence. Each vector represents the network's understanding and memory of the target's motion state up to the current time step.

[0187] Optionally, long-term temporal motion dependencies can be non-linear correlations between the motion states at historical moments and the current prediction. These dependencies might manifest as the target's lane-changing behavior several frames ago affecting the current position prediction, or the target's continuous acceleration trend being reflected in the historical velocity sequence. These patterns are revealed through the hidden state sequence, and their capture range is limited by a preset number.

[0188] Alternatively, the attention mechanism can be a weighted mechanism that allows the model to dynamically focus on the importance of different time steps when processing sequences. Specifically, the attention mechanism assigns different weights to different historical moments by calculating the importance score of each hidden state vector, enabling the model to focus on the keyframes that contribute most to the current prediction.

[0189] Optionally, weighting can be done by assigning a corresponding weight coefficient to each hidden state vector in the hidden state sequence based on its importance score. Specifically, the attention calculation module generates a weight vector with the same length as the hidden state sequence (i.e., a preset number), where a larger weight value indicates a greater contribution of historical information at that moment to the current correction vector.

[0190] Optionally, fusion can be a weighted summation or weighted average operation on the hidden state sequence according to the assigned weights. The fusion process involves multiplying each hidden state vector by its corresponding weight coefficient, then summing the weighted vectors to obtain a fixed-dimensional vector that integrates information from a preset number of historical moments.

[0191] Optionally, the attention-weighted motion feature can be a single vector after weighted fusion through an attention mechanism, used to represent the target's key motion intentions. The attention-weighted motion feature is a high-dimensional vector that condenses the core dynamic information from a predetermined number of historical motion state sequences, such as the rate of change of motion direction and acceleration / deceleration trends.

[0192] Optionally, the movement state correction vector can be a bias vector used to adjust the initial predicted movement state. The dimension of the movement state correction vector matches the state vector in the initial predicted movement state. By mapping the attention-weighted movement features to this dimension, the correction vector is obtained and directly superimposed on the initial prediction result.

[0193] Optionally, the process of determining the motion state correction vector may also include dimension mapping. The attention-weighted motion features may have a different dimension than the original state vector. A fully connected layer can be used to map the attention-weighted motion features to the same dimensional space as the initially predicted motion state, ensuring that the two can be directly added and fused.

[0194] In this embodiment, the method introduces an attention-based gated recurrent unit (GRU) network. This not only leverages the GRU's ability to capture long-term temporal motion dependencies but also adaptively focuses on key historical frames through an attention mechanism. This mechanism enables the motion state correction vector to accurately reflect the nonlinear motion characteristics of the target in complex scenes (such as sharp turns and sudden braking), thus providing more targeted corrections for the initial motion state prediction. This improves the accuracy of motion state prediction and enhances the robustness of multi-target tracking in occluded or complex motion scenarios.

[0195] As an optional implementation, the motion state correction vector is fused with the preliminary predicted motion state to obtain the predicted motion state, including: superimposing the motion state correction vector onto the state vector of the preliminary predicted motion state to obtain the predicted motion state, wherein the state vector of the preliminary predicted motion state includes at least one of the following: the center coordinates of the first environmental object at the current time, the movement speed of the first environmental object at the current time, and the size parameter of the first environmental object at the current time.

[0196] In this embodiment, if a motion state correction vector and a preliminary predicted motion state state vector are obtained, the motion state correction vector is directly superimposed on the preliminary predicted motion state state vector to obtain the final predicted motion state; wherein, the preliminary predicted motion state state vector includes at least one of the center coordinates, movement speed, or size parameters of the first environmental object at the current moment.

[0197] Optionally, the state vector can be a multidimensional data set describing the target's motion state and geometric properties in three-dimensional space. The state vector typically includes center coordinates. velocity components , acceleration component And the length of the goal ,Width ,high These parameters are used to fully characterize the spatial position and motion characteristics of the target.

[0198] Optionally, superimposing the movement state correction vector onto the state vector of the preliminary predicted movement state can be performed by performing a vector addition operation. Specifically, each element in the movement state correction vector is added to the corresponding element in the state vector of the preliminary predicted movement state; for example, the corrected center coordinates. ,in This is the coordinate correction value corresponding to the correction vector.

[0199] Optionally, the center coordinates of the first environmental object at the current moment can be the geometric center position of the target in three-dimensional space. Here, the center coordinates can be the geometric center point of the target's bounding box or leading edge, typically determined by... It consists of three coordinate values, used to indicate the specific location of the target in the vehicle coordinate system or the world coordinate system.

[0200] Optionally, the moving speed of the first environmental object at the current moment can be the target's rate of motion and direction relative to a reference frame (such as a vehicle or the ground). The moving speed is typically determined by... These represent the instantaneous velocities of the target along the X, Y, and Z axes, respectively, reflecting the speed and direction of the target's motion.

[0201] Optionally, the size parameter of the first environmental object at the current moment can be the geometric volume attribute of the target. The size parameter includes the target's length, width, and height, which describes the target's physical size in space. This is crucial for distinguishing different types of targets (such as cars and trucks) and calculating the intersection-over-union (IOU) ratio.

[0202] Optionally, the state vector includes at least one of the following: center coordinates, movement speed, and size parameters. This means that the fusion operation can correct only some state dimensions. Specifically, if only position prediction needs to be corrected, only the coordinate portion of the state vector can be superimposed; if the motion trend needs to be corrected, the speed portion can be superimposed. This flexibility allows the correction strategy to be adjusted according to different scenario requirements.

[0203] Optionally, the dimension of the movement state correction vector must be consistent with the dimension of the state vector of the initially predicted movement state. Specifically, before performing vector addition, it must be ensured that both have the same dimensional structure. For example, if the state vector is 8-dimensional, then the movement state correction vector should also be 8-dimensional to guarantee the validity of element-wise addition.

[0204] Optionally, the superposition operation may further include weighted fusion. In addition to direct addition, a scaling factor (such as learned weights) can be introduced. The moving state correction vector is multiplied by this factor before being superimposed onto the initial predicted moving state vector. This controls the magnitude of the nonlinear correction and prevents over-correction from causing prediction oscillations.

[0205] In this embodiment, the method described above achieves efficient fusion of the target motion state by directly superimposing the motion state correction vector output by the nonlinear temporal modeling network onto the preliminary predicted motion state output by the linear dynamics prediction model. This simple vector superposition method not only has low computational cost and is easy to deploy in real time on embedded devices, but also effectively combines the stability of the linear model with the nonlinear model's ability to capture complex motion patterns, improving the accuracy of the predicted motion state when the target undergoes nonlinear motions such as sharp turns and lane changes, thereby enhancing the overall performance of multi-target tracking.

[0206] As an optional implementation, the method further includes: in response to the identification of at least one second environmental object located around the vehicle at the current moment, acquiring image data and point cloud data containing the second environmental object at the current moment; and extracting feature information of the second environmental object at the current moment from the image data and point cloud data.

[0207] In this embodiment, after determining the association information based on the feature information of the predicted mobile device and the second environmental object at the current moment, the feature information of the second environmental object can be obtained through the following steps: identifying the second environmental object around the vehicle at the current moment, and acquiring image data and point cloud data containing the second environmental object. The feature information of the second environmental object can be extracted from the above image data and point cloud data.

[0208] Optionally, by detecting perception data around the vehicle, identifying at least one second environmental object, and triggering a data acquisition command, image data and point cloud data containing the second environmental object at the current moment are obtained. The image data can be a two-dimensional pixel matrix containing visual information about the vehicle's surrounding environment. The image data can be a color image or a grayscale image, recording visual features such as lighting, color, and texture in the scene to support subsequent appearance recognition and two-dimensional position detection.

[0209] Optionally, point cloud data can be a set of three-dimensional spatial coordinates containing the vehicle's surrounding environment. The point cloud data consists of a large number of points with spatial locations, which can accurately describe the geometry, distance, and height of objects in the scene, supporting subsequent target 3D localization and volume estimation.

[0210] Optionally, the process of detecting perception data around the vehicle and identifying at least one second environmental object can be achieved by determining the presence of a dynamic or static entity through sensor data fusion or a single-modal detection algorithm. Specifically, perception devices can be used to confirm the second environmental object around the vehicle at the current moment.

[0211] Optionally, the process of triggering a data acquisition command to obtain image data and point cloud data containing the second environmental object at the current moment can be achieved by simultaneously activating multiple sensors based on the recognition result and capturing data frames at the corresponding time. Specifically, when the recognition module outputs a signal indicating the presence of a second environmental object, the control system sends a synchronization trigger signal to the sensing device to ensure that the acquired image frames and point cloud frames are strictly aligned in timestamps and both contain the identified second environmental object, thus guaranteeing data consistency for subsequent feature extraction.

[0212] In the embodiments of this application, the above method achieves efficient utilization of computing resources by first identifying the object and then acquiring data in a targeted manner; at the same time, by simultaneously acquiring image data and point cloud data, it makes up for the lack of information dimension of a single sensor (such as the lack of depth information in images and the lack of texture information in point clouds), providing rich and complementary data support for subsequent accurate multi-target tracking and association, and improving the perception robustness of the system in complex environments.

[0213] As an optional implementation, acquiring image data and point cloud data containing a second environmental object at the current moment includes: at the current moment, using an image acquisition device deployed on the vehicle to acquire initial image data of the environment surrounding the vehicle, and using a radar device deployed on the vehicle to acquire initial point cloud data of the environment surrounding the vehicle, wherein the image content of the initial image data includes at least one second environmental object, and the point cloud content of the initial point cloud data includes at least one second environmental object; performing spatiotemporal calibration processing on the initial image data and the initial point cloud data to obtain calibrated initial image data and calibrated initial point cloud data; and performing denoising processing on the calibrated initial image data and the calibrated initial point cloud data respectively to obtain image data and point cloud data.

[0214] In this embodiment, if it is necessary to obtain the target data at the current moment, the original initial image data and initial point cloud data are first obtained simultaneously using the image acquisition device and the radar device, respectively, to ensure that both contain the second environmental object to be tracked. Then, the two heterogeneous data are subjected to spatiotemporal calibration processing of timestamp alignment and spatial coordinate transformation to eliminate synchronization deviation and position misalignment between sensors. Finally, the calibrated data is filtered and denoised to remove environmental noise and sensor interference, resulting in high-quality image data and point cloud data that can be used for feature extraction.

[0215] Optionally, the image acquisition device can be an optical sensor used to capture the visual scene around the vehicle. Specifically, the image acquisition device can be an in-vehicle high-definition camera capable of acquiring color or grayscale images of targets such as roads, pedestrians, and vehicles at a specific frame rate (e.g., 30 FPS) and resolution (e.g., 1920×1080), the image content including two-dimensional visual information of the targets. The in-vehicle high-definition camera can be a complementary metal-oxide-semiconductor image sensor (CMOS image sensor) or a charge-coupled device image sensor (CCD image sensor).

[0216] Optionally, the radar equipment can be an electromagnetic wave or laser emission and reception device used to detect the three-dimensional spatial distribution of surrounding objects. The radar equipment can be a lidar or millimeter-wave radar, which generates three-dimensional point cloud data containing target distance, angle, and reflection intensity by emitting laser beams or electromagnetic waves and receiving reflected signals. The point cloud content includes the target's geometry and spatial location.

[0217] Optionally, the initial image data can be raw visual image frames that have not undergone spatiotemporal alignment and denoising processing. The initial image data records the raw pixel information of the environment surrounding the vehicle, which may be affected by changes in illumination, sensor noise, or motion blur, but contains the raw visual features of the second environmental objects, such as appearance texture and two-dimensional contours.

[0218] Optionally, the initial point cloud data can be a raw 3D point set that has not undergone spatiotemporal alignment and denoising processing. The initial point cloud data consists of a large number of discrete 3D coordinate points, recording the geometry of the surrounding environment. It may contain ground noise, vegetation disturbance, or random sensor noise, but it contains the raw 3D spatial information of the second environmental object, such as its height and volume.

[0219] Optionally, spatiotemporal calibration can be a process of unifying data collected by different sensors to the same time reference and spatial coordinate system. Spatial calibration includes projecting the radar point cloud onto the image coordinate system or mapping image pixels to the point cloud coordinate system using an extrinsic parameter matrix determined by hand-eye calibration. Temporal calibration includes using high-precision clock synchronization or interpolation algorithms to ensure that image frames and point cloud frames correspond to the same physical moment, eliminating spatiotemporal misalignment caused by vehicle movement.

[0220] Optionally, the calibrated initial image data can be image data aligned after spatial coordinate transformation. After spatial calibration, the pixel positions in the image correspond precisely to the 3D points in the point cloud data, ensuring that the target region in the image can be accurately mapped to the corresponding point set in the point cloud, thereby supporting the fusion of multimodal information.

[0221] Optionally, the calibrated initial point cloud data can be point cloud data aligned after spatial coordinate transformation. After spatial calibration, the points in the point cloud data can accurately correspond to specific regions in the image, facilitating the extraction of corresponding height information or 3D contours from the point cloud using target detection boxes in the image, thus achieving complementarity between visual and geometric information.

[0222] Optionally, denoising can be a signal processing step to remove non-target noise interference from an image or point cloud. Specifically, for point cloud data, denoising can employ statistical filtering, radius filtering, or voxel grid filtering to remove outliers that are too far from the sensor, have too low density, or do not conform to the target's geometric features. For image data, denoising can employ Gaussian filtering, median filtering, or bilateral filtering to remove noise caused by lighting variations, sensor thermal noise, or compression artifacts.

[0223] Optionally, the image data can be a high-quality visual image that has undergone spatiotemporal calibration and denoising. This image data preserves the sharp edges and texture details of the target while removing background noise and interference, providing clean input for subsequent target detection algorithms and improving the localization accuracy of the detection box and the extraction quality of appearance features.

[0224] Optionally, the point cloud data can be a high-quality 3D point set that has undergone spatiotemporal calibration and denoising. The point cloud data preserves the complete geometry of the target and removes ground noise, vegetation interference, or random sensor noise, making the extracted target height, volume, and other physical attributes more accurate and reliable, thus helping to distinguish targets of different sizes.

[0225] Optionally, the image content of the initial image data includes at least one second environmental object, which means that the image visibly contains the target area to be tracked. In the initial image data, the second environmental object exists in the form of pixel blocks, and its position, color, and shape information are fully recorded. Even with slight occlusion or uneven lighting, the core visual features of the second environmental object can still be identified.

[0226] Optionally, the point cloud content of the initial point cloud data includes at least one second environmental object, meaning that the point cloud contains a discrete group of points that constitute the geometric structure of the target. Specifically, in the initial point cloud data, the second environmental object is represented by a dense set of three-dimensional coordinate points. These points constitute the three-dimensional outline and internal structure of the target. Even if some points are missing, the overall spatial distribution characteristics of the second environmental object can still be used for positioning and height measurement.

[0227] In this embodiment, the method described above ensures the consistency of multimodal data in time and space by acquiring initial images and point cloud data separately and rigorously performing spatiotemporal calibration and denoising processing. High-quality calibration and denoising data effectively eliminate the impact of sensor errors and environmental noise on subsequent target detection and tracking, laying a solid foundation for accurately extracting multidimensional feature information such as appearance, height, and confidence level. This improves the perception reliability and accuracy of the multi-target tracking system in complex autonomous driving scenarios.

[0228] As an optional implementation, the feature information includes the actual detection bounding box of the second environmental object, the actual appearance features of the second environmental object, the detection confidence of the second environmental object, and the actual height of the second environmental object. The feature information is extracted from the image data and point cloud data, including: extracting the actual detection bounding box, actual appearance features, and detection confidence from the image data; and extracting the actual height from the point cloud data.

[0229] In this embodiment, if image data and point cloud data are obtained, these two heterogeneous data are parsed respectively. Visual information (including location, appearance and reliability) is extracted from the image, and spatial geometric information (height) is extracted from the point cloud, thereby constructing a target feature set containing multi-dimensional attributes.

[0230] Optionally, the actual detection box can be a two-dimensional rectangular region enclosing the second environment object on the image plane. The actual detection box is typically defined by the coordinates of its top-left and bottom-right corners, or by its center coordinates, width, and height, and is used to indicate the approximate position and size of the target in the image, forming the basis for spatial constraints in subsequent association matching.

[0231] Optionally, the actual appearance features can be high-dimensional vectors describing the visual semantic attributes of the second environmental object. These actual appearance features are feature vectors (e.g., 256-dimensional) extracted from the target region cropped from the image using a convolutional neural network. They can characterize the target's category, color, texture, and other appearance attributes, assisting in target re-identification when occluded or after prolonged periods of non-matching. The convolutional neural network can be a residual network (ResNet) or a mobile network (MobileNet).

[0232] Optionally, the detection confidence score can be a probability assessment value of the object detection algorithm for the correctness of the recognition result. The detection confidence score is a floating-point number between 0 and 1, representing the degree of certainty with which the algorithm judges that the target does indeed exist within the detection box and that the category is correct. A high confidence score means the detection is more reliable, while a low confidence score may indicate a false detection or a vague target, and can be used to filter high-quality detection boxes.

[0233] Optionally, the actual height can be the vertical dimension of the second environmental object in three-dimensional space. The actual height is the physical length of the target along the Z-axis, typically obtained by calculating the difference between the maximum and minimum coordinates of the target's point cloud in the vertical direction, or estimated using the intersection of the lidar point cloud and the ground, to help distinguish targets of different sizes (such as pedestrians and vehicles).

[0234] Optionally, the process of extracting the actual detection boxes from the image data can be forward inference of an object detection algorithm. This involves inputting the image data into a pre-trained object detection model (such as the YOLO series, Faster R-CNN, etc.), and the model outputting bounding boxes that may contain the object and their corresponding class probabilities. Non-maximum suppression is then used to filter out the detection boxes that meet the criteria. The object detection model can be a single-stage object detection algorithm (You Only LookOnce, abbreviated as YOLO) or a Fast Region-based Convolutional Neural Network (abbreviated as Faster R-CNN).

[0235] Optionally, the process of extracting the actual appearance features from the image data can be a deep output of the feature extraction network. Specifically, after obtaining the actual detection box, the region of interest (ROI) corresponding to the detection box is input into a specific layer of the model (such as a feature layer before or after a fully connected layer) to extract a fixed-dimensional feature vector. This vector is robust to changes in illumination and angle.

[0236] Optionally, the process of extracting the detection confidence from the image data can be activated by a softmax function (Softmax) or a sigmoid function (Sigmoid) in the model output layer. Specifically, the detection model outputs a probability distribution for each category for each detection box, and takes the probability value of the largest category as the detection confidence for that detection box, reflecting the confidence in the existence of the target at the current moment.

[0237] Optionally, the process of extracting the actual height from the point cloud data can be a geometrical statistical analysis after point cloud clustering. First, a group of points belonging to the target is extracted from the point cloud using a clustering algorithm. Then, the difference between the maximum and minimum Z-coordinates of each point in this group is calculated, which represents the actual height of the target. For example, the clustering method can include density-based spatial clustering (DBSCAN) or Euclidean clustering.

[0238] Optionally, the actual detection bounding box, the actual appearance features, the detection confidence level, and the actual height together constitute the complete feature information of the second environmental object. This information provides the target's spatial location (detection bounding box), visual identity (appearance features), reliability (confidence level), and physical size attribute (height), respectively, providing rich data support for subsequent association matching and state prediction.

[0239] Optionally, the extraction process may also include feature alignment. Since images extract 2D features and point clouds extract 3D attributes, it may be necessary to standardize or normalize this information of different dimensions in subsequent use to ensure the consistency of the feature dimensions when calculating the association distance.

[0240] In this embodiment, the method fully leverages the complementary advantages of multi-sensor data to extract rich appearance and two-dimensional location information from images and precise three-dimensional height information from point clouds. This multimodal feature extraction approach not only improves the comprehensiveness of target description but also enhances the association discrimination capability in scenarios involving occlusion, distant objects, or similar appearances by introducing detection confidence and height information, thereby improving the overall robustness and accuracy of the multi-target tracking system. The technical solution of this embodiment is illustrated below with reference to preferred implementation methods.

[0241] With the rapid development of autonomous driving technology, environmental perception, as a core module of autonomous driving systems, directly determines vehicle decision-making and driving safety. Multi-target tracking, as a key component of environmental perception, needs to identify various targets (pedestrians, vehicles, obstacles, etc.) in road scenes in real time and accurately, and continuously track the target's position, movement history, trajectory, and status, providing reliable data support for vehicle path planning, obstacle avoidance control, and following other vehicles.

[0242] However, autonomous driving scenarios are complex and ever-changing, and multi-object tracking faces many technical challenges. Existing multi-object tracking algorithms are unable to meet the stringent requirements of autonomous driving, and mainly suffer from the following shortcomings:

[0243] Insufficient target motion modeling: In autonomous driving scenarios, targets (such as pedestrians crossing the road or vehicles changing lanes to overtake) exhibit complex nonlinear motion. Traditional prediction methods based on Kalman filtering can only fit linear motion, resulting in large prediction bias and tracking drift.

[0244] Tracking failure in occluded scenes: In urban roads, targets are easily obscured by other vehicles, buildings, and green belts. In this case, strong cues based on motion information and appearance features will fail, resulting in target tracking loss and frequent switching of tracking identifiers (IDs), which affects autonomous driving decisions.

[0245] Inadequate lifecycle management: Existing algorithms mostly use fixed thresholds to determine the disappearance of historical movement trajectories, which cannot adapt to situations such as temporary occlusion of targets or their appearance at a distance in autonomous driving scenarios. This can easily lead to the accidental deletion of historical movement trajectories or the retention of invalid historical movement trajectories, further reducing tracking stability.

[0246] Low utilization of multi-sensor information: Autonomous vehicles are generally equipped with multiple sensors, but existing tracking algorithms mostly only use information from a single sensor and do not fully integrate auxiliary clues such as lidar height information and detection confidence, resulting in insufficient correlation accuracy.

[0247] Therefore, there is an urgent need for a multi-target tracking method that can accurately model the nonlinear motion of targets, effectively cope with occlusion scenarios, and reasonably manage the life cycle of targets, in order to solve the above-mentioned technical pain points and improve the reliability and stability of autonomous driving environmental perception.

[0248] The methods of the embodiments of this application will be further illustrated below.

[0249] Figure 3 This is a flowchart of a multi-object tracking method for autonomous driving based on DynaSort, according to an embodiment of this application. Figure 3 As shown, the method may include the following steps.

[0250] Step S301: Multimodal data preprocessing.

[0251] In this embodiment, multi-sensor fusion is used to collect autonomous driving road scene data, and target detection related information is output after preprocessing. Step S301 may include steps S311, S312, S313, and S314.

[0252] Step S311: Multi-sensor data acquisition.

[0253] In this embodiment, data is collected simultaneously by a high-definition camera (which captures color images of the road scene) and a lidar (which captures the three-dimensional position, height, and point cloud data of the target) mounted on the autonomous vehicle. The camera frame rate is set to 30 FPS, and the lidar sampling frequency is set to 10 Hz.

[0254] Step S312: Multimodal data synchronization calibration.

[0255] In this embodiment, the camera and lidar are synchronized in time (using timestamp alignment) and calibrated in space (using hand-eye calibration) to eliminate time deviation and spatial offset in data acquisition, ensuring that the image information and point cloud information of the same target are accurately correlated.

[0256] Step S313: Denoising multimodal data.

[0257] In this embodiment, Gaussian filtering is used to denoise the lidar point cloud data, removing abnormal points caused by environmental noise and sensor noise, and retaining the effective target point cloud; median filtering is used to denoise the camera image, improving image clarity.

[0258] Step S314: Target detection preprocessing.

[0259] In this embodiment, the denoised image is input into the Eagle Eye-YOLO Object Detection Algorithm (Eagle Eye-YOLO for short), which outputs the target detection bounding box (x,y,w,h), appearance feature vector (256 dimensions), and detection confidence for each frame. The height information of the corresponding target is extracted from the LiDAR point cloud and associated with the detection bounding box and appearance feature vector to form complete target detection information for each frame.

[0260] Step S302: Construct the Dynamic Sorting Tracking Algorithm (DynaSort) multi-target tracking network.

[0261] In this embodiment, a DynaSort multi-target tracking network is constructed with "prediction-association-lifecycle management" as its core. Step S302 may include steps S321, S322, S323, and S324.

[0262] S321: Kalman Filter-Gated Recurrent Unit Fusion Prediction Algorithm (KF-GRU) prediction.

[0263] In this embodiment, to address the nonlinear motion problem of targets in autonomous driving scenarios, a Kalman Filter-Gated Recurrent Unit (GRU) fusion prediction algorithm (KF-GRU) is designed. This algorithm dynamically fuses Kalman Filter (KF) and Gated Recurrent Unit (GRU) attention mechanisms to achieve accurate prediction of the target's motion state. The specific implementation process is as follows:

[0264] Kalman filtering is responsible for the initial prediction of the target's motion state. It predicts the target's center coordinates, velocity, length, width, and other parameters in the next frame through the state transition equation (uniform motion model), obtaining the initial prediction result. The Gated Recurrent Unit-Attention Mechanism (GRU-Attention) module takes the motion state sequence of the target's previous 5 frames as input, captures the target's long-term temporal motion pattern through the Gated Recurrent Unit (GRU) network, and adaptively allocates the feature weights of each frame in combination with the attention mechanism, focusing on frames with significant changes in the target's motion state, and outputs a motion feature correction vector. The correction vector is fused with the initial prediction result of Kalman filtering to obtain the final target state prediction value, effectively fitting the complex nonlinear motion of targets such as pedestrians and vehicles, and reducing prediction bias.

[0265] S322: Joint association strategy.

[0266] In this embodiment, to address the issue of strong cues failing in occluded scenarios, a joint association strategy is constructed, fusing strong cues such as motion information and appearance features with weak cues such as height and confidence, thereby improving association accuracy. Specifically:

[0267] 1. Strong cue calculation: distance of motion information ( The Intersection Over Union (IOU) ratio is used for calculation. A higher IOU ratio indicates a smaller movement distance and a higher matching degree. The appearance feature distance ( The cosine similarity is calculated by subtracting the appearance feature vector from 1. The higher the cosine similarity, the more similar the appearances are and the higher the matching degree.

[0268] 2. Weak Clue Calculation: Height Distance ( The target height acquired by the lidar is normalized to the [0,1] interval, and the difference between the historical height of the historical movement trajectory and the current detected target height is calculated; confidence distance ( The detection confidence level is calculated by subtracting the detection confidence level from 1. The higher the confidence level, the more reliable the detection and the higher the matching degree.

[0269] 3. Weighted Fusion and Matching: The four types of cues are weighted and fused using the following formula to obtain the association distance matrix between the historical movement trajectory and the target; the Hungarian algorithm is used to match the association distance matrix according to the conditions to achieve accurate association between the historical movement trajectory and the currently detected target. Even in occluded scenes, weak cues can supplement the association basis and avoid tracking loss.

[0270] The above formula is shown below:

[0271]

[0272] in, These are weighting coefficients, and Preferred ; The distance traveled is the reciprocal of the Intersection over Union (IOU). The distance between appearance features is (1-cosine similarity). This is the height distance (the normalized difference in target height). The confidence distance is (1 - detection confidence).

[0273] S323: Lifecycle Management.

[0274] In this embodiment, to address the issue of unreasonable target lifecycle management, a dynamic lifecycle management mechanism is designed. This mechanism adaptively manages historical movement trajectories based on target matching status, confidence level, and motion continuity, as detailed below:

[0275] When creating a historical motion trajectory, it must meet the requirement of "detection confidence ≥ 0.7 + stable detection for 2 consecutive frames" to avoid creating invalid historical motion trajectories due to false detections. When the historical motion trajectory is maintained, if a match is successful, the lifecycle counter is reset (T=5). If there is no match but the motion state is continuous and the prediction error is ≤ 2 pixels, the counter is decremented by 1 to maintain the existence of the historical motion trajectory and to deal with temporary occlusion scenarios. When the historical motion trajectory disappears, the counter drops to 0 or there are 3 consecutive frames of no match and the prediction error is > 2 pixels, and the historical motion trajectory is deleted to avoid invalid historical motion trajectories occupying computing resources and reduce the number of tracking identifier (ID) switching times.

[0276] S324: Multi-target tracking.

[0277] In this embodiment, according to steps S321 to S323 above, a dynamic sorting tracking algorithm (DynaSort) multi-target tracking network can be constructed to perform multi-target tracking.

[0278] S303: Network training and parameter calibration.

[0279] In this embodiment, the preprocessed autonomous driving scenario dataset is input into the Dynamic Sorting Tracking Algorithm (DynaSort) multi-object tracking network for model training and parameter calibration. Step S303 may include steps S331, S332, and S333.

[0280] S331: Data input.

[0281] In this embodiment, the Carnegie Mellon University Multi-Object Tracking dataset (KITTI-MOT) for autonomous driving and the Waymo Open Dataset are used. The dataset covers complex scenarios such as urban roads, highways, rainy nights, and strong light, and includes targets such as pedestrians, vehicles, and non-motorized vehicles. The target tracking identifier (ID), location, historical movement trajectory and other information are labeled.

[0282] S332: Parameter iterative training.

[0283] In this embodiment, the Adam optimizer is used with an initial learning rate of 0.0005, a weight decay coefficient of 0.0001, and a momentum of 0.9 to accelerate model convergence. The training epochs are 80, the batch size is 8, and a gated recurrent unit (GRU) is used to accelerate training. During training, the evaluation metrics are multiple object tracking accuracy (MOTA), tracking identifier switching count (IDSwitches, IDSW), and multiple object tracking precision (MOTP).

[0284] S333: Model saving.

[0285] In this embodiment, after each training round, the model performance is evaluated using a validation set. If the multiple object tracking precision (MOTP) on the validation set does not improve for eight consecutive training epochs, training is stopped and the optimal model parameters are saved. For different autonomous driving scenarios (urban roads, highways), the weighting coefficients of the association strategy are calibrated to improve the model's scenario adaptability.

[0286] S304: Real-time multi-target tracking.

[0287] In this embodiment, the sensor data collected in real time by the autonomous vehicle is input into the trained Dynamic Sorting Tracking Algorithm (DynaSort) tracking model, and the following steps are performed to complete real-time tracking of multiple targets, providing support for autonomous driving decisions. Step S304 may include steps S341, S342, S343, and S344.

[0288] S341: Real-time data preprocessing.

[0289] In this embodiment, the data collected in real time by the camera and LiDAR are synchronously calibrated and denoised. The target detection box, appearance features, confidence score and height information are output by the Eagle Eye-YOLO Object Detection Algorithm (EagleEye-YOLO). The processing latency is ≤30ms.

[0290] S342: Multi-objective prediction.

[0291] In this embodiment, the Kalman Filter-Gated Cyclic Unit Fusion Prediction Algorithm (KF-GRU) prediction module takes into input the historical motion state sequence of the current historical trajectory and predicts the motion state (position, velocity) of the target in the next frame, with a prediction error ≤ 2 pixels.

[0292] S343: Multimodal feature association matching.

[0293] In this embodiment, based on the "strong clue + weak clue" joint association strategy, the association distance between the historical movement trajectory and the current detected target is calculated, and the optimal matching is achieved through the Hungarian algorithm, with a matching accuracy of ≥95%.

[0294] S344: Dynamic lifecycle management.

[0295] In this embodiment, the dynamic lifecycle management module updates the lifecycle counters of each historical movement trajectory based on the matching results, creates new historical movement trajectories, maintains valid historical movement trajectories, and deletes invalid historical movement trajectories. The number of tracking identifier (ID) switching times is reduced by more than 30% compared to traditional algorithms. It outputs in real time the unique tracking identifier (ID), real-time position (x, y, w, h), historical movement trajectory, movement speed, and status (normal, occluded, disappearing) of each target, feeding this data back to the autonomous driving decision-making system. This provides data support for functions such as path planning, obstacle avoidance control, and following other vehicles. The tracking frame rate is ≥25 FPS, meeting the real-time requirements of autonomous driving.

[0296] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0297] According to another aspect of the embodiments of this application, and the above... Figure 2 Corresponding to the embodiments of the vehicle information processing method shown, this specification also provides a vehicle information processing device.

[0298] Figure 4 This is a schematic diagram of an information processing device for objects surrounding a vehicle according to an embodiment of this application, such as... Figure 4 As shown, the information processing device 40 for the vehicle may include: a prediction module 401, configured to predict the predicted movement state of the first environmental object at the current moment based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period; a first determination module 402, configured to determine association information based on the predicted movement state and the feature information of the second environmental object at the current moment in response to the identification of at least one second environmental object located around the vehicle, wherein the association information is used to represent the degree of association between the predicted movement state of the first environmental object and the actual movement state of the second environmental object; a second determination module 403, configured to determine the association result between the first environmental object and the second environmental object using the association information; and an update module 404, configured to update the historical movement trajectory of the first environmental object based on the association result and the feature information of the second environmental object.

[0299] Figure 5 This is a schematic diagram of a vehicle according to an embodiment of this application, such as... Figure 5As shown, a vehicle 50 is also provided, including: a memory 51 storing an executable program; and a processor 52 for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0300] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application, such as... Figure 6 As shown, an electronic device 60 is also provided, including: a memory 61 storing an executable program; and a processor 62 for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0301] Embodiments of this application also provide a computer program product, including a computer program that is executed by a processor using the methods described in various embodiments of this application.

[0302] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection between units or modules can be electrical or other forms.

[0304] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0305] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0306] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a Universal Serial Bus Flash Drive (USB flash drive), a Read-Only Memory (ROM), a Random Access Memory (RAM), a portable hard drive, a magnetic disk, or an optical disk, and other media capable of storing program code.

[0307] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing information about objects in the environment surrounding a vehicle, characterized in that, include: Based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period, predict the predicted movement state of the first environmental object at the current moment. The system responds to the identification of at least one second environmental object located around the vehicle, and determines association information based on the predicted movement state and the feature information of the second environmental object at the current time, wherein the association information is used to represent the degree of association between the predicted movement state of the first environmental object and the actual movement state of the second environmental object; Using the association information, the association result between the first environment object and the second environment object is determined; Based on the association results and the feature information of the second environmental object, the historical movement trajectory of the first environmental object is updated.

2. The method according to claim 1, characterized in that, The determination of associated information based on the predicted movement state and the feature information of the second environmental object at the current moment includes: Based on the predicted movement state and the feature information of the second environmental object, a first sub-distance and a second sub-distance of the second environmental object are determined, wherein the influence of the first sub-distance on the matching accuracy of the historical movement trajectory and the second environmental object is greater than the influence of the second sub-distance on the matching accuracy of the historical movement trajectory and the second environmental object. The association information is determined based on the first sub-distance and the second sub-distance.

3. The method according to claim 2, characterized in that, The first sub-distance includes a movement distance, which represents the correlation between the predicted movement characteristics of the first environmental object at the current moment and the actual movement characteristics of the second environmental object. The predicted movement state includes a predicted bounding box of the first environmental object at the current moment predicted using the historical movement trajectory. The feature information includes the actual detection bounding box of the second environmental object. Determining the first sub-distance of the second environmental object based on the predicted movement state and the feature information includes: Determine the intersection-union ratio (IUGR) between the predicted bounding box and the actual detected bounding box, wherein the IUGR is used to represent the degree of overlap between the predicted bounding box and the actual detected bounding box; The reciprocal of the intersection-union ratio is determined as the moving distance; And / or, The first sub-distance includes an appearance distance, which represents the correlation between the predicted appearance features of the first environmental object at the current moment and the actual appearance features of the second environmental object. The predicted movement state includes the appearance feature vector of the first environmental object at any historical moment in the historical time. The feature information includes the appearance feature vector of the second environmental object. Determining the first sub-distance of the environmental object based on the predicted movement state and the feature information includes: Determine the similarity between the appearance feature vector of the first environmental object and the appearance feature vector of the second environmental object; The difference between the preset coefficient and the similarity is determined as the appearance distance.

4. The method according to claim 2, characterized in that, The second sub-distance includes an altitude distance and a confidence distance. The altitude distance represents the correlation between the predicted altitude of the first environmental object at the current time and the actual altitude of the second environmental object. The confidence distance represents the reliability of detecting the second environmental object at the current time. Determining the second sub-distance of the second environmental object based on the predicted movement state and the feature information includes: The height distance is determined based on the predicted movement state and the feature information; The confidence distance is determined based on the feature information of the second environmental object.

5. The method according to claim 4, characterized in that, The predicted movement state includes the height value of the first environmental object at any historical moment in history, the feature information includes the height value of the second environmental object, and the step of determining the height distance based on the predicted movement state and the feature information includes: The height values ​​of the first environment object and the second environment object are normalized respectively to obtain the normalized height values ​​of the first environment object and the normalized height values ​​of the second environment object. Determine the difference between the height value of the first environment object after normalization and the height value of the second environment object after normalization; The height distance is determined based on the difference.

6. The method according to claim 4, characterized in that, The feature information includes the detection confidence at the current moment, and determining the confidence distance based on the feature information of the second environmental object includes: The difference between the preset coefficient and the detection confidence level is determined as the confidence distance.

7. The method according to claim 2, characterized in that, The step of determining the association information based on the first sub-distance and the second sub-distance includes: Weight coefficients are assigned to the first sub-distance and the second sub-distance respectively, wherein the weight coefficient assigned to the first sub-distance is greater than the weight coefficient assigned to the second sub-distance; The first sub-distance assigned to the weight coefficient and the second sub-distance assigned to the weight coefficient are fused to obtain the association information.

8. The method according to claim 1, characterized in that, The step of determining the association result between the first environment object and the second environment object using the association information includes: A global matching strategy is invoked to associate the first environment object and the second environment object using the association information, thereby obtaining the association result. The global matching strategy represents the rule that minimizes the total cost of associating the first environment object and the second environment object from the association information.

9. The method according to claim 8, characterized in that, The invocation of the global matching strategy, utilizing the association information, associates the first environment object and the second environment object to obtain the association result, including: A bipartite graph is constructed using a first set of nodes with the historical movement trajectory of the first environmental object as nodes and a second set of nodes with the second environmental object as nodes. The edge weights between nodes in the first set of nodes and nodes in the second set of nodes in the bipartite graph are used to characterize the association information. Determine the edges between nodes in the first set of nodes and nodes in the second set of nodes that satisfy the objective of minimizing the edge weights in the bipartite graph; The association result is determined based on the edge.

10. The method according to claim 1, characterized in that, The step of updating the historical movement trajectory of the first environmental object based on the association result and the feature information of the second environmental object includes: Based on the association results, the historical movement trajectory of the first environmental object and the second environmental object are classified respectively to obtain the classification results of the historical movement trajectory and the classification results of the second environmental object; The historical movement trajectory is updated based on the classification results of the historical movement trajectory, the classification results of the second environmental object, and the feature information.

11. The method according to claim 10, characterized in that, The step of classifying the historical movement trajectory of the first environmental object and the second environmental object according to the association result to obtain the classification result of the historical movement trajectory and the classification result of the second environmental object includes: Based on the association results, the classification result of the historical movement trajectory is determined to include at least one of the following: the historical movement trajectory that has been matched at the current time, and the historical movement trajectory that has not been matched at the current time; Based on the association results, the classification result of the second environment object is determined to include at least one of the following: unmatched second environment object, matched second environment object.

12. The method according to claim 11, characterized in that, The updating of the historical movement trajectory based on the classification results of the historical movement trajectory, the classification results of the second environmental object, and the feature information includes at least one of the following: Reset the lifecycle state of the matched historical movement trajectory; Based on the mobility continuity of the unmatched historical movement trajectory, determine the lifecycle state of the unmatched historical movement trajectory; in response to the lifecycle state of the unmatched historical movement trajectory satisfying the maintenance condition, update the lifecycle counter of the unmatched historical movement trajectory; in response to the lifecycle state of the unmatched historical movement trajectory not satisfying the maintenance condition, delete the unmatched historical movement trajectory. From the feature information, the detection confidence and continuous occurrence status of the unmatched second environmental object are determined; in response to the detection confidence being greater than or equal to the confidence threshold, and the continuous occurrence status representing the moment when the unmatched second environmental object appears consecutively for a first number of targets, a historical movement trajectory corresponding to the unmatched second environmental object is created, and the lifecycle counter of the created historical movement trajectory is updated.

13. The method according to claim 12, characterized in that, The method further includes: The historical movement trajectory is deleted if at least one of the following conditions exists: The lifecycle counter for the historical movement trajectory is reduced to a counting threshold; The historical movement trajectory shows a second number of consecutive occurrences of the second target without matching the second environmental object, and the error in determining the predicted movement state using the historical movement trajectory is greater than the error threshold.

14. The method according to claim 1, characterized in that, The method of predicting the predicted movement state of at least one first environmental object located around the vehicle at the current moment based on the historical movement trajectory of at least one first environmental object within a historical time period includes: Based on the movement state sequence in the historical movement trajectory, a preliminary predicted movement state of the first environmental object at the current moment is generated by a linear dynamics prediction model, and a temporal feature extraction is performed on the movement state sequence by a nonlinear temporal modeling network to generate a movement state correction vector of the first environmental object. The predicted movement state is obtained by fusing the movement state correction vector with the preliminary predicted movement state.

15. The method according to claim 14, characterized in that, The nonlinear temporal modeling network is a gated recurrent unit network based on an attention mechanism. The step of extracting temporal features from the movement state sequence using the nonlinear temporal modeling network to generate a movement state correction vector for the first environmental object includes: The motion state sequence of a preset number of historical moments in the historical motion trajectory is input into the gated recurrent unit network. The gated recurrent unit is used to encode the motion state sequence with long-term temporal features to generate a hidden state sequence containing long-term temporal motion dependencies. The attention mechanism is used to assign weights to the hidden state sequence, and the weighted hidden state sequence is fused to obtain attention-weighted movement features. The attention-weighted movement features are determined as the movement state correction vector.

16. The method according to claim 14, characterized in that, The step of fusing the movement state correction vector with the preliminary predicted movement state to obtain the predicted movement state includes: The movement state correction vector is superimposed on the state vector of the preliminary predicted movement state to obtain the predicted movement state, wherein the state vector of the preliminary predicted movement state includes at least one of the following: the center coordinates of the first environmental object at the current time, the movement speed of the first environmental object at the current time, and the size parameter of the first environmental object at the current time.

17. The method according to any one of claims 1 to 16, characterized in that, The method further includes: In response to the current moment when at least one second environmental object is identified around the vehicle, image data and point cloud data containing the second environmental object are acquired at the current moment; The feature information of the second environmental object at the current moment is extracted from the image data and the point cloud data.

18. The method according to claim 17, characterized in that, The step of acquiring the image data and point cloud data containing the second environmental object at the current moment includes: At the current moment, initial image data of the environment surrounding the vehicle is acquired using the image acquisition device deployed on the vehicle, and initial point cloud data of the environment surrounding the vehicle is acquired using the radar device deployed on the vehicle, wherein the image content of the initial image data includes at least one second environmental object, and the point cloud content of the initial point cloud data includes at least one second environmental object. Spatiotemporal calibration processing is performed on the initial image data and the initial point cloud data to obtain calibrated initial image data and calibrated initial point cloud data; The calibrated initial image data and the calibrated initial point cloud data are respectively subjected to denoising processing to obtain the image data and the point cloud data.

19. The method according to claim 17, characterized in that, The feature information includes the actual detection bounding box of the second environmental object, the actual appearance features of the second environmental object, the detection confidence of the second environmental object, and the actual height of the second environmental object. Extracting the feature information from the image data and the point cloud data includes: Extract the actual detection bounding box, the actual appearance features, and the detection confidence from the image data; The actual height is extracted from the point cloud data.

20. An information processing device for objects in the environment surrounding a vehicle, characterized in that, include: The prediction module is used to predict the predicted movement state of the first environmental object at the current moment based on the historical movement trajectory of at least one first environmental object located around the vehicle within a historical time period. The first determining module is configured to respond to the identification of at least one second environmental object located around the vehicle, and determine association information based on the predicted movement state and the feature information of the second environmental object at the current time, wherein the association information is used to represent the degree of association between the predicted movement state of the first environmental object and the actual movement state of the second environmental object; The second determining module is used to determine the association result between the first environment object and the second environment object using the association information; An update module is used to update the historical movement trajectory of the first environment object based on the association result and the feature information of the second environment object.

21. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 19.

22. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 19.

23. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 19.

Images