Object speed prediction method and related apparatus

Abstract

An object speed prediction method and a related apparatus. The object speed prediction method comprises: acquiring a first observation vector of an object, the first observation vector indicating the observed value of at least one attribute of the object at time T; adjusting a motion model on the basis of the first observation vector, so that the motion model matches in dimensionality with the first observation vector; using the first observation vector as an input of the motion model to output a predicted vector; and obtaining the speed predicted value of the object at time T+1 on the basis of the predicted vector. In the present application, the motion model can be adjusted on the basis of the first observation vector, thereby improving the robustness and stability of the motion model, and preventing catastrophic collapse of the motion model due to data loss.

Description

A speed prediction method of an object and related device

[0001] The present application claims priority to the Chinese patent application No. 202410949561X, filed on July 15, 2024, and entitled "A speed prediction method of an object and related device", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular to a speed prediction method of an object and related device. BACKGROUND

[0003] One of the core functions of an autonomous driving system is to accurately perceive and predict moving targets in the surrounding environment, such as pedestrians, vehicles, etc. To achieve this function, the system mainly relies on perception devices such as lidar and camera. These perception devices can provide information about the spatial size, category and motion speed of moving targets. By learning and reasoning about this information through a perception model, the autonomous driving system can predict the spatial size, motion trajectory and speed of moving targets, thereby endowing the autonomous driving system with general perception ability in a variable environment. Therefore, a method for predicting the speed of moving targets is needed. SUMMARY

[0004] The present application provides a speed prediction method of an object and related device, which can obtain the speed of a moving target through the prediction of a motion model.

[0005] In a first aspect, the present application provides a speed prediction method of an object, mainly comprising: obtaining a first observation vector of the object, the first observation vector indicating an observation value of at least one attribute of the object at a T-th moment, T being an integer greater than or equal to 1; adjusting a motion model according to the first observation vector, so that the dimension of an input vector processed by the motion model matches the dimension of the first observation vector; inputting the first observation vector into the motion model to output a prediction vector, the prediction vector indicating a prediction value of the at least one attribute of the object at a T+1-th moment, the prediction value of the at least one attribute at the T+1-th moment including a speed prediction value at the T+1-th moment; and obtaining the speed prediction value of the object at the T+1-th moment based on the prediction vector.

[0006] In the technical solution of the present application, the speed of the object can be obtained through the prediction of the motion model. The motion model can adjust the motion model according to the first observation vector, so that the dimension of the input vector processed by the motion model matches the dimension of the first observation vector, facilitating the calculation of the motion model. That is, the motion model can be adaptively adjusted according to different input vectors. Therefore, when some data of the first observation vector is missing, the motion model can be adjusted accordingly. The adjusted motion model can predict the speed of the first observation vector with some missing data, thereby improving the robustness and stability of the motion model and preventing catastrophic collapse of the motion model due to data loss.

[0007] In a possible implementation, the motion model is modeled according to a Kalman equation set.

[0008] In this implementation, the internal algorithm of the motion model is implemented based on a Kalman equation set. Compared with the linear error caused by the interpolation calculation method, the motion model can perform model updating according to the prediction vector, the observation vector, and the Kalman gain. Therefore, the motion model will not produce linear error due to the error of the observation vector. At the same time, the motion model has good noise suppression capability. The Kalman filter is equivalent to a physical filtering system, and the predicted speed value has high accuracy. Therefore, the motion model in the present application can realize speed prediction for any input vector of the object, and has the characteristics of high accuracy and strong robustness.

[0009] In a possible implementation, the adjusting of the motion model according to the first observation vector includes adjusting a state transition matrix in the motion model according to the first observation vector, the state transition matrix being a matrix multiplied by the first observation vector in the Kalman equation set.

[0010] In this implementation, since the state transition matrix is a matrix multiplied by the first observation vector, the state transition matrix needs to be adjusted to match the dimensions of the state transition matrix and the first observation vector, thereby meeting the dimension requirement of the calculation. Furthermore, the adjusted state transition matrix of the motion model can be applied to all observation vectors, so that the motion model can be an input vector of any dimension, thereby ensuring the robustness and stability of the motion model.

[0011] In a possible implementation, the adjusting of the state transition matrix in the motion model according to the first observation vector includes extracting a corresponding vector in the state transition matrix before adjustment according to the attribute corresponding to the observation value in the first observation vector, to obtain the state transition matrix after adjustment.

[0012] In the implementation, a method for adjusting a state transition matrix is provided. According to attributes corresponding to observation values in a first observation vector, vectors corresponding to the attributes are extracted from a state transition matrix before adjustment, to obtain a state transition matrix after adjustment. In this way, not only is the first observation vector and the state transition matrix kept matching in dimension, facilitating calculation, but also each vector in the state transition matrix is matched with the first observation vector, ensuring the correctness of the calculation result.

[0013] In a possible implementation, the state transition matrix before adjustment is obtained according to a full-quantity vector. The full-quantity vector represents a vector of observation values of all attributes of an object in the input vector set, and the all attributes represent all attributes corresponding to the multiple input vectors in the input vector set.

[0014] In the implementation, the state transition matrix is obtained according to a full-quantity vector. The full-quantity vector contains all attributes in the input vector set. Therefore, the state transition matrix before adjustment is matched with the full-quantity vector, and the all attributes in the observation vector are part or all of the all attributes in the full-quantity vector. Therefore, the state transition matrix before adjustment can be applied to all observation vectors after adjustment, so that the motion model can be any input, avoiding the collapse of the motion model caused by the lack of some data in the input observation vector.

[0015] In a possible implementation, the above obtaining a speed prediction value of the object based on the prediction vector comprises: obtaining at least one second observation vector, each second observation vector indicating an observation value of at least one attribute of the object at the T+1 time; and matching the prediction vector and the at least one second observation vector to obtain a speed prediction value of the object at the T+1 time.

[0016] In the implementation, considering the uncertainty of the prediction vector and the uncertainty of the second observation vector, the prediction vector and the second observation vector are matched. When the prediction vector and the second observation vector are matched, the speed prediction value of the object is extracted from the prediction vector. If the prediction vector and the second observation vector are matched, it indicates that the accuracy of the prediction vector meets the requirements, and the speed prediction value corresponding to the prediction vector is a reliable prediction result.

[0017] In a possible implementation, the matching of the prediction vector and the at least one second observation vector to obtain the speed prediction value of the object at the T+1 moment comprises: matching the prediction vector and the at least one second observation vector according to a first matching strategy to obtain a first matching result; the first matching result is that there is a second observation vector matching the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment; the first matching result is that there is no second observation vector matching the prediction vector, and the prediction vector and a plurality of second observation vectors are matched according to a second matching strategy to obtain a second matching result, the matching accuracy of the second matching strategy is higher than that of the first matching strategy; and the second matching result is that there is a second observation vector matching the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment.

[0018] In the implementation, a matching method of a prediction vector and a second observation vector is provided. A first matching result of the prediction vector and the second observation vector is calculated according to a first matching strategy with lower accuracy; when the prediction vector and the second observation vector match, a speed prediction value of an object is obtained from the prediction vector; when the prediction vector and the second observation vector do not match, a second matching result of the prediction vector and the second observation vector is calculated according to a second matching strategy with higher accuracy, and when the prediction vector and the second observation vector match, a speed prediction value of an object is obtained from the prediction vector. That is, the matching method is equivalent to a two-stage matching method, the first matching strategy with lower accuracy is used for matching first, and when the prediction vector and the second observation vector do not match, the second matching strategy with higher accuracy is used for matching, wherein the first matching strategy with lower accuracy has the characteristics of small calculation amount and fast calculation speed, and can improve the matching efficiency of the matching process; the second matching strategy with higher accuracy has the characteristic of high calculation accuracy, and can improve the matching success rate and reduce the possibility of missing matching, thereby better establishing the correlation and consistency of the same object in multiple time sequences.

[0019] In a possible implementation, the method further comprises: when the second matching result is that there is no second observation vector matching the prediction vector, the object corresponding to the second observation vector is marked as a new object, and / or the object corresponding to the first observation value is marked as lost.

[0020] In the implementation, after the matching processing of the first matching strategy and the second matching strategy, for the second observation vector that still fails to match successfully, the object corresponding to the second observation vector can be marked as a new object, and the trajectory tracking of the new object is started. Optionally, the object can also be marked as lost, and the state of the object at the T+1 moment is recorded.

[0021] In a possible implementation, the method further includes: the first matching result is that there is a second observation vector matching the prediction vector, and the ID of the detected object corresponding to the second observation vector matching the prediction vector is marked according to the ID of the object.

[0022] In this implementation, the second observation vector that matches successfully is assigned to the ID of the object, so that the trajectory tracking of the object can be implemented.

[0023] In a possible implementation, the method further includes: adjusting the update step of the motion model based on the interval time of multiple time instants of the detection data of the object, the multiple time instants including the Tth time instant and the (T+1)th time instant.

[0024] In this implementation, a dynamic step update mechanism of the motion model is provided, and the sensitivity of the speed prediction under different observation vector sampling frequencies is solved. According to the interval time between multiple frames of data, the parameter of the update step of the motion model is adjusted, which is a mechanism of updating the motion model with a dynamic step and has higher sensitivity. In different sampling frequency scenarios, the motion model can always maintain high prediction sensitivity, improve the sensitivity and accuracy of the speed prediction of the object in different speed intervals, improve the sensitivity of the motion model to speed, and can distinguish different speeds. The speed prediction of the low-speed target is more accurate and has higher sensitivity.

[0025] In a possible implementation, the above adjusting the update step of the motion model based on the interval time of multiple time instants of the detection data of the object includes: when the sampling frequency of the detection data is a single frequency, setting the reciprocal of the sampling frequency as the update step of the motion model; and when the sampling frequency of the detection data is a mixed frequency, setting the interval time between the Tth time instant and the (T+1)th time instant as the update step of the motion model.

[0026] In this implementation, a method for determining the update step of the motion model under different sampling frequencies is provided. The sampling of multiple frames of data can be a single frequency or a mixed frequency. Through different update step mechanisms of the motion model, the update step of the motion model can be adjusted according to various actual situations.

[0027] In a possible implementation, the method further includes: adjusting the prediction step of the motion model based on the interval time of multiple time instants of the detection data of the object, the multiple time instants including the Tth time instant and the (T+1)th time instant.

[0028] In the implementation, the prediction step of the motion model can also be adjusted according to the interval time between the frames of the observation vectors, so that the prediction step of the motion model is dynamically adjusted, the matching of the prediction step and the sampling frequency is realized, the time error of the observation vectors and the prediction vectors is reduced, and the matching accuracy of the prediction vectors and the observation vectors is improved.

[0029] In a possible implementation, the first observation vector is used to indicate the geometric information and / or the speed information of the object, the geometric information is obtained based on attribute information of a bounding box of the object output by the 3D detection model, and the speed information is obtained based on the speed prediction model.

[0030] In the implementation, the geometric information and / or the speed information can be included in the first observation vector. For example, the geometric information is obtained based on attribute information of a bounding box of the object output by the 3D detection model, and the speed information is obtained based on the speed prediction model.

[0031] In a possible implementation, the speed prediction value is used as a basis for driving decision of the autonomous vehicle, and / or the speed prediction value is used as a speed label value of the object at the T+1 moment.

[0032] In the implementation, the speed prediction value can have more use scenarios, which are not limited by the embodiments of the application.

[0033] In a second aspect, the application provides an object speed prediction device, comprising an acquisition module and a processing module. The acquisition module is configured to acquire a first observation vector of an object, the first observation vector being generated based on an observation value of at least one attribute of the object at a T moment, T being an integer greater than 1. The processing module is configured to adjust a motion model according to the first observation vector, so that the dimension of an input vector processed by the motion model matches the dimension of the first observation vector; and output a prediction vector by taking the first observation vector as an input of the motion model, the prediction vector indicating a prediction value of at least one attribute of the object at a T+1 moment, the prediction value of the at least one attribute including a speed prediction value; and obtain a speed prediction value of the object at the T+1 moment based on the prediction vector.

[0034] In a possible implementation, the motion model is modeled according to a Kalman equation.

[0035] In a possible implementation, the processing module is specifically configured to adjust a state transition matrix in the motion model according to the first observation vector, the state transition matrix being a matrix in the Kalman equation group that is multiplied by the first observation vector.

[0036] In a possible implementation, the processing module is specifically configured to: according to the attributes corresponding to the observation values in the first observation vector, extract vectors corresponding to respective attributes from the state transition matrix before adjustment, to obtain the state transition matrix after adjustment.

[0037] In a possible implementation, the state transition matrix before adjustment is obtained according to a full-quantity vector, the full-quantity vector representing a vector of observation values of all attributes of the object in the input vector set, the all attributes representing all attributes corresponding to the multiple input vectors in the input vector set.

[0038] In a possible implementation, the processing module is specifically configured to: obtain at least one second observation vector, each second observation vector indicating an observation value of at least one attribute of the object at the T+1 moment; and match the prediction vector and the at least one second observation vector to obtain a speed prediction value of the object at the T+1 moment.

[0039] In a possible implementation, the processing module is specifically configured to: match the prediction vector and the at least one second observation vector according to a first matching strategy to obtain a first matching result; the first matching result is that there is a second observation vector matched with the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment; the first matching result is that there is no second observation vector matched with the prediction vector, and the prediction vector and multiple second observation vectors are matched according to a second matching strategy to obtain a second matching result, the matching accuracy of the second matching strategy being higher than that of the first matching strategy; and the second matching result is that there is a second observation vector matched with the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment.

[0040] In a possible implementation, the processing module is further configured to: the second matching result is that there is no second observation vector matched with the prediction vector, and the object corresponding to the second observation vector is marked as a new object, and / or the object corresponding to the first observation value is marked as lost.

[0041] In a possible implementation, the processing module is further configured to: the first matching result is that there is a second observation vector matched with the prediction vector, and the ID of the detection object corresponding to the second observation vector matched with the prediction vector is marked according to the ID of the object.

[0042] In a possible implementation, the processing module is further configured to: adjust the update step of the motion model based on interval times of multiple moments of the detection data, the multiple moments including the T moment and the T+1 moment.

[0043] In a possible implementation, the processing module is specifically configured to: when the sampling frequency of the detection data is a single frequency, set the reciprocal of the sampling frequency as the update step length of the motion model; and when the frequency of the detection data is a mixed frequency, set the interval time between the Tth moment and the T+1th moment as the update step length of the motion model.

[0044] In a possible implementation, the processing module is further configured to: based on the interval time of multiple moments of the detection data, adjust the prediction step length of the motion model, the multiple moments including the Tth moment and the T+1th moment.

[0045] In a possible implementation, the first observation vector is used to indicate the geometric information and / or the speed information of the object, the geometric information is obtained based on the attribute information of the bounding box of the object output by the 3D detection model, and the speed information is obtained based on the speed prediction model.

[0046] In a possible implementation, the speed prediction value is used as a basis for driving decision of the autonomous vehicle; and / or, the speed prediction value is used as a speed label value of the object at the T+1th moment.

[0047] In a third aspect, the present application provides an object speed prediction device, comprising at least one processor and an interface; the at least one processor acquires program instructions through the interface; the at least one processor is configured to execute the program instructions to implement the method described in the first aspect. Exemplarily, the device can be but is not limited to a chip.

[0048] In a fourth aspect, the present application provides an electronic device, comprising: at least one memory configured to store a program; and at least one processor configured to execute the program stored in the memory; wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect.

[0049] In a fifth aspect, the present application provides a vehicle, characterized in that the vehicle comprises the electronic device described in the fourth aspect.

[0050] In a sixth aspect, the present application provides a computer readable storage medium, which stores a computer program, and when the computer program runs on a processor, the processor executes the method described in the first aspect.

[0051] In a seventh aspect, the present application provides a computer program product, which, when running on a processor, causes the processor to execute the method described in the first aspect.

[0052] It can be understood that the beneficial effects of the above-mentioned second aspect to seventh aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0053] FIG. 1 is a schematic diagram of detection results of a laser radar and a camera in an autonomous vehicle according to an embodiment of the present application;

[0054] FIG. 2 is a schematic diagram of parameters and coordinate systems of a moving target according to an embodiment of the present application;

[0055] FIG. 3 is a schematic diagram of identity ID hopping and loss of a moving target according to an embodiment of the present application;

[0056] FIG. 4 is a schematic diagram of a first application scenario of a speed prediction method of an object according to an embodiment of the present application;

[0057] FIG. 5 is a schematic diagram of a second application scenario of a speed prediction method of an object according to an embodiment of the present application;

[0058] FIG. 6 is a schematic diagram of a flow of a speed prediction method of an object according to an embodiment of the present application;

[0059] FIG. 7 is a schematic diagram of a multi-source input vector set according to an embodiment of the present application;

[0060] FIG. 8 is a schematic diagram of an example of adjusting a state transition matrix according to an embodiment of the present application;

[0061] FIG. 9 is a schematic diagram of two-stage matching of a speed prediction method of an object according to an embodiment of the present application;

[0062] FIG. 10 is a schematic diagram of a method for determining an update step of a motion model in different frequency sampling scenarios according to an embodiment of the present application;

[0063] FIG. 11 is a schematic diagram of a speed prediction framework of an object according to an embodiment of the present application;

[0064] FIG. 12 is a schematic diagram of a flow of predicting a speed of an object by a motion model according to an embodiment of the present application;

[0065] FIG. 13 is a schematic diagram of a structure of a speed prediction apparatus of an object according to an embodiment of the present application;

[0066] FIG. 14 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application;

[0067] FIG. 15 is a schematic diagram of a structure of another speed prediction apparatus of an object according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] The term “and / or” used in the present document is used to describe an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The symbol “ / ” in the present document represents an or relationship of associated objects, for example, A / B represents A or B.

[0069] The terms “first” and “second” and the like in the description and claims of the present document are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, rather than to describe a specific order of the response messages.

[0070] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application is not to be construed as preferred or advantageous over other embodiments or design solutions. Rather, the use of “exemplary” or “for example” is intended to present concepts in a particular manner. The words “exemplary” and “for example” are used in the embodiments of the present application to mean serving as an example, instance, or illustration.

[0071] In the description of the embodiments of the present application, unless otherwise specified, “a plurality of” means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0072] In order to facilitate understanding of the scheme provided by the embodiments of the present application, some terms related to the present scheme are briefly introduced.

[0073] Trajectory prediction: first obtain detection data of the current driving environment in a period of time, for example, point cloud data obtained by a laser radar, detect each frame of point cloud data to obtain a detection box corresponding to each moving target in each frame of point cloud data, then determine which moving target in the previous frame of point cloud data each moving target in each frame of point cloud data corresponds to, and further determine the position of the same moving target in different frames of point cloud data, to obtain the trajectory of the moving target in the period of time. Thus, the vehicle can predict the possible action of the moving target according to the trajectory of the moving target. Then, measures such as braking are taken according to the possible action of the moving target to ensure driving safety.

[0074] Moving target ID (identity document, identity) prediction: the vehicle first predicts the position of each detection box in the current frame according to each detection box in the previous frame of point cloud data, that is, obtains each trajectory box. Then, the trajectory box obtained by prediction is matched with the detection box in the current frame to determine the detection box corresponding to each trajectory box. Further, it can be determined which moving target in the previous frame the moving target in the current frame corresponds to, and the moving target ID in the current frame is predicted.

[0075] Observation vector: the vector representation of one or more attributes of a detection object obtained after processing the detection data collected by a data collection device such as radar, camera, etc. For example, the observation vector is (w1, h1, l1), which represents the vector representation of the observed values of the length, width and height of the detection frame obtained by the detection data. w1 represents the observed value of the width of the detection frame, h1 represents the observed value of the height of the detection frame, and l1 represents the observed value of the length of the detection frame.

[0076] Prediction vector: the vector representation of the predicted value of one or more attributes of the object by the motion model. For example, the prediction vector is (w2, h2, l2), which represents the vector representation of the predicted values of the length, width and height of the trajectory frame by the motion model. w2 represents the predicted value of the width of the trajectory frame, h2 represents the predicted value of the height of the trajectory frame, and l2 represents the predicted value of the length of the trajectory frame.

[0077] Motion model: a motion model is a mathematical model that describes the position, velocity and acceleration of an object or system over time. The commonly used motion model is the relativistic motion model based on relativity. In engineering and scientific fields, motion model is very important, which can be used to predict the future motion state of an object or system. Common motion models include uniform motion model and uniform acceleration motion model, etc. In an example, the motion model can be a motion model established based on Kalman filtering principle. In this example, the Kalman model is a mathematical model established based on Kalman equation set. The input of the motion model is the observation vector of the current frame or the current time, and the output is the prediction vector of the next frame or the next time. After the prediction vector of the next frame or the next time is matched with the multiple observation vectors of the next frame or the next time, the model is updated according to the matched observation vector of the next frame or the next time and the prediction vector of the next frame or the next time, including updating the covariance matrix, the observation matrix and the state transition matrix of the running model.

[0078] Object: any object existing in nature. Taking autonomous driving technology as an example, the object refers to pedestrians, vehicles, roads, trees, etc. that can be automatically recognized by autonomous driving vehicles. Taking monitoring technology as an example, the object can be people, furniture, equipment, etc. According to whether the object is moving, the object can be divided into stationary objects and moving objects. In the field of autonomous driving technology, stationary objects are also called stationary targets, and moving objects are also called moving targets.

[0079] In the field of target tracking technology, the moving target speed prediction algorithm has wide application in automatic driving equipment and vehicles, and is one of the basic technologies of automatic driving. The main perception equipment of the automatic driving vehicle is a laser radar and a camera (as shown in FIG. 1), the laser radar mainly perceives the surrounding environment through laser point cloud, and the camera mainly perceives the surrounding environment through video stream. Whether it is laser point cloud or video stream, the perception attributes of the moving target mainly include the spatial size of the moving target, the category of the moving target, and the motion speed of the moving target. At the algorithm and software level, by means of learning and reasoning of the perception model, such as 3d detection model, multi-target tracking model, instance segmentation model, speed motion model, etc., the 3d spatial size, motion trajectory, instance category, motion speed, etc. of the moving target can be predicted from the laser point cloud and the video stream, thereby giving the automatic driving system general perception ability in the open set world.

[0080] Among them, the speed of the moving target is one of the important perception attributes of the moving target, and in the field of automatic driving, speed prediction has many application scenarios, for example, when the vehicle with the automatic driving system drives on the road, it needs to predict the speed of the surrounding vehicles, thereby providing safer automatic driving decisions. For another example, in the data automatic labeling and data true value system, the speed value of the moving target in the data needs to be labeled, which all need to apply the speed prediction algorithm. The speed prediction algorithm has wide application demand and application value.

[0081] In the embodiments of the present application, taking a vehicle as an example, the parameters and coordinate system of the moving target are exemplarily described. In the vehicle coordinate system, as shown in FIG. 2, a vehicle driving has 3-axis direction movement degrees of freedom, which are x-axis, y-axis and z-axis, wherein the x-axis is the direction of the vehicle advancing forward, the y-axis is the left direction perpendicular to the x-axis, and the z-axis is the direction perpendicular to the ground downward. Theoretically, the vehicle has 3-direction speeds, which are x-axis direction speed Vx, y-axis direction speed Vy and z-axis direction speed Vz. Since the actual road is flat, the movement amount of the z-axis direction is only the vibration amplitude of the vehicle suspension, and the speed value Vz is close to 0 and has no actual application value. Alternatively, in the embodiments of the present application, only the speed values Vx and Vy of the X-axis and Y-axis can be predicted, and the speed prediction of the z-axis is no longer focused on. After the vehicle coordinate system is bound with the vehicle, the relative rotation angle θ of the vehicle coordinate system and the radar coordinate system can be used as the rotation information of the vehicle, such as rotation angular displacement, rotation speed, rotation acceleration, etc., such as the rotation angular displacement of the vehicle coordinate system and the radar coordinate system around the z-axis shown in FIG. 2.

[0082] In one solution, a first-order interpolation method is adopted, and the calculation principle is: first, the 3d bbox center point coordinates (x, y, z) of the moving target are obtained, and then the speed is calculated by time stamp interpolation, as shown in formula (1). The calculation method of this technical solution is simple, and the calculation reasoning speed is fast.

[0083] In this solution, there are mainly three technical problems: (1) low precision: the speed value is calculated by 3d bbox geometric center interpolation, and the precision depends on the geometric precision of the 3d box. The center point coordinate precision of the general 3d box is low, so the speed value precision of the interpolation calculation is also low; (2) weak correlation: the same instance target lacks correlation and consistency in multiple frames of time; (3) poor sensitivity: the speed prediction method belongs to coarse-grained, and the speed perception is not sensitive to low-speed and ultra-low-speed targets (approximately static targets), and the speed prediction value of low-speed and ultra-low-speed targets is generally zero, which has large error.

[0084] In order to further illustrate the technical problems existing in the solution, the following will be further illustrated in combination with FIG. 3. As shown in FIG. 3, for the same moving target, it is necessary to find the same target in different time frames, and in the above solution, due to the poor precision of the 3d space geometric size of the moving target, the frequent switching of the identity ID of the same target, etc., the speed prediction precision is poor, so as to provide real-time and efficient decision for the automatic driving vehicle.

[0085] For example, as shown in FIG. 3, the vehicle with identity ID 6, at t=1, the identity ID is marked as 6; at t=2, the identity ID is marked as 9, the identity ID switching phenomenon; at t=3, the identity ID is marked as "none", the identity ID loss phenomenon, which means that the target tracking fails; at t=4, the identity ID is re-marked as 6, the identity ID is associated with t=1 and t=4, the time sequence correlation of the vehicle is weak, which is not conducive to speed prediction and the like.

[0086] That is, the poor precision of the 3d space geometric size of the moving target will lead to low speed prediction precision, the frequent switching of the identity ID of the same moving target will lead to weak time sequence correlation, and the different scanning frequencies of the laser radar will put higher requirements on the speed prediction sensitivity to adapt to different laser radar scanning frequencies. Low precision, weak correlation and poor sensitivity are the main obstacles of the existing speed prediction algorithm in the field of automatic driving (such as intelligent vehicle), therefore, a speed prediction algorithm with high precision, strong correlation and high sensitivity is needed.

[0087] In another solution, a deep neural network model is used to fit and predict the speed of a moving target. The prediction principle is that the model learns the labeled attributes of the true value data, which provides the spatial dimensions of the 3D box, the category, the speed value, etc. After the model learns the speed value features, it has the ability to infer the speed value, and can predict and obtain the speed value by inferring unknown data.

[0088] In this solution, the neural network fitting has high accuracy; based on the differentiable agent model, an end-to-end solution can be made. However, the true value data labeling cost for training the model is expensive, and high-precision true value is required to train the model; the neural network structure is complex, and the training cost is high; the inference deployment requires special hardware, which has high power consumption and cost.

[0089] Therefore, the embodiments of the present application provide a speed prediction method of an object, which can realize the speed prediction of the object through physical principles, rather than interpolating the speed, thereby reducing the original linear error caused by interpolating the speed and improving the accuracy of the speed prediction of the object. The technical solution of the embodiments of the present application solves the problem of how to avoid the linear error caused by interpolating the speed in the case that the original spatial information accuracy is not enough, and improves the speed prediction accuracy.

[0090] The embodiments of the present application propose a multi-dimensional adaptive motion model based on multi-source input, construct an adaptive motion equation with multi-physical quantity input, fit and predict the speed value through the physical motion model, reduce the noise interference, and thereby reduce the linear error; at the same time, the motion model can be applied to various input vectors, and the deployment flexibility of the motion model is improved.

[0091] In some implementations, by performing two-stage matching on the observation vector and the prediction vector, the correlation of the time sequence is fully utilized in multiple frames of time sequence, and the consistency of the speed value of the same instance target is improved. In some implementations, the prediction step and the update step of the motion model are dynamically adjusted, so that the low-speed target and the ultra-low-speed target (approximately static) can be better distinguished in different scenes when the sampling frequency is different (low speed, high speed), and the sensitivity of the speed prediction is improved.

[0092] In order to further illustrate the technical effects of the technical solutions of the embodiments of the present application, the application scenarios of the technical solutions of the embodiments of the present application are exemplarily explained below. It should be noted that the technical solutions of the embodiments of the present application can be applied to any application scenario of predicting the speed of a moving target, and the examples of the application scenarios provided by the embodiments of the present application do not limit the application scenarios of the technical solutions of the embodiments of the present application.

[0093] Exemplarily, FIG. 4 shows a schematic diagram of a first application scenario of the object speed prediction method provided by the embodiments of the present application. As shown in FIG. 4, in the application scenario of automatic driving perception decision, the technical solution of the embodiments of the present application can be applied to a vehicle deployed with an automatic driving system, for speed prediction of surrounding vehicles or moving targets. The technical solution of the embodiments of the present application has the characteristics of high-precision, high-sensitivity and strong correlation of speed prediction, and can provide efficient decision basis for emergency braking, obstacle avoidance and the like of the vehicle.

[0094] Exemplarily, FIG. 5 shows a schematic diagram of a second application scenario of the object speed prediction method provided by the embodiments of the present application. As shown in FIG. 5, in the application scenario of automatic driving data truth system, the technical solution of the embodiments of the present application can also be applied to an automatic driving data truth labeling system. The automatic driving data truth labeling system needs to provide high-quality labeling data, which puts high requirements on the capabilities of various models. The method proposed in the present patent can provide high-precision speed prediction values, can bring improvement in labeling efficiency and labeling quality to the automatic driving data truth labeling system, and has obvious application value.

[0095] It needs to be further explained that, exemplarily, the product using the technical solution of the embodiments of the present application can be an intelligent vehicle deployed with an intelligent driving system, or an automatic driving data truth labeling cloud system. For the intelligent vehicle deployed with the intelligent driving system, the technical solution of the embodiments of the present application can be deployed on the computing node of the vehicle-mounted device, and through software algorithm, the high-precision speed prediction capability described in the embodiments of the present application can be provided, which can be used for perception decision in the automatic driving scenario. For the automatic driving data truth labeling cloud system, the technical solution of the embodiments of the present application can be deployed in the cloud reasoning system, and through the cloud server and algorithm tool, the high-precision speed prediction capability described in the embodiments of the present application can be provided, which can bring improvement in labeling efficiency and labeling quality to the automatic driving data truth labeling cloud system.

[0096] Exemplarily, FIG. 6 shows a flowchart of the object speed prediction method provided by the embodiments of the present application. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capability. Exemplarily, the method can be executed by an object speed prediction device, wherein the device can be realized by software and / or hardware, and can be configured in an automatic driving system, typically, can be configured on a vehicle. As shown in FIG. 6, the object speed prediction method mainly includes the following steps:

[0097] In step S610, a first observation vector of the object is obtained, the first observation vector being generated based on an observation value of at least one attribute of the object at a Tth moment, T being an integer greater than or equal to 1. The object includes a static target and a dynamic target. The present application takes the dynamic target as an example for illustration, but this does not constitute a limitation on the scope of the object.

[0098] In step S620, the motion model is adjusted according to the first observation vector, so that the dimension of an input vector processed by the motion model matches the dimension of the first observation vector.

[0099] In a possible implementation, the motion model is modeled according to a Kalman equation set.

[0100] In a possible implementation, a state transition matrix in the motion model is adjusted according to the first observation vector, the state transition matrix being a matrix in the Kalman equation set that is multiplied by the first observation vector.

[0101] In this implementation, since the state transition matrix is the matrix that is multiplied by the first observation vector, the state transition matrix needs to be adjusted to achieve the matching of the state transition matrix and the first observation vector in the dimension, so as to meet the dimension requirement of the calculation. Furthermore, the state transition matrix of the motion model after the adjustment can be applied to all observation vectors, so that the motion model can be any input vector, thereby ensuring the robustness and stability of the motion model.

[0102] In a possible implementation, the state transition matrix before the adjustment is obtained according to a full-quantity vector, the full-quantity vector representing a vector in the input vector set that is used to indicate observation values of all attributes of the object, the all attributes representing all attributes corresponding to the multiple input vectors in the input vector set.

[0103] In this implementation, the state transition matrix is obtained according to the full-quantity vector, and the full-quantity vector contains all attributes in the input vector set. Therefore, the state transition matrix before the adjustment matches the full-quantity vector, and the all attributes in the observation vector are part or all of the all attributes in the full-quantity vector. Thus, the state transition matrix before the adjustment can be applied to all observation vectors after the adjustment, so that the motion model can be any input, thereby avoiding the collapse of the motion model due to the lack of some data in the observation vector of the input.

[0104] Generally, in the actual data detection process, the observation vector of the object can be obtained from multiple sources, and the detection data obtained from each source can be inconsistent in data content. Taking an autonomous vehicle as an example, the autonomous vehicle can detect data through multiple sensors, and the observation vector of the object obtained by the autonomous vehicle can be data containing different attributes, such as the center point coordinates of the object detected by some sensors and the length, width, and height data of the object detected by some sensors. This requires that the motion model can perform prediction work according to multiple forms of observation vectors. In the technical solution of the present application, an input vector set is constructed according to multiple sources of observation vectors. According to the input vector set, a full vector containing all attributes is determined, and according to the full vector, a state transition matrix is constructed, which can make the state transition matrix applicable to all observation vectors, so that the motion model can be any input, thereby ensuring the stability of the motion model.

[0105] In a possible implementation, according to the attributes corresponding to the observation values in the first observation vector, vectors corresponding to each attribute are extracted from the state transition matrix before adjustment, to obtain the state transition matrix after adjustment.

[0106] In this implementation, a method for adjusting a state transition matrix is provided. According to the first observation vector, an attribute set of all attributes is obtained, and according to the attribute set, vectors corresponding to each attribute in the attribute set are extracted from the state transition matrix before adjustment, to obtain the state transition matrix after adjustment. In this way, not only is it ensured that the first observation vector and the state transition matrix remain matched in dimension, which is convenient for calculation, but also each vector in the state transition matrix is matched with the first observation vector, which ensures the correctness of the calculation result.

[0107] In an example, as shown in FIG. 7, the input vector set is obtained according to the geometric attributes and the speed attributes of the observation data. The geometric attributes include the position coordinates (x, y, z, θ) of the 2D or 3D bounding box, the length, width, and height (w, h, l), etc., and the speed attributes include the speed (vx, vy, vz) of the 2D or 3D bounding box. For details, refer to the related content of FIG. 2. According to the different sources of input vectors, multiple input vectors are obtained, and the set of the multiple input vectors is the input vector set. The multiple input vectors in the input vector set can be input vector 1 (x, y, z, θ), input vector 2 (w, h, l), input vector 3 (vx, vy, vz), …, and input vector N (x, y, z, θ, w, h, l, vx, vy, vz) as shown in FIG. 7.

[0108] In this example, as shown in FIG. 8, in order to facilitate analysis of the positions of different attributes in the input vector, the alignment processing obtains corresponding input vectors, such as input vector 1 (x, y, z, θ, 0, 0, 0, 0, 0, 0), input vector 2 (0, 0, 0, 0, w, h, l, 0, 0, 0), input vector 3 (0, 0, 0, 0, 0, 0, 0, vx, vy, vz), …, and input vector N (x, y, z, θ, w, h, l, vx, vy, vz) as shown in FIG. 8. The input vector N is a full vector. If there is no input vector containing all attributes, a full vector containing all attributes can be constructed.

[0109] In this example, as shown in FIG. 8, according to the full vector, a state transition matrix before adjustment is constructed to obtain a state transition matrix with a dimension of (10*10). The dimension in the embodiment of the present application is represented as (number of rows*number of columns). The state transition matrix with a dimension of (10*10) is multiplied by the full vector with a dimension of (10*1) to obtain a prediction vector with a dimension of (10*1) corresponding to the full vector.

[0110] In this example, as shown in FIG. 8, when the input vector is input vector 2 (0, 0, 0, 0, w, h, l, 0, 0, 0), the attribute set including all attributes is (w, h, l), and the corresponding positions are the 5th, 6th, and 7th rows in the full vector. The 5th, 6th, and 7th columns in the state transition matrix before adjustment are extracted to obtain a state transition matrix with a dimension of (10*3). The state transition matrix with a dimension of (10*3) is multiplied by the input vector 2 with a dimension of (3*1) to obtain a prediction vector with a dimension of (10*1) corresponding to the input vector 2.

[0111] In step S630, the first observation vector is taken as the input of the motion model, and a prediction vector is output. The prediction vector indicates the predicted value of at least one attribute of the object at the T+1 time, and the predicted value of the at least one attribute at the T+1 time includes the velocity prediction value at the T+1 time in the prediction vector.

[0112] In step S640, based on the prediction vector, the velocity prediction value of the object at the T+1 time is obtained.

[0113] In the embodiment of the present application, when the data is an image frame, the T time can be understood as the time corresponding to the Tth frame of data, and the T+1 time can be understood as the time corresponding to the T+1th frame of data. When the data is other forms of data, the T time can be understood as the time corresponding to the Tth time of data, and the T+1 time can be understood as the time corresponding to the T+1th time of data. The embodiment of the present application is discussed based on image frames, but does not constitute a limitation on the form of data processed by the embodiment of the present application.

[0114] In a possible implementation, according to the first matching strategy, the predicted vector and at least one second observed vector are matched to obtain a first matching result, each second observed vector indicating an observed value of at least one attribute of an object at the T+1 moment; and the first matching result is that there is a second observed vector matching the predicted vector, and a speed prediction value in the predicted vector is taken as a speed prediction value of the object at the T+1 moment.

[0115] In this implementation, the first matching strategy can be understood as a low-precision matching strategy, where the first matching strategy with lower precision has the characteristics of small calculation amount and fast calculation speed, and can improve the work efficiency of the matching process. The detection model can set a detection box for the object in the image, and use the detection box to express the object. For the position information and size information of the detection box, the first matching strategy with lower precision can obtain a higher matching result, and therefore a lower first threshold value can be set to obtain the matching result. When the first matching result is that there is a second observed vector matching the predicted vector, it indicates the correctness of the predicted vector, and therefore the speed prediction value in the predicted vector can be taken as the speed prediction value of the object at the T+1 moment.

[0116] In a possible implementation, the method further includes that the first matching result is that there is no second observed vector matching the predicted vector, the predicted vector and a plurality of second observed vectors are matched according to a second matching strategy to obtain a second matching result, the matching precision of the second matching strategy is higher than that of the first matching strategy; and the second matching result is that there is a second observed vector matching the predicted vector, and the speed prediction value in the predicted vector is taken as the speed prediction value of the object at the T+1 moment.

[0117] In this implementation, for the observed vectors that are not matched successfully by the first strategy, the second matching strategy is further set in the embodiments of the application. For the objects that are occluded or have ambiguous sizes caused by motion, the confidence in the target detection stage is usually low, that is, corresponding to a low-score detection box, a more accurate matching strategy and a trajectory box can be used for matching, which can improve the matching rate between the detection box and the trajectory box. When the second matching result is that there is a second observed vector matching the predicted vector, it indicates the correctness of the predicted vector, and therefore the speed prediction value in the predicted vector can be taken as the speed prediction value of the object at the T+1 moment. The second matching strategy can improve the matching success rate and reduce the possibility of missing matching. Thus, the correlation and consistency of the same object in multiple frames of time can be better established.

[0118] In the embodiments of the application, the matching process of the observed vector and the predicted vector needs to match the geometric attributes and the speed attributes of the detection box. The matching process of the geometric attributes of the detection box is exemplarily described below.

[0119] For example, the electronic device can divide the detection boxes into high-score detection boxes and low-score detection boxes according to the confidence. For the high-score detection boxes, a low-precision matching strategy is used to match each trajectory box, for example, matching according to the size of the overlapping region. For the low-score detection boxes, a high-precision matching strategy is used to match each trajectory box, for example, matching according to the distance between the center points of the low-score detection box and the trajectory box and the diagonal length of the minimum bounding box, or matching according to the overlapping region, the distance between the center points, and the aspect ratio of the low-score detection box and the trajectory box. The high-precision matching strategy considers more geometric factors than the low-precision strategy, and thus has higher matching precision.

[0120] It should be understood that the electronic device can divide the detection boxes into two or more levels according to the confidence, and the number of specific classifications of the detection boxes is not limited in the present application.

[0121] In this way, for objects that are occluded or have ambiguous sizes caused by motion, the confidence in the target detection stage is usually low, i.e., corresponding to low-score detection boxes, a more accurate matching strategy and trajectory box can be used for matching, which can improve the matching rate between the detection box and the trajectory box. For relatively clear and complete objects, the position information and size information of the detection box obtained in the target detection stage are relatively accurate, and the confidence is also high, i.e., corresponding to high-score detection boxes, a low-precision matching strategy with less computational complexity and trajectory boxes can be used for matching, which can obtain a high matching rate and reduce the computational complexity. Furthermore, the target tracking method provided by the present application can improve the matching rate between the detection box and the trajectory box while reducing the total computational complexity and improving the matching efficiency.

[0122] Alternatively, the electronic device can first use a low-precision matching strategy to match the high-score detection boxes with the trajectory boxes, and then use a high-precision matching strategy to match the low-score detection boxes with the trajectory boxes that do not match the high-score detection boxes. Furthermore, the low-score detection boxes only need to match the remaining trajectory boxes that do not match the high-score detection boxes, which also reduces the computational complexity and improves the matching efficiency.

[0123] For example, FIG. 9 shows a schematic diagram of a two-stage matching method of the object speed prediction method provided by the embodiments of the present application. As shown in FIG. 9, the matching process corresponding to the first matching strategy is also referred to as one-stage matching, and the matching process corresponding to the second matching strategy is also referred to as two-stage matching, and thus the matching method of the embodiments of the present application is also referred to as a two-stage matching method.

[0124] As shown in FIG. 9, according to the observation vector b0 of the T+1th moment, the motion model predicts to obtain the prediction vector a1 of the T+1th frame; according to the first matching strategy, the prediction vector a1 of the T+1th frame is matched with the observation vector b1 of the T+1th frame, there is a matched prediction vector b1, then the speed prediction value of the object in the T+1th frame is extracted from the prediction vector, and the tracking ID of the object corresponding to the observation vector b0 is assigned to the object corresponding to the observation vector b1, so as to realize the tracking of the object.

[0125] As shown in FIG. 9, according to the observation vector b1 of the T+1th frame, the motion model predicts to obtain the prediction vector a2 of the T+2th frame; according to the first matching strategy, the prediction vector a2 of the T+2th frame is matched with the observation vector b2 of the T+2th frame, there is no matched prediction vector b2; according to the second matching strategy, the prediction vector a2 of the T+2th frame is matched with the observation vector b2 of the T+2th frame, there is a matched prediction vector b2, then the speed prediction value of the object in the T+2th frame is extracted from the prediction vector, and the tracking ID of the object corresponding to the observation vector b1 is assigned to the object corresponding to the observation vector b2, so as to realize the tracking of the object.

[0126] As shown in FIG. 9, according to the observation vector b2 of the T+2th frame, the motion model predicts to obtain the prediction vector a3 of the T+3th frame; according to the first matching strategy, the prediction vector a3 of the T+3th frame is matched with the observation vector b3 of the T+3th frame, there is a matched prediction vector b3, then the speed prediction value of the object in the T+3th frame is extracted from the prediction vector, and the tracking ID of the object corresponding to the observation vector b2 is assigned to the object corresponding to the observation vector b3, so as to realize the tracking of the object.

[0127] Therefore, assuming that there is only a one-stage matching process of the first matching strategy, it can be found that the observation vector b2 of the T+2th frame is a matching failure. Therefore, the two-stage matching using the second matching strategy improves the matching success rate of the object, thereby improving the correlation between different time sequences and improving the accuracy of target tracking.

[0128] In other examples, the multi-frame data can also be simultaneously subjected to the one-stage matching of the first matching strategy, and the unmatched observation vector is subjected to the two-stage matching using the second matching strategy.

[0129] In a possible implementation, the method further includes: the second matching result is that there is no second observation vector matched with the prediction vector, the object corresponding to the second observation vector is marked as a new object, and / or the object corresponding to the first observation value is marked as lost.

[0130] In the implementation, for the second observation vector that has not been matched successfully after the matching processing of the first matching strategy and the second matching strategy, the object corresponding to the second observation vector can be marked as a new object, and the trajectory tracking of the new object is started. Optionally, the object can also be marked as lost, and the state of the object at the T+1 moment is recorded.

[0131] In a possible implementation, the method further includes: the first matching result is that there is a second observation vector matching the prediction vector, and the ID of the object corresponding to the second observation vector matching the prediction vector is marked according to the ID of the object.

[0132] In the implementation, the second observation vector matched successfully is assigned with the ID of the object, and the trajectory tracking of the object can be implemented.

[0133] In a possible implementation, the method further includes: updating the motion model at the T+1 moment according to the prediction vector and the second observation vector matching the prediction vector; and adjusting the parameter of the update step of the motion model according to the interval time between frames of the multiple frames of data, the multiple frames of data including the data at the T+1 moment and the previous frame of data or multiple frames of data, and the update step indicating the interval time between the completion of the update of the motion model at the T+1 moment and the next model update.

[0134] In the implementation, a dynamic step update mechanism of the motion model is provided, and the sensitivity of the speed prediction under different sampling frequencies of the observation vector is solved. The parameter of the update step of the motion model is adjusted according to the interval time between frames of the multiple frames of data, which is a mechanism of updating the motion model with a dynamic step and has higher sensitivity. In different sampling frequency scenarios, the motion model can always maintain high prediction sensitivity, the sensitivity and precision of the speed prediction of the dynamic target in different speed ranges are improved, the sensitivity of the motion model to the speed is improved, and different speeds can be distinguished, the speed prediction of the low-speed target is more accurate, and the sensitivity is higher.

[0135] In one example, it is assumed that the current update step of the motion model is suitable for the general photographing frequency. In one scenario, there are more low-speed targets in the image, and a higher frequency is generally used for photographing. If the update step of the motion model is unchanged, when the motion model predicts multiple frames of data, the same state transition matrix, observation matrix, and covariance matrix can be used, which can cause the motion model to incorrectly identify a low-speed object as a stationary object with a speed of zero, and thus the sensitivity of the identified object speed is low. According to the interval time between the multiple frames of data, the parameter of the update step of the motion model is adjusted in the technical solution of the present application, so that when the photographing frequency is high, the update step of the motion model is short, and when the motion model predicts multiple frames of data, different state transition matrices, observation matrices, and covariance matrices are used for each frame, and the motion model can well identify the speed of a low-speed object, and thus the sensitivity of the identified object speed is high.

[0136] In one possible implementation, the frequency of the multiple frames of data is obtained according to the interval time between the frames of the multiple frames of data, and the parameter of the update step of the motion model is adjusted according to the frequency of the multiple frames of data.

[0137] In this implementation, a method for adjusting the parameter of the update step of the motion model is provided. The frequency of the multiple frames of data is obtained according to the interval time between the frames of the multiple frames of data, and the parameter of the update step of the motion model is adjusted according to the frequency of the multiple frames of data. The update step of the motion model can be adapted to the frequency of the multiple frames of data, so as to dynamically change the update step of the motion model, adapt to the frequency of the multiple frames of data, and improve the prediction accuracy.

[0138] In one possible implementation, when the frequency of the multiple frames of data is a single frequency, the reciprocal of the frequency of the multiple frames of data is set as the update step of the motion model, and when the frequency of the multiple frames of data is a mixed frequency, the interval time between the T+1 time data and the previous frame of data is set as the update step of the motion model.

[0139] In this implementation, a method for determining the update step of the motion model under different sampling frequencies is provided. The sampling of the multiple frames of data can be a single frequency or a mixed frequency, and through different update step mechanisms of the motion model, the update step of the motion model can be adjusted according to various actual situations.

[0140] In one example, as shown in FIG. 10, assuming that the frequency of the multi-frame data is frep, the inverse of the frequency of the multi-frame data is 1 / frep. When the single frequency is 1 Hz, the update step s of the motion model is 1; when the single frequency is 2 Hz, the update step s of the motion model is 1 / 2=0.5; when the single frequency is 5 Hz, the update step s of the motion model is 1 / 5=0.2; when the single frequency is 10 Hz, the update step s of the motion model is 1 / 10=0.1. When the frequency of the multi-frame data is a mixed frequency, the update step of the motion model needs to be set according to the interval time of each two frames. When the interval time of each two frames is Δt, the update step s of the motion model is Δt.

[0141] In one possible implementation, the method further comprises adjusting a parameter of a prediction step of the motion model according to an interval time between frames of the multi-frame observation vectors, the multi-frame data comprising observation vectors of a T-th time and a previous frame or multiple previous frames thereof, the prediction step indicating an interval time between a T-th time corresponding to a prediction vector output by the motion model and a T+1-th time corresponding to the prediction vector.

[0142] In this implementation, the prediction step of the motion model can also be adjusted according to the interval time between the frames of the observation vectors, so that the prediction step of the motion model is dynamically adjusted, the matching of the prediction step and the sampling frequency is realized, the time error of the observation vectors and the prediction vectors is reduced, and the matching accuracy of the prediction vectors and the observation vectors is improved.

[0143] In one possible implementation, the geometric information comprises at least one of a center point coordinate, a yaw angle, a length, a width, and a height, and the speed information comprises at least one of a speed or an angular speed in a direction of each coordinate axis in a coordinate system related to the center point coordinate.

[0144] In order to more clearly express the technical solutions provided by the embodiments of the present application and possible implementation manners thereof, the following further describes the entire working process of the motion model.

[0145] For example, FIG. 11 shows a schematic diagram of a speed prediction framework of an object provided by the embodiments of the present application. As shown in FIG. 11, the speed prediction framework of the object comprises a multi-source input vector module, a multi-frame time sequence module, a motion model, and a matching and association module.

[0146] Wherein, as shown in FIG. 11, a device such as a radar or a camera detects a plurality of detection object laser point clouds; a 3D detection model performs data processing according to the laser point clouds to obtain an input vector set; the input vector set is divided into 3D BBOX (Bounding boxes) center point coordinates, 3D BBOX speed values, 3D BBOX center point coordinates and 3D BBOX speed values, …, 3D BBOX geometric dimensions, etc. according to different data sources; under the control of a multi-frame timing module, a motion model predicts according to the observation vector (a subset of the input vector set) of the current frame frame by frame to output a predicted vector of the next frame; a matching and association module matches the predicted vector and the observation vector of the next frame; when there is a matching successful observation vector, the speed prediction value of the object in the next frame is obtained from the predicted vector.

[0147] For example, FIG. 12 shows a flowchart of a motion model predicting the speed of an object according to an embodiment of the present application. As shown in FIG. 12, based on the speed prediction framework of FIG. 11, the process of the motion model predicting the speed of an object mainly includes three processes of prediction, matching, and updating.

[0148] As shown in FIG. 12, the prediction process of the motion model includes: obtaining a first observation vector of the Tth frame; inputting the first observation vector into the motion model, and the motion model outputs a predicted vector of the T+1th frame.

[0149] As shown in FIG. 12, the matching process includes: obtaining one or more second observation vectors of the T+1th frame; performing one-stage matching on the predicted vector of the T+1th frame and the one or more second observation vectors of the T+1th frame according to a first matching principle; judging whether the matching degree is greater than a first threshold value; when the judgment result is that the matching degree is greater than the first threshold value, obtaining the speed prediction value of the object at the T time from the predicted vector; when the judgment result is that the matching degree is not greater than the first threshold value, performing two-stage matching on the predicted vector of the T+1th frame and the one or more second observation vectors of the T+1th frame according to a second matching principle; judging whether the matching degree is greater than a second threshold value; when the judgment result is that the matching degree is greater than the second threshold value, obtaining the speed prediction value of the object at the T time from the predicted vector; when the judgment result is that the matching degree is not greater than the second threshold value, there are three possible processing methods: setting the object corresponding to the unmatched second observation vector as a new object to track the new object; marking the model object corresponding to the first observation vector as lost; deleting the predicted vector.

[0150] As shown in FIG. 12, the updating process of the motion model includes: after matching by the first matching strategy, when there is a second observation vector with a matching degree greater than a first threshold, updating the motion model according to the matched second observation vector and the predicted vector, updating the covariance matrix, the observation matrix, the state transition matrix, etc. of the motion model; after matching by the second matching strategy, when there is a second observation vector with a matching degree greater than a second threshold, updating the motion model according to the matched second observation vector and the predicted vector, updating the covariance matrix, the observation matrix, the state transition matrix, etc. of the motion model.

[0151] The modeling, prediction, matching, and updating processes of the motion model are further explained below.

[0152] (1) Modeling

[0153] In the embodiment of the present application, the velocity prediction method of the motion model is mainly based on a motion model based on Kalman principle. In order to more clearly understand the working principle of the motion model of the present application, the motion model based on Kalman principle is first explained below.

[0154] First, an input vector set is constructed. According to the spatial information of the moving target provided by the 3D target detector, the spatial geometric information (x, y, z, w, h, l, θ) or the velocity information (v x , v y , v z ) of the 3D bbox is used to construct a multi-source observation value input vector, such as the input vector set shown in FIG. 7.

[0155] Second, a state transition matrix is constructed. Based on the multi-source input vector set, the state transition matrix is constructed so that the state transition matrix can adaptively adjust its matrix dimension according to the dimension of the input vector (a subset of the input vector set).

[0156] Finally, the motion model is modeled according to the state transition matrix. The motion model is a multi-dimensional adaptive input vector motion model. Specifically, the motion model can be mathematically modeled based on the formulas (2), (3), and (4) of the Kalman principle.

[0157] State equation: X k+1 = φ k X k + U k Formula (2)

[0158] Observation equation: L k+1 = H k+1 X k+1 + V k+1 Formula (3)

[0159] Initial conditions: C0= Var{X0} Equation (4)

[0160] where X k+1 is the predicted vector, describing the predicted state of the moving object at time k+1 (or k+1 frame data); X k is the observation vector, describing the state of the moving object at time k (or k frame data); φ k is the state transition matrix, describing the state change from time k (or k frame data) to time k+1 (or k+1 frame data); U k is the control input vector or process noise, representing external inputs or random disturbances in the process that affect the state. L k+1 is the observation vector, representing the true measurement value at time k+1 (or k+1 frame data); H k+1 is the observation matrix, describing how the state vector X k+1 is mapped to the observation space; V k+1 is the observation noise, representing random errors in the measurement process. is the initial predicted vector of the state vector, representing the expected value of the state at the initial time (or initial frame data); C0 is the initial state covariance matrix, representing the uncertainty or error of the initial state estimate.

[0161] (2) Prediction

[0162] In the velocity prediction principle of the motion model, prediction is based on the predicted state vector of the moving object at the previous time (or previous frame) data, to predict the velocity of the moving object at the current time (or current frame) data. Prediction mainly includes: predicted vector, predicted covariance matrix and Kalman gain calculation. The calculation formula used in prediction is shown in Equations (5), (6) and (7).

[0163] Predicted vector:

[0164] Predicted covariance matrix:

[0165] Calculate Kalman gain:

[0166] where, is the observation vector, in the initial state, X k+1 is the predicted state vector of the object at time k+1 (or k+1 frame); Q k is the process noise covariance matrix, in the initial state, R k+1 is the observation noise covariance matrix; Covariance matrix for k+1 time (or k+1 frame).

[0167] In the embodiments of the present application, a multi-source input based adaptive motion model is provided, which can be applied to an input vector of multi-source observation values, and the speed is predicted by using the physical principle of the motion model instead of interpolating the speed. The speed of the object is predicted by using the multi-dimensional adaptive motion model, which can reduce the original linear error problem caused by the first-order interpolation calculation speed. According to the dimension of the input vector, the state transition matrix of the motion model is adaptively adjusted, so that the purpose of arbitrary input is achieved, and the motion model has the characteristics of high precision and strong robustness.

[0168] (3) Matching

[0169] The predicted vector of the current frame is matched and associated with the observation vector of the current frame. When there is a matching observation vector, the speed prediction value can be obtained from the predicted vector and used as the final output. The matching method is described in the first matching strategy and the second matching strategy of the present application, and will not be described here.

[0170] The matching method is a two-stage multi-target tracking association matching method. The input vector is divided into two times and sent into the matching module, and the association and matching are performed in two stages. When there is a matching observation vector, the final speed prediction value can be obtained from the predicted vector.

[0171] For example, for a multi-frame time sequence observation vector, the motion model is input according to the time sequence; the predicted vector output by the motion model can be (bbox size, tracking id, speed prediction value); after two-stage matching, when there is a matching observation vector, the speed prediction value of multiple frames can be obtained from the predicted vector; the speed prediction value of the instances on the same trajectory is refined (denoising, smoothing, etc.) in time sequence, and the speed prediction value curve or speed trajectory of the multi-frame input can be obtained.

[0172] In the embodiments of the present application, a two-stage matching based multi-target tracking speed prediction method is provided. The time sequence association is obtained by multi-target matching, and the matching success rate is improved based on two-stage matching, which strengthens the time sequence association, thereby solving the problem of poor time sequence association. That is, the speed prediction is based on tracking in the embodiments of the present application, which fully utilizes the time sequence association, improves the matching rate of the same instance target in different time sequences, and makes the consistency of the speed value of the same instance target better and the time sequence association stronger.

[0173] (3) Update

[0174] The updating is based on matching the observation vector constructed based on the current time (or current frame) data and the state vector of the predicted moving target in the current time (or current frame) data, using the observation vector constructed based on the current time (or current frame) data and the state vector of the predicted moving target in the current time (or current frame) data to update the Kalman filter, and taking the speed prediction value contained in the matched state vector of the moving target in the current time (or current frame) data as the final output. The calculation formula used for updating is shown in formula (8) and formula (9):

[0175] Updating:

[0176] Calculate the estimated covariance matrix:

[0177] Wherein, X k+1 is the state vector of the predicted object in the current time (or current frame) data; L k+1 is the observation vector constructed based on the current time (or current frame) data, I is the unit matrix, K k+1 is the Kalman gain calculated in the prediction stage, H k+1 can be calculated by formula 3.

[0178] In addition, in each Kalman filter, the dimension of the state transition matrix φ k can be adjusted according to the dimension of the input observation vector. Wherein, the dimension of the state vector X k+1 may be but not limited to fixed, such as including the predicted value of all attributes. When adjusting the dimension of the state transition matrix φ k , the dimension of the state transition matrix φ k can be obtained through the dimension of the state vector X k .

[0179] Optionally, the prediction step can also be dynamically adjusted, the prediction step refers to the time interval between the predicted next frame and the current frame, and the calculation formula of the prediction step Δt is shown in formula (10).

[0180] Δt=t k+1 -t k Formula (9)

[0181] Wherein, t k is the time point of the current time (or current frame); t k+1 is the time point of the last time (or last frame).

[0182] In one example, according to the current input sequence data, the frequency of the input sequence data is determined by calculating the inter-frame interval in a time segment. Different sampling frequencies have different step lengths. For example, the interval step length of the motion model update is equal to the inverse of the sampling frequency Δt = 1 / freq. For example, the step lengths in the sampling scenarios of 1Hz, 2Hz, 5Hz, 10Hz and mixed frequency are calculated as follows: 1Hz sampling scenario: step length k = 1; 2Hz sampling scenario: step length k = 0.5; 5Hz sampling scenario: step length k = 0.2; 10Hz sampling scenario: step length k = 0.1; mixed sampling scenario: step length Δt = t k+1 -t k The dynamic step length is calculated, and the model prediction and model update process are performed based on the dynamic step length; the motion model is updated using the dynamic step length; and the motion model predicts the prediction vector of the current instance target. According to the method of the embodiment of the present application, the speed prediction value of the arbitrary sampling frequency input is obtained.

[0183] In the embodiment of the present application, a dynamic step length update mechanism is proposed to solve the sensitivity of speed prediction under different sampling frequencies. That is, the embodiment of the present application proposes a dynamic step length update motion model mechanism with higher sensitivity, so that the motion model can always maintain high prediction sensitivity under different sampling frequency scenarios, and the sensitivity and accuracy of the speed prediction of the dynamic target in different speed intervals are improved. Using the dynamic step length update model, the sensitivity of the motion model to speed is improved, which can distinguish different speeds and more accurately predict the speed of low-speed targets with higher sensitivity. For different sampling frequency scenarios, the motion model update step length (motion model update mechanism) is adjusted adaptively, which significantly improves the sensitivity of speed prediction, especially in low-speed and ultra-low-speed scenarios, the sensitivity of the motion model to speed prediction is improved qualitatively.

[0184] It should be further explained that the working process of the motion model of the embodiment of the present application can be further explained as follows: using the geometric information (x, y, z, w, h, l, θ) or speed information (v x , v y , v z), constructs a multi-source observation value input vector; based on the multi-source input vector, the state transition matrix perceives the dimension of the input vector, and adaptively adjusts the matrix dimension, thereby constructing a multi-dimensional adaptive motion model; the input vector is sent into the motion model twice respectively, and correlation and matching are performed in two stages: in the first stage, all input vectors are matched with the prediction vector of the adaptive motion model, and the prediction vector of the motion model includes the size of the bounding box, the tracking id and the speed prediction value; in the second stage, the prediction vector that is not matched in the first stage is matched and correlated with the input vector again; after the correlation and matching in the two stages are completed, the speed prediction value that is matched is output, and the final speed prediction value of the current frame of the moving target is obtained; after all the matching of the current frame is completed, the dynamic step is calculated, the dynamic step is updated to the motion model, and the prediction vector and correlation matching of the next frame are performed; the above steps are looped until the speed prediction of all frames is completed.

[0185] In the embodiment of the application, how to avoid the linear error caused by the interpolation calculation speed in the case of insufficient original spatial information precision is solved. The embodiment of the application proposes a multi-dimensional adaptive motion model based on multi-source input, constructs an adaptive motion equation with multi-physical quantity input, fits and predicts the speed value through the physical motion model, reduces the noise interference, thereby reducing the linear error, and at the same time, can be applied to various input vector application scenarios, and improves the deployment flexibility.

[0186] In the embodiment of the application, how to improve the correlation of the time sequence in the multi-frame time sequence is solved. The embodiment of the application proposes a two-stage multi-target tracking and speed prediction framework, adopts a two-stage correlation matching framework, realizes a unified framework of multi-target tracking and speed prediction, fully utilizes the time sequence correlation, and improves the correlation of the speed prediction of the same target.

[0187] In the embodiment of the application, how to distinguish low-speed targets and super-low-speed targets (approximately static) under different sampling frequencies (low speed, high speed) is solved. The embodiment of the application proposes a dynamic step updating mechanism, adopts differentiated parameters to realize speed prediction from coarse to fine, and improves the prediction sensitivity.

[0188] That is, based on the above description, it can be seen that the embodiments of the present application and their possible implementation at least propose three improvements: (1) a multi-dimensional adaptive motion model based on multi-source input is proposed, an observation vector is constructed from multiple input source data, and the absence of any input data will not have a large impact on the model, and the robustness is stronger; the adaptive motion model is used, and the multi-source input vector is better for the stability of the motion model; (2) a two-stage multi-target tracking and speed prediction framework is proposed, and the two-stage data matching association method improves the data association matching rate; the correlation of multi-target tracking and the speed prediction accuracy are improved, the time sequence correlation is fully utilized, and the speed prediction consistency and correlation are improved; (3) a dynamic step motion model updating mechanism is proposed, which adaptively adjusts the motion model update step (motion model updating mechanism) for different sampling frequency scenarios; the sensitivity of speed prediction is significantly improved, especially in low speed and ultra-low speed scenarios.

[0189] In order to further illustrate the beneficial effects brought by the technical solutions of the embodiments of the present application, the embodiments of the present application evaluate the method proposed in the embodiments of the present application on a real business test set. The real business set is selected to verify the beneficial effects achieved by the method provided by the embodiments of the present application.

[0190] In the embodiments of the present application, the method proposed in the embodiments of the present application is evaluated from three dimensions of sampling frequency, speed interval and multi-time sequence multi-category. The sampling frequency is selected as 10Hz non-key frame and 2Hz key frame for evaluation set. The speed interval is selected as Static (static), Slow (low speed), Fast (high speed) for evaluation. Multi-time sequence multi-category is selected as CAR (car), TRUCK (truck), PED (pedestrian) and CYCLIST (bicycle) for evaluation.

[0191] The test results of the 10Hz non-key frame evaluation set are described.

[0192] The actual measurement is performed on the real 10Hz non-key frame data evaluation set, and the comparison results are shown in Tables 1, 2, 3 and 4. Tables 1 and 3 are the evaluation results of the comparative example of the existing method, and Tables 2 and 4 are the evaluation results of the method proposed in the embodiments of the present application.

[0193] From Tables 1, 2, 3 and 4, it can be seen that the method proposed in the embodiments of the present application has improved speed absolute value accuracy and angle value accuracy in the main categories (car / truck / pedestrian / bicycle) and the main evaluation range (BEV<=90m / 60m / 30m). The greater the accuracy, the greater the relative error.

[0194] The speed value accuracy is improved by 4.71%, and the angle value accuracy is improved by 11.25%. The BEV (Bird's Eye View) refers to observing a scene from a bird's eye view, and is widely used in automatic driving, intelligent transportation and the like.

[0195] Table 1: Speed absolute value accuracy of 10HZ non-key frame data test set (comparative example)

[0196] Table 2: Speed absolute value accuracy of 10HZ non-key frame data test set (embodiment of the application)

[0197] Table 3: Angular velocity absolute value accuracy of 10HZ non-key frame data test set (comparative example)

[0198] Table 4: Angular velocity absolute value accuracy of 10HZ non-key frame data test set (embodiment of the application)

[0199] The test results of the 2Hz key frame evaluation set are described.

[0200] The 2Hz key frame is a difficult example evaluation set constructed on real business data, and the embodiment of the application actually measures the 2Hz key frame evaluation set. The comparison results are shown in Tables 5, 6, 7 and 8. Tables 5 and 7 are the evaluation results of the existing method, and Tables 6 and 8 are the evaluation results of the method proposed in the embodiment of the application.

[0201] The speed absolute value accuracy and the angle value accuracy of the method proposed in the embodiment of the application are improved in the main categories (car / truck / pedestrian / bicycle) and the main evaluation range (BEV<=90m / 60m / 30m). The speed value accuracy is improved by 9.29%, and the angle value accuracy is improved by 7.13%.

[0202] Table 5: Speed absolute value accuracy of 2HZ key frame data test set (comparative example)

[0203] Table 6: Speed absolute value accuracy of 2HZ key frame data test set (embodiment of the application)

[0204] Table 7: Angular velocity absolute value accuracy of 2HZ key frame data test set (comparative example)

[0205] Table 8: Angular velocity absolute value accuracy of 2HZ key frame data test set (embodiment of the application)

[0206] The evaluation results and beneficial effects on the evaluation set of different speed intervals (Static, Slow, Fast) are shown in Table 9. Compared with the speed prediction error of the comparative example of the existing method, the method proposed in the embodiment of the application reduces the Static target speed error by 52.67%, the Slow target speed error by 45.56%, the Fast target speed error by 44.93%, and the average speed error by 46.3%.

[0207] Table 9: Speed prediction error in different speed intervals

[0208] The evaluation results and beneficial effects on the evaluation set of multi-time sequence and multi-category (car / truck / pedestrian / bicycle) are shown in Table 10 and Table 11. Compared with the speed prediction accuracy of the comparative example of the existing method, the beneficial improvements of the method proposed in the embodiment of the application are as follows: the speed absolute value accuracy is improved by 21.85%, and the speed angle value accuracy is improved by 30.66%.

[0209] IOU (intersection over union) is a matching algorithm that uses the intersection over union of the overlapping area to represent the matching rate. Threshold: IOU = 0.7, indicating that the matching rate is more than 70%, which is a successful match. Threshold: pedestrian distance = 0.5, indicating that the distance between pedestrians is less than 50%, which is a successful match.

[0210] Table 10: Speed absolute value (3D BBOX threshold: IOU = 0.7, pedestrian distance = 0.5)

[0211] Table 11: Speed angle value (3D BBOX threshold: IOU = 0.7, pedestrian distance = 0.5)

[0212] The above results all show that the method proposed in the embodiment of the application can effectively improve the speed prediction accuracy in different scenarios, achieving higher accuracy, stronger correlation, and higher sensitivity algorithm performance.

[0213] It can be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application. In addition, the technical features described in each of the above-described embodiments or involved in the embodiments can be combined according to the actual situation, and the combined scheme is still within the protection scope of the application.

[0214] Based on the method in the above embodiment, the embodiment of the application further provides an object speed prediction device.

[0215] Exemplarily, FIG. 13 shows a structural schematic diagram of an object speed prediction apparatus provided by an embodiment of the present application. As shown in FIG. 13, the object speed prediction apparatus 1300 can include an acquisition module 1310 and a processing module 1320.

[0216] The acquisition module 1310 is configured to acquire a first observation vector of the object, the first observation vector indicating an observation value of at least one attribute of the object at a Tth moment, T being an integer greater than or equal to 1; the processing module 1320 is configured to adjust a motion model according to the first observation vector, so that a dimension of an input vector processed by the motion model matches a dimension of the first observation vector; and output a prediction vector as an input of the motion model, the prediction vector indicating a prediction value of the at least one attribute of the object at a (T+1)th moment, the prediction value of the at least one attribute including a speed prediction value; and obtain the speed prediction value of the object at the (T+1)th moment based on the prediction vector.

[0217] In a possible implementation, the motion model is modeled according to a Kalman equation set.

[0218] In a possible implementation, the processing module 1320 is specifically configured to adjust a state transition matrix in the motion model according to the first observation vector, the state transition matrix being a matrix in the Kalman equation set that is multiplied by the first observation vector.

[0219] In a possible implementation, the processing module 1320 is specifically configured to extract a vector corresponding to each attribute from a state transition matrix before adjustment, to obtain a state transition matrix after adjustment, according to the attribute to which an observation value in the first observation vector corresponds.

[0220] In a possible implementation, the state transition matrix before adjustment is obtained according to a full-quantity vector, the full-quantity vector representing a vector in an input vector set that is used to indicate observation values of all attributes of the object, the all attributes representing all attributes corresponding to multiple input vectors in the input vector set.

[0221] In a possible implementation, the processing module 1320 is specifically configured to acquire at least one second observation vector, each second observation vector being generated based on an observation value of at least one attribute of one detected object at the (T+1)th moment; and perform matching on the prediction vector and the at least one second observation vector, to obtain the speed prediction value of the object at the (T+1)th moment.

[0222] In a possible implementation, the processing module 1320 is specifically configured to: match the prediction vector and the at least one second observation vector according to a first matching strategy to obtain a first matching result; the first matching result is that there is a second observation vector matching the prediction vector, and a speed prediction value in the prediction vector is taken as a speed prediction value of the object at the T+1 moment; the first matching result is that there is no second observation vector matching the prediction vector, and the prediction vector and a plurality of second observation vectors are matched according to a second matching strategy to obtain a second matching result, a matching accuracy of the second matching strategy is higher than a matching accuracy of the first matching strategy; and the second matching result is that there is a second observation vector matching the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment.

[0223] In a possible implementation, the processing module 1320 is further configured to: the second matching result is that there is no second observation vector matching the prediction vector, and the object corresponding to the second observation vector is marked as a new object, and / or the object corresponding to the first observation value is marked as lost.

[0224] In a possible implementation, the processing module 1320 is further configured to: the first matching result is that there is a second observation vector matching the prediction vector, and the ID of the detection object corresponding to the second observation vector matching the prediction vector is marked according to the ID of the object.

[0225] In a possible implementation, the processing module 1320 is further configured to: adjust the update step of the motion model based on interval times of a plurality of moments of the detection data, the plurality of moments including the T moment and the T+1 moment.

[0226] In a possible implementation, the processing module 1320 is specifically configured to: when the sampling frequency of the detection data is a single frequency, set the reciprocal of the sampling frequency as the update step of the motion model; and when the frequency of the plurality of frames of data is a hybrid frequency, set the interval time between the T moment and the T+1 moment as the update step of the motion model.

[0227] In a possible implementation, the processing module 1320 is further configured to: adjust the prediction step of the motion model based on interval times of a plurality of moments of the detection data, the plurality of moments including the T moment and the T+1 moment.

[0228] In a possible implementation, the first observation vector is used to indicate geometric information and / or speed information of the object, the geometric information is obtained based on attribute information of a bounding box of the object output by a 3D detection model, and the speed information is obtained based on a speed prediction model.

[0229] In a possible implementation, the speed prediction value is used as a basis for driving decisions of the autonomous vehicle; and / or, the speed prediction value is used as a speed label value of the object at the T+1 moment.

[0230] It should be understood that the above device is used to execute the method in the above embodiments, and the corresponding program modules in the device, the implementation principles and technical effects of which are similar to the description in the above method, and the working process of the device can refer to the corresponding process in the above method, which will not be described here.

[0231] Based on the method in the above embodiments, the embodiments of the present application further provide an electronic device. As shown in FIG. 14, the electronic device 1400 includes a bus 1402, a processor 1404, a memory 1406 and a communication interface 14015. The processor 1404, the memory 1406 and the communication interface 14015 communicate through the bus 1402. The electronic device 1400 can be, but is not limited to, a device configured with an automatic driving function, such as a vehicle, etc. It should be understood that the present application does not limit the number of processors and memories in the electronic device 1400.

[0232] The bus 1402 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 14, but it does not mean that there is only one bus or only one type of bus. The bus 1404 can include a path for transmitting information between various components (e.g., the memory 1406, the processor 1404, the communication interface 14015) of the electronic device 1400.

[0233] The processor 1404 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0234] The memory 1406 can include volatile memory (such as random access memory (RAM)), and the processor 1404 can further include non-volatile memory (such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0235] The memory 1406 stores executable program code, and the processor 1404 executes the executable program code to respectively implement the functions of the foregoing described modules in FIG. 6, so as to implement all or part of the steps of the method in the foregoing embodiments. That is, the memory 1406 stores instructions for executing all or part of the steps of the method in the foregoing embodiments.

[0236] Alternatively, the memory 1406 stores executable program code, and the processor 1404 executes the executable program code to respectively implement the functions of the foregoing described object speed prediction apparatus 600, so as to implement all or part of the steps of the method in the foregoing embodiments. That is, the memory 1406 stores instructions for executing all or part of the steps of the method in the foregoing embodiments.

[0237] The communication interface 1403 uses a transceiver module such as, but not limited to, a network interface card and a transceiver to implement communication between the electronic device 1400 and other devices or communication networks.

[0238] Based on the method in the foregoing embodiments, the embodiments of the present application further provide a vehicle, which is provided with the electronic device 1400 in the foregoing embodiments.

[0239] Based on the method in the foregoing embodiments, the embodiments of the present application further provide an object speed prediction apparatus. Please refer to FIG. 15, which is a structural schematic diagram of another object speed prediction apparatus provided by the embodiments of the present application. As shown in FIG. 15, the object speed prediction apparatus 1500 includes one or more processors 1501 and interface circuit 1502. Optionally, the object speed prediction apparatus 1500 can also include a bus 1503. Wherein:

[0240] The processor 1501 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 1501. The processor 1501 described above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method and step disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0241] The interface circuit 1502 can be used for transmitting or receiving data, instructions or information. The processor 1501 can process the data, instructions or other information received by the interface circuit 1502, and can send the processed information out through the interface circuit 1502.

[0242] Optionally, the object speed prediction device 1500 further includes a memory, which can include a read-only memory and a random access memory, and provide operation instructions and data for the processor. Part of the memory can also include a non-volatile random access memory (NVRAM).

[0243] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which can be stored in an operating system).

[0244] Optionally, the interface circuit 1502 can be used to output the execution result of the processor 1501.

[0245] It should be noted that the functions of the processor 1501 and the interface circuit 1502 respectively can be realized by hardware design, software design or combination of hardware and software, which is not limited here.

[0246] It should be understood that each step of the above method embodiment can be completed by the logic circuit in the form of hardware or the instruction in the form of software in the processor.

[0247] Based on the method in the above embodiments, the embodiments of the present application provide a computer readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device cluster including at least one computing device, the computing device cluster executes the method in the above embodiments. Illustratively, the computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device, including one or more available media or data storage devices of a data center, etc. The available medium can be a magnetic medium, (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state hard drive), etc.

[0248] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product including instructions, when the instructions are executed by a computing device cluster including at least one computing device, the computing device cluster executes the method in the above embodiments.

[0249] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0250] The method steps in the embodiments of the present application can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0251] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as a coaxial cable, an optical fiber, a digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0252] It can be understood that the various numerical numbers involved in the embodiments of the present application are only used for differentiation for convenience of description, and do not limit the scope of the embodiments of the present application.

[0253] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of predicting the velocity of an object, characterized by, The method comprises: obtaining a first observation vector of an object, the first observation vector indicating an observation value of at least one attribute of the object at a Tth moment, T being an integer greater than or equal to 1; adjusting a motion model according to the first observation vector, so that a dimension of an input vector processed by the motion model matches a dimension of the first observation vector; inputting the first observation vector into the motion model to output a prediction vector, the prediction vector indicating a prediction value of at least one attribute of the object at a (T+1)th moment, the prediction value of the at least one attribute at the (T+1)th moment including a speed prediction value at the (T+1)th moment; obtaining the speed prediction value of the object at the (T+1)th moment based on the prediction vector.

2. The method of claim 1, wherein, The motion model is modeled according to a Kalman equation set.

3. The method of claim 2, wherein, The adjusting of the motion model according to the first observation vector comprises: adjusting a state transition matrix in the motion model according to the first observation vector, the state transition matrix being a matrix in the Kalman equation set that is multiplied by the first observation vector.

4. The method of claim 3, wherein, The adjusting of the state transition matrix in the motion model according to the first observation vector comprises: extracting a vector in the state transition matrix before adjustment to obtain a state transition matrix after adjustment according to an attribute corresponding to an observation value in the first observation vector.

5. The method according to claim 3 or 4, characterized in that, The state transition matrix before adjustment is obtained according to a full-quantity vector, the full-quantity vector representing a vector in a set of input vectors for indicating observation values of all attributes of the object, the all attributes representing all attributes corresponding to multiple input vectors in the set of input vectors.

6. The method according to any one of claims 1 to 5, characterized in that, The obtaining of the speed prediction value of the object at the (T+1)th moment based on the prediction vector comprises: obtaining at least one second observation vector, each second observation vector indicating an observation value of at least one attribute of an object at the (T+1)th moment; matching the prediction vector and the at least one second observation vector to obtain the speed prediction value of the object at the (T+1)th moment.

7. The method of claim 6, wherein, The matching of the prediction vector and the at least one second observation vector to obtain the speed prediction value of the object at the (T+1)th moment comprises: matching the prediction vector and the at least one second observation vector according to a first matching strategy to obtain a first matching result; the first matching result being that there is a second observation vector matching the prediction vector, and the speed prediction value in the prediction vector being taken as the speed prediction value of the object at the (T+1)th moment; the first matching result being that there is no second observation vector matching the prediction vector, and the prediction vector and the multiple second observation vectors being matched according to a second matching strategy to obtain a second matching result, the matching accuracy of the second matching strategy being higher than that of the first matching strategy; the second matching result being that there is a second observation vector matching the prediction vector, and the speed prediction value in the prediction vector being taken as the speed prediction value of the object at the (T+1)th moment.

8. The method of claim 7, wherein, The method further comprises: The second matching result is that there is no second observation vector matching the prediction vector, the object corresponding to the second observation vector is marked as a new object, and / or the object corresponding to the first observation value is marked as lost.

9. The method according to any one of claims 7-8, characterized in that, Further comprising: The first matching result is that there is a second observation vector matching the prediction vector, and the ID of the detection object corresponding to the second observation vector matching the prediction vector is marked according to the ID of the object.

10. The method according to any one of claims 1 to 9, characterized in that, Further comprising: Adjusting the update step of the motion model based on the interval time of multiple time points of the detection data of the object, the multiple time points including the Tth time point and the T+1th time point.

11. The method of claim 10, wherein, The adjusting the update step of the motion model based on the interval time of multiple time points of the detection data of the object includes: When the sampling frequency of the detection data of the object is a single frequency, the reciprocal of the sampling frequency is set as the update step of the motion model; When the sampling frequency of the detection data of the object is a mixed frequency, the interval time of the Tth time point and the T+1th time point is set as the update step of the motion model.

12. The method according to any one of claims 1 to 11, characterized in that, Further comprising: Adjusting the prediction step of the motion model based on the interval time of multiple time points of the detection data of the object, the multiple time points including the Tth time point and the T+1th time point.

13. The method according to any one of claims 1 to 12, characterized in that, The first observation vector is used to indicate the geometric information and / or the speed information of the object, the geometric information is obtained based on the attribute information of the bounding box of the object output by the 3D detection model, and the speed information is obtained based on the speed prediction model.

14. The method according to any one of claims 1 to 13, characterized in that, The speed prediction value is used as a basis for driving decision of an autonomous vehicle; and / or, the speed prediction value is used as a speed label value of the object at the T+1th time point.

15. A speed prediction device of an object, characterized by, Comprising: An acquisition module is configured to acquire a first observation vector of an object, the first observation vector indicating an observation value of at least one attribute of the object at a Tth time point, T being an integer greater than or equal to 1; A processing module is configured to adjust a motion model according to the first observation vector, so that the dimension of an input vector processed by the motion model matches the dimension of the first observation vector, the motion model being modeled according to a Kalman equation set; and input the first observation vector into the motion model to output a prediction vector, the prediction vector indicating a prediction value of at least one attribute of the object at a T+1th time point, the prediction value of the at least one attribute at the T+1th time point including a speed prediction value at the T+1th time point; and obtain a speed value of the object at the T+1th time point based on the prediction vector.

16. The apparatus of claim 15, wherein, The motion model is modeled according to a Kalman equation set.

17. The apparatus of claim 16, wherein, The processing module is specifically configured to: Adjust a state transition matrix in the motion model according to the first observation vector, the state transition matrix being a matrix in the Kalman equation set that is multiplied by the first observation vector.

18. The apparatus of claim 17, wherein, The processing module is specifically configured to: Extract a vector in the state transition matrix before adjustment according to an attribute corresponding to the observation value in the first observation vector to obtain a state transition matrix after adjustment.

19. The apparatus of claim 17 or 18, wherein, The state transition matrix before adjustment is obtained according to a full quantity vector, and the full quantity vector represents a vector of observation values in the input vector set for indicating all attributes of the object, and the all attributes represent all attributes corresponding to multiple input vectors in the input vector set.

20. The apparatus of any of claims 15-19, wherein, The processing module is specifically configured to: obtain at least one second observation vector, each second observation vector indicating an observation value of at least one attribute of an object at a T+1 moment; match the prediction vector and the at least one second observation vector to obtain a speed prediction value of the object at the T+1 moment.

21. The apparatus of claim 20, wherein, The processing module is specifically configured to: match the prediction vector and the at least one second observation vector according to a first matching strategy to obtain a first matching result; the first matching result is that there is a second observation vector matched with the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment; the first matching result is that there is no second observation vector matched with the prediction vector, and the prediction vector and the multiple second observation vectors are matched according to a second matching strategy to obtain a second matching result, and a matching accuracy of the second matching strategy is higher than that of the first matching strategy; the second matching result is that there is a second observation vector matched with the prediction vector, and the speed prediction value in the prediction vector is taken as the speed prediction value of the object at the T+1 moment.

22. The apparatus of claim 21, wherein, The processing module is further configured to: the second matching result is that there is no second observation vector matched with the prediction vector, and the object corresponding to the second observation vector is marked as a new object, and / or the object corresponding to the first observation value is marked as lost.

23. The apparatus of any of claims 21-22, wherein, The processing module is further configured to: the first matching result is that there is a second observation vector matched with the prediction vector, and the ID of the detection object corresponding to the second observation vector matched with the prediction vector is marked according to the ID of the object.

24. The apparatus of any one of claims 15-23, wherein, The processing module is further configured to: adjust an update step length of the motion model based on interval times of multiple moments of detection data of the object, and the multiple moments include the T moment and the T+1 moment.

25. The apparatus of claim 24, wherein, The processing module is specifically configured to: when a sampling frequency of the detection data of the object is a single frequency, set a reciprocal of the sampling frequency as the update step length of the motion model; when the sampling frequency of the detection data of the object is a mixed frequency, set interval times of the T moment and the T+1 moment as the update step length of the motion model.

26. The apparatus of any one of claims 15-25, wherein, The processing module is further configured to: adjust a prediction step length of the motion model based on interval times of multiple moments of detection data of the object, and the multiple moments include the T moment and the T+1 moment.

27. The apparatus of any of claims 15-26, wherein, The first observation vector is used to indicate geometric information and / or speed information of the object, the geometric information is obtained based on attribute information of a bounding box of the object output by a 3D detection model, and the speed information is obtained based on a speed prediction model.

28. The apparatus of any of claims 15-27, wherein, The speed prediction value is used as a basis for driving decisions of the autonomous vehicle; and / or, the speed prediction value is used as a speed label value of the object at the T+1 moment.

29. A speed prediction device for an object, characterized by comprise at least one processor and an interface; the at least one processor obtains program instructions through the interface; the at least one processor is configured to execute the program instructions to implement the method of any one of claims 1-14.

30. An electronic device, comprising: comprise: at least one memory configured to store a program; at least one processor configured to execute the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to execute the method of any one of claims 1-14.

31. A vehicle characterized by comprise the electronic device of claim 30.

32. A computer readable storage medium storing a computer program, which, when executed on a processor, causes the processor to execute the method of any one of claims 1-14.

33. A computer program product, characterised in that, when the computer program product is executed on a processor, causes the processor to execute the method of any one of claims 1-14.

Images