Target trajectory prediction method and device based on attitude perception

Abstract

The invention discloses a target trajectory prediction method and device based on attitude perception. The method comprises the following steps: acquiring state information of a target; determining attitude information of the target based on the state information; according to the state information and attitude information of the target determined based on the state information, speculating the probability that the target may move according to each preset motion mode; and predicting the trajectory of the target according to the probability. According to the invention, the state information and the attitude information of the target are effectively utilized to deduce or update the probability of the preset motion mode, and then the trajectory of the target is predicted by utilizing the probability, so that the accuracy of predicting the trajectory of the target is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method and device for predicting target trajectory based on attitude perception. Background Art

[0002] Trajectory prediction of targets (such as vehicles, cyclists, and pedestrians) is an important topic in the field of vehicle or traffic intelligence. For autonomous driving software systems, the target trajectory prediction module is generally located between the perception module and the planning module. It receives information from the perception module and then provides prediction information to the planning module, thereby helping the planning module set actions to achieve safe driving and control of the vehicle.

[0003] Currently popular target trajectory prediction methods include the following two categories: 1) Methods based on physical models. IMM is a representative method of this type. It uses a variety of kinematic models (such as constant velocity CV, constant acceleration CA, constant turn CT, etc.) to describe the possible motion patterns of the target, and uses the Kalman filter to further improve the robustness and accuracy of the prediction results. The disadvantage of this type of method is that the target information used is limited, mainly the target state information within the past few seconds, and the accuracy is low; 2) Methods based on deep learning. The models of this type of method are developing rapidly. The disadvantage is that a large amount of data is required for model training and it is highly dependent on data. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a method and device for target trajectory prediction based on posture perception that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of the present invention, a method for predicting target trajectory based on posture perception is provided, the method comprising:

[0006] Get the target's status information;

[0007] Determining posture information of the target based on the state information;

[0008] Inferring the probability of the target moving according to each preset motion mode according to the state information and the posture information of the target determined based on the state information; and

[0009] A trajectory of the target is predicted based on the probability.

[0010] In some implementations, inferring the probability that the target moves according to each preset motion mode according to the state information and the posture information of the target determined based on the state information includes:

[0011] Inputting the state information into a tracking trajectory prediction model to obtain an initial probability of predicting that the target moves according to each preset motion mode;

[0012] The initial probability is updated using the posture information to obtain the probability.

[0013] In some implementations, the tracking trajectory model includes an interacting multiple model.

[0014] In some implementations, inputting the state information into a tracking trajectory prediction model to obtain an initial probability that the target may move according to each preset motion mode includes:

[0015] According to the correlation coefficient between each model in the interactive multi-model, each state estimation value and covariance matrix output by each model at the previous time point are mixed respectively; wherein the covariance matrix is ​​obtained based on the difference between the state estimation value and the target measurement value, and each model corresponds to each preset motion mode respectively;

[0016] Inputting the mixed result and the measured value at the current time point into the Kalman filter of each model respectively to obtain updated state estimation values;

[0017] Calculating the likelihood value of each model based on the mixed result, and obtaining the updated confidence of each model according to the likelihood value;

[0018] Based on the updated state estimation values ​​and the confidence levels, the comprehensive state estimation value at the current time point and the initial probability of each model are obtained.

[0019] In some implementations, determining the posture information of the target based on the state information includes:

[0020] obtaining the speed of the target;

[0021] Determining a horizontal velocity component and a vertical velocity component of the velocity of the target in a predetermined coordinate system;

[0022] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0023] In some implementations, determining the posture information of the target based on the state information includes:

[0024] Obtaining a relative speed of the target relative to the vehicle;

[0025] Determining a horizontal velocity component and a vertical velocity component of the relative velocity in a relative coordinate system;

[0026] Determining the heading angle of the target according to the horizontal velocity component and the vertical velocity component; and / or

[0027] Get the speed of the vehicle;

[0028] Determining an absolute speed of the target according to the speed of the vehicle and a relative speed of the target relative to the vehicle;

[0029] Determining a horizontal velocity component and a vertical velocity component of the absolute velocity in an absolute coordinate system;

[0030] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0031] In some implementations, the speed of the vehicle or the target is obtained from a motion sensor, or the speed of the vehicle or the target is obtained from Internet of Vehicles communications.

[0032] In some implementations, the heading angle of the target is determined based on an inverse tangent function or a dot product formula of the horizontal velocity component and the vertical velocity component.

[0033] In some implementations, determining the posture information of the target based on the state information includes:

[0034] Acquire an image of the target from an image sensor;

[0035] Inputting the image into a trained deep learning model;

[0036] The orientation angle of the target in the image is detected according to the deep learning model.

[0037] In some implementations, detecting the orientation angle of the corresponding target in the image according to the deep learning model includes:

[0038] Identify objects in images;

[0039] Assigning a label frame to the target, so that the label frame surrounds the target following the target contour;

[0040] The heading angle of the target is determined based on the orientation of the annotation bounding box.

[0041] In some implementations, the target includes a pedestrian or a vehicle, and detecting the orientation angle of the corresponding target in the image according to the deep learning model includes:

[0042] Identify pedestrian faces or vehicle wheels in images;

[0043] The orientation angle of the pedestrian's face or the orientation angle of the vehicle's wheels is detected according to the deep learning model.

[0044] In some implementations, updating the initial probability using the posture information to obtain the probability includes:

[0045] Taking the target as a starting point, dividing the area in front of the target into a plurality of angular sub-areas with reference to the moving direction of the target;

[0046] Determine the angle sub-region of the corner indentation;

[0047] The initial probability of each preset motion mode is updated based on the determined angle sub-region to obtain the probability.

[0048] In some implementations, updating the initial probability of each preset motion mode based on the determined angle sub-region includes obtaining the probability including:

[0049] A probability update formula for each preset motion mode is specified for each angle sub-region;

[0050] The initial probability is input into the probability update formula to which the determined angle sub-region belongs to obtain the probability.

[0051] In some embodiments, the probability update formula is configured to reduce a first initial probability of a first preset motion mode and increase a second initial probability of a second preset motion mode based on a determined angle sub-region, wherein the amount of reduction in the first initial probability is added to the second initial probability according to a predetermined coefficient.

[0052] In some embodiments, predicting the trajectory of the target according to the probability includes:

[0053] Determine a maximum value among the probabilities of the preset motion patterns, and use the preset motion pattern corresponding to the maximum value as the prediction result; and / or

[0054] synthesizing the preset motion models according to corresponding probabilities to obtain a synthesized motion model as a prediction result; and / or

[0055] The state of the target is predicted based on the probability and the tracking trajectory model.

[0056] According to another aspect of the present invention, a tracking trajectory prediction device based on posture perception is provided, the device comprising:

[0057] A state acquisition module, adapted to acquire state information of a target;

[0058] A posture determination module, which determines the posture information of the target based on the state information;

[0059] A probability inference module, adapted to infer the probability of the target moving according to each preset motion mode according to the state information and the posture information of the target determined based on the state information;

[0060] A trajectory prediction module is adapted to predict the trajectory of the target according to the probability.

[0061] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes a target trajectory prediction method based on posture perception according to any one of the above items.

[0062] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the target trajectory prediction method based on posture perception according to any of the above items is implemented.

[0063] It can be seen from the above that according to the above technical solution disclosed in the present invention, the posture information of the target is effectively utilized, especially for updating the probabilities of multiple preset motion modes, thereby improving the accuracy of predicting the target trajectory.

[0064] In particular, the above technical solution can greatly improve the accuracy of prediction of each motion mode. For example, the probability of predicting a certain motion mode can be increased from 60% to 90%.

[0065] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0067] Figure 1 A schematic diagram of a process flow of a method for predicting a target trajectory based on posture perception according to some embodiments of the present invention is shown;

[0068] Figure 2 A schematic flow chart showing the steps of implementing interactive multi-models according to some embodiments of the present invention is shown;

[0069] Figure 3 A schematic diagram showing the use of millimeter wave radar to sense attitude information according to some embodiments of the present invention is shown;

[0070] Figure 4A schematic diagram showing the use of an image sensor to perceive posture information according to some embodiments of the present invention;

[0071] Figure 5 Schematic diagrams of three models in the IMM model according to some embodiments of the present invention are shown;

[0072] Figure 6 A schematic diagram showing dividing the front of a target into several areas according to some embodiments of the present invention;

[0073] Figure 7 A schematic diagram showing a region where a target falls according to a direction angle perceived by the target is determined according to some embodiments of the present invention;

[0074] Figure 8 A schematic diagram showing the structure of a target trajectory prediction device based on posture perception according to some embodiments of the present invention is shown;

[0075] Fig. 9 A schematic structural diagram of an electronic device according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0076] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0077] Specifically, Figure 1 A flow chart of a method for predicting a target trajectory based on posture perception according to some embodiments of the present invention is shown. The method is executed by a vehicle (ego) controller and includes the following steps:

[0078] Step S110, obtaining the state information of the target;

[0079] Step S120, determining the posture information of the target based on the state information;

[0080] Step S130, inferring the probability that the target may move according to each preset motion mode according to the state information and the posture information of the target determined based on the state information; and

[0081] Step S140: predicting the trajectory of the target according to the probability.

[0082] The technical solution disclosed in this embodiment improves the accuracy of trajectory prediction by inferring and predicting the possibility of occurrence of each preset motion mode based on the posture information of the target, and predicting the trajectory of the target according to the magnitude of the possibility.

[0083] In some optional embodiments, in step S130, inferring the probability that the target may move according to each preset motion mode according to the state information and the posture information of the target determined based on the state information includes:

[0084] Inputting the state information into a tracking trajectory prediction model to obtain an initial probability that the target may move according to each preset motion mode;

[0085] The initial probability is updated using the posture information to obtain the probability.

[0086] Specifically, the above-mentioned status information includes real-time video, image and / or motion parameters of the target, etc. The above-mentioned tracking trajectory prediction model can be a physical model or a deep learning module, and there is no specific restriction. The motion mode can include: linear motion (such as uniform linear motion, accelerated and decelerated linear motion), left turn motion (such as left turn motion with different turning radii, uniform left turn motion, accelerated and decelerated left turn motion), right turn motion (such as right turn motion with different turning radii, uniform right turn motion, accelerated and decelerated right turn motion).

[0087] The above-mentioned posture information mainly includes various angle information of the target.

[0088] The step of updating the initial probability based on the posture information in step S130 to obtain the updated probability of each motion mode is to associate the posture information with the motion mode. In some optional embodiments, the probability of the corresponding motion mode can be updated according to the matching degree of the posture information, such as the orientation angle of the target or its part, and each motion mode. For example, the orientation angle of the target is associated with the turn, and if the orientation angle is toward the left or right, the probability of the corresponding turn direction is increased.

[0089] In step S140, the trajectory of the target is predicted according to the probability. The trajectory can be predicted according to the above model, or the probability can be returned to the tracking trajectory prediction model to obtain a comprehensive trajectory prediction result, thereby improving the accuracy of trajectory prediction.

[0090] In some specific embodiments, the tracking trajectory model includes an interacting multi-model IMM.

[0091] Interactive multi-model IMM is a state estimation model that continues the idea of ​​early multi-model algorithms and adds a posteriori information to update the model probability. A motion model set is set to include multiple tracking models, each model corresponds to a motion mode, and each model has a corresponding filter and probability, and the filter is preferably a Kalman filter. When the motion state of the target changes, different models are converted to each other through the state moment transfer matrix. When the target state matches the current model, the model will be assigned a higher probability. Conversely, when the model does not match the target state, a lower probability will be assigned to the current model. The filter corresponding to each model is responsible for completing the state estimation and model probability update functions, and then the filter outputs of multiple models are weighted to obtain the fused state estimation output, the covariance and probability of the posteriori state estimation error. Following this approach, the current state estimate and error covariance are used to complete the recursion at the next moment.

[0092] Further, in some embodiments, the step S130 of inputting the state information into the tracking trajectory prediction model to obtain the initial probability of predicting that the target may move according to each preset motion mode specifically includes:

[0093] Step S131, according to the correlation coefficient between each model in the interacting multi-model, each state estimation value and covariance matrix output by each model at the previous time point are mixed respectively.

[0094] The covariance matrix is ​​obtained based on the difference between the state estimation value and the target measurement value, and each of the models corresponds to each of the motion modes.

[0095] Step S132, inputting the mixed result and the measurement value at the current time point into the Kalman filter of each model respectively to obtain updated state estimation values.

[0096] Step S133, calculating the likelihood value of each of the models based on the mixed result, and obtaining the updated confidence of each of the models according to the likelihood value.

[0097] Step S134, based on the updated state estimation values ​​and the confidence levels, a comprehensive state estimation value at the current time point is obtained, and based on the confidence levels, an initial probability of each model is obtained.

[0098] Specifically, combined Figure 2 As shown, the implementation of the interactive multi-model includes the following steps:

[0099] S131 mixing, adding the state estimation value and covariance matrix output from each model at the previous time point respectively according to the correlation coefficient between each model, to obtain the mixed state estimation value and covariance matrix.

[0100] S132 model estimation, that is, the state estimation value, covariance matrix obtained by the above mixture and the observation value obtained according to the sensor at the current time point are input into the Kalman filter of each model to obtain the prediction and subsequent update value, thereby obtaining the state estimation and covariance matrix of each model output at the current time point.

[0101] S133 probability estimation, first, calculate the likelihood value of each model based on the mixed state estimate and covariance matrix. The likelihood value is an indicator used to judge the quality of the model prediction value. Then, based on the likelihood value, update the confidence of each model. The model with a larger likelihood value can obtain a greater confidence. The sum of the confidences of each model at each time point is 1.

[0102] S134 fusion output, the estimated values ​​of all models are merged with their confidence as the proportion to generate the final estimated state of the target. At the same time, the confidence values ​​are integrated to obtain the initial probability of each model.

[0103] It should be noted that when the tracking trajectory model includes an interactive multi-model IMM, combined with Figure 2 As shown by the dotted line, the present invention also introduces posture information, uses the posture information to update the above confidence again to obtain the confidence corresponding to the probability, and then uses the above S134 fusion output step to fuse the confidence corresponding to the probability and the state estimation of each model at the current time point to obtain the updated final estimated state.

[0104] In some specific embodiments, determining the posture information of the target based on the state information in step S120 includes:

[0105] obtaining the speed of the target;

[0106] Determining a horizontal velocity component and a vertical velocity component of the velocity of the target in a predetermined coordinate system;

[0107] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0108] The target speed in this embodiment can be obtained from a detection sensor fixed on the ground, such as a speed measuring instrument installed on a road, etc. The posture information at this time is the posture relative to the detection sensor, and the orientation angle is equivalent to the orientation angle of the detection sensor.

[0109] In some embodiments, determining the posture information of the target based on the state information in step S120 includes:

[0110] Obtaining a relative speed of the target relative to the vehicle;

[0111] Determining a horizontal velocity component and a vertical velocity component of the relative velocity in a relative coordinate system;

[0112] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0113] This embodiment can be used in the tracking analysis of the vehicle in front of the vehicle. The orientation angle at this time is the orientation angle of the target relative to the vehicle, which is a relative angle. Of course, it can also be used for the vehicle to track moving pedestrians or other objects.

[0114] In some optional embodiments, determining the posture information of the target based on the state information in step S120 further includes:

[0115] Get the speed of the vehicle;

[0116] Determining an absolute speed of the target according to the speed of the vehicle and a relative speed of the target relative to the vehicle;

[0117] Determining a horizontal velocity component and a vertical velocity component of the absolute velocity in an absolute coordinate system;

[0118] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0119] Combination Figure 3 As shown in the figure, taking vehicle trajectory prediction and tracking as an example, the vehicle-mounted laser radar or millimeter wave radar can be used to sense the speed of other vehicles. The ego vehicle (the vehicle behind) can sense the speed v of the target vehicle (the vehicle in front) x and v y , and then calculate the direction angle of the vehicle speed It should be noted that the speed directly perceived by the ego vehicle is the relative speed of the target vehicle. To calculate the absolute speed of the target vehicle, it is necessary to know the speed and yaw rate of the ego vehicle, which can be obtained through the communication bus of the following vehicle, such as the CAN bus.

[0120] In some embodiments, the speed of the vehicle or the target is obtained from a motion sensor, or the speed of the vehicle or the target is obtained from vehicle network communication.

[0121] According to this embodiment, the above-mentioned calculation can be performed using edge computing such as motion sensors or vehicle network communications. For example, the speed of the target can be calculated based on the vehicle data and relative data obtained from the motion sensors or vehicle network communications.

[0122] In some embodiments, the heading angle of the target is determined according to an arc tangent function or a dot product formula of the horizontal velocity component and the vertical velocity component.

[0123] Figure 3 The method of calculating the heading angle according to the inverse tangent function is shown; the calculation of the dot product can be used to measure the angle between two vectors, thereby indirectly calculating the heading angle. The dot product is defined as the sum of the products of the corresponding elements of the two vectors. The result is a scalar equal to the product of the vector size and the cosine value of the angle. Therefore, based on the determination of the vectors of the horizontal velocity component and the vertical velocity component, the angle between them can be obtained.

[0124] In some embodiments, determining the posture information of the target based on the state information includes:

[0125] Acquire an image of the target from an image sensor;

[0126] Inputting the image into a trained deep learning model;

[0127] The orientation angle of the target in the image is detected according to the deep learning model.

[0128] Combination Figure 4 As shown, the image sensor can be combined with the existing perception algorithm to directly obtain the direction of the target (vehicle, pedestrian, wheel or face). Of course, vision-based perception usually uses deep learning methods, but compared with deep learning algorithms that directly predict trajectories, vision algorithms are usually more mature and targeted optimization algorithms are better. The above deep learning models include but are not limited to social-LSTM, vector-Net, Dense TNT, etc.

[0129] In addition to the two methods mentioned in the above embodiments, a combination of laser radar and multiple image sensors can also be used to obtain the direction of the target.

[0130] In some specific embodiments, detecting the orientation angle of the corresponding target in the image according to the deep learning model includes:

[0131] Identify objects in images;

[0132] Assigning a label frame to the target, so that the label frame surrounds the target following the target contour;

[0133] The heading angle of the target is determined based on the orientation of the annotation bounding box.

[0134] According to this embodiment, a labeled frame is assigned to the target in the deep learning model, and the orientation of the labeled frame is directed to the direction of the target, so that the orientation angle of the target can be directly identified according to the orientation of the frame of the deep learning model.

[0135] In some embodiments, the target includes a pedestrian or a vehicle, and detecting the orientation angle of the corresponding target in the image according to the deep learning model includes:

[0136] Identify pedestrian faces or vehicle wheels in images;

[0137] The orientation angle of the pedestrian's face or the orientation angle of the vehicle's wheels is detected according to the deep learning model.

[0138] In this embodiment, if the target is a pedestrian or a vehicle, the orientation angle of the pedestrian or the vehicle is determined according to the identified facial orientation of the pedestrian and the orientation of the vehicle wheels.

[0139] In some embodiments, in step S130, using the posture information to update the initial probability to obtain the probability includes:

[0140] Taking the target as a starting point, dividing the area in front of the target into a plurality of angular sub-areas with reference to the moving direction of the target;

[0141] Determine the angle sub-region of the corner indentation;

[0142] The initial probability of each preset motion mode is updated based on the determined angle sub-region to obtain the probability.

[0143] Combination Figure 5 As shown in FIG1 , the IMM model is set to use three motion modes, namely, constant acceleration CA, constant speed left turn CRL and constant speed right turn CRR, where CA is a motion mode of linear acceleration; assuming that the IMM model obtains the initial probability after calculation, and its value is shown in Table 1, indicating that the target has a 20% probability of uniformly accelerated linear motion, a 20% probability of turning left (with a certain turning radius), and a 60% probability of turning right.

[0144] Table 1

[0145] Model Probability CA <![CDATA[P CA =0.2]]> CRL <![CDATA[P CRL =0.2]]> CRR <![CDATA[P CRR =0.6]]>

[0146] At this time, in order to update the initial probability using posture information, combined with Figure 6 As shown, the area in front of the vehicle is divided into multiple angle sub-areas (5 in the figure), and the direction is determined based on the result of the perceived direction angle. Figure 7 As shown, its direction falls into Area-5, that is, judging from the direction, the vehicle is likely to turn right.

[0147] Then, according to the above-mentioned orientation analysis result, the probability of the motion mode is updated. The updating algorithm can be predefined based on prior knowledge such as expert knowledge base or mathematical statistics method. Table 2 is an example of the probability updating formula.

[0148] Table 2

[0149]

[0150] The updated probabilities are shown in Table 3.

[0151] Table 3

[0152]

[0153]

[0154] Therefore, with the assistance of posture perception results, the probability of the right turn model in IMM increased from 60% to 90%. Practice has proved that this probability is more in line with actual scenarios.

[0155] Therefore, in some optional embodiments, updating the initial probability of each preset motion mode based on the determined angle sub-region includes obtaining the probability including:

[0156] A probability update formula for each preset motion mode is specified for each angle sub-region;

[0157] The initial probability is input into the probability update formula to which the determined angle sub-region belongs to obtain the probability.

[0158] It should be noted that the probability update formula can be determined based on experience, can be defined in advance according to different scenarios and actual needs, and can be kept in a preset data table to be queried, extracted and applied in actual application.

[0159] In some embodiments, the probability update formula is configured to reduce a first initial probability of a first preset motion mode and increase a second initial probability of a second preset motion mode based on a determined angle sub-region, wherein the amount of reduction in the first initial probability is added to the second initial probability according to a predetermined coefficient.

[0160] As shown in Table 2 and Table 3 above, the total probability value is preset to 1. If the probability of a certain motion mode decreases, then according to the rule that the total value remains unchanged, the probability of another motion mode is increased according to the rule determined by the formula.

[0161] Specifically, changing the initial probability corresponding to the corresponding motion mode and other motion modes to obtain the probability includes: determining a probability update formula using prior knowledge such as an expert knowledge base or a statistical algorithm; inputting the initial probability into the probability update formula to obtain the probability. The probability update formula is a formula with the initial probability as an independent variable and the probability as a dependent variable, and may include operations such as addition, subtraction, multiplication, and division.

[0162] In some embodiments, predicting the trajectory of the target according to the probability includes:

[0163] Determine a maximum value among the probabilities of the preset motion patterns, and use the preset motion pattern corresponding to the maximum value as a prediction result;

[0164] Alternatively, each preset motion model is synthesized according to corresponding probabilities to obtain a synthesized motion model as a prediction result;

[0165] Alternatively, the state of the target is predicted based on the probability and the tracking trajectory model.

[0166] Specifically, when the trajectory prediction model is the interactive multi-model IMM, the fusion output of step S114 is used to obtain the estimated state of the target based on the probability and the tracking trajectory model prediction, and the estimated state includes parameters such as the target's position, speed, acceleration and direction (heading angle).

[0167] According to another aspect of the present invention, see Figure 8 As shown, a target trajectory prediction device 800 based on posture perception is provided, and the device 800 includes:

[0168] A state acquisition module 810, adapted to acquire state information of a target;

[0169] a posture determination module 820 adapted to determine the posture information of the target based on the state information;

[0170] A probability inference module 830, adapted to infer the probability that the target may move according to each preset motion mode according to the state information and the posture information of the target determined based on the state information;

[0171] The trajectory prediction module 840 is adapted to predict the trajectory of the target according to the probability.

[0172] According to the above-mentioned device 800 disclosed in this embodiment, by introducing posture information, especially the orientation angle of the target, the accuracy of model prediction can be greatly improved, and better results can be achieved.

[0173] In some embodiments, the probability inference module 830 is further adapted to:

[0174] Inputting the state information into a tracking trajectory prediction model to obtain an initial probability of predicting that the target moves according to each preset motion mode;

[0175] The initial probability is updated using the posture information to obtain the probability.

[0176] In some embodiments, the tracking trajectory model comprises an interacting multiple model.

[0177] In some embodiments, the probability inference module 830 is further adapted to:

[0178] According to the correlation coefficient between each model in the interactive multi-model, each state estimation value and covariance matrix output by each model at the previous time point are mixed respectively; wherein the covariance matrix is ​​obtained based on the difference between the state estimation value and the target measurement value, and each model corresponds to each preset motion mode respectively;

[0179] Inputting the mixed result and the measured value at the current time point into the Kalman filter of each model respectively to obtain updated state estimation values;

[0180] Calculating the likelihood value of each model based on the mixed result, and obtaining the updated confidence of each model according to the likelihood value;

[0181] Based on the updated state estimation values ​​and the confidence levels, the comprehensive state estimation value at the current time point and the initial probability of each model are obtained.

[0182] In some embodiments, the posture determination module 820 is further adapted to:

[0183] obtaining the speed of the target;

[0184] Determining a horizontal velocity component and a vertical velocity component of the velocity of the target in a predetermined coordinate system;

[0185] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0186] In some embodiments, the posture determination module 820 is further adapted to:

[0187] Obtaining a relative speed of the target relative to the vehicle;

[0188] Determining a horizontal velocity component and a vertical velocity component of the relative velocity in a relative coordinate system;

[0189] Determining the heading angle of the target according to the horizontal velocity component and the vertical velocity component; and / or

[0190] Get the speed of the vehicle;

[0191] Determining an absolute speed of the target according to the speed of the vehicle and a relative speed of the target relative to the vehicle;

[0192] Determining a horizontal velocity component and a vertical velocity component of the absolute velocity in an absolute coordinate system;

[0193] The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

[0194] In some embodiments, the device is further adapted to:

[0195] The speed of the vehicle or the target is obtained from a motion sensor, or the speed of the vehicle or the target is obtained from vehicle network communication.

[0196] In some embodiments, the device is further adapted to:

[0197] The heading angle of the target is determined according to an arc tangent function or a dot product formula of the horizontal velocity component and the vertical velocity component.

[0198] In some embodiments, the posture determination module 820 is further adapted to:

[0199] Acquire an image of the target from an image sensor;

[0200] Inputting the image into a trained deep learning model;

[0201] The orientation angle of the target in the image is detected according to the deep learning model.

[0202] In some embodiments, detecting the orientation angle of the corresponding target in the image according to the deep learning model includes:

[0203] Identify objects in images;

[0204] Assigning a label frame to the target, so that the label frame surrounds the target following the target contour;

[0205] The heading angle of the target is determined based on the orientation of the annotation bounding box.

[0206] In some embodiments, the target includes a pedestrian or a vehicle, and detecting the orientation angle of the corresponding target in the image according to the deep learning model includes:

[0207] Identify pedestrian faces or vehicle wheels in images;

[0208] The orientation angle of the pedestrian's face or the orientation angle of the vehicle's wheels is detected according to the deep learning model.

[0209] In some embodiments, the probability inference module 830 updates the initial probability using the posture information to obtain the probability, including:

[0210] Taking the target as a starting point, dividing the area in front of the target into a plurality of angular sub-areas with reference to the moving direction of the target;

[0211] Determine the angle sub-region of the corner indentation;

[0212] The initial probability of each preset motion mode is updated based on the determined angle sub-region to obtain the probability.

[0213] In some embodiments, updating the initial probability of each preset motion mode based on the determined angle sub-region includes obtaining the probability including:

[0214] A probability update formula for each preset motion mode is specified for each angle sub-region;

[0215] The initial probability is input into the probability update formula to which the determined angle sub-region belongs to obtain the probability.

[0216] In some embodiments, the probability update formula is configured to reduce a first initial probability of a first preset motion mode and increase a second initial probability of a second preset motion mode based on a determined angle sub-region, wherein the amount of reduction in the first initial probability is added to the second initial probability according to a predetermined coefficient.

[0217] In some embodiments, the trajectory prediction module 840 is further adapted to:

[0218] Determine a maximum value among the probabilities of the preset motion patterns, and use the preset motion pattern corresponding to the maximum value as the prediction result; and / or

[0219] synthesizing the preset motion models according to corresponding probabilities to obtain a synthesized motion model as a prediction result; and / or

[0220] The state of the target is predicted based on the probability and the tracking trajectory model.

[0221] It should be noted that the specific implementation of the above-mentioned device embodiments can be carried out with reference to the specific implementation of the above-mentioned corresponding method embodiments, which will not be repeated here.

[0222] It should be noted that:

[0223] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general-purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for the purpose of disclosing the best mode of the present invention.

[0224] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0225] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0226] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0227] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0228] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the target trajectory prediction device based on attitude perception according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0229] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the above-mentioned target trajectory prediction method based on posture perception in any of the above-mentioned method embodiments.

[0230] The figure shows a schematic diagram of the structure of an embodiment of an electronic device in the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronics, including the electronic equipment in a vehicle.

[0231] like Fig. 9 As shown, the electronic device may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

[0232] Wherein: the processor 902, the communication interface 904, and the memory 906 communicate with each other through the communication bus 908. The communication interface 904 is used to communicate with other devices such as network elements of a client or other servers. The processor 902 is used to execute the program 910, which can specifically execute the relevant steps in the above-mentioned embodiment of the target trajectory prediction method based on posture perception for the electronic device.

[0233] Specifically, the program 910 may include program codes, which include computer operation instructions.

[0234] The processor 902 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the onboard image processing board may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0235] The memory 906 is used to store the program 910. The memory 906 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0236] The program 910 can be specifically used to enable the processor 902 to perform operations corresponding to the above-mentioned target trajectory prediction method embodiment based on posture perception.

[0237] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A target trajectory prediction method based on posture perception, the method comprising: Get the target's status information; Determining posture information of the target based on the state information; Inferring the probability of the target moving according to each preset motion mode according to the state information and the posture information of the target determined based on the state information; as well as A trajectory of the target is predicted based on the probability.

2. The method according to claim 1, characterized in that: Inferring the probability of the target moving according to each preset motion mode according to the state information and the posture information of the target determined based on the state information includes: Inputting the state information into a tracking trajectory prediction model to obtain an initial probability of predicting that the target moves according to each preset motion mode; The initial probability is updated using the posture information to obtain the probability.

3. The method according to claim 2, characterized in that The tracking trajectory model includes an interacting multi-model.

4. The method according to claim 3, characterized in that Inputting the state information into the tracking trajectory prediction model to obtain the initial probability that the target may move according to each preset motion mode includes: According to the correlation coefficient between each model in the interactive multi-model, each state estimation value and covariance matrix output by each model at the previous time point are mixed respectively; wherein the covariance matrix is ​​obtained based on the difference between the state estimation value and the target measurement value, and each model corresponds to each preset motion mode respectively; Inputting the mixed result and the measured value at the current time point into the Kalman filter of each model respectively to obtain updated state estimation values; Calculating the likelihood value of each model based on the mixed result, and obtaining the updated confidence of each model according to the likelihood value; Based on the updated state estimation values ​​and the confidence levels, the comprehensive state estimation value at the current time point and the initial probability of each model are obtained.

5. The method according to any one of claims 1 to 4, characterized in that Determining the posture information of the target based on the state information includes: obtaining the speed of the target; Determining a horizontal velocity component and a vertical velocity component of the velocity of the target in a predetermined coordinate system; The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

6. The method according to claim 5, characterized in that Determining the posture information of the target based on the state information includes: Obtaining a relative speed of the target relative to the vehicle; Determining a horizontal velocity component and a vertical velocity component of the relative velocity in a relative coordinate system; Determining the heading angle of the target according to the horizontal velocity component and the vertical velocity component; and / or Get the speed of the vehicle; Determining an absolute speed of the target according to the speed of the vehicle and a relative speed of the target relative to the vehicle; Determining a horizontal velocity component and a vertical velocity component of the absolute velocity in an absolute coordinate system; The heading angle of the target is determined according to the horizontal velocity component and the vertical velocity component.

7. The method according to claim 5, characterized in that The speed of the vehicle or the target is obtained from a motion sensor, or the speed of the vehicle or the target is obtained from vehicle network communication.

8. The method according to claim 5, characterized in that The heading angle of the target is determined according to an arc tangent function or a dot product formula of the horizontal velocity component and the vertical velocity component.

9. The method according to any one of claims 1 to 4, characterized in that: Determining the posture information of the target based on the state information includes: Acquire an image of the target from an image sensor; Inputting the image into a trained deep learning model; The orientation angle of the target in the image is detected according to the deep learning model.

10. The method according to claim 9, characterized in that Detecting the orientation angle of the corresponding target in the image according to the deep learning model includes: Identify objects in images; Assigning a label frame to the target, so that the label frame surrounds the target following the target contour; The heading angle of the target is determined based on the orientation of the annotation bounding box.

11. The method according to claim 9, characterized in that The target includes a pedestrian or a vehicle, and the orientation angle of the corresponding target in the image detected by the deep learning model includes: Identify pedestrian faces or vehicle wheels in images; The orientation angle of the pedestrian's face or the orientation angle of the vehicle's wheels is detected according to the deep learning model.

12. The method according to any one of claims 6-8, 10 and 11, characterized in that: Using the posture information to update the initial probability, the probability is obtained including: Taking the target as a starting point, dividing the area in front of the target into a plurality of angular sub-areas with reference to the moving direction of the target; Determine the angle sub-region of the corner indentation; The initial probability of each preset motion mode is updated based on the determined angle sub-region to obtain the probability.

13. The method according to claim 12, characterized in that Updating the initial probability of each preset motion mode based on the determined angle sub-region includes obtaining the probability including: A probability update formula for each preset motion mode is specified for each angle sub-region; The initial probability is input into the probability update formula to which the determined angle sub-region belongs to obtain the probability.

14. The method according to claim 13, characterized in that The probability update formula is configured to reduce a first initial probability of a first preset motion mode and increase a second initial probability of a second preset motion mode based on a determined angle sub-region, wherein the reduction amount of the first initial probability is added to the second initial probability according to a predetermined coefficient.

15. The method according to any one of claims 1 to 4, characterized in that Predicting the trajectory of the target according to the probability includes: Determine a maximum value among the probabilities of the preset motion patterns, and use the preset motion pattern corresponding to the maximum value as the prediction result; and / or synthesizing the preset motion models according to corresponding probabilities to obtain a synthesized motion model as a prediction result; and / or A state of the target is determined based on the probability and the tracking trajectory model.

16. A tracking trajectory prediction device based on posture perception, the device comprising: A state acquisition module, adapted to acquire state information of a target; A posture determination module, which determines the posture information of the target based on the state information; A probability inference module, adapted to infer the probability of the target moving according to each preset motion mode according to the state information and the posture information of the target determined based on the state information; A trajectory prediction module is adapted to predict the trajectory of the target according to the probability.

17. An electronic device comprising: processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the target trajectory prediction method based on posture perception according to any one of claims 1-15.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the target trajectory prediction method based on posture perception according to any one of claims 1-15.

Images