Target trajectory acquisition method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for acquiring a target trajectory. The method comprises: based on environmental perception data collected by a first sensor and / or a second sensor at a current time, identifying a target state of a target and obtaining an actual target state at the current time; obtaining a previous reference target state corresponding to a previous time; predicting the previous reference target state based on a time interval between the current time and the previous time to obtain a predicted target state; weightedly fusing the predicted target state and the actual target state to obtain a fused target state; generating a trajectory for the target based on the fused target state, so as to fuse the actual target state perceived by each sensor frame by frame in the time dimension to realize time synchronization between multiple sensors, avoid the situation where inaccurate positioning is caused by the time difference between each sensor, and improve the accuracy of the finally generated trajectory.

Description

Technical Field

[0001] The present application relates to the technical field of trajectory analysis, and in particular to a method, device and storage medium for acquiring a target trajectory. Background Art

[0002] With the continuous development of sensing technology, single sensor systems have gradually shown limitations in complex application scenarios, especially in the field of target detection. In order to improve the accuracy of the detection system, the technology of combining multiple sensors has been applied in more and more fields.

[0003] However, when fusing multi-sensor information, high-precision target recognition results cannot be obtained at all times. For example, the timestamps of different sensors are not synchronized, data frames of different sensors are lost, the calibration errors of different sensors are too large, etc., which will reduce the accuracy of target recognition results. Summary of the invention

[0004] In order to solve the above technical problems, the present application at least provides a target trajectory acquisition method, device and storage medium.

[0005] In a first aspect, the present application provides a method for acquiring a target trajectory, the method comprising: identifying a target state of a target based on environmental perception data collected by a first sensor and / or environmental perception data collected by a second sensor at a current time, and obtaining an actual target state of the target at the current time; wherein the target state includes position parameters and motion parameters of the target; obtaining a preceding reference target state corresponding to the current time, the preceding reference target state refers to a target state of the target calculated at a preceding time, the preceding time being earlier than the current time; calculating a time interval between the current time and the preceding time, and predicting the position parameters and motion parameters of the target at the current time based on the time interval and the preceding reference target state, and obtaining a predicted target state; weightedly fusing the predicted target state and the actual target state to obtain a fused target state of the target at the current time; generating a trajectory for the target based on the fused target state, and taking the actual target state or the fused target state as a preceding reference target state at a subsequent time, the subsequent time being later than the current time.

[0006] In one embodiment, the position parameters and motion parameters of the target at the current time are predicted based on the time interval and the previous reference target state to obtain the predicted target state, including: setting a state change weight value based on the time interval, the time interval is proportional to the state change weight value; calculating the product between the state change weight value and the previous reference target state to obtain the state change value; calculating the sum of the previous reference target state and the state change value to obtain the predicted target state.

[0007] In one embodiment, the actual target state includes a first actual target state and / or a second actual target state; based on the environmental perception data collected by the first sensor at the current time and / or the environmental perception data collected by the second sensor, the target state of the target is identified to obtain the actual target state of the target at the current time, including: based on the environmental perception data collected by the first sensor at the current time, the target state is identified to obtain the first actual target state; and / or, based on the environmental perception data collected by the second sensor at the current time, the target state of the target is identified to obtain the second actual target state; weighted fusion of the predicted target state and the actual target state is performed to obtain the fused target state of the target at the current time, including: if the actual target state includes the first actual target state, then obtain The fusion weight corresponding to the first sensor is obtained to obtain the first fusion weight, and the predicted target state and the first actual target state are weightedly fused according to the first fusion weight to obtain the fused target state of the target at the current time; if the actual target state includes the second actual target state, the fusion weight corresponding to the second sensor is obtained to obtain the second fusion weight, and the predicted target state and the second actual target state are weightedly fused according to the second fusion weight to obtain the fused target state of the target at the current time; if the actual target state includes the first actual target state and the second actual target state, the predicted target state, the first actual target state and the second actual target state are weightedly fused according to the first fusion weight and the second fusion weight to obtain the fused target state of the target at the current time.

[0008] In one embodiment, the first sensor includes a lidar sensor and the second sensor includes an image sensor.

[0009] In one embodiment, trajectory generation for targets is performed based on fused target states, including: taking a target identified at a previous time as a previous target, and taking a target identified at a current time as a current target; obtaining a predicted target state at the current time corresponding to the fused target state of each previous target, and obtaining a fused target state of each current target calculated at the current time; calculating a correlation between the predicted target state corresponding to each previous target and the fused target state corresponding to each current target, using the correlation as an association weight, and constructing an association weight matrix; determining a matching relationship between each previous target and each current target when the sum of the association weights in the association weight matrix is ​​the largest; and for successfully matched current targets and previous targets, adding the fused target state of the successfully matched current target to the trajectory of the corresponding matched previous target.

[0010] In one embodiment, the position parameters include a target marking box; calculating the correlation between each predicted target state and each fused target state, including: calculating the overlapping area between the target marking box of the preceding target in the predicted target state and the target marking box of the current target in the fused target state; obtaining the correlation between the predicted target state and the fused target state based on the overlapping area, and the overlapping area is proportional to the correlation.

[0011] In one embodiment, the correlation between the predicted target state and the fused target state is obtained based on the overlapping area, including: if the overlapping area is less than a preset overlapping threshold, the correlation between the predicted target state and the fused target state is set to a preset minimum value; if the overlapping area is not less than the preset overlapping threshold, the correlation between the predicted target state and the fused target state is calculated according to the positive proportional relationship between the overlapping area and the correlation.

[0012] In one embodiment, the method further includes: if there is a preceding target that does not match any current target, then the cumulative number of lost frames corresponding to the preceding target is increased by one; if the cumulative number of lost frames corresponding to the preceding target is greater than a preset frame number threshold, then it is determined that the preceding target is lost and the preceding target is deleted.

[0013] According to a second aspect of the present application, there is provided a target trajectory acquisition device, which includes: a current state recognition module, which is used to identify the target state of the target based on the environmental perception data collected by the first sensor and / or the environmental perception data collected by the second sensor at the current time, and obtain the actual target state of the target at the current time; wherein the target state includes the position parameters and motion parameters of the target; a previous state acquisition module, which is used to obtain the previous reference target state corresponding to the current time, and the previous reference target state refers to the target state of the target calculated at the previous time, and the previous time is earlier than the current time; a previous state prediction module, which is used to calculate the time interval between the current time and the previous time, and predict the position parameters and motion parameters of the target at the current time based on the time interval and the previous reference target state, and obtain the predicted target state; a state fusion module, which is used to perform weighted fusion of the predicted target state and the actual target state to obtain the fused target state of the target at the current time; a trajectory generation module, which is used to generate a trajectory for the target based on the fused target state, and use the actual target state or the fused target state as the previous reference target state of the subsequent time, and the subsequent time is later than the current time.

[0014] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned target trajectory acquisition method.

[0015] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned target trajectory acquisition method when the program instructions are executed by a processor.

[0016] The above scheme identifies the target state of the target based on the environmental perception data collected by the first sensor at the current time and / or the environmental perception data collected by the second sensor, and obtains the actual target state of the target at the current time; obtains the previous reference target state corresponding to the current time, and the previous reference target state refers to the target state of the target calculated at the previous time; calculates the time interval between the current time and the previous time, and obtains the predicted target state based on the time interval and the previous reference target state; performs weighted fusion on the predicted target state and the actual target state to obtain the fused target state of the target at the current time; generates a trajectory for the target based on the fused target state, and uses the actual target state or the fused target state as the previous reference target state at the subsequent time, so as to fuse the actual target state perceived by each sensor frame by frame in the time dimension to achieve time synchronization between multiple sensors, avoid inaccurate positioning caused by the time difference between each sensor, and improve the accuracy of the finally generated trajectory.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0019] Figure 1 is a schematic diagram of sensor target perception exemplarily shown in one embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0021] Figure 3 is a flow chart of a method for obtaining a target trajectory shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of target state fusion shown in an exemplary embodiment of the present application;

[0023] Figure 5 is a block diagram of a target trajectory acquisition device shown in an exemplary embodiment of the present application;

[0024] Figure 6 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] Accurate and stable multi-sensor fusion results are the basis for subsequent target tracking. However, when fusing multi-sensor information, the accuracy of target recognition results is reduced due to reasons such as asynchronous timestamps of different sensors, data frame loss of different sensors, and large calibration errors of different sensors.

[0030] For example, Figure 1 The schematic diagram of sensor target perception is exemplarily shown in an embodiment of the present application. The sensor includes a laser radar sensor and an image sensor. After the laser radar sensor and the image sensor are calibrated, three frames of perception data are continuously collected respectively. The recognition results of the three frames of perception data collected by the laser radar sensor and the image sensor are as follows: Figure 1 As shown in the figure, the scene actually includes vehicle target 1, vehicle target 2, vehicle target 3, and vehicle target 4. The vehicle target detected by the image sensor is the solid rectangular frame in the figure. After calibration, the radar target in the lidar coordinate system can be mapped to the image coordinate system to form a rectangular frame. Figure 1 The vehicle target detected by the lidar sensor is the dotted rectangular box in the figure.

[0031] As shown in the perception results corresponding to the first frame of perception data, the lidar sensor and the image sensor both detected all vehicle targets normally. After a period of time, the sensor detection results are shown in the perception results corresponding to the second frame of perception data and the perception results corresponding to the third frame of perception data. Since the lidar sensor and the image sensor are not synchronized in time and there is a missed detection problem, in the second frame, only the lidar sensor recognizes vehicle targets 2 and 4, while vehicle targets 1 and 3 lose the lidar detection results. In the third frame, only the image sensor recognizes vehicle targets 1, 3, and 4, while vehicle target 2 loses the image detection result.

[0032] If the laser radar detection result and the image detection result of each frame are directly fused according to the existing technology, the accuracy of the target recognition result will be reduced.

[0033] Based on this, the present application at least provides a target trajectory acquisition method, device and storage medium.

[0034] The target trajectory acquisition method provided in the embodiment of the present application is described below.

[0035] Please refer to Figure 2 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a sensor 210 deployed relative to a target detection area, the sensor 210 includes at least a first sensor 211 and a second sensor 212, and the sensor 210 and a server 220 are connected to each other in communication.

[0036] The first sensor 211 and the second sensor 212 are used to collect environmental perception data of the target detection area. The first sensor 211 and the second sensor 212 include but are not limited to image sensors, laser radar sensors, depth sensors, etc.

[0037] Server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0038] In one example, the server 220 may perform target trajectory generation processing for the environmental perception data obtained from the sensor 210 to obtain the trajectory of the target. Of course, the server 220 may store the trajectory of the target locally, transmit it back to the sensor 210, or transmit it to other terminals.

[0039] In one example, a target application is installed and running in the sensor 210, and the target application may be an application that provides target recognition and / or target trajectory generation functions. The server 220 may be a background server of the target application, and is used to provide background services for the client of the target application.

[0040] In the target trajectory acquisition method provided in the embodiment of the present application, the execution subject of each step can be the sensor 210, such as the target application installed and running in the sensor 210, or the server 220, or the sensor 210 and the server 220 interact and cooperate to execute, that is, part of the steps of the method are executed by the sensor 210 and the other steps are executed by the server 220.

[0041] See also Figure 3 , Figure 3 is a flow chart of a target trajectory acquisition method shown in an exemplary embodiment of the present application. The target trajectory acquisition method can be applied to Figure 2 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0042] like Figure 3 As shown, the target trajectory acquisition method at least includes steps S310 to S350, which are described in detail as follows:

[0043] Step S310: Based on the environmental perception data collected by the first sensor and / or the environmental perception data collected by the second sensor at the current time, identify the target state of the target and obtain the actual target state of the target at the current time; wherein the target state includes the position parameters and motion parameters of the target.

[0044] At least a first sensor and a second sensor are deployed in the target detection area. The first sensor and the second sensor respectively collect environmental perception data. By performing target recognition on the environmental perception data collected by the first sensor and the second sensor respectively, the actual target state of the target can be obtained.

[0045] Since data frames may be lost in sensors and timestamps of data frames may not be synchronized between sensors, the environmental perception data received at the current time may be environmental perception data collected by the first sensor, or environmental perception data collected by the second sensor, or may contain environmental perception data collected by both the first sensor and the second sensor.

[0046] In response to any sensor identifying a target, the acquisition time of the environmental perception data corresponding to the identified target is taken as the current time, and the target state of the identified target is taken as the actual target state of the target at the current time.

[0047] Among them, the targets include but are not limited to vehicles, ships, aircraft, animals, etc. The target state includes the position parameters and motion parameters of the target. The position parameters are used to describe the spatial information of the target in the target detection area, and the motion parameters are used to describe the speed, movement direction and other information of the target.

[0048] Similarly, the actual target state may include a target state obtained by performing target recognition based on environmental perception data collected by the first sensor, and / or a target state obtained by performing target recognition based on environmental perception data collected by the second sensor.

[0049] Step S320: Obtain a previous reference target state corresponding to the current time, where the previous reference target state refers to a target state of the target calculated at a previous time, and the previous time is earlier than the current time.

[0050] Get the target state of the target calculated at the previous time and get the previous reference target state.

[0051] It should be noted that when starting to acquire the target trajectory, since the target state of the target is not calculated at the previous time, the target state can be initialized to a default value, and the default value is generally 0, so that the initialized target state is used as the previous reference target state of the current time.

[0052] Optionally, the fused target state obtained by identifying the environmental perception data collected at the previous time can be used as the previous reference target state; or the actual target state can be calculated based on the environmental perception data collected at the previous time. The previous time can be the time when the environmental perception data frame was received last time, or it can be the time between n environmental perception data frames, where n is an integer greater than or equal to 1, and the number of previous reference target states obtained can be one or more, which is not limited in this application.

[0053] Step S330: Calculate the time interval between the current time and the previous time, and predict the position parameters and motion parameters of the target at the current time based on the time interval and the previous reference target state to obtain the predicted target state.

[0054] After obtaining the previous reference target state, the target state change can be predicted according to the time interval between the current time and the previous time, that is, the position parameters and motion parameters of the target at the current time are predicted to obtain the predicted target state.

[0055] In Example 1, based on the position parameters and motion parameters of the target in the previous reference target state, the motion parameters can be set unchanged, the position change of the target after the time interval can be estimated, and the position parameters and motion parameters of the target at the current time can be obtained.

[0056] Example 2: The motion parameters of the target in multiple previous reference target states can be used to analyze the trend of motion parameter changes, thereby predicting the change in the motion parameters of the target within the time interval, and based on the change in the motion parameters of the target within the time interval, the position change of the target after the time interval can be estimated, as well as the motion parameters of the target after the time interval, to ultimately obtain the position parameters and motion parameters of the target at the current time.

[0057] In Example 3, the type of the target can also be obtained. If the target is a uniformly moving target, the prediction strategy of Example 1 is used to predict the position parameters and motion parameters of the target at the current time; if the target is not a uniformly moving target, the prediction strategy of Example 2 is used to predict the position parameters and motion parameters of the target at the current time.

[0058] Of course, in addition to the embodiments listed above, other methods can be used to predict the position parameters and motion parameters of the target at the current time according to actual conditions. For example, a pre-trained neural network model with target state prediction function can be used to predict the position parameters and motion parameters of the target at the current time. This application does not limit this.

[0059] Step S340: Perform weighted fusion on the predicted target state and the actual target state to obtain the fused target state of the target at the current time.

[0060] Specifically, the corresponding elements of the predicted target state and the actual target state are weightedly fused according to the weighted weights to obtain the fused target state of the target at the current time.

[0061] The weighted weights used in weighted fusion may be pre-set based on experience or may be flexibly calculated.

[0062] For example, the weighted weight is set according to the size of the time interval. If the time interval is larger, it means that the predicted target state is less credible, and the weighted weight of the predicted target state is smaller; conversely, if the time interval is smaller, it means that the predicted target state is more credible, and the weighted weight of the predicted target state is larger.

[0063] For another example, the recognition accuracy corresponding to the actual target state is obtained. The recognition accuracy can be determined based on the data quality of the environmental perception data, the recognition algorithm used for target recognition, etc. If a pre-trained neural network model with target recognition function is used to recognize the target of the environmental perception data, the neural network model will also output the confidence of the actual target state, and the confidence will be used as the recognition accuracy corresponding to the actual target state. After the recognition accuracy corresponding to the actual target state is determined, the weighted weight is set. If the recognition accuracy is greater, the actual target state is more credible, and the weighted weight of the actual target state is greater; conversely, if the recognition accuracy is smaller, the predicted target state is less credible, and the weighted weight of the actual target state is smaller.

[0064] Of course, the weighted weights used for weighted fusion of the predicted target state and the actual target state may also be set in combination with the time interval and the recognition accuracy corresponding to the actual target state.

[0065] Step S350: Generate a trajectory for the target based on the fused target state, and use the actual target state as a preceding reference target state at a subsequent time, where the subsequent time is later than the current time.

[0066] The predicted target state and the actual target state are weightedly fused, and the fused target state obtained by weighted fusion is used as the final target state of the target at the current time, so as to generate the trajectory of the target using the fused target state.

[0067] Specifically, after obtaining the fused target state of the target at each time, the trajectory of the target is obtained.

[0068] The present application predicts the target state through the time interval between the current environment perception data frame and the previous one, thereby weightedly fusion of the predicted target state and the actual target state identified by the current environment perception data frame, so as to fuse the actual target state perceived by each sensor frame by frame in the time dimension to achieve time synchronization between multiple sensors, avoid inaccurate positioning caused by the time difference between each sensor, and improve the accuracy of the final generated trajectory.

[0069] Next, some embodiments of the present application are described in detail.

[0070] The environmental perception data collected by each sensor is obtained, and the target state, specifically the position and / or motion state of the target, can be obtained by performing target recognition on the environmental perception data.

[0071] In order to realize the unity of coordinate systems of different sensors, multiple sensors need to be calibrated so that the target states sensed by multiple sensors can be unified into the same coordinate system through calibration parameters.

[0072] Taking image sensors and lidar sensors as examples, image sensors can give the position of the target in the image coordinate system based on the collected images; lidar sensors can convert the target points detected in the radar coordinate system into the image coordinate system after calibration based on the collected point cloud data to generate the target position and speed information.

[0073] For example, the target state of the kth target obtained by recognizing the environmental perception data of the image sensor can be expressed as , X1 and Y1 are the horizontal and vertical coordinates of the target in the image coordinate system, and t is the acquisition time of the environmental perception data of the image sensor.

[0074] The target state of the kth target obtained by recognizing the environmental perception data of the lidar sensor can be expressed as , and are the horizontal and vertical coordinates of the target in the radar coordinate system, and is the speed information, and t is the acquisition time of the environment perception data of the LiDAR sensor. Convert to the image coordinate system and get the target state after conversion: , X2 and Y2 are the horizontal and vertical coordinates of the converted target in the image coordinate system.

[0075] Based on the actual target states detected by each sensor, state prediction and fusion are performed frame by frame.

[0076] In some embodiments, step S330 predicts the position parameters and motion parameters of the target at the current time based on the time interval and the previous reference target state to obtain the predicted target state, including the following steps S331 to S333.

[0077] Step S331: setting a state change weight value based on a time interval, where the time interval is proportional to the state change weight value.

[0078] Optionally, the time interval may be directly used as the state change weight value, which is not limited in the present application.

[0079] Alternatively, the time interval is substituted into a preset direct proportional relationship function to obtain a state change weight value.

[0080] Step S332: Calculate the product between the state change weight value and the previous reference target state to obtain the state change value.

[0081] Step S333: Calculate the sum of the previous reference target state and the state change value to obtain the predicted target state.

[0082] For example, the previous reference target state of the kth target is expressed as , the time interval is expressed as , if the time interval is directly used as the state change weight value, the formula for calculating the predicted target state can be expressed as the following formula 1:

[0083] (Formula 1)

[0084] in, represents the predicted target state of the kth target.

[0085] Of course, the above state prediction method is only a simple example. In actual application scenarios, considering the available computing resources, the accuracy requirements of state prediction, etc., other state prediction algorithms can be flexibly selected for state prediction. For example, when there are many available computing resources and the accuracy requirements of state prediction are high, a pre-trained neural network model can be used for state prediction.

[0086] In some embodiments, the target state of the target is identified based on the environmental perception data collected by the first sensor at the current time to obtain the first actual target state; and / or the target state of the target is identified based on the environmental perception data collected by the second sensor at the current time to obtain the second actual target state, and the actual target state includes the first actual target state and / or the second actual target state. Then, the predicted target state and the actual target state are weighted fused to obtain the fused target state of the target at the current time, which includes at least the following situations:

[0087] Case 1: If the actual target state includes the first actual target state, the fusion weight corresponding to the first sensor is obtained to obtain the first fusion weight, and the predicted target state and the first actual target state are weightedly fused according to the first fusion weight to obtain the fused target state of the target at the current time.

[0088] Among them, the first fusion weight can be flexibly set according to the type of the first sensor, the recognition accuracy of the actual target state obtained by environmental perception data, the size of the time interval, etc., or it can be set in advance based on experience, which is not limited in this application.

[0089] The predicted target state and the first actual target state are weightedly fused according to the first fusion weight to obtain a fused target state of the target at the current time.

[0090] Specifically, taking the first sensor including a laser radar sensor as an example, the relevant formula for state fusion can be expressed as the following formula 2:

[0091] (Formula 2)

[0092] in, represents the first fusion weight, represents the predicted target state of the kth target, represents the first actual target state of the kth target detected by the lidar sensor, represents the fused target state of the k-th target.

[0093] Case 2: If the actual target state includes the second actual target state, the fusion weight corresponding to the second sensor is obtained to obtain the second fusion weight, and the predicted target state and the second actual target state are weightedly fused according to the second fusion weight to obtain the fused target state of the target at the current time.

[0094] Among them, the second fusion weight can be flexibly set according to the type of the second sensor, the recognition accuracy of the actual target state obtained by environmental perception data, the size of the time interval, etc., or it can be set in advance based on experience, which is not limited in this application.

[0095] The predicted target state and the second actual target state are weightedly fused according to the second fusion weight to obtain a fused target state of the target at the current time.

[0096] Specifically, taking the second sensor including an image sensor as an example, the relevant formula for state fusion can be expressed as the following formula 3:

[0097] (Formula 3)

[0098] in, represents the second fusion weight, represents the predicted target state of the kth target, represents the second actual target state of the kth target detected by the image sensor, represents the fused target state of the k-th target.

[0099] Case 3: If the actual target state includes the first actual target state and the second actual target state, the predicted target state, the first actual target state and the second actual target state are weightedly fused according to the first fusion weight and the second fusion weight to obtain the fused target state of the target at the current time.

[0100] Specifically, the above formula 2 and formula 3 can be combined to perform weighted fusion on the predicted target state, the first actual target state and the second actual target state to obtain the fused target state of the target at the current time.

[0101] It should be noted that the above embodiment is described by taking two sensors as an example. More sensors can be deployed in actual scenarios, so more combinations will appear, and this application does not limit this. Also, in actual application scenarios, other types of sensors can be selected as the first sensor or the second sensor, or more sensors can be selected, such as a third sensor deployed in the target detection area, and this application does not limit this.

[0102] In some implementations, after the fused target state at the current time is obtained, the fused target state is also used to update the previous reference target state, so as to facilitate the fusion of the target state at the subsequent time.

[0103] In some embodiments, after obtaining the fusion target state at the current time, it is detected whether the target position in the fusion target state is within the target detection area. If it is within the target detection area, the iterative execution of steps S310 to S350 continues. If it is not within the target detection area, the target state information related to the target is deleted and the state is no longer updated.

[0104] For an example, see Figure 4 , Figure 4 A schematic diagram of target state fusion is shown as an exemplary embodiment of the present application. Figure 4 As shown, including:

[0105] Step S401: Acquire the environmental perception data collected at the current time, and identify the actual target state of the target at the current time;

[0106] Step S402: Acquire a previous reference target state, and perform state prediction on the previous reference target state based on the time interval between the current time and the previous time to obtain a predicted target state;

[0107] Among them, corresponding to the first frame of environmental perception data, the target state is initialized, and the initialization result is used as the previous reference target state.

[0108] Step S403: determining the source type of the environmental perception data collected at the current time, if it is from the first sensor, executing step S404, if it is from the second sensor, executing step S405;

[0109] Step S404: performing weighted fusion on the predicted target state and the first actual target state identified by the first sensor according to the first fusion weight;

[0110] Step S405: performing weighted fusion on the predicted target state and the second actual target state identified by the second sensor according to the second fusion weight;

[0111] Step S406: updating the previous reference target state according to the fusion target state obtained by weighted fusion;

[0112] Step S407: Detect whether the target position in the fused target state is within the target detection area. If it is within the target detection area, continue to iterate step S401 and subsequent steps. If it is not within the target detection area, execute step S408.

[0113] Step S408: Delete the target status information related to the target, and do not update the status of the target any more.

[0114] Through the above steps, the actual target states that can be perceived by all sensors are fused in the time dimension, the time synchronization of different sensors is achieved, and the target state is enhanced, which can effectively prevent the accuracy of target tracking from being affected by time asynchrony or data frame loss.

[0115] In some implementations, step S350 generates a trajectory for the target based on the fused target state, including the following steps S351 to S355.

[0116] Step S351: The target identified at the previous time is regarded as the previous target, and the target identified at the current time is regarded as the current target.

[0117] Step S352: Obtain the predicted target state at the current time corresponding to the fused target state predicted based on each previous target, and obtain the fused target state of each current target calculated at the current time.

[0118] For example, for the previous target k, let the fusion target state of the previous target k be expressed as ,in, The calculation process of can refer to the above steps S310 to S350. Based on the fusion target state of each previous target, the predicted target state at the current time is predicted. The relevant formula is expressed as the following formula 4:

[0119] (Formula 4)

[0120] in, represents the predicted target state corresponding to the previous target k, Indicates the time interval between the fused target state of the previous target and the fused target state of the current target.

[0121] It should be noted that other methods may also be used to predict the predicted target state of the fused target state of the previous target at the current time, which is similar to the prediction process in step S330, and this application does not limit this.

[0122] Step S353: Calculate the correlation between the predicted target state corresponding to each previous target and the fused target state corresponding to each current target, use the correlation as the correlation weight, and construct an association weight matrix.

[0123] Exemplarily, the difference in position parameters and / or motion parameters of the target between the predicted target state and the fused target state is analyzed to determine the probability that the predicted target state and the fused target state belong to the same target.

[0124] Exemplarily, a perception data block corresponding to the target of the predicted target state is intercepted from the environmental perception data, and a perception data block corresponding to the target of the fused target state is intercepted from the environmental perception data, and the similarity between the two perception data blocks is compared to determine the probability that the predicted target state and the fused target state belong to the same target.

[0125] For example, the position parameter includes a target marking frame, and the correlation between each predicted target state and each fused target state is calculated in step S353, including:

[0126] Step S3531: Calculate the overlapping area between the target mark frame of the previous target in the predicted target state and the target mark frame of the current target in the fused target state;

[0127] Step S3532: Based on the overlapping area, the correlation between the predicted target state and the fused target state is obtained, and the overlapping area is proportional to the correlation.

[0128] For example, the target marking box area of ​​the i-th preceding target is recorded as , the target marking box area of ​​the jth current target is recorded as , the intersection-and-union ratio between the i-th previous target and the j-th current target is calculated based on the overlapping area, and the calculation formula is expressed as the following formula 5:

[0129] (Formula 5)

[0130] in, Represents the intersection-and-union ratio between the i-th previous target and the j-th current target.

[0131] The correlation between the predicted target state and the fused target state is obtained according to the intersection-and-union ratio. The larger the value of the intersection-and-union ratio, the higher the correlation.

[0132] Optionally, in order to prevent two targets with less overlap from being incorrectly associated, if the overlapping area is less than a preset overlap threshold, the correlation between the predicted target state and the fused target state is set to a preset minimum value; if the overlapping area is not less than the preset overlap threshold, the correlation between the predicted target state and the fused target state is calculated according to the positive proportional relationship between the overlapping area and the correlation.

[0133] The minimum value setting method can be expressed as the following formula 6:

[0134] (Formula 6)

[0135] in, is the preset overlap threshold, The default minimum value.

[0136] The correlation degree is used as the correlation weight to construct the correlation weight matrix.

[0137] For example, the previous goals include , current goals include , then the constructed association weight matrix can be expressed as:

[0138]

[0139] Step S354: Determine the matching relationship between each previous target and each current target when the sum of the associated weights in the associated weight matrix is ​​maximum.

[0140] Exemplarily, according to the constructed association weight matrix, the optimal match can be solved by using the maximum weight matching algorithm (Kuhn-Munkres, KM). Specifically, when the sum of the association weights is the largest, it is considered to be the optimal matching result.

[0141] Of course, during the matching process, it is necessary to follow that a predecessor target can only match one current target at most.

[0142] Step S355: For the successfully matched current target and the previous target, the fused target state of the successfully matched current target is added to the trajectory of the corresponding matched previous target.

[0143] Optionally, if there is a preceding target that does not match any current target, the cumulative number of lost frames corresponding to the preceding target is increased by one; if the cumulative number of lost frames corresponding to the preceding target is greater than a preset frame number threshold, it is determined that the preceding target is lost and the preceding target is deleted.

[0144] And, if there is a current target that does not match any previous target, the current target is considered to be a newly added target, and a trajectory of the current target is created.

[0145] The target trajectory acquisition method provided by the present application identifies the target state of the target based on the environmental perception data collected by the first sensor at the current time and / or the environmental perception data collected by the second sensor, and obtains the actual target state of the target at the current time; obtains the previous reference target state corresponding to the current time, and the previous reference target state refers to the target state of the target calculated at the previous time; calculates the time interval between the current time and the previous time, and obtains the predicted target state based on the time interval and the previous reference target state; performs weighted fusion on the predicted target state and the actual target state to obtain the fused target state of the target at the current time; generates a trajectory for the target based on the fused target state, and uses the actual target state or the fused target state as the previous reference target state at the subsequent time, so as to fuse the actual target state perceived by each sensor frame by frame in the time dimension to achieve time synchronization between multiple sensors, avoid the inaccurate positioning caused by the time difference between each sensor, and improve the accuracy of the trajectory finally generated.

[0146] Figure 5 FIG. 1 is a block diagram of a target trajectory acquisition device shown in an exemplary embodiment of the present application. Figure 5 As shown, the exemplary target trajectory acquisition device 500 includes: a current state recognition module 510, a previous state acquisition module 520, a previous state prediction module 530, a state fusion module 540 and a trajectory generation module 550. Specifically:

[0147] The current state recognition module 510 is used to identify the target state of the target based on the environmental perception data collected by the first sensor and / or the environmental perception data collected by the second sensor at the current time, and obtain the actual target state of the target at the current time; wherein the target state includes the position parameters and motion parameters of the target;

[0148] A previous state acquisition module 520 is used to acquire a previous reference target state corresponding to the current time, where the previous reference target state refers to a target state calculated at a previous time, where the previous time is earlier than the current time;

[0149] A previous state prediction module 530 is used to calculate the time interval between the current time and the previous time, and predict the position parameters and motion parameters of the target at the current time based on the time interval and the previous reference target state to obtain the predicted target state;

[0150] The state fusion module 540 is used to perform weighted fusion on the predicted target state and the actual target state to obtain the fused target state of the target at the current time;

[0151] The trajectory generation module 550 is used to generate a trajectory for the target based on the fused target state, and use the actual target state or the fused target state as a preceding reference target state at a subsequent time, where the subsequent time is later than the current time.

[0152] It should be noted that the target trajectory acquisition device provided in the above embodiment and the target trajectory acquisition method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the target trajectory acquisition device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0153] See also Figure 6 , Figure 6 6 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 600 includes a memory 601 and a processor 602, and the processor 602 is used to execute the program instructions stored in the memory 601 to implement the steps in any of the above-mentioned target trajectory acquisition method embodiments. In a specific implementation scenario, the electronic device 600 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 600 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0154] Specifically, the processor 602 is used to control itself and the memory 601 to implement the steps in any of the above-mentioned target trajectory acquisition method embodiments. The processor 602 can also be called a central processing unit (CPU). The processor 602 may be an integrated circuit chip with signal processing capabilities. The processor 602 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 602 can be implemented by an integrated circuit chip.

[0155] See also Figure 7 , Figure 7The computer-readable storage medium 700 stores program instructions 710 that can be executed by a processor, and the program instructions 710 are used to implement the steps in any of the above-mentioned target trajectory acquisition method embodiments.

[0156] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0157] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0159] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A method for acquiring a target trajectory, characterized in that: A first sensor and a second sensor are deployed for a target detection area, and the method includes: Based on the environmental perception data collected by the first sensor and / or the environmental perception data collected by the second sensor at the current time, identifying the target state of the target, and obtaining the actual target state of the target at the current time; wherein the target state includes position parameters and motion parameters of the target; Acquire a previous reference target state corresponding to the current time, where the previous reference target state refers to a target state of the target calculated at a previous time, and the previous time is earlier than the current time; Calculate the time interval between the current time and the previous time, predict the position change and motion parameter change of the target after the time interval based on the previous reference target state, obtain the position parameter and motion parameter of the target at the current time, and use them as the predicted target state; Performing weighted fusion on the predicted target state and the actual target state to obtain a fused target state of the target at the current time; A trajectory is generated for the target based on the fused target states at various times, and the actual target state or the fused target state is used as a preceding reference target state at a subsequent time, where the subsequent time is later than the current time.

2. The method according to claim 1, characterized in that The step of predicting the position change and motion parameter change of the target after the time interval based on the previous reference target state, obtaining the position parameter and motion parameter of the target at the current time, and using them as the predicted target state, includes: Setting a state change weight value based on the time interval, wherein the time interval is proportional to the state change weight value; Calculate the product of the state change weight value and the previous reference target state to obtain a state change value; The sum of the preceding reference target state and the state change value is calculated to obtain a predicted target state.

3. The method according to claim 1, characterized in that The actual target state includes a first actual target state and / or a second actual target state; the identifying the target state of the target based on the environmental perception data collected by the first sensor and / or the environmental perception data collected by the second sensor at the current time, and obtaining the actual target state of the target at the current time, includes: Identify the target state of the target based on the environmental perception data collected by the first sensor at the current time to obtain a first actual target state; and / or identify the target state of the target based on the environmental perception data collected by the second sensor at the current time to obtain a second actual target state; The weighted fusion of the predicted target state and the actual target state to obtain the fused target state of the target at the current time includes: If the actual target state includes the first actual target state, obtaining a fusion weight corresponding to the first sensor to obtain a first fusion weight, performing weighted fusion on the predicted target state and the first actual target state according to the first fusion weight to obtain a fused target state of the target at the current time; If the actual target state includes the second actual target state, obtaining a fusion weight corresponding to the second sensor to obtain a second fusion weight, performing weighted fusion on the predicted target state and the second actual target state according to the second fusion weight to obtain a fused target state of the target at the current time; If the actual target state includes the first actual target state and the second actual target state, the predicted target state, the first actual target state and the second actual target state are weightedly fused according to the first fusion weight and the second fusion weight to obtain the fused target state of the target at the current time.

4. The method according to claim 3, characterized in that The first sensor includes a lidar sensor, and the second sensor includes an image sensor.

5. The method according to claim 1, characterized in that Generating a trajectory for the target based on the fused target state at each time includes: The target identified at the previous time is regarded as the previous target, and the target identified at the current time is regarded as the current target; Obtaining a predicted target state at the current time corresponding to the predicted fused target state based on each preceding target, and obtaining a fused target state of each current target calculated at the current time; Calculate the correlation between the predicted target state corresponding to each previous target and the fused target state corresponding to each current target, use the correlation as the correlation weight, and construct an association weight matrix; Determine a matching relationship between each of the preceding targets and each of the current targets when the sum of the associated weights in the associated weight matrix is ​​maximum; For the successfully matched current target and predecessor target, the fused target state of the successfully matched current target is added to the trajectory of the corresponding matched predecessor target.

6. The method according to claim 5, characterized in that The position parameter includes a target marking frame; the calculating the correlation between the predicted target state corresponding to each preceding target and the fused target state corresponding to each current target includes: Calculating an overlapping area between a target marking frame of the preceding target in the predicted target state and a target marking frame of the current target in the fused target state; The correlation degree between the predicted target state and the fused target state is obtained based on the overlapping area, and the overlapping area is proportional to the correlation degree.

7. The method according to claim 6, characterized in that The obtaining, based on the overlapping area, a correlation between the predicted target state and the fused target state comprises: If the overlapping area is smaller than a preset overlapping threshold, setting the correlation between the predicted target state and the fused target state to a preset minimum value; If the overlapping area is not less than the preset overlapping threshold, the correlation degree between the predicted target state and the fused target state is calculated according to the positive proportional relationship between the overlapping area and the correlation degree.

8. The method according to claim 5, characterized in that The method further comprises: If there is a previous target that does not match any current target, the number of lost cumulative frames corresponding to the previous target is increased by one; If the accumulated number of lost frames corresponding to the preceding target is greater than a preset frame number threshold, it is determined that the preceding target is lost, and the preceding target is deleted.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images