Predictive track processing method and apparatus, electronic device, and readable medium

Abstract

The application discloses a track prediction processing method and device, electronic equipment and readable medium. Fusion data including radar data sensed by a radar and video data shot by a video shooting device is acquired, and the fusion data represents a track fusion relationship between a radar target in the radar data and a video target in corresponding video data; the radar target and the video target having the track fusion relationship are associated in track, and a first association result is obtained; a to-be-predicted radar target with a failed association in the first association result is determined, and whether the to-be-predicted radar target meets a preset track prediction condition is determined; when the preset condition is met, a future track of the to-be-predicted radar target is predicted, and a predicted track is obtained; the predicted track is associated with the video target, a second association result is obtained, and the predicted track is processed according to the second association result. The scheme of the application can improve the authenticity of the track in the scene and guarantee the accuracy of traffic service data depending on the track.

Description

Technical Field

[0001] This invention relates to the field of trajectory verification technology, and in particular to a method, apparatus, electronic device, and readable medium for predicting trajectories. Background Technology

[0002] In real-world road scenarios, millimeter-wave radar is typically deployed at intersections to detect road targets. However, since millimeter-wave radar is not sensitive enough to detect targets at low speeds or in congested conditions, predictive methods are required to ensure the continuity of the detected targets' tracks.

[0003] In real-world road scenarios, environmental factors may lead to the presence of abnormal targets in radar or video, which reduces the accuracy of the trajectory in the scenario. After prediction processing, this may affect the operational data of millimeter-wave radar in traffic applications and fail to guarantee the accuracy of the predicted trajectory.

[0004] Therefore, when the accuracy of predicted flight paths cannot be guaranteed, a method for verifying predicted targets is needed. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and readable medium for predicting flight paths to ensure the accuracy of the predicted paths.

[0006] According to one aspect of the present invention, fused data including radar data sensed by radar and video data captured by a video capturing device is obtained, wherein the fused data characterizes the track fusion relationship between a radar target in the radar data and a corresponding video target in the video data;

[0007] The radar target is correlated with video targets that have track fusion relationships to obtain the first correlation result;

[0008] The first association result is determined to be a radar target to be predicted that has failed to be associated. It is then determined whether the radar target to be predicted meets the preset trajectory prediction conditions. If the preset conditions are met, the future trajectory of the radar target to be predicted is predicted to obtain the predicted trajectory.

[0009] The predicted trajectory is associated with the video target to obtain a second association result, and the predicted trajectory is processed based on the second association result.

[0010] Optionally, the step of associating the radar target with video targets having a track fusion relationship to obtain a first association result includes:

[0011] By converting each radar target and each video target to the same position reference frame, the radar track of the radar target and the video track of the video target are obtained respectively.

[0012] The association between radar targets and video targets is determined based on the radar track and the video track.

[0013] Optionally, determining the association result of the radar target and the video target based on the radar track and the video track includes:

[0014] For each radar target, execute:

[0015] Determine the degree of overlap between the radar track corresponding to the current radar target and the video track with the aforementioned track fusion relationship;

[0016] If a video track with an overlap higher than a preset value exists, the association result between the current radar target and the video target corresponding to the video track is determined to be a successful association; otherwise, the association result of the current radar target is determined to be a failed association.

[0017] Optionally, determining whether the radar target to be predicted meets the preset trajectory prediction conditions includes:

[0018] Determine whether the target lifecycle of the radar track of the radar target to be predicted has reached a set value;

[0019] And / or,

[0020] Whether the target speed of the radar target to be predicted reaches the set value.

[0021] Optionally, before associating the predicted trajectory with the video target, the method further includes:

[0022] Determine whether the number of prediction frames for the predicted trajectory has reached a set value;

[0023] If the set value is reached, then if the set value is not reached, determine whether the radar target to be predicted is the lead vehicle target in the channelized lane;

[0024] If the target is the lead vehicle, determine whether the video recognition of the predicted trajectory is interfered with;

[0025] If there is no interference, the predicted trajectory is associated with the video target.

[0026] Optionally, determining whether video recognition of the predicted trajectory is interfered with includes:

[0027] Determine whether the video data acquired by the video capturing device has stopped updating; if it has stopped updating, determine that the video recognition is being interfered with.

[0028] And / or,

[0029] Determine whether there is a large recognition box in the video data; if so, determine that the video recognition is interfered with.

[0030] And / or,

[0031] The number of radar targets in the radar data is compared with the number of video targets in the video data. If the number is less than a set value, interference is detected.

[0032] Optionally, associating the predicted trajectory with the video target to obtain a second association result includes:

[0033] The ID of each radar target and each video target is determined, wherein the ID of the radar target is determined by the radar and the ID of the video target is determined by the video capturing device;

[0034] Identify video targets that have the track fusion relationship with the radar target to be predicted in the fused data;

[0035] Determine whether the radar target to be predicted and the video target still have the track fusion relationship;

[0036] If the track fusion relationship is not present, the second association result is determined based on the position difference between the predicted track and the video target.

[0037] If the track fusion relationship exists, the second association result is determined based on the ID of the video target and the ID of the radar target to be predicted.

[0038] Optionally, determining the second association result based on the positional difference between the predicted trajectory and the video target includes:

[0039] An elliptical gate corresponding to the video target is established, and the elliptical gate is used to represent the positional difference with the video target;

[0040] Determine whether the predicted trajectory is within the elliptical gate. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

[0041] Optionally, determining the second association result based on whether the ID of the video target matches the ID of the radar target to be predicted includes:

[0042] The ID of each radar target and the ID of each video target included in the fused data are determined, wherein the ID of each radar target is determined by the radar and the ID of each video target is determined by the video capturing device;

[0043] Determine whether the ID of the radar target to be predicted and the ID of the video target both exist in the ID of the radar target and the ID of the video target. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

[0044] Optionally, processing the predicted trajectory based on the second association result includes:

[0045] If the second association result is an association failure, the number of association failure frames is recorded, and the predicted track is deleted when the number of association failure frames reaches a set value.

[0046] If the second association result is an association failure, then the number of association failure frames is cleared and the predicted trajectory is output.

[0047] According to another aspect of the present invention, a predictive trajectory processing apparatus is provided, comprising:

[0048] The fusion data acquisition unit is used to acquire fusion data including radar data sensed by radar and video data captured by video shooting device. The fusion data represents the track fusion relationship between the radar target in the radar data and the corresponding video target in the video data.

[0049] The track association unit is used to associate the radar target with video targets that have track fusion relationship to obtain a first association result;

[0050] The trajectory prediction unit is used to determine that the first association result is a radar target to be predicted that has failed to be associated, to determine whether the radar target to be predicted meets the preset trajectory prediction conditions, and when the preset conditions are met, to predict the future trajectory of the radar target to be predicted and obtain the predicted trajectory.

[0051] The processing unit is configured to associate the predicted trajectory with the video target to obtain a second association result, and process the predicted trajectory based on the second association result.

[0052] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0053] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the predictive trajectory processing method according to any embodiment of the present invention.

[0054] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the predictive trajectory processing method according to any embodiment of the present invention.

[0055] The technical solution of this invention involves acquiring fused data, including radar data sensed by radar and video data captured by a video shooting device. The fused data represents the trajectory fusion relationship between a radar target in the radar data and a corresponding video target in the video data. The radar target and the video target with the trajectory fusion relationship are then correlated to obtain a first correlation result. The first correlation result is determined to be a radar target to be predicted that has failed to be correlated. It is then determined whether the radar target to be predicted meets preset trajectory prediction conditions. If the preset conditions are met, the future trajectory of the radar target to be predicted is predicted to obtain a predicted trajectory. The predicted trajectory is correlated with the video target to obtain a second correlation result. The predicted trajectory is then processed based on the second correlation result. This invention solves the problem that abnormal targets or targets generated during prediction processing affect the accuracy of millimeter-wave radar's operational data in traffic applications. By verifying reliable video targets and predicted targets, combined with video quality judgment methods, abnormal targets can be effectively identified, thereby improving the accuracy of trajectories in the scenario and ensuring the accuracy of trajectory-dependent traffic operational data.

[0056] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a predicted trajectory processing method provided in Embodiment 1 of the present invention;

[0059] Figure 2 This is a schematic diagram illustrating a flight path association applicable to Embodiment 1 of the present invention;

[0060] Figure 3 This is a schematic diagram of the structure of a predictive trajectory processing device provided in Embodiment 2 of the present invention;

[0061] Figure 4This is a schematic diagram of the structure of an electronic device that implements the predicted trajectory processing method of the present invention. Detailed Implementation

[0062] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0064] Example 1

[0065] Figure 1 This is a flowchart of a predicted trajectory processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to verifying predicted targets when abnormal targets are present in radar or video. This method can be executed by a predicted trajectory processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0066] S110. Acquire fused data including radar data sensed by radar and video data captured by video shooting device, wherein the fused data represents the track fusion relationship between the radar target in the radar data and the corresponding video target in the video data.

[0067] In this embodiment of the invention, the radar mentioned is a millimeter-wave radar. Millimeter-wave radar is typically deployed at urban road intersections to identify passing vehicles. For example, it is used at urban intersections, installed on traffic light poles, with a stop line distance of 50-100 meters. Because millimeter-wave radar is not sensitive enough to detect targets in low-speed, congested conditions, predictive methods are needed to ensure trajectory continuity.

[0068] Fusion data refers to the fusion of detection tracks from millimeter-wave radar and video data. In real road scenarios, due to environmental factors, there may be abnormal targets in the radar or video, which reduces the accuracy of the trajectory in the scene. This invention aims to solve the problem that abnormal targets or targets generated by predictive processing affect the accuracy of business data of millimeter-wave radar in traffic applications.

[0069] S120. The radar target is associated with the video target that has a track fusion relationship to obtain the first association result.

[0070] In this embodiment of the invention, the step of converting each radar target and each video target to the same position reference frame to obtain the radar track of the radar target and the video track of the video target respectively includes:

[0071] The association between radar targets and video targets is determined based on the radar track and the video track.

[0072] The positioning of the tracks detected by millimeter-wave radar is based on the radar's coordinate system, while the positioning of the video tracks in the video data is based on the coordinate system of the video shooting device. Therefore, it is necessary to calibrate the two, such as converting the video tracks to the radar's coordinate system, so that the tracks can be compared and correlated intuitively.

[0073] In this embodiment of the invention, determining the association result of the radar target and the video target based on the radar track and the video track includes:

[0074] For each radar target, execute:

[0075] Determine the degree of overlap between the radar track corresponding to the current radar target and the video track with the aforementioned track fusion relationship;

[0076] If a video track with an overlap higher than a preset value exists, the association result between the current radar target and the video target corresponding to the video track is determined to be a successful association; otherwise, the association result of the current radar target is determined to be a failed association.

[0077] Since radar tracks and video tracks have a unified position reference frame after calibration, video tracks with a high degree of overlap with radar tracks can be considered to originate from the same target, thus successfully associating radar tracks and video tracks. If they do not overlap, it is possible that the radar target has been lost, making it impossible to associate with video tracks, and track prediction is required.

[0078] S130. Determine that the first association result is a radar target to be predicted that failed to be associated. Determine whether the radar target to be predicted meets the preset trajectory prediction conditions. If the preset conditions are met, predict the future trajectory of the radar target to be predicted to obtain the predicted trajectory.

[0079] In this embodiment of the invention, determining whether the radar target to be predicted meets the preset trajectory prediction conditions includes:

[0080] Determine whether the target lifecycle of the radar track of the radar target to be predicted has reached a set value;

[0081] And / or,

[0082] Whether the target speed of the radar target to be predicted reaches the set value.

[0083] The target lifecycle can be determined by whether the number of frames in the video reaches a set value. Since there is an upper limit to the detection and prediction of flight paths, paths exceeding this frame limit need to be retrieved. Similarly, if the frame count has reached the set value, the path needs to be retrieved to determine that it does not meet the prediction conditions. Because radar cannot identify static targets, it is necessary to determine whether the target's velocity reaches a set value to judge whether it is a valid target and, consequently, whether the prediction conditions are met.

[0084] In this embodiment of the invention, before associating the predicted trajectory with the video target, the method further includes:

[0085] Determine whether the number of prediction frames for the predicted trajectory has reached a set value;

[0086] If the set value is reached, then if the set value is not reached, determine whether the radar target to be predicted is the lead vehicle target in the channelized lane;

[0087] If the target is the lead vehicle, determine whether the video recognition of the predicted trajectory is interfered with;

[0088] If there is no interference, the predicted trajectory is associated with the video target.

[0089] If trajectory prediction is required, specific conditions need to be determined during the prediction process to decide whether trajectory prediction can continue. First, it needs to be determined whether the number of frames for predicting the trajectory has reached a set value. If the number of frames for prediction has reached the set value, the predicted target will be deleted and recycled, meaning that further processing is not possible.

[0090] Channelized lanes refer to the method of using traffic signs, markings, and traffic islands at intersections to guide vehicle and pedestrian traffic to their respective lanes. Channelized traffic involves marking lanes on the road or using green belts to separate lanes according to traffic volume, allowing vehicles of different types and speeds to travel in designated directions without interfering with each other, much like water flowing in a channel. By rationally arranging traffic islands, traffic signs, and markings (elements) at intersections, vehicles traveling in different directions and at different speeds are designated to travel in lanes with clearly defined tracks, avoiding mutual interference and thus reducing the possibility of collisions between vehicles and between vehicles and pedestrians, thereby improving traffic safety and capacity.

[0091] Since radar cannot identify static targets, it compensates by prediction. However, prediction is prone to distortion, so video is used to confirm the accuracy of the prediction by determining whether there is a leading vehicle in the unchannelized lane.

[0092] Since the lead vehicle in a channelized lane is not obstructed, it has a very high probability of being detected given the limited video detection capabilities. Therefore, detecting the lead vehicle can ensure the effectiveness of the target.

[0093] In addition, millimeter-wave radar can also perform traffic flow statistics. This is done by setting up loops and counting the number of vehicles passing through them. Since some vehicles may change lanes or turn, deviating from their original tracks, these vehicles can be considered "false targets" for traffic flow statistics. Therefore, only the first vehicle in the channelized lane is considered. By identifying whether a vehicle is a valid target before it passes through the loop, the accuracy of traffic flow statistics can be improved.

[0094] In this embodiment of the invention, determining whether the video recognition of the predicted trajectory is interfered with includes:

[0095] Determine whether the video data acquired by the video capturing device has stopped updating; if it has stopped updating, determine that the video recognition is being interfered with.

[0096] And / or,

[0097] Determine whether there is a large recognition box in the video data; if so, determine that the video recognition is interfered with.

[0098] And / or,

[0099] The number of radar targets in the radar data is compared with the number of video targets in the video data. If the number is less than a set value, interference is detected.

[0100] In the process of predicting flight paths, it is necessary to determine whether video recognition is interfered with, in order to determine in real time whether the video target is a valid target and whether prediction can continue. The determination of whether video recognition is interfered with is mainly based on the following conditions: 1. If the video data stops updating, it means the video target is lost and cannot be predicted; 2. If the video target has a large recognition area, such as being obscured by a large vehicle, the video target no longer corresponds to the original video target, and therefore accurate prediction is no longer possible; 3. The number of dynamic targets identified by the radar is compared with the number of targets in the video. If it is less than a set value, it is determined to be interference.

[0101] S140. Associate the predicted trajectory with the video target to obtain a second association result, and process the predicted trajectory according to the second association result.

[0102] The second correlation result represents the correlation between video targets and radar targets at the same time. The accuracy of the radar target used for trajectory prediction is determined by the second correlation result.

[0103] In this embodiment of the invention, associating the predicted trajectory with the video target to obtain a second association result includes:

[0104] The ID of each radar target and each video target is determined, wherein the ID of the radar target is determined by the radar and the ID of the video target is determined by the video capturing device;

[0105] Identify video targets that have the track fusion relationship with the radar target to be predicted in the fused data;

[0106] Determine whether the radar target to be predicted and the video target still have the track fusion relationship;

[0107] If the track fusion relationship is not present, the second association result is determined based on the position difference between the predicted track and the video target.

[0108] If the track fusion relationship exists, the second association result is determined based on the ID of the video target and the ID of the radar target to be predicted.

[0109] When the millimeter-wave radar and video recording device detect a target, they assign an ID to the identified target. For example, the target is numbered according to its location, such as radar targets 1, 2, 3, 4, and video targets A, B, C, D. Let's assume the ID of the radar target to be predicted before prediction is 3, and the ID of the corresponding video target is C.

[0110] In this embodiment of the invention, determining the second association result based on the position difference between the predicted trajectory and the video target includes:

[0111] An elliptical gate corresponding to the video target is established, and the elliptical gate is used to represent the positional difference with the video target;

[0112] Determine whether the predicted trajectory is within the elliptical gate. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

[0113] In this embodiment of the invention, determining the second association result based on whether the ID of the video target matches the ID of the radar target to be predicted includes:

[0114] The ID of each radar target and the ID of each video target included in the fused data are determined, wherein the ID of each radar target is determined by the radar and the ID of each video target is determined by the video capturing device;

[0115] Determine whether the ID of the radar target to be predicted and the ID of the video target both exist in the ID of the radar target and the ID of the video target. If yes, the second association result is a successful association; if no, the second association result is a failed association.

[0116] Figure 2 This is a schematic diagram of a track association provided in Embodiment 2 of the present invention, as shown below. Figure 2 As shown, after obtaining the predicted trajectory and the video target corresponding to the radar target to be predicted, it is determined whether the trajectory has been fused. If not, the position difference between the predicted trajectory and the video target is determined through an elliptical gate. The position difference is then determined to meet a set value based on whether the predicted trajectory falls within the elliptical gate. If the position difference meets the set value, the predicted trajectory and the video target can be considered the same target, the prediction is valid, and the second association result is a successful association; otherwise, the second association result is a failed association. If fused, it is determined that the ID of the radar target to be predicted and the ID of the video target both exist in the IDs of the radar target and the video target. For example, if the radar target IDs are 1, 2, 3, 4, the video target IDs are A, B, C, D, and the radar target ID to be predicted is 1, corresponding to video target A, and the predicted trajectory has been fused, it is determined whether the current video target A still exists. If it does, the prediction is valid, and the second association result is a successful association; otherwise, the second association result is a failed association.

[0117] In this embodiment of the invention, the predicted trajectory is processed based on the second association result, including:

[0118] Determine whether the radar target to be predicted and the video target are successfully associated. If the association is successful, the number of frames that failed to associate is cleared to zero, and the predicted track is output to obtain a more accurate track for the predicted target. If the association fails, the number of failed frames is recorded. If the set value is reached, the track will be deleted. Since the video may be affected by interference, such as obstacles or weather, some frames may be lost. Therefore, if the number of failed frames is less than the set value, it can be regarded as an error. The track will only be recovered if the set value is higher than the set value.

[0119] In summary, the solution of this invention solves the problem that abnormal targets or targets generated by predictive processing affect the accuracy of traffic data from millimeter-wave radar in traffic applications. By verifying predicted targets within the channelization using reliable video targets, combined with video quality judgment methods, abnormal targets can be effectively identified and deleted, thereby improving the accuracy of tracks in the scenario and ensuring the accuracy of traffic business data that depends on the track.

[0120] Example 2

[0121] Figure 3 This is a schematic diagram of a predictive trajectory processing device provided in Embodiment 2 of the present invention.

[0122] like Figure 3 As shown, the device includes:

[0123] The fusion data acquisition unit 310 is used to acquire fusion data including radar data sensed by radar and video data captured by video shooting device. The fusion data represents the track fusion relationship between the radar target in the radar data and the corresponding video target in the video data.

[0124] The track association unit 320 is used to associate the radar target with a video target that has a track fusion relationship to obtain a first association result;

[0125] The trajectory prediction unit 330 is used to determine that the first association result is a radar target to be predicted that has failed to be associated, to determine whether the radar target to be predicted meets the preset trajectory prediction conditions, and when the preset conditions are met, to predict the future trajectory of the radar target to be predicted and obtain the predicted trajectory.

[0126] The processing unit 340 is used to associate the predicted trajectory with the video target to obtain a second association result, and to process the predicted trajectory based on the second association result.

[0127] Optionally, the track association unit 320 is used to perform:

[0128] By converting each radar target and each video target to the same position reference frame, the radar track of the radar target and the video track of the video target are obtained respectively.

[0129] The association between radar targets and video targets is determined based on the radar track and the video track.

[0130] Optionally, when executing the process of determining the association results of radar targets and video targets based on the radar tracks and video tracks, the track association unit 320 performs the following for each radar target:

[0131] Determine the degree of overlap between the radar track corresponding to the current radar target and the video track with the aforementioned track fusion relationship;

[0132] If a video track with an overlap higher than a preset value exists, the association result between the current radar target and the video target corresponding to the video track is determined to be a successful association; otherwise, the association result of the current radar target is determined to be a failed association.

[0133] Optionally, when performing the step of determining whether the radar target to be predicted meets the preset trajectory prediction conditions, the trajectory prediction unit 330 specifically performs the following:

[0134] Determine whether the target lifetime of the radar track of the radar target to be predicted has reached a set value;

[0135] And / or,

[0136] Whether the target speed of the radar target to be predicted reaches the set value.

[0137] Optionally, the trajectory prediction unit 330, before associating the predicted trajectory with the video target, is further configured to perform:

[0138] Determine whether the number of prediction frames for the predicted trajectory has reached a set value;

[0139] If the set value is reached, then if the set value is not reached, determine whether the radar target to be predicted is the lead vehicle target in the channelized lane;

[0140] If the target is the lead vehicle, determine whether the video recognition of the predicted trajectory is interfered with;

[0141] If there is no interference, the predicted trajectory is associated with the video target.

[0142] Optionally, when performing the determination of whether video recognition of the predicted trajectory is interfered with, the trajectory prediction unit 330 specifically performs the following:

[0143] Determine whether the video data acquired by the video capturing device has stopped updating; if it has stopped updating, determine that the video recognition is being interfered with.

[0144] And / or,

[0145] Determine whether there is a large recognition box in the video data; if so, determine that the video recognition is interfered with.

[0146] And / or,

[0147] The number of radar targets in the radar data is compared with the number of video targets in the video data. If the number is less than a set value, interference is detected.

[0148] Optionally, when processing unit 340 performs the step of associating the predicted trajectory with the video target to obtain a second association result, it specifically performs the following:

[0149] The ID of each radar target and each video target is determined, wherein the ID of the radar target is determined by the radar and the ID of the video target is determined by the video capturing device;

[0150] Identify video targets that have the track fusion relationship with the radar target to be predicted in the fused data;

[0151] Determine whether the radar target to be predicted and the video target still have the track fusion relationship;

[0152] If the track fusion relationship is not present, the second association result is determined based on the position difference between the predicted track and the video target.

[0153] If the track fusion relationship exists, the second association result is determined based on the ID of the video target and the ID of the radar target to be predicted.

[0154] Optionally, when processing unit 340 determines the second association result based on the position difference between the predicted trajectory and the video target, it specifically performs the following:

[0155] An elliptical gate corresponding to the video target is established, and the elliptical gate is used to represent the positional difference with the video target;

[0156] Determine whether the predicted trajectory is within the elliptical gate. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

[0157] Optionally, when processing unit 340 determines the second association result based on whether the ID of the video target matches the ID of the radar target to be predicted, it specifically performs the following:

[0158] The ID of each radar target and the ID of each video target included in the fused data are determined, wherein the ID of each radar target is determined by the radar and the ID of each video target is determined by the video capturing device;

[0159] Determine whether the ID of the radar target to be predicted and the ID of the video target both exist in the ID of the radar target and the ID of the video target. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

[0160] Optionally, when processing the predicted trajectory based on the second association result, processing unit 340 specifically performs the following:

[0161] If the second association result is an association failure, the number of association failure frames is recorded, and the predicted track is deleted when the number of association failure frames reaches a set value.

[0162] If the second association result is an association failure, then the number of association failure frames is cleared and the predicted trajectory is output.

[0163] The predicted trajectory processing device provided in the embodiments of the present invention can execute the predicted trajectory processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0164] Example 3

[0165] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0166] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0167] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0168] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the predictive trajectory processing method.

[0169] In some embodiments, the trajectory prediction processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the trajectory prediction processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the trajectory prediction processing method by any other suitable means (e.g., by means of firmware).

[0170] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0175] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0176] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for predicting flight paths, characterized in that, include: Acquire fused data including radar data sensed by radar and video data captured by video shooting device, wherein the fused data represents the track fusion relationship between the radar target in the radar data and the corresponding video target in the video data; The radar target is correlated with video targets that have track fusion relationships to obtain the first correlation result; The first association result is determined to be a radar target to be predicted that has failed to be associated. It is then determined whether the radar target to be predicted meets the preset trajectory prediction conditions. If the preset trajectory prediction conditions are met, the future trajectory of the radar target to be predicted is predicted to obtain the predicted trajectory. The predicted trajectory is associated with the video target to obtain a second association result, and the predicted trajectory is processed according to the second association result; Before associating the predicted trajectory with the video target, the method further includes: Determine whether the number of prediction frames for the predicted trajectory has reached a set value; If the set value is reached, the radar target to be predicted will be deleted and retrieved. If the set value is not reached, determine whether the radar target to be predicted is the lead vehicle target in the channelized lane; If the target is the lead vehicle, determine whether the video recognition of the predicted trajectory is interfered with; If there is no interference, the predicted trajectory is associated with the video target; The step of determining whether the video recognition of the predicted trajectory is interfered with includes: Determine whether the video data acquired by the video capturing device has stopped updating; if it has stopped updating, determine that the video recognition is being interfered with. And / or, Determine whether there is a large recognition box in the video data; if so, determine that the video recognition is interfered with. And / or, The number of radar targets in the radar data is compared with the number of video targets in the video data. If the number is less than a set value, interference is detected.

2. The method according to claim 1, characterized in that, The step of associating the radar target with video targets having a track fusion relationship to obtain a first association result includes: By converting each radar target and each video target to the same position reference frame, the radar track of the radar target and the video track of the video target are obtained respectively. The association between radar targets and video targets is determined based on the radar track and the video track.

3. The method according to claim 2, characterized in that, The determination of the association result between the radar target and the video target based on the radar track and the video track includes: For each radar target, execute: Determine the degree of overlap between the radar track corresponding to the current radar target and the video track with the aforementioned track fusion relationship; If a video track with an overlap higher than a preset value exists, the association result between the current radar target and the video target corresponding to the video track is determined to be a successful association; otherwise, the association result of the current radar target is determined to be a failed association.

4. The method according to claim 2, characterized in that, Determining whether the radar target to be predicted meets the preset trajectory prediction conditions includes: Determine whether the target lifecycle of the radar track of the radar target to be predicted has reached a set value; And / or, Whether the target speed of the radar target to be predicted reaches the set value.

5. The method according to claim 1, characterized in that, The step of associating the predicted trajectory with the video target to obtain a second association result includes: The ID of each radar target and each video target is determined, wherein the ID of the radar target is determined by the radar and the ID of the video target is determined by the video capturing device; Identify video targets that have the track fusion relationship with the radar target to be predicted in the fused data; Determine whether the radar target to be predicted and the video target still have the track fusion relationship; If the track fusion relationship is not present, the second association result is determined based on the position difference between the predicted track and the video target. If the track fusion relationship exists, the second association result is determined based on the ID of the video target and the ID of the radar target to be predicted.

6. The method according to claim 5, characterized in that, Determining the second association result based on the position difference between the predicted trajectory and the video target includes: An elliptical gate corresponding to the video target is established, and the elliptical gate is used to represent the positional difference with the video target; Determine whether the predicted trajectory is within the elliptical gate. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

7. The method according to claim 5, characterized in that, The step of determining the second association result based on whether the ID of the video target matches the ID of the radar target to be predicted includes: The ID of each radar target and the ID of each video target included in the fused data are determined, wherein the ID of each radar target is determined by the radar and the ID of each video target is determined by the video capturing device; Determine whether the ID of the radar target to be predicted and the ID of the video target both exist in the ID of the radar target and the ID of the video target. If yes, the second association result is a successful association; otherwise, the second association result is a failed association.

8. The method according to claim 1, characterized in that, The step of processing the predicted trajectory based on the second association result includes: If the second association result is an association failure, the number of association failure frames is recorded, and the predicted track is deleted when the number of association failure frames reaches a set value. If the second association result is an association failure, then the number of association failure frames is cleared and the predicted trajectory is output.

9. A predictive trajectory processing device, characterized in that, include: The fusion data acquisition unit is used to acquire fusion data including radar data sensed by radar and video data captured by video shooting device. The fusion data represents the track fusion relationship between the radar target in the radar data and the corresponding video target in the video data. The track association unit is used to associate the radar target with video targets that have track fusion relationship to obtain a first association result; The trajectory prediction unit is used to determine that the first association result is a radar target to be predicted that has failed to be associated, to determine whether the radar target to be predicted meets the preset trajectory prediction conditions, and when the preset trajectory prediction conditions are met, to predict the future trajectory of the radar target to be predicted and obtain the predicted trajectory. The processing unit is configured to associate the predicted trajectory with the video target to obtain a second association result, and process the predicted trajectory based on the second association result; The trajectory prediction unit, before associating the predicted trajectory with the video target, is further configured to perform: Determine whether the number of prediction frames for the predicted trajectory has reached a set value; If the set value is reached, the radar target to be predicted will be deleted and retrieved. If the set value is not reached, determine whether the radar target to be predicted is the lead vehicle target in the channelized lane; If the target is the lead vehicle, determine whether the video recognition of the predicted trajectory is interfered with; If there is no interference, the predicted trajectory is associated with the video target; When the trajectory prediction unit performs the step of determining whether the video recognition of the predicted trajectory is interfered with, it specifically performs the following: Determine whether the video data acquired by the video capturing device has stopped updating; if it has stopped updating, determine that the video recognition is being interfered with. And / or, Determine whether there is a large recognition box in the video data; if so, determine that the video recognition is interfered with. And / or, The number of radar targets in the radar data is compared with the number of video targets in the video data. If the number is less than a set value, interference is detected.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the predictive trajectory processing method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the predictive trajectory processing method according to any one of claims 1-8.

Images