Roadside fusion tracking method and system based on Leiye frame fusion

Abstract

The invention discloses a roadside fusion tracking method and system based on Leiye frame fusion, and relates to the technical field of intelligent traffic. The method comprises the steps that when a certain MEC server triggers a fusion instruction and a timer of the MEC server reaches trigger time, image sensing data and radar sensing data in current cache data are acquired; performing frame fusion on the image sensing data and the radar sensing data to generate a thunder fusion frame; and carrying out target continuous tracking based on the Reighting fusion frame and the historical fusion data to obtain a tracking result, updating the historical fusion data, calculating to obtain a complete credible interval and a basic credible interval of the MEC server, and outputting the Reighting fusion frame and the tracking result at the same time. According to the method, the perception results of radar vision are fused, then multi-target tracking is carried out based on the fusion result, the continuous tracking effect of the target is good, the perception precision is high, the universality is high, the detection effect is good under the conditions of a static state and a long distance, and the problems of ghosting, splitting, position jumping and the like are effectively solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a roadside fusion tracking method and system based on radar-visual frame fusion. Background Art

[0002] At present, the roadside perception of vehicle-road collaboration mainly relies on MEC (Multi-Access Edge Computing). MEC realizes the perception, fusion and tracking of traffic participants through roadside intelligent devices such as cameras, millimeter-wave radars, traffic lights, and lidars. Among them, radar and camera detection have their own strengths. Millimeter-wave radars have poor detection effects on static objects, and cameras have poor detection effects on the position and speed of distant traffic participants. In addition, the common radar and visual detection in the existing technology are carried out separately, resulting in problems such as the splitting and fusion of detection objects and low accuracy of detection results during the perception process of MEC. In addition, the accuracy and reliability of MEC perception are different for different detection ranges, and the fusion tracking algorithm of the existing technology cannot identify and distinguish the perception results of the near end and the far end. In the scenario of continuous deployment of multiple MECs, it cannot provide reliable data support for continuous road perception coverage. Summary of the invention

[0003] The present application provides a roadside fusion tracking method and system based on radar-visual frame fusion, which solves the problems in the prior art that the detection objects cannot be split and fused during the perception process of MEC, the detection result accuracy is not high, the near-end and far-end perception results cannot be identified and distinguished, and multiple MECs cannot provide reliable data support for continuous road perception coverage when deployed continuously.

[0004] The specific technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a roadside fusion tracking method based on radar-visual frame fusion, comprising:

[0006] When a certain MEC server triggers a fusion instruction, it is determined whether the timer of the MEC server reaches the trigger time. If so, the image perception data and radar perception data in the current cache data are obtained, otherwise, the waiting continues; wherein, the timers of all the MEC servers are time-aligned timers after synchronous setting, the image perception data and the radar perception data are image perception results and radar perception results after data structure unification processing, and the image perception data includes the latest frame data of each camera sensor, and the radar perception data includes the latest frame data of each radar sensor;

[0007] Perform frame fusion on the image perception data and the radar perception data to generate a radar-visual fusion frame; wherein the radar-visual fusion frame includes a fusion frame number, a fusion timestamp, and a fusion object array, the fusion object array includes a fusion object and an attribute value of the fusion object, and the attribute value of the fusion object is obtained by fusion calculation based on the image perception data and the radar perception data and according to a radar projection algorithm and an optimal optimization algorithm;

[0008] Continuously track the target based on the radar-visual fusion frame and the historical fusion data to obtain the tracking result, and update the historical fusion data according to the radar-visual fusion frame and the tracking result, output the radar frame-visual fusion and the tracking result, and calculate the complete trust interval and basic trust interval of the MEC server based on the projection result of the radar projection algorithm; wherein the tracking result includes the target ID and the target history record.

[0009] In some embodiments of the present application, the image perception results include camera target type, target box pixel coordinate information, camera target position, camera target speed and camera target heading angle, and the radar perception results include radar target position coordinates, radar target speed and radar target heading angle; wherein, when the image perception results and the radar perception results are processed in a unified data structure, the target box pixel coordinate information and the radar target position coordinates are converted to the same coordinate system.

[0010] In some embodiments of the present application, the radar projection algorithm comprises the following steps:

[0011] The upper left point coordinates and lower right point coordinates of the camera box corresponding to a camera target are calculated according to the target box pixel coordinate information of a camera target. The calculation formula is:

[0012]

[0013] Among them, x c.b.lefttop The horizontal coordinate of the upper left point of the camera box, y c.b.lefttop The ordinate of the upper left point of the camera box; x c.p.downcenter is the horizontal coordinate of the center point of the lower edge of the camera target, y c.p.downcenter The ordinate of the center point of the lower edge of the camera target; x c.b.rightdown The horizontal coordinate of the lower right point of the camera box, y c.b.rightdown L is the vertical coordinate of the lower right point of the camera box; b.size H is the length of the camera box; b.size The height of the camera box;

[0014] Perform radar projection based on the coordinates of the lower left point and the upper right point of the camera box and the coordinates of the radar target position to determine whether the radar sensing object of the radar sensor exists in the camera box;

[0015] If the horizontal coordinate of the radar target position coordinate is not less than the horizontal coordinate of the upper left point of a camera box, and is not greater than the horizontal coordinate of the lower right point of the camera box, and the vertical coordinate of the radar target position coordinate is not greater than the vertical coordinate of the upper left point of the camera box, and is not less than the vertical coordinate of the lower right point of the camera box, then it is determined that the radar perception object exists in the camera box and the radar projection is successful, otherwise it is determined that the radar projection fails;

[0016] When the radar projection is successful, determining the number of camera perception objects corresponding to the radar perception objects in the camera box;

[0017] If the number of the camera-perceived objects is 1, the radar-perceived object of the radar sensor is matched with the camera-perceived object in the camera box to generate a new fusion object;

[0018] If the number of the camera-perceived objects is greater than 1, the camera-perceived object with the highest matching degree with the radar-perceived object is determined based on the camera target type, the target box pixel coordinate information, the camera target speed and the camera target heading angle of each camera-perceived object in the camera box, and based on the optimal optimization algorithm, and the camera-perceived object is matched with the radar-perceived object of the radar sensor to generate a new fusion object;

[0019] If the number of the camera-perceived objects is less than 1, the camera-perceived object with the highest matching degree with the radar-perceived object is determined based on the camera target type, the target box pixel coordinate information, the camera target speed and the camera target heading angle of each camera-perceived object in other camera boxes that have not been successfully projected, and based on the optimal optimization algorithm, and the camera-perceived object is matched with the radar-perceived object of the radar sensor to generate a new fusion object;

[0020] When radar projection fails, the discrete radar-visual fusion frames are re-fused based on the optimal optimization algorithm to obtain re-fused radar-visual fusion frames.

[0021] In some embodiments of the present application, calculating the matching degree between the camera-perceived object and the radar-perceived object based on the optimal optimization algorithm specifically includes:

[0022] According to the camera target type of the camera perception object, the target box pixel coordinate information, the camera target speed and the camera target heading angle, the matching degree between the camera perception object and the radar perception object is calculated, and the calculation formula is:

[0023] M c =ω c.t Z c.t +ω c.center Z c.center +ω c.v Z c.v +ω c.h Z c.h

[0024] Among them, M c Indicates the matching degree between the camera perception object and the radar perception object; Z c.t ω represents the attribute value of the camera target type; c.t Represents the weight of the attribute value of the camera target type; Z c.center Represents the attribute value of the camera target center point, Z c.center =1 means the distance difference between the camera target center point and the radar target position is not greater than the distance difference threshold, Z c.center =0 means that the distance difference between the camera target center point and the radar target position is greater than the distance difference threshold; ω c.center Represents the weight of the attribute value of the camera target center point; Z c.v The attribute value representing the camera's target speed, Z c.v =1 means the difference between the camera target speed and the radar target speed is not greater than the speed difference threshold, Z c.v =0 means the difference between the camera target speed and the radar target speed is greater than the speed difference threshold; ω c.v The weight of the attribute value representing the camera target speed; Z c.h Indicates the attribute value of the camera target heading angle, Z c.h =1 means the difference between the camera target heading angle and the radar target heading angle is not greater than the heading angle difference threshold, Z c.h =0 means that the difference between the camera target heading angle and the radar target heading angle is greater than the heading angle difference threshold; ω c.h The weight of the attribute value representing the camera's target heading angle.

[0025] In some embodiments of the present application, calculating the attribute value of the fusion object according to the optimal optimization algorithm specifically includes:

[0026] The attribute value of the fused object is calculated according to the radar target position coordinates, the radar target speed, the radar target heading angle of the radar perception object and the camera target position, the camera target speed, and the camera target heading angle of the camera perception object matched therewith, and the calculation formula is:

[0027] A f.value =ω f.p Z f.p +ω f.v Z f.v +ω f.h Z f.h

[0028] Among them, A f.value Represents the attribute value of the fused object; Z f.p Represents the attribute value of the fusion target position, Z f.p =1 means the radar perceives the object projection successfully, Z f.p =0 means radar fails to perceive the object projection; ω f.p Represents the weight of the attribute value of the fusion target position; Z f.v Indicates the attribute value of the fusion target speed, Z f.v =1 means the difference between the camera target speed and the radar target speed is not greater than the speed difference threshold, Z f.v =0 means the difference between the camera target speed and the radar target speed is greater than the speed difference threshold; ω f.v Indicates the weight of the attribute value of the fusion target speed; Z f.h Indicates the attribute value of the fusion target heading angle, Z f.h =1 means the difference between the camera target heading angle and the radar target heading angle is not greater than the heading angle difference threshold, Z f.h =0 means that the difference between the camera target heading angle and the radar target heading angle is greater than the heading angle difference threshold; ω f.h Indicates the weight of the attribute value of the fused target heading angle.

[0029] In some embodiments of the present application, the fusion object array further includes the object type, object position, object speed, object heading angle, and target ID of the fusion object, and the target is continuously tracked based on the radar-visual fusion frame and the historical fusion data to obtain the tracking result, and the historical fusion data is updated according to the radar-visual fusion frame and the tracking result, specifically including:

[0030] Tracking and matching the fused object in the radar-vision fusion frame with the historical fusion object in the historical fusion data according to the object type, the object position, the object speed, and the object heading angle;

[0031] If the fusion object in the thunder-vision fusion frame successfully matches one of the historical fusion objects in the historical fusion data, updating the fusion object array of the historical fusion object according to the thunder-vision fusion frame;

[0032] If the fused object in the radar-vision fusion frame fails to match all the historical fused objects in the historical fusion data, a new target ID is generated for the fused object, and the fused object is added to the historical fusion data.

[0033] In some embodiments of the present application, the complete trust interval and the basic trust interval of the MEC server are calculated based on the projection result of the radar projection algorithm, specifically including:

[0034] Based on the projection result of the radar projection algorithm, a completely credible value of each historical fusion object in the updated historical fusion data is calculated, if the historical fusion object is generated by fusion when the radar perception object projection is successful, then the completely credible value of the historical fusion object is 1; if the historical fusion object is generated by fusion when the radar perception object projection fails, then the completely credible value of the historical fusion object is 0;

[0035] The fully credible interval of the MEC server is determined according to all the historical fusion objects whose fully credible values ​​are 1, and the basic credible interval of the MEC server is determined according to all the historical fusion objects whose fully credible values ​​are 0.

[0036] In some embodiments of the present application, the completely credible interval is a set of points whose completely credible values ​​are 1, and the basic credible interval is a set of points whose completely credible values ​​are 0. When determining the completely credible interval and the basic credible interval, the following steps are included:

[0037] Let S be any subset of Euclidean space. The smallest convex set of S is called the convex hull of S, denoted by conv(S). Let P = {P1, P2, ..., P n}=conv(S), which is solved as follows:

[0038] (1) Select the point H with the largest x coordinate from all the points, use it as the base point, and add the base points H to the set P in sequence; if there are multiple points with the largest x coordinates, select the point with the largest y coordinate; exclude points with the same coordinates;

[0039] (2) The remaining points P i The vector formed by the base point H <H,P i >, calculate the angle between them and the x-axis, sort them, select the point with the largest angle, record it as P1, and put P1 into the set P in sequence; if there are multiple points with the largest angle, select the point with the largest y coordinate;

[0040] (3) Select the latest two points from the set P to form a vector <P n-1 , P n>, from the remaining points P i and the latest point P of set P n The vector composed <P i , P n >, calculate its difference with the vector <P n-1 , P n >, and sort them, select the point with the largest angle, and record it as P i , put P i Put them into the set P; if there are multiple points with the largest angle, select the point with the largest y coordinate;

[0041] (4) Repeat step (3) until the set P is traversed and the selected P i Let be the vertices of the convex hull of set P;

[0042] (5) The polygon formed by all convex hull vertices in set P is determined as the trusted interval of the MEC server, and the area of ​​the polygon in set P is calculated to obtain the coverage area of ​​the trusted interval of the MEC server.

[0043] In a second aspect, the embodiment of the present application provides a roadside fusion tracking system based on radar-visual frame fusion, including:

[0044] A fusion trigger module is used to determine whether the timer of a MEC server reaches the trigger time when a fusion instruction is triggered by a certain MEC server. If so, the image perception data and radar perception data in the current cache data are obtained, otherwise, the waiting continues; wherein the timers of all the MEC servers are time-aligned timers after synchronous setting, the image perception data and the radar perception data are image perception results and radar perception results after data structure unification processing, and the image perception data includes the latest frame data of each camera sensor, and the radar perception data includes the latest frame data of each radar sensor;

[0045] A frame fusion module, used for performing frame fusion on the image perception data and the radar perception data to generate a radar-visual fusion frame; wherein the radar-visual fusion frame includes a fusion frame number, a fusion timestamp, and a fusion object array, the fusion object array includes a fusion object and an attribute value of the fusion object, and the attribute value of the fusion object is obtained by fusion calculation based on the image perception data and the radar perception data and according to a radar projection algorithm and an optimal optimization algorithm;

[0046] A tracking and credible interval calculation module is used to continuously track the target based on the radar-visual fusion frame and the historical fusion data, obtain the tracking result, update the historical fusion data according to the radar-visual fusion frame and the tracking result, output the radar frame-visual fusion and the tracking result, and calculate the complete credible interval and basic credible interval of the MEC server based on the projection result of the radar projection algorithm; wherein the tracking result includes the target ID and the target history record.

[0047] In some embodiments of the present application, the fused object array further includes the object type, object position, object speed, object heading angle and target ID of the fused object, and the tracking and trustworthy interval calculation module includes:

[0048] a target continuous tracking unit, for tracking and matching the fused object in the radar vision fusion frame with the historical fused objects in the historical fusion data according to the object type, the object position, the object speed and the object heading angle; if the fused object in the radar vision fusion frame successfully matches one of the historical fused objects in the historical fusion data, then updating the data of the fused object array of the historical fusion object according to the radar vision fusion frame; if the fused object in the radar vision fusion frame fails to match all the historical fused objects in the historical fusion data, then generating a new target ID for the fused object and adding the fused object to the historical fusion data;

[0049] A credible interval calculation unit is used to calculate the fully credible value of each historical fusion object in the updated historical fusion data based on the projection result of the radar projection algorithm, if the historical fusion object is generated by fusion when the radar perception object projection is successful, then the fully credible value of the historical fusion object is 1, if the historical fusion object is generated by fusion when the radar perception object projection fails, then the fully credible value of the historical fusion object is 0, and is used to determine the fully credible interval of the MEC server according to all the historical fusion objects with a fully credible value of 1, and determine the basic credible interval of the MEC server according to all the historical fusion objects with a fully credible value of 0.

[0050] In a third aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory;

[0051] The processor is coupled to the memory, the memory is used to store a computer program, and the computer program is executed by the processor, so that the computer device implements the roadside fusion tracking method based on radar-visual frame fusion as described in the first aspect.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the roadside fusion tracking method based on radar-visual frame fusion as described in the first aspect is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes program instructions. When the program instructions are executed on a computer or a processor, the computer or the processor executes the roadside fusion tracking method based on radar-visual frame fusion as described in the first aspect.

[0054] The innovative features of the embodiments of the present application include but are not limited to the following:

[0055] 1. Frame fusion of image perception data of visual perception target detection and radar perception data of radar perception target detection is one of the innovative points of the embodiment of the present application. It has high perception accuracy and makes full use of the advantages of radar and camera to make up for each other's shortcomings. After using the optimal optimization algorithm, indicators such as target type, speed, position, and heading angle are more controllable than those of the prior art.

[0056] 2. Continuous multi-target tracking using the radar-visual fusion frames after frame fusion is one of the innovative features of the embodiments of the present application. The continuous tracking of targets is effective and has high universality. The detection effect is good in static and long-distance conditions, and the problems of ghosting, splitting, position jump, etc. in the prior art are effectively solved.

[0057] 3. This method can realize the calculation of the trusted coverage area of ​​the MEC server and determine the completely trusted interval and basic trusted interval of the MEC server. It is one of the innovative points of the embodiment of the present application and can evaluate the fusion effect of the MEC server, which can further improve the driving safety of intelligent connected vehicles.

[0058] The beneficial effects of the embodiments of the present application are as follows:

[0059] Through the image perception results of visual perception target detection and the radar perception results of radar perception target detection, the image perception results and radar perception results are unified in data structure and then cached. When the single-point fusion of the MEC server is triggered and the timer reaches the trigger time, the latest frame data of the cached camera sensor and radar sensor are obtained, and the acquired image perception data and radar perception data are frame-fused to generate a radar-visual fusion frame. The fused radar-visual fusion frame is used for continuous multi-target tracking, and the corresponding historical fusion targets are associated or new fusion targets are added. On this basis, target trajectory tracking is performed, which can effectively solve the problems of ghosting, splitting, position jump, etc. in the prior art. The continuous tracking effect of the target is good, and the perception accuracy is high, the universality is high, and the detection effect is good in static state and long distance. At the same time, the method can also calculate the trusted coverage area of ​​the MEC server, which can evaluate the fusion effect of the MEC server and further improve the driving safety of intelligent connected vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 A schematic diagram of a flow chart of a roadside fusion tracking method based on radar-visual frame fusion provided in an embodiment of the present application;

[0062] Figure 2 A schematic block diagram of the composition of a roadside fusion tracking system based on radar-visual frame fusion provided in an embodiment of the present application;

[0063] Figure 3 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The terms "including" and "having" in the embodiments of the present application and the accompanying drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0066] The embodiment of the present application discloses a roadside fusion tracking method based on radar-visual frame fusion, which is applied to MEC (Multi-Access Edge Computing) servers and is mainly applied to scenarios where multiple MEC servers are continuously deployed. The MEC server interacts with multiple roadside sensors and intelligent networked vehicles. By deploying local MEC servers at multiple networked intersections, the image perception results output by camera sensors and the radar perception results output by radar sensors are collected at the road end, and the image perception results and radar perception results are locally fused (referred to as "radar-visual fusion") and the fused multi-target tracking is performed locally. The fused tracking results calculated locally are then published through RSU (Road Side Unit), so as to use the continuous multi-target fusion tracking results to assist intelligent networked vehicles in making correct driving decisions, improve the driving safety and traffic efficiency of the vehicle, and determine the confidence area of ​​the MEC server to achieve fusion effect evaluation. In addition, the fused tracking data can also be uploaded to the cloud server to support applications such as holographic intersection display.

[0067] Figure 1 1 is a flow chart of a roadside fusion tracking method based on radar-visual frame fusion provided by an embodiment of the present application. Figure 1 The roadside fusion tracking method based on radar-visual frame fusion is described in detail.

[0068] like Figure 1 As shown, the roadside fusion tracking method based on radar frame fusion includes the following steps:

[0069] Step S110: When a certain MEC server triggers a fusion instruction, it is determined whether the timer of the MEC server reaches the trigger time.

[0070] In an embodiment of the present application, timers of all MEC servers deployed on the roadside are set synchronously, that is, the timers of all MEC servers are time-aligned timers after synchronous setting, so as to ensure that the timers of all MEC servers are aligned and the fusion frequency is consistent.

[0071] In a scenario where multiple MEC servers are deployed continuously, when any single-point MEC server triggers a fusion instruction, it is necessary to determine whether its timer has reached the trigger time. If not, proceed from step S110 to step S120. If so, proceed from step S110 to step S130.

[0072] Step S120: Continue to wait until the timer of the MEC server reaches the trigger time.

[0073] When the timer of the MEC server reaches the trigger time, the process proceeds from step S110 or step S120 to step S130.

[0074] Step S130: Obtain image perception data and radar perception data in the current cache data.

[0075] In the embodiment of the present application, each networked intersection is equipped with a camera sensor and a radar sensor. The camera sensor outputs the image perception result of the visual perception target detection, and the radar sensor outputs the radar perception result of the radar perception target detection. The MEC server communicates with the camera sensor and the radar sensor through optical fiber and a switch to obtain the image perception result and the radar perception result. After that, the MEC server unifies the data structure of the acquired image perception result and radar perception result, and caches the processed image perception result and radar perception result, so as to use the cached data for subsequent fusion tracking calculation.

[0076] Specifically, the image perception results include but are not limited to the camera target type, target box pixel coordinate information, camera target position, camera target speed, and camera target heading angle. The target box pixel coordinate information includes the coordinates of the center point of the lower edge of the camera target. The radar perception results include but are not limited to the radar target position coordinates, radar target speed, and radar target heading angle. When the image perception results and radar perception results are unified in data structure processing, the target box pixel coordinate information and the radar target position coordinates are converted to the same coordinate system, for example, the coordinates of both are converted to the road coordinate system.

[0077] The roadside fusion tracking method based on radar-visual frame fusion adopts frame fusion based on radar data frames and visual data frames. The image perception results and radar perception results in this application refer to each frame perception result of the camera sensor and radar sensor respectively. Since the output frequencies of the radar and the camera are inconsistent, and the output frequency of the camera is generally higher than the output frequency of the radar, this application uses the cached data of the radar and the camera for subsequent fusion tracking calculations, which solves the problem of inconsistent output frequencies of the radar and the camera, and it caches the latest frame data of each sensor, reducing the read operation on the server database, reducing the pressure on the database, and speeding up the response speed. Among them, it should be noted that the image perception data includes the latest frame data of each camera sensor, and the radar perception data includes the latest frame data of each radar sensor. The "latest frame data" mentioned above refers to all new frame data between the last frame fusion time point and the current frame fusion time point.

[0078] Step S140: Frame fusion of the image perception data and the radar perception data to generate a radar-visual fusion frame.

[0079] In the embodiment of the present application, the radar-visual fusion frame includes a fusion frame number, a fusion timestamp, and a fusion object array. The fusion frame number is set incrementally, the fusion timestamp is the time when the radar-visual fusion is currently completed, the fusion object array includes the fusion object and the attribute value of the fusion object, the fusion object is a new object generated after matching the visual and radar perception objects, and the attribute value of the fusion object is obtained by fusion calculation based on image perception data and radar perception data and radar projection (RadarProjection) algorithm and optimal optimization algorithm (Best Optimized).

[0080] When the image perception data and the radar perception data are frame-fused, three situations may occur after the radar projection is successful: the first is that the radar perception object corresponds to multiple perception objects in the camera visual perception object frame (box); the second is that the radar perception object corresponds to one perception object in the camera box; the third is that the number of camera perception objects corresponding to the radar perception object in the camera box is less than 1. Specifically, the radar projection algorithm refers to: judging whether the radar perception object perceived by the radar is within the camera perception object perceived by the vision, if there is one, the radar perception object perceived by the radar coincides with the camera perception object perceived by the vision, if there are multiple, then the most suitable camera perception object is selected based on key features such as type, center point, speed, heading angle, etc., if not, then the most suitable one is selected from other visual perception object frames that have not been successfully projected based on key features such as type, center point, speed, heading angle, etc.

[0081] In some embodiments, the radar projection algorithm includes the following steps:

[0082] A. Calculate the upper left and lower right coordinates of the camera box corresponding to a camera target based on the target box pixel coordinate information of a camera target. The calculation formula is:

[0083]

[0084] Among them, x c.b.lefttop The horizontal coordinate of the upper left point of the camera box, y c.b.lefttop The ordinate of the upper left point of the camera box; x c.p.downcenter is the horizontal coordinate of the center point of the lower edge of the camera target, y c.p.downcenter The ordinate of the center point of the lower edge of the camera target; x c.b.rightdown The horizontal coordinate of the lower right point of the camera box, y c.b.rightdown L is the vertical coordinate of the lower right point of the camera box; b.size H is the length of the camera box; b.size The height of the camera box.

[0085] B. Perform radar projection based on the coordinates of the lower left point, the upper right point and the radar target position of the camera box to determine whether the radar sensor's radar perception object exists in the camera box.

[0086] Specifically, the radar projection is judged to be successful based on the horizontal and vertical coordinates of the radar target position coordinates and the lower left and upper right coordinates of the camera box.

[0087] C. If the horizontal coordinate of the radar target position coordinate is not less than the horizontal coordinate of the upper left point of a camera box, and not greater than the horizontal coordinate of the lower right point of the camera box, and the vertical coordinate of the radar target position coordinate is not greater than the vertical coordinate of the upper left point of the camera box, and not less than the vertical coordinate of the lower right point of the camera box, that is If the radar projection is successful, the radar perception object exists in the camera box; otherwise, the radar projection is failed, and the radar perception object does not exist in the camera box.

[0088] D. When the radar projection is successful, determine the number of camera-perceived objects corresponding to the radar-perceived objects in the camera box.

[0089] E. If the number of camera-perceived objects is 1, match the radar-perceived object of the radar sensor with the camera-perceived object in the camera box to generate a new fusion object.

[0090] F. If the number of camera-perceived objects is greater than 1, the camera-perceived object with the highest matching degree with the radar-perceived object is determined based on the camera target type, target box pixel coordinate information, camera target speed, and camera target heading angle of each camera-perceived object in the camera box, and the optimal optimization algorithm is used to match it with the radar-perceived object of the radar sensor to generate a new fusion object.

[0091] G. If the number of camera-perceived objects is less than 1, the camera-perceived object with the highest matching degree with the radar-perceived object is determined based on the camera target type, target box pixel coordinate information, camera target speed and camera target heading angle of each camera-perceived object in other camera boxes that have not been successfully projected, and the camera-perceived object is matched with the radar-perceived object of the radar sensor to generate a new fusion object.

[0092] H. When radar projection fails, the discrete radar-visual fusion frames are re-fused based on the optimal optimization algorithm to obtain the re-fused radar-visual fusion frames.

[0093] In the above radar projection algorithm, the matching degree between the camera perception object and the radar perception object is calculated based on the optimal optimization algorithm, specifically including: according to the camera target type, target box pixel coordinate information, camera target speed and camera target heading angle of the camera perception object, the matching degree between the camera perception object and the radar perception object is calculated, and the calculation formula is:

[0094] M c =ω c.t Z c.t +ω c.center Z c.center +ω c.v Z c.v +ω c.h Z c.h

[0095] Among them, M c Indicates the matching degree between the camera perception object and the radar perception object; Z c.t ω represents the attribute value of the camera target type; c.t Represents the weight of the attribute value of the camera target type; Z c.center Represents the attribute value of the camera target center point, Z c.center =1 means the distance difference between the camera target center point and the radar target position is not greater than the distance difference threshold, Z c.center =0 means that the distance difference between the camera target center point and the radar target position is greater than the distance difference threshold; ω c.center Represents the weight of the attribute value of the camera target center point; Z c.v The attribute value representing the camera's target speed, Z c.v=1 means the difference between the camera target speed and the radar target speed is not greater than the speed difference threshold, Z c.v =0 means the difference between the camera target speed and the radar target speed is greater than the speed difference threshold; ω c.v The weight of the attribute value representing the camera target speed; Z c.h Indicates the attribute value of the camera target heading angle, Z c.h =1 means the difference between the camera target heading angle and the radar target heading angle is not greater than the heading angle difference threshold, Z c.h =0 means that the difference between the camera target heading angle and the radar target heading angle is greater than the heading angle difference threshold; ω c.h The weight of the attribute value representing the camera's target heading angle.

[0096] In other embodiments, the attribute value of the fused object is calculated according to the optimal optimization algorithm, specifically including: according to the radar target position coordinates, radar target speed, radar target heading angle of the radar perception object and the camera target position, camera target speed, and camera target heading angle of the camera perception object matched therewith, the attribute value of the fused object is calculated, and the calculation formula is:

[0097] A f.value =ω f.p Z f.p +ω f.v Z f.v +ω f.h Z f.h

[0098] Among them, A f.value Represents the attribute value of the fused object; Z f.p Represents the attribute value of the fusion target position, Z f.p =1 means the radar perceives the object projection successfully, Z f.p =0 means radar fails to perceive the object projection; ω f.p Represents the weight of the attribute value of the fusion target position; Z f.v Indicates the attribute value of the fusion target speed, Z f.v =1 means the difference between the camera target speed and the radar target speed is not greater than the speed difference threshold, Z f.v =0 means the difference between the camera target speed and the radar target speed is greater than the speed difference threshold; ω f.v Indicates the weight of the attribute value of the fusion target speed; Z f.h Indicates the attribute value of the fusion target heading angle, Z f.h =1 means the difference between the camera target heading angle and the radar target heading angle is not greater than the heading angle difference threshold, Z f.h =0 means that the difference between the camera target heading angle and the radar target heading angle is greater than the heading angle difference threshold; ω f.h Indicates the weight of the attribute value of the fused target heading angle.

[0099] In an embodiment of the present application, the same attributes perceived by radar and vision (i.e., position coordinates, speed, and heading angle) are recalculated based on the weights of various factors through an optimal optimization algorithm to obtain the attribute values ​​of the fused object, thereby evaluating the target similarity in subsequent target tracking, thereby making the tracking association matching degree of the roadside fusion tracking method based on radar-visual frame fusion higher.

[0100] Step S150: Continuously track the target based on the radar-visual fusion frame and the historical fusion data to obtain the tracking result, and update the historical fusion data according to the radar-visual fusion frame and the tracking result, output the radar frame-visual fusion and tracking result, and calculate the complete credible interval and basic credible interval of the MEC server based on the projection result of the radar projection algorithm.

[0101] In the embodiment of the present application, the tracking result includes the target ID and the target history record. In the specific implementation process, the tracking result can be set as a Key-Value Map data structure, where the Key is the target ID and the Value is the history record (array) of the target in all fused frames.

[0102] Furthermore, the fusion object array also includes the object type, object position, object speed, object heading angle and target ID of the fusion object. The fusion object in the newly generated radar fusion frame is matched with the existing tracking results in the historical fusion data based on the tracking algorithm. If the matching result of the fusion object is a new object, a target ID is regenerated for the fusion object and the fusion object is added to the Map data structure. If the matching result of the fusion object is a historical fusion object, the target ID remains unchanged and the array is updated.

[0103] In some embodiments, the specific implementation process of matching based on the tracking algorithm to achieve continuous target tracking includes the following steps:

[0104] 1. According to the object type, object position, object speed and object heading angle, the fused object in the radar fusion frame is tracked and matched with the historical fusion object in the historical fusion data.

[0105] The tracking algorithm in this embodiment is matched based on four key features: object type, object position, object speed, and object heading angle. Specifically, when matching based on object type, the array in the target history record Map data structure that is inconsistent with the object type of the fused object is filtered, and object position matching is performed based on this filtering, and the array in the target history record Map data structure whose object position difference with the fused object is greater than the distance difference threshold is filtered, and object heading angle matching is performed based on the object position filtering, and the array in the target history record Map data structure whose object heading angle difference with the fused object is greater than the heading angle difference threshold is filtered, and object speed matching is performed based on the object heading angle filtering, and the array in the target history record Map data structure whose object speed difference with the fused object is greater than the speed difference threshold is filtered. After matching and filtering the object type, object position, object speed, and object heading angle, if only one historical fusion object can be determined, the match is determined to be successful, otherwise the match is determined to be failed.

[0106] 2. If the fusion object in the radar-vision fusion frame successfully matches one of the historical fusion objects in the historical fusion data, the fusion object array of the historical fusion object is updated according to the radar-vision fusion frame.

[0107] When the fusion object in the radar-vision fusion frame successfully matches one of the historical fusion objects in the historical fusion data, the matching result of the fusion object is determined to be the historical fusion object, and the target ID remains unchanged. The original target ID of the historical fusion object is used, and the array of the historical fusion object in the Map data structure is updated using the newly fused radar-vision fusion frame.

[0108] 3. If the fused object in the radar-vision fusion frame fails to match all historical fusion objects in the historical fusion data, a new target ID is generated for the fused object and the fused object is added to the historical fusion data.

[0109] When the fusion object in the radar fusion frame fails to match all the historical fusion objects in the historical fusion data, that is, the matching result of the fusion object is determined to be a new object, a target ID is regenerated, and the fusion object is added to the Map data structure based on the newly generated target ID.

[0110] The present application utilizes the above-mentioned tracking algorithm to realize the detection of tracking results. By detecting the tracking results, it is possible to ensure the consistency of attributes such as object type and target ID of the tracked object, as well as the continuity of attributes such as object position, trajectory, and object speed.

[0111] In addition, the roadside fusion tracking method based on radar frame fusion in this application can not only realize continuous multi-target fusion tracking, but also calculate the trusted coverage area of ​​the MEC server, that is, determine the complete trusted interval and basic trusted interval of the MEC server, and realize the evaluation of the fusion effect.

[0112] In some other embodiments, the specific calculation process of the trusted coverage area of ​​the MEC server includes the following steps:

[0113] a. Based on the projection results of the radar projection algorithm, the fully credible value of each historical fusion object in the updated historical fusion data is calculated. If the historical fusion object is generated by fusion when the radar perception object projection is successful, the fully credible value of the historical fusion object is 1. If the historical fusion object is generated by fusion when the radar perception object projection fails, the fully credible value of the historical fusion object is 0.

[0114] The fully trusted value of the historical fusion object is TrustObj CRFusion , then if the historical fusion object is generated by fusion when the radar perception object projection is successful, TrustObj CRFusion =1; if the historical fusion object is generated by fusion when the radar perception object projection fails, TrustObj CRFusion =0.

[0115] b. Determine the fully trusted interval of the MEC server based on all historical fusion objects with a fully trusted value of 1, and determine the basic trusted interval of the MEC server based on all historical fusion objects with a fully trusted value of 0.

[0116] Totally trusted interval = {TrustObj CRFusion =1 point};

[0117] Basic Trust Interval = {TrustObj CRFusion =0 point}.

[0118] When determining the credible interval, let S be any subset of the Euclidean space, and the smallest convex set of S is called the convex hull of S, denoted by conv(S).

[0119] Let P = {P1, P2, ..., P n}=conv(S), which is solved as follows:

[0120] (1) Select the point H with the largest x coordinate from all the points, use it as the base point, and add the base points H to the set P in sequence; if there are multiple points with the largest x coordinates, select the point with the largest y coordinate; exclude points with the same coordinates;

[0121] (2) The remaining points Pi The vector formed by the base point H <H,P i >, calculate the angle between them and the x-axis, sort them, select the point with the largest angle, record it as P1, and put P1 into the set P in sequence; if there are multiple points with the largest angle, select the point with the largest y coordinate;

[0122] (3) Select the latest two points from the set P to form a vector <P n-1 , P n >, from the remaining points P i and the latest point P of set P n The vector composed <P i , P n >, calculate its difference with the vector <P n-1 , P n >, and sort them, select the point with the largest angle, and record it as P i , put P i Put them into the set P; if there are multiple points with the largest angle, select the point with the largest y coordinate;

[0123] (4) Repeat step (3) until the set P is traversed and the selected P i Let be the vertices of the convex hull of set P.

[0124] (5) The polygon formed by all convex hull vertices in set P is determined as the trusted interval of the MEC server, and the area of ​​the polygon in set P is calculated to obtain the coverage area of ​​the trusted interval of the MEC server.

[0125] It should be noted and understood that the convex hull vertex conv_p is determined counterclockwise above, and the clockwise direction is similar, so it will not be repeated here.

[0126] The roadside fusion tracking method based on radar-visual frame fusion in the embodiment of the present application uses the image perception results of visual perception target detection and the radar perception results of radar perception target detection, unifies the data structure of the image perception results and the radar perception results, and then caches the data. When the single-point fusion of the MEC server is triggered and the timer reaches the trigger time, the latest frame data of the cached camera sensor and the radar sensor are obtained, and the acquired image perception data and the radar perception data are frame-fused to generate a radar-visual fusion frame, and the fused radar-visual fusion frame is used to perform continuous multi-target tracking, associate the corresponding historical fusion targets or add new fusion targets, and then track the target trajectory on this basis, which can effectively solve the problems of ghosting, splitting, position jump, etc. in the prior art. The continuous tracking effect of the target is good, and the perception accuracy is high, the universality is high, and the detection effect is good in the static state and long-distance conditions. At the same time, the method can also calculate the trusted coverage area of ​​the MEC server, and can evaluate the fusion effect of the MEC server, which can further improve the driving safety of intelligent connected vehicles.

[0127] Corresponding to the above-mentioned roadside fusion tracking method embodiment based on radar-visual frame fusion, the present application embodiment also provides a roadside fusion tracking system based on radar-visual frame fusion. The roadside fusion tracking system based on radar-visual frame fusion includes a unit for executing the above-mentioned roadside fusion tracking method embodiment based on radar-visual frame fusion, and the system can be configured in a server. Specifically, Figure 2 As shown, the roadside fusion tracking system 200 based on radar-visual frame fusion includes a fusion trigger module 210 , a frame fusion module 220 , and a tracking and credible interval calculation module 230 .

[0128] The fusion trigger module 210 is used to determine whether the timer of the MEC server reaches the trigger time when a certain MEC server triggers the fusion instruction. If so, the image perception data and radar perception data in the current cache data are obtained, otherwise continue to wait. The timers of all MEC servers are time-aligned timers after synchronous setting, and the image perception data and radar perception data are image perception results and radar perception results after data structure unification processing, and the image perception data includes the latest frame data of each camera sensor, and the radar perception data includes the latest frame data of each radar sensor.

[0129] The frame fusion module 220 is used to fuse the image perception data and the radar perception data to generate a radar-visual fusion frame. The radar-visual fusion frame includes a fusion frame number, a fusion timestamp, and a fusion object array. The fusion object array includes a fusion object and an attribute value of the fusion object. The attribute value of the fusion object is obtained by fusion calculation based on the image perception data and the radar perception data and according to the radar projection algorithm and the optimal optimization algorithm.

[0130] The tracking and credible interval calculation module 230 is used to continuously track the target based on the radar-visual fusion frame and the historical fusion data, obtain the tracking result, update the historical fusion data according to the radar-visual fusion frame and the tracking result, output the radar-visual fusion frame and the tracking result, and calculate the complete credible interval and basic credible interval of the MEC server based on the projection result of the radar projection algorithm. The tracking result includes the target ID and the target history record.

[0131] In some embodiments, the fused object array also includes the object type, object position, object speed, object heading angle and target ID of the fused object, and the tracking and credible interval calculation module 230 includes a target continuous tracking unit 231 and a credible interval calculation unit 232 .

[0132] The target continuous tracking unit 231 is used to track and match the fused object in the radar-vision fusion frame with the historical fused object in the historical fusion data according to the object type, object position, object speed and object heading angle. If the fused object in the radar-vision fusion frame successfully matches one of the historical fusion objects in the historical fusion data, the fusion object array of the historical fusion object is updated according to the radar-vision fusion frame. If the fused object in the radar-vision fusion frame fails to match all the historical fusion objects in the historical fusion data, a new target ID is generated for the fusion object and the fusion object is added to the historical fusion data.

[0133] The credible interval calculation unit 232 is used to calculate the fully credible value of each historical fusion object in the updated historical fusion data based on the projection result of the radar projection algorithm. If the historical fusion object is generated by fusion when the radar perception object projection is successful, the fully credible value of the historical fusion object is 1. If the historical fusion object is generated by fusion when the radar perception object projection fails, the fully credible value of the historical fusion object is 0. It is used to determine the fully credible interval of the MEC server based on all historical fusion objects with a fully credible value of 1, and to determine the basic credible interval of the MEC server based on all historical fusion objects with a fully credible value of 0.

[0134] It should be noted that the specific implementation process of each module in the roadside fusion tracking system based on radar frame fusion can refer to the corresponding description in the aforementioned embodiment of the roadside fusion tracking method based on radar frame fusion. For the sake of brief description, it will not be repeated here.

[0135] In addition, the aforementioned roadside fusion tracking system based on radar-visual frame fusion can also be implemented in the form of a computer program, which can be run on a computer device. The embodiment of the present application also provides a computer device, which can be an independent server or a server cluster composed of multiple servers. Specifically, Figure 3As shown, the computer device 300 includes a processor 310, a memory, and a network interface 340 connected via a system bus 350, the processor 310 is coupled to the memory, wherein the memory may include a non-volatile storage medium 330 and an internal memory 320, further, the non-volatile storage medium 330 may store an operating system 331 and a computer program 332. The computer program 332 includes program instructions, and when the program instructions are executed, the processor 310 may execute the roadside fusion tracking method based on radar-visual frame fusion in the aforementioned embodiment.

[0136] It should be noted that the processor 310 is used to provide computing and control capabilities to support the operation of the entire computer device 300. The internal memory 320 provides an environment for the operation of the computer program 332 in the non-volatile storage medium 330. The network interface 340 is used to communicate with other devices over the network. Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 300 to which the solution of the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In addition, the processor 310 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0138] It can be understood by a person skilled in the art that all or part of the processes in the method of the above embodiment can be implemented by instructing the relevant hardware through a computer program product. The computer program product includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above-mentioned roadside fusion tracking method based on radar-visual frame fusion.

[0139] Therefore, another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the aforementioned embodiment of the roadside fusion tracking method based on radar-visual frame fusion is implemented.

[0140] Another embodiment of the present application provides a computer program product, which includes program instructions. When the program instructions are executed on a computer or a processor, the computer or the processor executes the method described in the above-mentioned embodiment of the roadside fusion tracking method based on radar-visual frame fusion.

[0141] Those skilled in the art will understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. In addition, the modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be changed accordingly and located in one or more devices different from the present embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple sub-modules.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A roadside fusion tracking method based on radar-visual frame fusion, characterized in that: include: When a certain MEC server triggers a fusion instruction, it is determined whether the timer of the MEC server reaches the trigger time. If so, the image perception data and radar perception data in the current cache data are obtained, otherwise, the waiting continues; wherein, the timers of all the MEC servers are time-aligned timers after synchronous setting, the image perception data and the radar perception data are image perception results and radar perception results after data structure unification processing, and the image perception data includes the latest frame data of each camera sensor, and the radar perception data includes the latest frame data of each radar sensor; Perform frame fusion on the image perception data and the radar perception data to generate a radar-visual fusion frame; wherein the radar-visual fusion frame includes a fusion frame number, a fusion timestamp, and a fusion object array, the fusion object array includes a fusion object and an attribute value of the fusion object, and the attribute value of the fusion object is obtained by fusion calculation based on the image perception data and the radar perception data and according to a radar projection algorithm and an optimal optimization algorithm; Continuously track the target based on the radar-visual fusion frame and the historical fusion data to obtain the tracking result, and update the historical fusion data according to the radar-visual fusion frame and the tracking result, output the radar frame-visual fusion and the tracking result, and calculate the complete trust interval and basic trust interval of the MEC server based on the projection result of the radar projection algorithm; wherein the tracking result includes the target ID and the target history record.

2. The roadside fusion tracking method based on radar-visual frame fusion according to claim 1 is characterized in that: The image perception result includes camera target type, target box pixel coordinate information, camera target position, camera target speed and camera target heading angle, and the radar perception result includes radar target position coordinates, radar target speed and radar target heading angle; wherein, when the image perception result and the radar perception result are processed in a unified data structure, the target box pixel coordinate information and the radar target position coordinates are converted into the same coordinate system.

3. The roadside fusion tracking method based on radar-visual frame fusion according to claim 2 is characterized in that: The radar projection algorithm comprises the following steps: The upper left point coordinates and lower right point coordinates of the camera box corresponding to a camera target are calculated according to the target box pixel coordinate information of a camera target. The calculation formula is: Among them, x c.b.lefttop The horizontal coordinate of the upper left point of the camera box, y c.b.lefttop The ordinate of the upper left point of the camera box; x c.p.downcenter is the horizontal coordinate of the center point of the lower edge of the camera target, y c.p.downcenter The ordinate of the center point of the lower edge of the camera target; x c.b.rightdown The horizontal coordinate of the lower right point of the camera box, y c.b.rightdown L is the vertical coordinate of the lower right point of the camera box; b.size H is the length of the camera box; b.size The height of the camera box; Perform radar projection based on the coordinates of the lower left point and the upper right point of the camera box and the coordinates of the radar target position to determine whether the radar sensing object of the radar sensor exists in the camera box; If the horizontal coordinate of the radar target position coordinate is not less than the horizontal coordinate of the upper left point of a camera box, and is not greater than the horizontal coordinate of the lower right point of the camera box, and the vertical coordinate of the radar target position coordinate is not greater than the vertical coordinate of the upper left point of the camera box, and is not less than the vertical coordinate of the lower right point of the camera box, then it is determined that the radar perception object exists in the camera box and the radar projection is successful, otherwise it is determined that the radar projection fails; When the radar projection is successful, determining the number of camera perception objects corresponding to the radar perception objects in the camera box; If the number of the camera-perceived objects is 1, the radar-perceived object of the radar sensor is matched with the camera-perceived object in the camera box to generate a new fusion object; If the number of the camera-perceived objects is greater than 1, the camera-perceived object with the highest matching degree with the radar-perceived object is determined based on the camera target type, the target box pixel coordinate information, the camera target speed and the camera target heading angle of each camera-perceived object in the camera box, and based on the optimal optimization algorithm, and the camera-perceived object is matched with the radar-perceived object of the radar sensor to generate a new fusion object; If the number of the camera-perceived objects is less than 1, the camera-perceived object with the highest matching degree with the radar-perceived object is determined based on the camera target type, the target box pixel coordinate information, the camera target speed and the camera target heading angle of each camera-perceived object in other camera boxes that have not been successfully projected, and based on the optimal optimization algorithm, and the camera-perceived object is matched with the radar-perceived object of the radar sensor to generate a new fusion object; When radar projection fails, the discrete radar-visual fusion frames are re-fused based on the optimal optimization algorithm to obtain re-fused radar-visual fusion frames.

4. The roadside fusion tracking method based on radar-visual frame fusion according to claim 3 is characterized in that: Calculating the matching degree between the camera perception object and the radar perception object based on the optimal optimization algorithm specifically includes: According to the camera target type of the camera perception object, the target box pixel coordinate information, the camera target speed and the camera target heading angle, the matching degree between the camera perception object and the radar perception object is calculated, and the calculation formula is: M c =ω c.t Z c.t +ω c.center Z c.center +ω c.v Z c.v +ω c.h Z c.h Among them, M c Indicates the matching degree between the camera perception object and the radar perception object; Z c.t ω represents the attribute value of the camera target type; c.t Represents the weight of the attribute value of the camera target type; Z c.center Represents the attribute value of the camera target center point, Z c.center =1 means the distance difference between the camera target center point and the radar target position is not greater than the distance difference threshold, Z c.center =0 means that the distance difference between the camera target center point and the radar target position is greater than the distance difference threshold; ω c.center Represents the weight of the attribute value of the camera target center point; Z c.v The attribute value representing the camera's target speed, Z c .v=1 means the difference between the camera target speed and the radar target speed is not greater than the speed difference threshold, Z c .v=0 means that the difference between the camera target speed and the radar target speed is greater than the speed difference threshold; ω c.v The weight of the attribute value representing the camera target speed; Z c.h Indicates the attribute value of the camera target heading angle, Z c .h=1 means the difference between the camera target heading angle and the radar target heading angle is not greater than the heading angle difference threshold, Z c .h=0 means that the difference between the camera target heading angle and the radar target heading angle is greater than the heading angle difference threshold; ω c.h The weight of the attribute value representing the camera's target heading angle.

5. The roadside fusion tracking method based on radar-visual frame fusion according to claim 3 is characterized in that: Calculating the attribute value of the fusion object according to the optimal optimization algorithm specifically includes: The attribute value of the fused object is calculated according to the radar target position coordinates, the radar target speed, the radar target heading angle of the radar perception object and the camera target position, the camera target speed, and the camera target heading angle of the camera perception object matched therewith, and the calculation formula is: A f.value =ω f.p Z f.p +ω f.v Z f.v +ω f.h Z f.h Among them, A f.value Represents the attribute value of the fused object; Z f.p Indicates the attribute value of the fused target position. Zf.p=1 means that the radar perceives the object projection successfully. f .p=0 means radar fails to perceive the object projection; ω f.p Represents the weight of the attribute value of the fusion target position; Z f.v Indicates the attribute value of the fusion target speed, Z f .v=1 means the difference between the camera target speed and the radar target speed is not greater than the speed difference threshold, Z f .v=0 means that the difference between the camera target speed and the radar target speed is greater than the speed difference threshold; ω f.v Indicates the weight of the attribute value of the fusion target speed; Z f .h represents the attribute value of the fusion target heading angle, Z f .h=1 means the difference between the camera target heading angle and the radar target heading angle is not greater than the heading angle difference threshold, Z f .h=0 means that the difference between the camera target heading angle and the radar target heading angle is greater than the heading angle difference threshold; ω f.h Indicates the weight of the attribute value of the fused target heading angle.

6. The roadside fusion tracking method based on radar-visual frame fusion according to claim 1 is characterized in that: The fusion object array also includes the object type, object position, object speed, object heading angle and target ID of the fusion object. The target is continuously tracked based on the radar-visual fusion frame and the historical fusion data to obtain the tracking result, and the historical fusion data is updated according to the radar-visual fusion frame and the tracking result, specifically including: Tracking and matching the fused object in the radar-vision fusion frame with the historical fusion object in the historical fusion data according to the object type, the object position, the object speed, and the object heading angle; If the fusion object in the thunder-vision fusion frame successfully matches one of the historical fusion objects in the historical fusion data, updating the fusion object array of the historical fusion object according to the thunder-vision fusion frame; If the fused object in the radar-vision fusion frame fails to match all the historical fused objects in the historical fusion data, a new target ID is generated for the fused object, and the fused object is added to the historical fusion data.

7. The roadside fusion tracking method based on radar-visual frame fusion according to claim 1 is characterized in that: The complete trust interval and basic trust interval of the MEC server are calculated based on the projection result of the radar projection algorithm, specifically including: Based on the projection result of the radar projection algorithm, a completely credible value of each historical fusion object in the updated historical fusion data is calculated, if the historical fusion object is generated by fusion when the radar perception object projection is successful, then the completely credible value of the historical fusion object is 1, if the historical fusion object is generated by fusion when the radar perception object projection fails, then the completely credible value of the historical fusion object is 0; According to the historical fusion objects whose fully credible values ​​are all 1, the fully credible interval of the MEC server is determined, and according to the historical fusion objects whose fully credible values ​​are all 0, the basic credible interval of the MEC server is determined.

8. The roadside fusion tracking method based on radar-visual frame fusion according to claim 7 is characterized in that: The completely credible interval is a set of points whose completely credible values ​​are 1, and the basic credible interval is a set of points whose completely credible values ​​are 0. When determining the completely credible interval and the basic credible interval, the following steps are included: Let S be any subset of Euclidean space. The smallest convex set of S is called the convex hull of S, denoted by conv(S). Let P = {P1, P2, ..., P n }=conv(S), which is solved as follows: (1) Select the point H with the largest x coordinate from all the points, use it as the base point, and add the base points H to the set P in sequence; if there are multiple points with the largest x coordinates, select the point with the largest y coordinate; exclude points with the same coordinates; (2) The remaining points P i The vector formed by the base point H <H,P i >, calculate the angle between them and the x-axis, sort them, select the point with the largest angle, record it as P1, and put P1 into the set P in sequence; if there are multiple points with the largest angle, select the point with the largest y coordinate; (3) Select the latest two points from the set P to form a vector <P n-1 , P n >, from the remaining points P i and the latest point P of set P n The vector composed <P i , P n >, calculate its difference with the vector <P n-1 , P n >, and sort them, select the point with the largest angle, and record it as P i , put P i Put them into the set P; if there are multiple points with the largest angle, select the point with the largest y coordinate; (4) Repeat step (3) until the set P is traversed and the selected P i Let be the vertices of the convex hull of set P; (5) The polygon formed by all convex hull vertices in set P is determined as the trusted interval of the MEC server, and the area of ​​the polygon in set P is calculated to obtain the coverage area of ​​the trusted interval of the MEC server.

9. A roadside fusion tracking system based on radar-visual frame fusion, characterized in that: include: A fusion trigger module is used to determine whether the timer of a MEC server reaches the trigger time when a fusion instruction is triggered by a certain MEC server. If so, the image perception data and radar perception data in the current cache data are obtained, otherwise, the waiting continues; wherein the timers of all the MEC servers are time-aligned timers after synchronous setting, the image perception data and the radar perception data are image perception results and radar perception results after data structure unification processing, and the image perception data includes the latest frame data of each camera sensor, and the radar perception data includes the latest frame data of each radar sensor; A frame fusion module, used for performing frame fusion on the image perception data and the radar perception data to generate a radar-visual fusion frame; wherein the radar-visual fusion frame includes a fusion frame number, a fusion timestamp, and a fusion object array, the fusion object array includes a fusion object and an attribute value of the fusion object, and the attribute value of the fusion object is obtained by fusion calculation based on the image perception data and the radar perception data and according to a radar projection algorithm and an optimal optimization algorithm; A tracking and credible interval calculation module is used to continuously track the target based on the radar-visual fusion frame and the historical fusion data, obtain the tracking result, update the historical fusion data according to the radar-visual fusion frame and the tracking result, output the radar frame-visual fusion and the tracking result, and calculate the complete credible interval and basic credible interval of the MEC server based on the projection result of the radar projection algorithm; wherein the tracking result includes the target ID and the target history record.

10. The roadside fusion tracking system based on radar-visual frame fusion according to claim 9, characterized in that: The fused object array also includes the object type, object position, object speed, object heading angle and target ID of the fused object. The tracking and trustworthy interval calculation module includes: a target continuous tracking unit, for tracking and matching the fused object in the radar vision fusion frame with the historical fused objects in the historical fusion data according to the object type, the object position, the object speed and the object heading angle; if the fused object in the radar vision fusion frame successfully matches one of the historical fused objects in the historical fusion data, then updating the data of the fused object array of the historical fusion object according to the radar vision fusion frame; if the fused object in the radar vision fusion frame fails to match all the historical fused objects in the historical fusion data, then generating a new target ID for the fused object and adding the fused object to the historical fusion data; A credible interval calculation unit is used to calculate the fully credible value of each historical fusion object in the updated historical fusion data based on the projection result of the radar projection algorithm, if the historical fusion object is generated by fusion when the radar perception object projection is successful, then the fully credible value of the historical fusion object is 1, if the historical fusion object is generated by fusion when the radar perception object projection fails, then the fully credible value of the historical fusion object is 0, and is used to determine the fully credible interval of the MEC server according to all the historical fusion objects with a fully credible value of 1, and determine the basic credible interval of the MEC server according to all the historical fusion objects with a fully credible value of 0.

Images