A post-fusion method and device for roadside sensor

Abstract

The present invention relates to the field of vehicle-road collaboration technology, and specifically to a post-fusion method and device for roadside sensors, the method comprising: first acquiring the target detected by each roadside sensor, and performing coordinate conversion processing and time alignment processing on the target, then performing address matching and association processing on the target after the coordinate conversion processing and time alignment processing, so as to acquire and update the tracking information of the target; finally, determining the main target from the targets detected by each roadside sensor, and using the main target as a fusion reference to update the target information of other roadside sensors except the main target, so as to perform fusion processing on each roadside sensor. The present invention realizes effective fusion between multiple sensors, realizes fusion with sensors such as radar without dismantling the existing capture machine, significantly enhances the compatibility of the digital twin intersection system, and reduces the renovation cost of the intersection.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-road cooperative technology, and in particular to a post-fusion method and device for roadside sensors. Background Art

[0002] In recent years, with the rapid development of autonomous driving and vehicle-road collaborative technologies, roadside perception systems in urban traffic environments have also been significantly improved. Roadside perception systems are equipped with more types and higher-precision sensors, such as millimeter-wave radars, 3D lidars, and smart cameras.

[0003] The application of roadside sensors can perceive the surrounding environment more accurately, thus making it possible to realize digital twin technology. In the field of transportation, digital twin technology can be used to simulate real traffic scenarios, help users better optimize traffic management, and improve the safety and efficiency of autonomous vehicles. However, to realize digital twin technology, it is also necessary to rely on the target output of the integrated radar and vision machine. The integrated radar and vision machine is an advanced device that integrates radar and visual sensors. It can simultaneously obtain the radar echo information and image information of the target, and output the attributes and precise location coordinates of the target.

[0004] In the above scheme, although the integrated radar and vision device plays a key role in the digital twin technology, the use of the integrated radar and vision device to implement the digital twin technology may not be able to maximize the use of the original traffic monitoring and management equipment, so there may be certain difficulties in the transformation of existing intersections. Summary of the invention

[0005] In view of this, the present invention provides a post-fusion method and device for roadside sensors to solve the problem that the use of a radar-vision integrated device to implement digital twin technology may not be able to maximize the use of existing traffic monitoring and management equipment, and there may be certain difficulties in modifying existing intersections.

[0006] In a first aspect, the present invention provides a post-fusion method of a roadside sensor, the method comprising:

[0007] Acquire targets detected by various roadside sensors, and perform coordinate conversion and time alignment processing on the targets;

[0008] Performing address matching and association processing on the target after the coordinate conversion processing and the time alignment processing to obtain and update the tracking information of the target;

[0009] A main target is determined from the targets detected by the various roadside sensors, and the target information of other roadside sensors except the main target is updated by taking the main target as a fusion reference, so as to perform fusion processing on the various roadside sensors.

[0010] In an optional implementation, before acquiring the targets detected by each roadside sensor and performing coordinate conversion processing and time alignment processing on the targets, the method further includes:

[0011] Determine the camera to be calibrated in each roadside sensor, and obtain the camera trajectory set point set output by the camera to be calibrated;

[0012] Based on the camera trajectory point set, generating a camera trajectory probability map;

[0013] According to the distribution mutual information of the camera trajectory probability map, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

[0014] In an optional implementation, the acquiring the camera trajectory point set output by the camera to be calibrated includes:

[0015] Obtaining a two-dimensional pixel coordinate trajectory output by the camera to be calibrated within a collection period;

[0016] Based on the two-dimensional pixel coordinate trajectory, a camera trajectory point set of the camera to be calibrated is generated.

[0017] In an optional implementation, generating a camera trajectory probability map based on the camera trajectory point set includes:

[0018] Based on the camera intrinsic parameter mapping of the camera to be calibrated, mapping the camera trajectory set point set to a radar coordinate system, and obtaining a mapping result;

[0019] A mixed Gaussian model is established for the camera trajectory point set, and a camera trajectory probability map is generated.

[0020] In an optional implementation, the calibrating and optimizing the extrinsic parameters of the camera to be calibrated according to the distribution mutual information of the camera trajectory probability map includes:

[0021] According to the numerical range of the camera trajectory probability map, establishing the marginal distribution and the joint distribution of the camera trajectory probability map;

[0022] Based on the marginal distribution and the joint distribution and the mapping result, calculating the distribution mutual information of the camera trajectory probability map;

[0023] Based on the distribution mutual information of the camera trajectory probability map and the mapping result, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

[0024] In an optional implementation, the coordinate conversion processing and time alignment processing of the target includes:

[0025] The target is converted into a unified coordinate system, and time alignment processing is performed on the target to adjust the time stamp of the target detected by each roadside sensor to the target time point.

[0026] In an optional implementation, performing address matching and association processing on the detection target after the coordinate conversion processing and the time alignment processing to obtain and update the tracking information of the detection target includes:

[0027] Performing address matching and association processing on the detection target and the tracking target after the coordinate conversion processing and the time alignment processing, and obtaining a matching relationship between the detection target and the tracking target;

[0028] Acquire and update the tracking information of the detection target according to the matching relationship between the detection target and the tracking target;

[0029] For the detection targets for which the address matching and association processing are unsuccessful, a new tracking target is regenerated:

[0030] The tracking targets are matched in pairs to optimize the matching of the address matching and association processing results.

[0031] In an optional implementation manner, the obtaining and updating the tracking information of the target further includes:

[0032] Determining the heading angle of the detected target according to the lateral and longitudinal velocities of the detected target;

[0033] Acquiring a motion state of the detection target according to the heading angle;

[0034] According to the motion state, the tracking information of the detection target is updated.

[0035] In an optional implementation manner, determining the main target from the targets detected by the roadside sensors includes:

[0036] In a case where each of the roadside sensors includes a camera and a plurality of radars, taking a target detected by the one camera as a main target;

[0037] In the case where each of the roadside sensors includes a plurality of cameras and a plurality of radars, a target detected by a camera having the largest number of detection times among the plurality of cameras is taken as a main target.

[0038] In a second aspect, the present invention provides a post-fusion device for a roadside sensor, the device comprising:

[0039] A coordinate conversion and time alignment module, used to obtain targets detected by various roadside sensors, and perform coordinate conversion and time alignment processing on the targets;

[0040] A tracking information updating module, used for performing address matching and association processing on the target after the coordinate conversion processing and the time alignment processing, so as to obtain and update the tracking information of the target;

[0041] The fusion processing module is used to determine a main target from the targets detected by the various roadside sensors, and use the main target as a fusion reference to update the target information of other roadside sensors except the main target, so as to perform fusion processing on the various roadside sensors.

[0042] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes a post-fusion method of a roadside sensor according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute a post-fusion method of a roadside sensor according to the first aspect or any corresponding embodiment thereof.

[0044] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute a post-fusion method of a roadside sensor according to the first aspect or any corresponding embodiment thereof.

[0045] The technical solution provided by the present invention may include the following beneficial effects:

[0046] The present invention first obtains the target detected by each roadside sensor, and performs coordinate conversion and time alignment processing on the target, and then performs address matching and association processing on the target after the coordinate conversion and time alignment processing to obtain and update the tracking information of the target; finally, the main target is determined from the targets detected by each roadside sensor, and the main target is used as a fusion reference to update the target information of other roadside sensors except the main target, so as to perform fusion processing on each roadside sensor. The present invention realizes effective fusion between multiple sensors through main target selection, address matching and association processing, ensures the consistency and accuracy of different sensor data, improves the stability and reliability of target tracking and identification, realizes fusion with radar and other sensors without dismantling the existing capture machine, and significantly enhances the compatibility of the digital twin intersection system, which can maximize the use of existing resources, reduce the renovation cost of the intersection, and reduce resource waste and cost expenditure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a schematic flow chart of a post-fusion method of a roadside sensor according to an embodiment of the present invention;

[0049] Figure 2 is a flow chart of another post-fusion method of a roadside sensor according to an embodiment of the present invention;

[0050] Figure 3 is a structural block diagram of a post-fusion device of a roadside sensor according to an embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] According to an embodiment of the present invention, an embodiment of a post-fusion method of a roadside sensor is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] In this embodiment, a post-fusion method of a roadside sensor is provided. Figure 1 is a flow chart of a post-fusion method of a roadside sensor according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0055] Step S101, obtaining the targets detected by each roadside sensor, and performing coordinate conversion and time alignment processing on the targets. The targets include detection targets and tracking targets.

[0056] Furthermore, firstly, this embodiment obtains the target information detected by each roadside sensor (such as a camera, radar, etc.), that is, the detected target, and the target information includes the position, speed, size, etc. of the target. Secondly, since different roadside sensors may use different coordinate systems (such as a camera coordinate system, a radar coordinate system, a WGS-84 coordinate system, etc.), it is necessary to convert the coordinates of all detected targets into a unified coordinate system for subsequent processing. In addition, since the sampling frequency and response time of different roadside sensors may be different, it is necessary to align the timestamps of the detected targets to ensure that data processing is performed at the same time point.

[0057] The coordinate transformation processing of this embodiment ensures the spatial consistency of data from different roadside sensors, providing a basis for subsequent data fusion; the time alignment processing eliminates the impact of time differences on data processing, thereby improving the accuracy and real-time performance of data processing.

[0058] Step S102: performing address matching and association processing on the target after the coordinate conversion processing and the time alignment processing, so as to obtain and update the tracking information of the target.

[0059] Furthermore, after coordinate conversion and time alignment, this embodiment performs ID matching (i.e., address matching) on ​​the detection target and the tracking target after time alignment, and updates the information of the tracking target if the IDs are the same; and then performs GNN association on the detection target and the tracking target to achieve the optimal matching relationship.

[0060] The address matching and association processing of this embodiment realizes the deep fusion of multi-sensor data, improves the integrity and accuracy of target information, and updates tracking information to help better understand the movement state and trajectory of the target, providing a more reliable basis for traffic management and decision-making.

[0061] Step S103 , determining a main target from the targets detected by the various roadside sensors, and using the main target as a fusion reference to update target information of other roadside sensors except the main target, so as to perform fusion processing on the various roadside sensors.

[0062] Furthermore, among the detection targets of multiple roadside sensors, this embodiment determines a main target according to certain criteria (such as the number of detections, etc.), and uses the main target as a fusion reference to update the detection targets of other sensors, that is, the detection targets of other sensors are fused based on the main target.

[0063] This embodiment determines the main target and uses it as a reference for fusion processing, which ensures the consistency and reliability of the fusion result. The fusion processing eliminates the redundancy and conflict between multi-sensor data and improves the efficiency and accuracy of data processing.

[0064] In summary, the present invention first obtains the target detected by each roadside sensor, and performs coordinate conversion and time alignment processing on the target, and then performs address matching and association processing on the target after the coordinate conversion and time alignment processing to obtain and update the tracking information of the target; finally, the main target is determined from the targets detected by each roadside sensor, and the main target is used as a fusion reference to update the target information of other roadside sensors except the main target, so as to perform fusion processing on each roadside sensor. The present invention realizes effective fusion between multiple sensors through main target selection, address matching and association processing, ensures the consistency and accuracy of different sensor data, improves the stability and reliability of target tracking and identification, realizes fusion with radar and other sensors without dismantling the existing capture machine, and significantly enhances the compatibility of the digital twin intersection system, which can maximize the use of existing resources, reduce the renovation cost of the intersection, and reduce resource waste and cost expenditure.

[0065] In this embodiment, another post-fusion method of roadside sensors is provided. Figure 2 is a flow chart of another post-fusion method of a roadside sensor according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0066] Step S201 : determining the camera to be calibrated in each roadside sensor, and obtaining the camera trajectory point set output by the camera to be calibrated.

[0067] In an optional implementation, the step S201 of “obtaining a camera trajectory point set output by the camera to be calibrated” includes:

[0068] Obtaining the two-dimensional pixel coordinate trajectory output by the camera to be calibrated during the acquisition period;

[0069] Based on the two-dimensional pixel coordinate trajectory, a camera trajectory point set of the camera to be calibrated is generated.

[0070] Furthermore, post-fusion (also called result fusion) refers to the fusion of independent detection results from different roadside sensors (radar and camera in this case). Unlike pre-fusion (raw data fusion), post-fusion occurs in the later stage of data processing, that is, after each roadside sensor has completed the preliminary detection, identification and processing of the target.

[0071] This embodiment assumes that the installation position of the sensors to be calibrated in the roadside sensors is fixed, the internal parameters of the sensors to be calibrated are known, there are common detection areas between the sensors to be calibrated, and the common detection area accounts for >50% of the detection area union, and the common detection area of ​​the sensors to be calibrated is a relatively flat area, such as a road surface area, the radar calibration is completed, and the initial inaccurate external parameters are known. Therefore, before implementing post-fusion, this embodiment first performs offline calibration on the camera. When performing offline calibration, this embodiment first implements data collection of the sensors to be calibrated. The cameras to be calibrated collect data in the same time period (the same time period data is about 1 minute to 3 minutes). After the corresponding target detection and multi-target tracking algorithms, trajectory-level observation results are obtained. The camera to be calibrated outputs the two-dimensional pixel coordinate trajectory of the detected target in the image plane (this step does not require precise synchronization of the sensors, and about >80% of the material time period overlaps). Afterwards, this embodiment accumulates the target trajectory of the camera to be calibrated during the acquisition period to form a camera trajectory point set:

[0072] ;

[0073] in, represents the camera trajectory point set, represents the target trajectory point set of the camera to be calibrated, Indicates the trajectory The pixel coordinates of the points, Represents the union of the target trajectory point sets of each camera to be calibrated.

[0074] That is to say, when performing offline calibration, this embodiment first uses the camera to be calibrated to capture dynamic targets in the scene within a specific time period (about 1 minute to 3 minutes), and uses target detection and multi-target tracking algorithms to process these data to extract the target's motion trajectory. On the basis of target detection, the multi-target tracking algorithm further processes the target information in the continuous frames to identify and track the movement trajectory of each target in the image, and finally outputs the two-dimensional pixel coordinate trajectory of each target on the image plane. The two-dimensional pixel coordinate trajectory is a sequence of pixel points that changes with time and can reflect the movement of the target in the image. Afterwards, the trajectory of each target (i.e., the sequence of pixel coordinates that changes with time) is sorted, and these trajectories are merged into a large point set, where each point represents the pixel coordinates of a certain target at a certain moment. The camera trajectory set point set can be regarded as a point cloud in a two-dimensional space, which contains the motion information of all targets during the acquisition period.

[0075] Step S202: Generate a camera trajectory probability map based on the camera trajectory point set.

[0076] In an optional implementation, step S202 includes:

[0077] Based on the camera intrinsic parameter mapping of the camera to be calibrated, the camera trajectory set point set is mapped to the radar coordinate system, and the mapping result is obtained;

[0078] A mixed Gaussian model is established for the camera trajectory point set, and a camera trajectory probability map is generated.

[0079] Furthermore, this embodiment assumes that the intrinsic parameter mapping of the camera to be calibrated is known, denoted as , which represents the mapping function from the camera coordinate system to the radar coordinate system. Map the trajectory point set obtained in the previous step to the radar coordinate system:

[0080] ;

[0081] in, represents a point in the radar coordinate system, , Respectively represent the initial rotation matrix and translation vector between the camera system and the radar system, It means that the points in the camera trajectory set are projected to the radar points, and finally the camera trajectory points that cross the image pixel boundary are removed.

[0082] Afterwards, this embodiment targets the camera trajectory set point set , establish a mixed two-dimensional Gaussian model, and the mean of each Gaussian component corresponds to the camera trajectory set point set The coordinates of a point in the Gaussian component are set to , for dimensional unit matrix, is the standard deviation parameter, which can be set according to the density of the trajectory and the image resolution. The Gaussian component weight is uniformly distributed , then the Gaussian mixture distribution is:

[0083] ;

[0084] in, represents the Gaussian mixture distribution form, represents a random variable that conforms to the Gaussian distribution, V represents the Gaussian component covariance matrix, Indicates the trajectory points.

[0085] In this embodiment, based on the established mixed Gaussian model, the probability value of each pixel index on the pixel plane can be calculated, and the probability value is scaled and discretized. For example, for a camera, the probability value is:

[0086] ;

[0087] After traversing all pixel indices, the camera trajectory probability map can be generated Similarly, the trajectory probability map of the radar in the camera plane can be generated .

[0088] That is to say, after obtaining the camera trajectory point set of the camera to be calibrated, this embodiment uses the known camera intrinsic parameters and the extrinsic parameters between the camera and the radar (rotation matrix and translation vector) to project the trajectory point set in the camera coordinate system to the radar coordinate system, and removes points that exceed the effective observation range of the radar or the image pixel boundary to ensure that the projected point set is valid and interpretable. Afterwards, a mixed two-dimensional Gaussian model is established for the camera trajectory point set, and the mean of each Gaussian component corresponds to a point in the point set. According to the established mixed Gaussian model, all pixel points on the image (or radar plane) are traversed, the probability value of each pixel point is calculated, and the calculated probability value is scaled and discretized to generate a trajectory probability map. The trajectory probability map intuitively shows the probability distribution of the target motion trajectory, which can be used for further analysis and decision support.

[0089] Step S203: calibrate and optimize the extrinsic parameters of the camera to be calibrated according to the distribution mutual information of the camera trajectory probability map.

[0090] In an optional implementation, step S203 includes:

[0091] According to the numerical range of the camera trajectory probability map, the marginal distribution and the joint distribution of the camera trajectory probability map are established;

[0092] Based on the marginal distribution and the joint distribution and the mapping result, calculating the distribution mutual information of the camera trajectory probability map;

[0093] Based on the distribution mutual information of the camera trajectory probability map and the mapping result, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

[0094] Furthermore, after obtaining the camera trajectory probability map, it can be known that the numerical range of the trajectory probability map generated by the camera is , the numerical histogram can be directly re-stated based on the numerical range of the camera trajectory probability map, and then the histogram statistics can be normalized to obtain the probability map distribution as follows:

[0095] ;

[0096] in, represents the indicator function, represents the number of pixels, X represents the random variable of each pixel value in the trajectory probability map, and k represents the value range of the trajectory probability map. According to the above formula, the marginal distribution of the camera probability map can be obtained as follows:

[0097]

[0098] The probability graph random variable of the camera is and ,in, represents the probability distribution of a random variable X, Represents the probability distribution of the random variable Y of the probability graph. Similarly, the joint distribution form can be established as follows:

[0099] ;

[0100] in, and represents the values ​​that two different discrete random variables can take, represents two different random variables.

[0101] The distribution mutual information of the camera trajectory probability map can be calculated from the marginal distribution and the joint distribution through the following formula:

[0102] ;

[0103] in, represents the distribution mutual information of the camera trajectory probability map, represents the joint probability density function of (x, y), Represents the product of the probability density functions of the marginal distributions.

[0104] At the end of the offline camera calibration, this embodiment uses a numerical method to calculate the gradient of the mutual information with respect to the extrinsic parameters:

[0105] ;

[0106] ;

[0107] Among them, this embodiment defines an external parameter set , and uses gradient algorithm to optimize , is the learning rate, which can be adjusted dynamically and optimized to the parameter Convergence completes the calibration process. Represents the gradient of the mutual information with respect to the extrinsic parameter, which can be approximated by numerical methods.

[0108] That is to say, after obtaining the camera trajectory probability map, this embodiment first extracts numerical information from the camera trajectory probability map, performs histogram statistics on the extracted values, and then normalizes the histogram statistics results so that the sum of the statistical values ​​of all intervals is 1, thereby obtaining a probability distribution map, including marginal distribution (distribution of a single random variable) and joint distribution (joint distribution of multiple random variables). Afterwards, the marginal distribution and joint distribution are used to calculate the mutual information of the camera trajectory probability map distribution according to the definition formula of mutual information. Mutual information is a measure of the mutual dependence of two variables. In camera calibration, mutual information can be used to evaluate the correlation between camera parameters (such as external parameters) and observed data. The larger the mutual information, the stronger the correlation between the camera parameters and the observed data. Finally, this embodiment uses a numerical method to calculate the gradient of the mutual information relative to the external parameter parameters. The gradient indicates the direction of adjustment of the external parameter parameters to maximize the mutual information. The gradient ascent algorithm is used to optimize the external parameter parameters. In each iteration, the external parameter parameters are updated according to the gradient direction, and the learning rate is adjusted to control the update step size. When the update amount of the external parameter is less than a preset threshold or the change amount of the mutual information is less than a threshold, this embodiment considers that the algorithm has converged and the calibration process is completed.

[0109] Step S204: acquiring the target detected by each roadside sensor, and performing coordinate conversion and time alignment processing on the target.

[0110] In an optional implementation, the "performing coordinate conversion and time alignment processing on the target" in step S204 includes:

[0111] The target is converted into a unified coordinate system, and time alignment is performed on the target to adjust the time stamps of the targets detected by each roadside sensor to the target time point.

[0112] Furthermore, when performing coordinate conversion processing, this embodiment takes the first radar in the roadside sensor as the station center, and converts the detection targets of all sensors to the coordinate system of the station center, that is, converts the detection target WGS-84 coordinate system to the ENU coordinate system, and converts the WGS-84 coordinate system to the position of the ECEF coordinate system:

[0113] ;

[0114] Among them, lat is longitude, lon is latitude, alt is altitude, e is ellipsoid eccentricity, and N is the radius of curvature of the reference ellipsoid.

[0115] Convert the ECEF coordinate system to the ENU coordinate system:

[0116] ;

[0117] ;

[0118] in, , , Respectively represent the position of the target in the ECEF coordinate system, , Indicates the location of the origin (center of the station), and It is the longitude and latitude at 84 coordinates.

[0119] That is to say, the coordinate conversion processing of this embodiment includes two geographic coordinate system conversion processes, namely, conversion from the WGS-84 coordinate system to the ECEF (Earth Center Earth Fixed) coordinate system, and then conversion from the ECEF coordinate system to the ENU (East-North-Up) coordinate system with a specific point as the station center. Through two coordinate conversions, the conversion from the globally applicable WGS-84 coordinate system to the local ENU coordinate system with a specific point as the origin is realized, which can accurately describe the position of the target in the local area.

[0120] After the coordinate conversion process is performed, this embodiment uses a multi-term fitting method to obtain the velocity vector for the sensor target detection list according to the fusion trigger time CycleTime, and then implements time alignment. The function of time alignment is:

[0121] + ;

[0122] It is expressed in matrix as:

[0123] ;

[0124] The corresponding matrix expression is:

[0125] ;

[0126] in, represents the time alignment result, represents the coefficient of the fitted curve, represents the sample attributes, m represents the number of samples, and n represents the dimension.

[0127] The loss function of the time alignment function is expressed as:

[0128] ;

[0129] in, Indicates the loss result, represents the x attribute value of the jth sample (the independent variable of the fitting equation), Represents an observed quantity.

[0130] The derivative of the loss function is equal to 0 to obtain the coefficient matrix:

[0131] ;

[0132] That is to say, since the timestamp of each sensor's detection target is different, if multiple sensors are targeting one target, the timestamp of each sensor's detection target needs to be adjusted to a time point to determine whether the targets detected by different sensors are the same target, thereby achieving a fusion effect. The sensor target detection list is a cache list of tracking targets. Since the frequencies of different sensors are different, the targets detected by the sensors need to be cached, and the fusion operation is performed on the targets in the list at the triggering stage of fusion.

[0133] First, the sensor of this embodiment detects the target in a continuous time period and generates a target detection list. This list contains the position information of the target at each time point. Then, the position data of the target is fitted using a multi-term fitting method to estimate the velocity vector of the target at each time point through the fitted polynomial function. Since different sensors or different detection periods may produce data that are not aligned in time, time alignment is required. This embodiment ensures that all sensor data are consistent in time through time alignment processing, so that subsequent data analysis and processing are more convenient. In order to determine the optimal coefficient of the polynomial fitting, this embodiment defines a loss function (also called a cost function or an error function), which measures the difference between the fitting polynomial and the actual data. Once the coefficients of the fitting polynomial are obtained, the expression of the velocity vector can be obtained by differentiation. The velocity vector represents the moving direction and velocity of the target at each time point. That is, this embodiment obtains the velocity vector through the multi-term fitting method, and uses matrix expression to achieve time alignment, ensuring that the detection targets of different sensors are compared and fused at the same time point.

[0134] Step S205 , performing address matching and association processing on the target after the coordinate conversion processing and the time alignment processing, so as to obtain and update the tracking information of the target.

[0135] In an optional implementation, the step S205 includes:

[0136] Performing address matching and association processing on the detection target and the tracking target after the coordinate conversion processing and the time alignment processing, and obtaining a matching relationship between the detection target and the tracking target;

[0137] Acquire and update the tracking information of the detection target according to the matching relationship between the detection target and the tracking target;

[0138] For the detection target whose address matching and association processing are unsuccessful, a new tracking target is regenerated:

[0139] The tracking targets are matched in pairs to optimize the matching of the address matching and association processing results.

[0140] In an optional implementation, the "obtaining and updating the tracking information of the target" in step S205 further includes:

[0141] Determine the heading angle of the detected target according to the lateral and longitudinal speeds of the detected target;

[0142] According to the heading angle, obtaining the motion state of the detection target;

[0143] According to the motion state, the tracking information of the detected target is updated.

[0144] Furthermore, after the time alignment process, this embodiment performs ID matching on the time-aligned tracking list and detection list targets (the detection list is the input of the fusion, that is, the sensor detection target set; the tracking list refers to the tracking target set within the fusion, which is the target set that has been fused before). If the ID is the same, the tracking target information is updated, and then the detection target and the tracking target are associated using the GNN (Global Nearest Neighbour) method to achieve the best matching relationship. If they match, the tracking target information is updated. The expression formula for GNN association is as follows:

[0145] ;

[0146] in, is a binary variable, 0 means no association, 1 means association, Represents the distance between target i and target j.

[0147] For a moving target, this embodiment determines the heading angle of the target relative to the quadrant of the sensor based on the lateral and longitudinal speeds. The expression for obtaining the heading angle is as follows:

[0148] ;

[0149] in, ρ Indicates the quadrant.

[0150] The fusion algorithm uses the dual conditions of filtering smoothing and historical trajectory to determine the motion state of the target. If it is judged to be a static target, the target heading angle will remain consistent with the initial state or the previous frame state.

[0151] That is to say, after time alignment, the targets in the tracking list and the detection list are synchronized in time. At this time, this embodiment performs ID matching (i.e., the above-mentioned address matching) on ​​the targets in the tracking list and the detection list after time alignment. If there are targets with the same ID in the two lists, the latest detection information (such as position, speed, etc.) of the tracked target is directly updated. After ID matching, for unmatched targets (whether unmatched targets in the tracking list or newly appeared targets in the detection list), this embodiment uses the GNN method for association, and finds the optimal matching relationship based on the distance (or similarity) between possible target pairs. Once a matching relationship is found through the GNN method, the tracking information of the target, including position, speed, acceleration, etc., is updated to reflect the latest detection results.

[0152] When updating the tracking information of the target, this embodiment also determines the position of the target relative to the sensor (i.e., the quadrant in which it is located) for each moving target based on its horizontal and vertical speeds, and calculates the heading angle accordingly. The heading angle is the angle of the target's moving direction relative to a reference direction (such as the true north direction of the sensor). For static targets (i.e., those that are judged to be not moving or moving very slowly), their heading angles are locked in the initial state or the state of the previous frame.

[0153] Step S206 , determining a main target from the targets detected by the various roadside sensors, and using the main target as a fusion reference to update target information of other roadside sensors except the main target, so as to perform fusion processing on the various roadside sensors.

[0154] In an optional implementation, the “determining a main target from the targets detected by each roadside sensor” in step S206 includes:

[0155] In the case where each roadside sensor includes a camera and a plurality of radars, a target detected by the camera is taken as a main target;

[0156] In the case where each roadside sensor includes a plurality of cameras and a plurality of radars, a target detected by a camera having the largest number of detection times among the plurality of cameras is regarded as a main target.

[0157] Furthermore, in the single-camera multi-radar scenario, this embodiment selects the camera target as the main target. In the multi-camera multi-radar scenario, the main target is selected from the targets of multiple cameras, and the target with the most detection times is selected as the main target. If the detection times of multiple targets are the same, the target of the first camera is selected as the main target. For targets that are not matched, this embodiment regenerates new tracking targets, and finally matches the tracking targets pairwise to achieve the best fusion effect.

[0158] That is, in the single-camera multi-radar scenario, since there is only one camera, the target detected by the camera naturally becomes the main target, and the ID of the target detected by the camera is set to MainID, and the index number of the camera is set to MainDevice. In the multi-camera multi-radar scenario, first, count the number of times the target detected by each camera is detected by different sensors (including radar), and then select the target with the most detections as the main target. Second, if the number of detections of multiple targets is the same, select the target of the first camera as the main target according to the index order of the cameras (or predefined priority) to ensure the consistency and predictability of the selection process.

[0159] For those targets that have not been successfully matched with other sensor targets (whether detected by the camera or the radar), this embodiment regenerates new tracking targets for them. These newly generated tracking targets will be subsequently processed as independent tracking entities until they are successfully matched with targets detected by other sensors. This embodiment uses the main target as a fusion benchmark, matches all tracking targets in pairs, and updates the target information of other sensors based on the matching results of the tracking targets, so that the data of all roadside sensors are adjusted and corrected around the fusion benchmark, thereby achieving post-fusion of all roadside sensors. And for targets that have not been matched, this embodiment regenerates new tracking targets and matches them in pairs to ensure that all targets can be effectively tracked and fused.

[0160] In summary, the present invention first obtains the target detected by each roadside sensor, and performs coordinate conversion and time alignment processing on the target, and then performs address matching and association processing on the target after the coordinate conversion and time alignment processing to obtain and update the tracking information of the target; finally, the main target is determined from the targets detected by each roadside sensor, and the main target is used as a fusion reference to update the target information of other roadside sensors except the main target, so as to perform fusion processing on each roadside sensor. The present invention realizes effective fusion between multiple sensors through main target selection, address matching and association processing, ensures the consistency and accuracy of different sensor data, improves the stability and reliability of target tracking and identification, realizes fusion with radar and other sensors without dismantling the existing capture machine, and significantly enhances the compatibility of the digital twin intersection system, which can maximize the use of existing resources, reduce the renovation cost of the intersection, and reduce resource waste and cost expenditure.

[0161] In this embodiment, a post-fusion device of a roadside sensor is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0162] This embodiment provides a post-fusion device of a roadside sensor, such as Figure 3 As shown, including:

[0163] The coordinate conversion and time alignment module 301 is used to obtain the target detected by each roadside sensor and perform coordinate conversion and time alignment processing on the target;

[0164] The tracking information updating module 302 is used to perform address matching and association processing on the target after the coordinate conversion processing and the time alignment processing, so as to obtain and update the tracking information of the target;

[0165] The fusion processing module 303 is used to determine a main target from the targets detected by the various roadside sensors, and use the main target as a fusion reference to update target information of other roadside sensors except the main target, so as to perform fusion processing on the various roadside sensors.

[0166] In some optional embodiments, the device is also used for:

[0167] Determine the camera to be calibrated in each roadside sensor, and obtain the camera trajectory point set output by the camera to be calibrated;

[0168] Based on the camera trajectory point set, a camera trajectory probability map is generated;

[0169] According to the distribution mutual information of the camera trajectory probability map, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

[0170] In some optional embodiments, the device is also used for:

[0171] Obtaining the two-dimensional pixel coordinate trajectory output by the camera to be calibrated during the acquisition period;

[0172] Based on the two-dimensional pixel coordinate trajectory, a camera trajectory point set of the camera to be calibrated is generated.

[0173] In some optional embodiments, the device is also used for:

[0174] Based on the camera intrinsic parameter mapping of the camera to be calibrated, the camera trajectory set point set is mapped to the radar coordinate system, and the mapping result is obtained;

[0175] A mixed Gaussian model is established for the camera trajectory point set, and a camera trajectory probability map is generated.

[0176] In some optional embodiments, the device is also used for:

[0177] According to the numerical range of the camera trajectory probability map, the marginal distribution and the joint distribution of the camera trajectory probability map are established;

[0178] Based on the marginal distribution and the joint distribution and the mapping result, calculating the distribution mutual information of the camera trajectory probability map;

[0179] Based on the distribution mutual information of the camera trajectory probability map and the mapping result, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

[0180] In some optional implementations, the coordinate conversion and time alignment module 301 is further used to:

[0181] The target is converted into a unified coordinate system, and time alignment is performed on the target to adjust the time stamps of the targets detected by each roadside sensor to the target time point.

[0182] In some optional implementations, the tracking information updating module 302 is further used to:

[0183] Performing address matching and association processing on the detection target and the tracking target after the coordinate conversion processing and the time alignment processing, and obtaining a matching relationship between the detection target and the tracking target;

[0184] Acquire and update the tracking information of the detection target according to the matching relationship between the detection target and the tracking target;

[0185] For the detection target whose address matching and association processing are unsuccessful, a new tracking target is regenerated:

[0186] The tracking targets are matched in pairs to optimize the matching of the address matching and association processing results.

[0187] In some optional implementations, the coordinate conversion and time alignment module 301 is further used to:

[0188] Determine the heading angle of the detected target according to the lateral and longitudinal speeds of the detected target;

[0189] According to the heading angle, obtaining the motion state of the detection target;

[0190] According to the motion state, the tracking information of the detected target is updated.

[0191] In some optional implementations, the fusion processing module 303 is further used to:

[0192] The determining of a main target from the targets detected by the roadside sensors includes:

[0193] In the case where each roadside sensor includes a camera and a plurality of radars, a target detected by the camera is taken as a main target;

[0194] In the case where each roadside sensor includes a plurality of cameras and a plurality of radars, a target detected by a camera having the largest number of detection times among the plurality of cameras is regarded as a main target.

[0195] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0196] In this embodiment, a post-fusion device of a roadside sensor is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0197] In summary, the present invention first obtains the target detected by each roadside sensor, and performs coordinate conversion and time alignment processing on the target, and then performs address matching and association processing on the target after the coordinate conversion and time alignment processing to obtain and update the tracking information of the target; finally, the main target is determined from the targets detected by each roadside sensor, and the main target is used as a fusion reference to update the target information of other roadside sensors except the main target, so as to perform fusion processing on each roadside sensor. The present invention realizes effective fusion between multiple sensors through main target selection, address matching and association processing, ensures the consistency and accuracy of different sensor data, improves the stability and reliability of target tracking and identification, realizes fusion with radar and other sensors without dismantling the existing capture machine, and significantly enhances the compatibility of the digital twin intersection system, which can maximize the use of existing resources, reduce the renovation cost of the intersection, and reduce resource waste and cost expenditure.

[0198] The embodiment of the present invention also provides a computer device having the above Figure 3 A post-fusion device of a roadside sensor is shown.

[0199] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0200] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0201] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0202] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0204] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0205] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0206] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0207] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A post-fusion method for roadside sensors, characterized in that: The post-fusion is to fuse the independent detection results of different roadside sensors, and the method includes: Acquire targets detected by various roadside sensors, and perform coordinate conversion and time alignment processing on the targets; Performing address matching and association processing on the target after the coordinate conversion processing and the time alignment processing to obtain and update the tracking information of the target; Determine a main target from the targets detected by the various roadside sensors, and use the main target as a fusion reference to update target information of other roadside sensors except the main target, so as to perform fusion processing on the various roadside sensors; The determining of the main target from the targets detected by the roadside sensors includes: In a case where each of the roadside sensors includes a camera and a plurality of radars, taking a target detected by the one camera as a main target; In a case where each of the roadside sensors includes a plurality of cameras and a plurality of radars, taking a target detected by a camera having the largest number of detections among the plurality of cameras as a main target; The updating of target information of other roadside sensors except the main target by taking the main target as a fusion reference includes: The main target is used as a fusion benchmark, all tracked targets are matched in pairs, and target information of other roadside sensors except the main target is updated according to the matching results of the tracked targets, so that the data of all roadside sensors are adjusted and corrected based on the fusion benchmark.

2. The method according to claim 1, characterized in that Before acquiring the targets detected by each roadside sensor and performing coordinate conversion processing and time alignment processing on the targets, the method further includes: Determine the camera to be calibrated in each roadside sensor, and obtain the camera trajectory set point set output by the camera to be calibrated; Based on the camera trajectory point set, generating a camera trajectory probability map; According to the distribution mutual information of the camera trajectory probability map, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

3. The method according to claim 2, characterized in that The step of obtaining a camera trajectory point set output by the camera to be calibrated includes: Obtaining a two-dimensional pixel coordinate trajectory output by the camera to be calibrated within a collection period; Based on the two-dimensional pixel coordinate trajectory, a camera trajectory point set of the camera to be calibrated is generated.

4. The method according to claim 2, characterized in that: The step of generating a camera trajectory probability map based on the camera trajectory point set includes: Based on the camera intrinsic parameter mapping of the camera to be calibrated, mapping the camera trajectory set point set to a radar coordinate system, and obtaining a mapping result; A mixed Gaussian model is established for the camera trajectory point set, and a camera trajectory probability map is generated.

5. The method according to claim 4, characterized in that The step of calibrating and optimizing the extrinsic parameters of the camera to be calibrated according to the distribution mutual information of the camera trajectory probability map includes: According to the numerical range of the camera trajectory probability map, establishing the marginal distribution and the joint distribution of the camera trajectory probability map; Based on the marginal distribution and the joint distribution and the mapping result, calculating the distribution mutual information of the camera trajectory probability map; Based on the distribution mutual information of the camera trajectory probability map and the mapping result, the extrinsic parameters of the camera to be calibrated are calibrated and optimized.

6. The method according to claim 1, characterized in that The coordinate conversion processing and time alignment processing of the target include: The target is converted into a unified coordinate system, and time alignment processing is performed on the target to adjust the time stamp of the target detected by each roadside sensor to the target time point.

7. The method according to claim 1, characterized in that The address matching and association processing is performed on the target after the coordinate conversion processing and the time alignment processing to obtain and update the tracking information of the target, including: Performing address matching and association processing on the detection target and the tracking target after the coordinate conversion processing and the time alignment processing, and obtaining a matching relationship between the detection target and the tracking target; Acquire and update the tracking information of the detection target according to the matching relationship between the detection target and the tracking target; For the detection targets for which the address matching and association processing are unsuccessful, a new tracking target is regenerated: The tracking targets are matched in pairs to optimize the matching of the address matching and association processing results.

8. The method according to claim 7, characterized in that The obtaining and updating the tracking information of the target further includes: Determining the heading angle of the detected target according to the lateral and longitudinal velocities of the detected target; Acquiring a motion state of the detection target according to the heading angle; According to the motion state, the tracking information of the detection target is updated.

9. A post-fusion device for a roadside sensor, characterized in that: The post-fusion is to fuse the independent detection results of different roadside sensors, and the device includes: A coordinate conversion and time alignment module, used to obtain targets detected by various roadside sensors, and perform coordinate conversion and time alignment processing on the targets; A tracking information updating module, used for performing address matching and association processing on the target after the coordinate conversion processing and the time alignment processing, so as to obtain and update the tracking information of the target; A fusion processing module, used to determine a main target from the targets detected by the various roadside sensors, and use the main target as a fusion reference to update target information of other roadside sensors except the main target, so as to perform fusion processing on the various roadside sensors; The fusion processing module is further used for: In a case where each of the roadside sensors includes a camera and a plurality of radars, taking a target detected by the one camera as a main target; In a case where each of the roadside sensors includes a plurality of cameras and a plurality of radars, taking a target detected by a camera having the largest number of detections among the plurality of cameras as a main target; The main target is used as a fusion benchmark, all tracked targets are matched in pairs, and target information of other roadside sensors except the main target is updated according to the matching results of the tracked targets, so that the data of all roadside sensors are adjusted and corrected based on the fusion benchmark.

Images