Curve early warning method, system and terminal based on target detection
Visual twin images are generated through multi-sensor fusion technology, which reflects the motion state of moving targets and obstacles in the curved area in real time, and issues warning information when potential collision risks are detected, solving the problems of low detection accuracy of existing curved warning systems and inability to intuitively reflect the real-time motion state, achieving more efficient traffic safety monitoring.
Patent Information
- Application Number
- CN202510319360.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-18
- Publication Date
- 2025-06-03
AI Technical Summary
The existing curve warning system has low detection accuracy and poor environmental adaptability. The traditional warning method cannot intuitively reflect the real-time movement status of moving targets and obstacles, making it difficult for drivers to make correct judgments quickly.
Multi-sensor fusion technology is used to generate visual twin images of target curves, reflecting the motion state of moving targets and obstacles in real time, and sending out early warning messages when potential collision risks are detected.
Through multi-sensor fusion technology, the accuracy and robustness of target detection are improved, the real-time trajectory prediction capability is enhanced, and the dynamic early warning mechanism ensures that the driver can respond in a timely manner, effectively reducing the incidence of traffic accidents in curved areas.
Smart Images

Figure CN120089020A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent transportation systems, and particularly relates to a curve warning method, system and terminal based on object detection. Background Art
[0002] Curve areas are high-incidence areas of traffic accidents. Especially in situations where the line of sight is restricted or the road conditions are complex, it is difficult for drivers to timely detect obstacles or oncoming vehicles ahead, resulting in an increased risk of collision. Existing curve warning systems mostly rely on a single sensor (such as a camera or radar) for object detection, and there are problems such as low detection accuracy and poor environmental adaptability. In addition, traditional warning methods usually only prompt through simple warning information and cannot intuitively reflect the real-time motion states of moving objects and obstacles, making it difficult for drivers to quickly make correct judgments.
[0003] Therefore, the present invention provides a curve warning method, system and terminal based on object detection to solve the above problems. Summary of the Invention
[0004] In view of the above situation, to overcome the defects of the prior art, the present invention provides a curve warning method, system and terminal based on object detection. By using multi-sensor fusion technology, a visual twin image of the target curve is generated to reflect the real-time motion states of moving objects and obstacles, and a warning message is issued when a potential collision risk is detected, thereby effectively reducing the incidence of traffic accidents in curve areas.
[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:
[0006] In a first aspect, a curve warning method based on object detection, the method includes:
[0007] Generating a visual twin image of the target curve, the visual twin image reflecting the real-time motion state of a moving object relative to the road in the target curve section, and the visual twin image including a road part, lane lines, the real-time position of an obstacle, and the estimated moving trajectory of the obstacle;
[0008] Displaying the visual twin image on display devices on both sides of the target curve;
[0009] If the motion trajectory of the moving object overlaps with the position of the obstacle, or the moving trajectory of the moving object overlaps with the estimated moving trajectory of the obstacle, a warning message is issued, the warning message including a sound prompt message and an image prompt message, and the graphic prompt message including: rendering the visual twin image corresponding to the moving object on the display device on one side of the road in the same direction as the moving direction of the moving object.
[0010] A further improvement of the present application is that generating the visual twin image of the target curve includes: acquiring the raw environmental data collected by the heterogeneous sensor arrays between any two adjacent display devices on the same side of the road within the target curve section, where the heterogeneous sensor arrays include binocular cameras, millimeter-wave radars, and lidar; preprocessing the raw environmental data to obtain the preprocessed environmental data; identifying the road profile, lane lines, obstacles, and their position information based on the preprocessed environmental data; calculating the estimated movement trajectory of the obstacles based on the trajectory prediction algorithm; and fusing the road profile, lane lines, obstacle positions, and estimated movement trajectory into a visual twin image.
[0011] A further improvement of the present application is that preprocessing the raw environmental data to obtain the preprocessed environmental data includes: performing time synchronization and spatial alignment on the raw environmental data. Among them, all sensors are hardware-level time synchronized through the GNSS second pulse signal. The polar coordinate data of the millimeter-wave radar and the three-dimensional point cloud of the lidar are uniformly mapped to the global coordinate system with the curve curvature center as the origin by using the Lie group SE(3) transformation matrix. The inverse perspective transformation is performed on the images collected by the binocular cameras to eliminate the image distortion caused by the curve curvature and generate a bird's-eye view perspective; denoising and enhancing the raw environmental data. The statistical outlier removal algorithm is used for the lidar point cloud to remove the noise points generated by rain, fog, or dust; Doppler filtering is performed on the millimeter-wave radar data to suppress the clutter interference of stationary objects on both sides of the road; and the CLAHE algorithm is used for the binocular images to enhance the texture details of the lane lines.
[0012] A further improvement of the present application is that identifying the road profile, lane lines, obstacles, and their position information based on the preprocessed environmental data includes: using the bird's-eye view generated by the binocular cameras, and extracting the road profile and lane lines through a semantic segmentation model based on U-Net, where the input of the model is the undistorted bird's-eye view image and the output is the pixel-level road mask and lane line coordinates; performing a segmentation algorithm based on Euclidean clustering on the lidar point cloud data, combining the point cloud reflection intensity features to distinguish the obstacle categories, and calculating the three-dimensional bounding box and centroid coordinates of the obstacles in the global coordinate system; fusing the radial velocity and azimuth angle information of the millimeter-wave radar, and dynamically correcting the position and motion state of the obstacles through the extended Kalman filter to generate the real-time position information of the obstacles; storing the road profile, lane line coordinates, and obstacle position information in the JSON structured data format and interacting with the rendering engine of the visual twin image in real time.
[0013] A further improvement of the present application is that performing a segmentation algorithm based on Euclidean clustering on the lidar point cloud data, combining the point cloud reflection intensity features to distinguish the obstacle categories, and the determined obstacle categories include motor vehicles, non-motor vehicles, and non-mobile obstacles.
[0014] A further improvement of the present application lies in that calculating the predicted movement trajectory of the obstacle based on the trajectory prediction algorithm includes: if the obstacle is a motor vehicle, constructing a motion constraint equation based on the road curvature of the target curve and the vehicle kinematic model, fusing the radial velocity of the millimeter-wave radar and the pose information of the lidar point cloud through an extended Kalman filter, predicting the future trajectory of the vehicle in the global coordinate system, and using an LSTM network to model the long-term temporal dependence relationship of driving behavior to correct the trajectory prediction result; if the obstacle is a non-motor vehicle, modeling its interaction relationship with the surrounding environment and moving targets through a social perception multi-layer graph network, combining the probability distribution of pedestrian crossing hotspots, generating multi-modal trajectory prediction results, and using Monte Carlo sampling to screen out the top K candidate trajectories with the highest confidence; if the obstacle is a non-moving obstacle, detecting and confirming its stationary state through the temporal difference of the lidar point cloud, directly outputting its current position as a fixed trajectory point, and judging whether it is a temporarily stationary obstacle according to historical data; mapping the predicted trajectories of all the above obstacles to the spatio-temporal grid map in the global coordinate system, and synchronizing the prediction results to the visual twin image; updating the trajectory prediction results every preset time window, and matching the obstacle IDs of the front and rear frames to ensure the temporal consistency of the trajectories.
[0015] A further improvement of the present application lies in that fusing the road contour, lane lines, obstacle positions, and predicted movement trajectories into a visual twin image includes: establishing a virtual curve three-dimensional scene based on the global coordinate system, mapping the road contour and lane line coordinates into continuous spline curves with curvature constraints, and rendering them as semi-transparent blue ribbon layers; converting the obstacle position information into dynamic markers in the three-dimensional scene, using different geometric icons to distinguish obstacle categories, and updating their positions in real time through the centroid coordinates; discretizing the predicted movement trajectory into a trajectory point set according to the time series, generating a smooth trajectory line through Brazier curve fitting, and superimposing it in front of the movement direction of the corresponding obstacle, and the color of the trajectory line gradually changes from the first color to the second color with the prediction time window; introducing a lighting rendering model, dynamically adjusting the brightness and flicker frequency of the trajectory line according to the relative speed between the obstacle and the moving target, and the greater the speed difference, the higher the flicker frequency; projecting the fused three-dimensional scene onto a two-dimensional plane in real time through a lightweight rendering engine to generate a visual twin image matching the resolution of the display device, and ensuring the temporal consistency of the image frames and sensor data through a timestamp synchronization mechanism.
[0016] In a second aspect, a curve warning system based on object detection, the system includes:
[0017] An image processing module, configured to generate a visual twin image of the target curve, the visual twin image reflecting the real-time motion state of the moving target relative to the road in the target curve section, and the visual twin image including a road part, lane lines, real-time positions of obstacles, and predicted movement trajectories of obstacles.
[0018] The image processing module is further configured to display the visual twin image on display devices on both sides of the target curve;
[0019] The logic storage and control module determines whether to issue a warning message according to the data sent by the image processing module and feeds back the determination result to the image processing module. The conditions for determining whether to issue a warning message are as follows: if the movement trajectory of the moving target overlaps with the position of the obstacle, or the moving trajectory of the moving target overlaps with the predicted moving trajectory of the obstacle, a warning message is issued. The warning message includes a sound prompt message and an image prompt message. The graphic prompt message includes rendering the visual twin image corresponding to the moving target on the display device on one side of the road that is consistent with the moving direction of the moving target.
[0020] In a third aspect, a curve warning terminal based on target detection, the terminal includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above.
[0021] The present invention generates a visual twin image of a target curve through multi-sensor fusion technology, can reflect the motion states of moving targets and obstacles in real time and accurately, and prompts potential dangerous situations for drivers through dynamic rendering and warning mechanisms. Compared with the prior art, the present invention has the following advantages:
[0022] 1. Multi-sensor fusion, through a heterogeneous sensor array of binocular cameras, millimeter-wave radars and lidar, realizes the omnidirectional perception of the road environment, and improves the accuracy and robustness of target detection.
[0023] 2. Real-time trajectory prediction, based on the trajectory prediction algorithm, can accurately predict the future moving trajectory of obstacles, and further improve the reliability of the prediction results by combining driving behavior modeling and social perception technology.
[0024] 3. Intuitive visual twin image, through three-dimensional scene rendering and dynamic marking technology, the generated visual twin image can intuitively display the road contour, lane lines, obstacle positions and their predicted moving trajectories, helping drivers quickly understand the current road conditions.
[0025] 4. Dynamic warning mechanism, according to the relative motion states of moving targets and obstacles, dynamically adjusts the display mode of warning messages to ensure that drivers can make correct responses in time. Description of the Drawings
[0026] Figure 1Schematic flowchart of a curve warning method based on object detection according to the present invention;
[0027] Figure 2 Schematic structural diagram of a curve warning system based on object detection according to the present invention;
[0028] Figure 3 Schematic structural diagram of a curve warning terminal based on object detection according to the present invention;
[0029] Figure 4 Application diagram in an embodiment of a curve warning method based on object detection according to the present invention. Detailed implementation manners
[0030] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Those skilled in the art should understand that these implementation manners are only used to explain the technical principle of the present invention and are not intended to limit the protection scope of the present invention.
[0031] Curve areas are high-incidence areas of traffic accidents. Especially in situations where the line of sight is restricted or the road conditions are complex, it is difficult for drivers to timely detect obstacles or oncoming vehicles ahead, resulting in an increased risk of collision. Existing curve warning systems mostly rely on a single sensor (such as a camera or radar) for object detection, having problems such as low detection accuracy and poor environmental adaptability. In addition, traditional warning methods usually only provide simple warning information, unable to intuitively reflect the real-time motion states of moving objects and obstacles, making it difficult for drivers to quickly make correct judgments.
[0032] To address the above problems, the present application provides a curve warning method based on object detection. The method includes: generating a visual twin image of a target curve, the visual twin image reflecting the real-time motion state of a moving object relative to the road within the target curve section, where the visual twin image includes a road part, lane lines, the real-time position of an obstacle, and the estimated moving trajectory of the obstacle; displaying the visual twin image on display devices on both sides of the target curve; if the moving trajectory of the moving object overlaps with the position of the obstacle, or the moving trajectory of the moving object overlaps with the estimated moving trajectory of the obstacle, then issuing a warning message, the warning message including a sound prompt message and an image prompt message, and the graphic prompt message including: rendering the visual twin image corresponding to the moving object on the display device on the road side consistent with the moving direction of the moving object.
[0033] By using multi-sensor fusion technology to generate a visual twin image of a target curve, reflecting the motion states of moving objects and obstacles in real time, and issuing a warning message when detecting a potential collision risk, thereby effectively reducing the incidence of traffic accidents in curve areas.
[0034] Next, the technical solution will be described in detail in combination with specific embodiments.
[0035] Embodiment
[0036] Reference Figure 1 , a curve warning method based on object detection, the method includes steps S10 - S30:
[0037] S10. Generate a visual twin image of the target curve, the visual twin image reflects the real-time motion state of the moving target relative to the road (or ground) within the target curve section, and the visual twin image includes the road part, lane lines, real-time positions of obstacles, and the estimated moving trajectories of obstacles;
[0038] S20. Display the visual twin image on display devices (such as LED screens) on both sides of the target curve;
[0039] S30. If the motion trajectory of the moving target overlaps with the position of the obstacle, or the moving trajectory of the moving target overlaps with the estimated moving trajectory of the obstacle, then send a warning message, the warning message includes a sound prompt message (such as continuous beeping) and an image prompt message, and the graphic prompt message includes: rendering the visual twin image corresponding to the moving target on the display device on the side of the road consistent with the moving direction of the moving target.
[0040] Optionally, in an embodiment of the present application, in step S10, generating a visual twin image of the target curve includes steps S100 - S400:
[0041] S100. Obtain the original environmental data collected by the heterogeneous sensor array between any two adjacent display devices on the same side of the road within the target curve section, the heterogeneous sensor array includes a binocular camera, a millimeter-wave radar, and a lidar, and preprocess the original environmental data to obtain preprocessed environmental data;
[0042] S200. Identify the road contour, lane lines, obstacles, and their position information based on the preprocessed environmental data;
[0043] S300. Calculate the estimated moving trajectory of the obstacle based on the trajectory prediction algorithm;
[0044] S400. Integrate the road contour, lane lines, obstacle positions, and estimated moving trajectories into a visual twin image.
[0045] Optionally, in an embodiment of the present application, in step S100, during actual use, the set point of the display device is specifically set according to the specific conditions of the target curve. The binocular camera is deployed at the top of the display device at an inclined angle, the millimeter-wave radar is embedded in the road guardrail in a horizontal scanning mode, and the lidar is installed on a column at the center of the curve curvature.
[0046] Specifically, in step S100, it involves preprocessing the original environmental data to obtain preprocessed environmental data, including steps S101 - S102:
[0047] S101. Perform time synchronization and spatial alignment on the original environmental data. Among them, hardware-level time synchronization of all sensors (binocular camera, millimeter-wave radar, and lidar) is performed through the GNSS second pulse signal, and the error is controlled within ±1ms. The Lie group SE(3) transformation matrix is used to uniformly map the polar coordinate data of the millimeter-wave radar, the three-dimensional point cloud of the lidar, and the image of the binocular camera to the global coordinate system with the center of the curve curvature as the origin. Inverse perspective transformation is performed on the image collected by the binocular camera to eliminate the image distortion caused by the curve curvature and generate a bird's-eye view perspective;
[0048] Among them, time synchronization of the original environmental data is through the PPS (second pulse) signal output by the GNSS receiver, which triggers all sensors to collect data under the same time reference and determines the trigger time. The actual acquisition timestamp of the sensor is equal to the determined trigger time plus the sensor inherent delay calibration value (obtained through calibration).
[0049] Spatial alignment of the original environmental data is achieved through the Lie group SE(3) transformation. Specifically, the data of the millimeter-wave radar (polar coordinates), lidar (3D point cloud), and binocular camera (image) are uniformly mapped to the global coordinate system with the center of the curve curvature as the origin. The specific expression is:
[0050]
[0051] In the above expression, R represents a 3×3 rotation matrix (obtained by calibrating the sensor installation attitude), t represents a 3×1 translation vector (the position of the sensor relative to the origin of the global coordinate system), and the function of this expression is to transform the points in the local coordinate system of the sensor to the global coordinate system;
[0052] For example, assume that the lidar detects that the coordinates of a certain point in its own coordinate system are (2, 1, 0.5), and its calibrated SE(3) transformation matrix is:
[0053]
[0054] Its coordinates in the global coordinate system are:
[0055] p global = T·[2, 1, 0.5, 1] T = [10 - 1, 5 + 2, 0.5, 1] T = [9, 7, 0.5] T
[0056] S102. Denoise and enhance the original environmental data. For the lidar point cloud, use the statistical outlier removal algorithm to remove the noise points generated by rain, fog, or dust; for the millimeter-wave radar data, perform Doppler filtering to suppress the clutter interference of stationary objects on both sides of the road (such as guardrails, utility poles, tall trees, buildings, etc.); for the binocular images, use the CLAHE algorithm to enhance the lane line texture details.
[0057] Among them, filtering the clutter of stationary objects using the Doppler effect is determined by the following expression:
[0058]
[0059] Among them, f represents the Doppler frequency of the echo signal, f ego represents the Doppler frequency shift caused by the movement of the vehicle itself, f threshold represents the threshold (set according to the minimum detection speed).
[0060] Specifically, using the CLAHE algorithm to enhance the lane line texture details is achieved by: dividing the image into N×N blocks (such as 8×8), independently calculating the histogram for each block, and restricting the gray level distribution of the histogram not to exceed the threshold T (such as T = 0.01×total number of pixels). After mapping the cumulative distribution function (CDF), the results of each block are merged through bilinear interpolation. Among them, for each image block, the mapping function of histogram equalization is:
[0061]
[0062] Among them, I(x, y) represents the gray value of the original image pixel, L: the number of gray levels (such as 256), CDF min represents the minimum value of the cumulative distribution function.
[0063] Optionally, in an embodiment of the present application, in step S200, identifying the road contour, lane lines, obstacles, and their position information based on the preprocessed environmental data includes steps S201 - S204:
[0064] S201. Use the bird's-eye view generated by the binocular camera, and extract the road contour and lane lines through a semantic segmentation model based on U-Net. The input of the model is the undistorted bird's-eye view image (size H×W×3), and the output is the pixel-level road mask (H×W×1) and lane line coordinates;
[0065] Among them, U-Net is a convolutional neural network with an encoder-decoder structure, designed specifically for pixel-level semantic segmentation. Its core is to fuse shallow details and deep semantic features through skip connections, and it is applicable to the segmentation of roads and lane lines in bird's-eye view images of curves. Among them, the loss function is obtained through supervised training using cross-entropy loss, and its expression is:
[0066]
[0067] where N represents the total number of pixels, C represents the number of categories (such as roads, lane lines, background), y i,c represents the true class label (one-hot encoded) of the i-th pixel, and p i,c represents the probability that the model predicts the i-th pixel belongs to category c.
[0068] For example, assuming the resolution of the input bird's-eye view image is 512×512, the probability that a certain pixel in the road mask predicted by the model is "road" is 0.95, and the true label is 1, then its cross-entropy loss is:
[0069]
[0070] The average value of the losses of all pixels is the loss of the current training batch.
[0071] S202. Implement a segmentation algorithm based on Euclidean clustering on the lidar point cloud data, distinguish obstacle categories by combining the point cloud reflection intensity characteristics, and calculate the three-dimensional bounding box and centroid coordinates of the obstacles in the global coordinate system;
[0072] In actual operation, the determined obstacle categories are divided according to their moving states, specifically including motor vehicles, non-motor vehicles (including non-motor vehicles, pedestrians, etc.) and non-moving obstacles (such as vehicles parked within the lane lines, or rocks sliding in mountainous areas, etc.). The identified obstacles are as Figure 4 shown.
[0073] Specifically, Euclidean clustering is based on the spatial distance of the point cloud for segmentation, and combines the reflection intensity to distinguish obstacle categories (such as metal vehicles have high reflection intensity, pedestrians / vegetation have low reflection intensity). Among them, the Euclidean clustering process is as follows:
[0074] Traverse the point cloud, for each unlabeled point p i , search for neighboring points within the search radius r (such as r = 0.5m); if the number of neighboring points ≥ N min (such as 5), then form a new cluster and iterate to expand until no new points can be merged.
[0075] The process of classifying reflection intensity is as follows:
[0076] Calculate the mean value μ of the reflection intensity of the point cloud within the cluster I , and its expression is as follows:
[0077]
[0078] Set the threshold T of the motor vehicle based on the reflection intensity of the point cloud vehicle = 50, and the threshold T of the non-motor vehicle based on the reflection intensity of the point cloud pedestrian = 20, then the classification is:
[0079]
[0080] For example, assume that a certain cluster contains 100 points and the sum of the reflection intensity values is 6500, then:
[0081]
[0082] It is determined as a motor vehicle, and its centroid coordinates are calculated ((x, y, z) = (10.2, -3.5, 0.1) in the global coordinate system).
[0083] S203. Integrate the radial velocity and azimuth angle information of the millimeter-wave radar, and dynamically correct the position and motion state of the obstacle through the extended Kalman filter to generate the real-time position information of the obstacle;
[0084] Among them, the extended Kalman filter integrates the radial velocity of the millimeter-wave radar and the pose information of the lidar to correct the motion state (position, velocity) of the obstacle in real time. For example:
[0085] Assume that the state of an obstacle at the previous moment is x k-1 = [10, 5, 2, 1] T (position (10, 5), velocity (2, 1) m / s), and the millimeter-wave radar observes z k = [12.2m, 30°, 2.5m / s].
[0086] Prediction step: The predicted position by the uniform motion model is (10 + 2Δt, 5 + 1Δt) (assuming Δt = 0.1s, then the predicted position is (10.2, 5.1))).
[0087] Convert the radar observation value to Cartesian coordinates:
[0088] x = r cosθ = 12.2cos30° ≈ 10.56m
[0089] y = r sinθ = 12.2sin30° = 6.1m
[0090] (Radial velocity),
[0091] By weighting the prediction and the observation value with the Kalman gain, the corrected obstacle state (such as the position (10.5, 5.3)) is output, reducing the measurement errors of the lidar and the millimeter-wave radar.
[0092] S204. Store the road profile, lane line coordinates, and obstacle position information in the JSON structured data format and interact with the rendering engine of the visual twin image in real time to ensure data consistency and low latency.
[0093] Optionally, in an embodiment of the present application, in step S300, calculating the predicted movement trajectory of the obstacle based on the trajectory prediction algorithm includes steps S301 - S303:
[0094] S301. Predict the predicted movement trajectories of obstacles of different categories:
[0095] If the obstacle is a motor vehicle, a motion constraint equation is constructed based on the road curvature of the target curve and the vehicle kinematic model. The radial velocity of the millimeter-wave radar and the pose information of the lidar point cloud are fused through the extended Kalman filter to predict the future trajectory of the vehicle in the global coordinate system, and the long-term time series dependence of the driving behavior is modeled using the LSTM network to correct the trajectory prediction result;
[0096] Among them, modeling the long-term time series dependence of the driving behavior using the LSTM network to correct the trajectory prediction result includes the following methods:
[0097] Network input: historical trajectory sequence [x t-T ,..., x t and context features (such as lane departure and relative position of surrounding vehicles);
[0098] Network output: future N-step trajectory offset Δx t+1 ,..., Δx t+N ;
[0099] Loss function: mean square error (MSE) jointly optimizes position and speed prediction, and the expression is:
[0100]
[0101] λ represents the speed weight coefficient (such as 0.5).
[0102] For example: Assume the current state of the vehicle is (x = 10m, y = 5m, v = 5m / s, ψ (heading angle) = 30°), and the LSTM predicts the lateral offset Δy = [0.2, 0.5, 0.7, 0.8, 0.9, 1.0]m every 0.5 seconds within the next 3 seconds. Combine with the kinematic model to generate a trajectory and correct the pure physical prediction of the extended Kalman filter.
[0103] If the obstacle is a non-motor vehicle, model its interaction with the surrounding environment and moving targets through a social perception multi-layer graph network. Combine with the probability distribution of pedestrian crossing hotspots to generate multi-modal trajectory prediction results, and use Monte Carlo sampling to select the top K candidate trajectories with the highest confidence;
[0104] Among them, the social perception multi-layer graph network modeling includes the following steps:
[0105] Define the graph structure, including nodes: obstacles (pedestrians, bicycles, etc.) and key environmental elements (zebra crossings, intersections); and edges: interaction relationships (such as following, avoidance, conflict, etc.), and the weights are determined by the relative speed and distance.
[0106] Determine the node features, expressed as: h i = [x i , y i , v i , θ i , type i T , type i represents the one-hot encoded type (pedestrian / bicycle).
[0107] Message passing, expressed as: m ij = φ(h i , h j , ||p i - p j ||), where φ represents the MLP encoding of the interaction intensity.
[0108] Multi-modal trajectory generation and Monte Carlo sampling include the following steps:
[0109] Probability distribution modeling, outputting M candidate trajectories and their probabilities
[0110] Monte Carlo sampling, sampling K = 100 trajectories from the distribution and retaining the top 5 (K = 5) with the highest confidence.
[0111] Illustrative example: A pedestrian may walk towards a zebra crossing (trajectory A, probability 0.6) or bypass an obstacle (trajectory B, probability 0.3). After Monte Carlo sampling, select trajectories A and B as the prediction results.
[0112] If the obstacle is a non - moving obstacle, its stationary state is confirmed through the time - series difference detection of the lidar point cloud, and its current position is directly output as a fixed trajectory point. Then, it is determined whether it is a temporarily stationary obstacle based on historical data. For example, if the obstacle remains stationary for more than 10 minutes, it is marked as a stationary obstacle; otherwise, it is a temporary obstacle.
[0113] S302. Uniformly map the predicted trajectories of all the above - mentioned obstacles into the spatio - temporal grid map in the global coordinate system, and synchronize the prediction results to the visual twin image.
[0114] Among them, the specific method of the spatio - temporal grid map is as follows:
[0115] Define the rasterization rule, discretize the global coordinate system into spatio - temporal units of 0.5m×0.5m×0.1s, and record the probability that each unit is occupied by the trajectory.
[0116] Collision detection: If a certain grid is covered by the trajectories of multiple obstacles, a collision warning is triggered.
[0117] For example, when the predicted trajectory of motor vehicle A covers the grid (10, 5, 1.0 - 1.5s) at t = 1s, if the trajectory of vehicle B appears in the same grid, the system marks a conflict.
[0118] S303. Update the trajectory prediction results every preset time window, and match the obstacle IDs of the front and rear frames to ensure the temporal consistency of the trajectories.
[0119] Specifically, the Hungarian algorithm is used to associate the obstacle detection results of adjacent frames to ensure the invariance of the ID.
[0120] Optionally, in an embodiment of the present application, in step S400, when fusing the road profile, lane lines, obstacle positions, and estimated movement trajectories into the visual twin image, it includes steps S401 - S405:
[0121] Based on the global coordinate system, establish a virtual three - dimensional curved - road scene, map the road profile and lane - line coordinates into a continuous spline curve with curvature constraints, and render it as a semi - transparent blue ribbon - like layer.
[0122] Convert the obstacle position information into dynamic markers in the three - dimensional scene, use different geometric icons to distinguish the obstacle categories, and update their positions in real time through the centroid coordinates.
[0123] Discretize the estimated movement trajectory into a set of trajectory points according to the time series, generate a smooth trajectory line through Bezier curve fitting, and superimpose it in front of the movement direction of the corresponding obstacle. The color of the trajectory line gradually changes from the first color (green) to the second color (red) with the prediction time window.
[0124] Introduce a lighting rendering model, and dynamically adjust the brightness and flicker frequency of the trajectory line according to the relative speed between the obstacle and the moving target. The greater the speed difference, the higher the flicker frequency.
[0125] Project the fused three-dimensional scene onto a two-dimensional plane in real time through a lightweight rendering engine to generate a visual twin image that matches the resolution of the display device, and ensure the temporal consistency between the image frames and the sensor data through a timestamp synchronization mechanism.
[0126] The present invention generates a visual twin image of the target curve through multi-sensor fusion technology, can reflect the motion states of the moving target and the obstacle in real time and accurately, and prompts the driver of potential dangerous situations through a dynamic rendering and warning mechanism. Compared with the prior art, the present invention has the following advantages:
[0127] 1. Multi-sensor fusion: Through a heterogeneous sensor array of binocular cameras, millimeter-wave radars, and lidar, it realizes an all-round perception of the road environment, and improves the accuracy and robustness of target detection.
[0128] 2. Real-time trajectory prediction: Based on the trajectory prediction algorithm, it can accurately predict the future moving trajectory of the obstacle, and further improve the reliability of the prediction results by combining driving behavior modeling and social perception technology.
[0129] 3. Intuitive visual twin image: Through three-dimensional scene rendering and dynamic marking technology, the generated visual twin image can intuitively display the road contour, lane lines, obstacle positions, and their estimated moving trajectories, helping the driver quickly understand the current road conditions.
[0130] 4. Dynamic warning mechanism: According to the relative motion state between the moving target and the obstacle, dynamically adjust the display mode of the warning information to ensure that the driver can make a correct response in time.
[0131] Reference Figure 2 , a curve warning system based on target detection, the system at least includes:
[0132] An image processing module, used to generate a visual twin image of the target curve, the visual twin image reflects the real-time motion state of the moving target relative to the road in the target curve section, and the visual twin image includes a road part, lane lines, real-time positions of obstacles, and estimated moving trajectories of obstacles;
[0133] The image processing module is also used to display the visual twin image on a display device on both sides of the target curve;
[0134] The logical storage and control module determines whether to issue a warning message based on the data sent by the image processing module and feeds back the determination result to the image processing module. The conditions for determining whether to issue a warning message are as follows: If the movement trajectory of the moving target overlaps with the position of the obstacle, or the moving trajectory of the moving target overlaps with the estimated moving trajectory of the obstacle, a warning message is issued. The warning message includes a sound prompt message and an image prompt message. The graphic prompt message includes rendering the visual twin image corresponding to the moving target on the display device on one side of the road in the same direction as the moving direction of the moving target.
[0135] Among them, the image processing module can use the NVIDIA Jetson AGX Xavier module to provide an accelerated computing power of 32 TOPS and support the input of raw data from binocular cameras, lidar, and millimeter-wave radars. Preferably, the binocular camera: 1920×1080@30FPS; the lidar: 32 lines, 10Hz; the millimeter-wave radar: 77GHz, 20Hz.
[0136] The logical storage and control module can adopt an ARM Cortex-A72 quad-core module with a processing delay ≤50ms and equipped with CAN bus, Ethernet (TCP / IP), and Wi-Fi6 communications, with a main frequency of 2.0GHz.
[0137] Reference Figure 3 , a curve warning terminal based on target detection, the terminal includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above.
[0138] The various embodiments of the systems and technologies described above in this article can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs, the one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a dedicated or general programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.
[0139] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.
[0140] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0141] In order to provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0142] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.
[0143] A computer system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating blockchain.
[0144] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0145] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.
Claims
1. A curve warning method based on target detection, characterized in that: The method comprises: Generate a visual twin image of the target curve, wherein the visual twin image reflects the real-time motion state of the moving target relative to the road within the target curve interval, and the visual twin image includes the road portion, lane lines, real-time position of obstacles, and estimated movement trajectory of obstacles; Displaying the visual twin image on display devices on both sides of the target curve; If the motion trajectory of the moving target overlaps with the position of the obstacle, or the motion trajectory of the moving target overlaps with the estimated motion trajectory of the obstacle, a warning message is issued, and the warning message includes sound prompt information and image prompt information. The graphic prompt information includes: rendering the visual twin image corresponding to the moving target on a display device on the side of the road consistent with the moving direction of the moving target.
2. The curve warning method based on target detection according to claim 1, characterized in that: The generating of the visual twin image of the target curve includes: Acquire raw environmental data collected by a heterogeneous sensor array between any two adjacent display devices located on the same side of the road in a target curve section, wherein the heterogeneous sensor array includes a binocular camera, a millimeter-wave radar, and a laser radar, and pre-process the raw environmental data to obtain pre-processed environmental data; Identify road contours, lane lines, obstacles and their location information based on preprocessed environmental data; Calculate the estimated moving trajectory of the obstacle based on the trajectory prediction algorithm; Fusion of road contours, lane lines, obstacle locations, and estimated movement trajectories into visual twin images.
3. The curve warning method based on target detection according to claim 2 is characterized in that: The preprocessing of the original environmental data to obtain the preprocessed environmental data includes: The original environmental data is synchronized in time and space. All sensors are synchronized at the hardware level through the GNSS second pulse signal. The polar coordinate data of the millimeter wave radar and the three-dimensional point cloud of the lidar are uniformly mapped to the global coordinate system with the center of the curve curvature as the origin using the Lie group SE (3) transformation matrix. The images collected by the binocular camera are transformed by inverse perspective to eliminate the image distortion caused by the curve curvature and generate a bird's-eye view. The original environmental data is denoised and enhanced, and a statistical outlier removal algorithm is used on the lidar point cloud to remove noise points caused by rain, fog or dust. Doppler filtering is performed on the millimeter-wave radar data to suppress clutter interference from stationary objects on both sides of the road. The CLAHE algorithm is used on the binocular image to enhance the lane line texture details.
4. The curve warning method based on target detection according to claim 3 is characterized in that: The identifying of road contours, lane lines, obstacles and their location information based on pre-processed environmental data includes: Using the bird's-eye view image generated by the binocular camera, the road contour and lane lines are extracted through the U-Net-based semantic segmentation model. The model input is the dedistorted bird's-eye view image, and the output is the pixel-level road mask and lane line coordinates. A segmentation algorithm based on Euclidean clustering is implemented on the lidar point cloud data, and obstacle categories are distinguished by combining the point cloud reflection intensity characteristics, and the three-dimensional bounding box and centroid coordinates of the obstacle in the global coordinate system are calculated; By integrating the radial velocity and azimuth information of the millimeter-wave radar, the position and motion state of the obstacle are dynamically corrected through the extended Kalman filter to generate the real-time position information of the obstacle; The road contour, lane line coordinates and obstacle location information are stored in JSON structured data format and interact with the rendering engine of the visual twin image in real time.
5. The method for curve warning based on target detection according to claim 4, characterized in that: The laser radar point cloud data is segmented using an Euclidean clustering-based segmentation algorithm, and obstacle categories are distinguished in combination with the point cloud reflection intensity characteristics. The determined obstacle categories include motor vehicles, non-motor vehicles, and non-mobile obstacles.
6. The method for curve warning based on target detection according to claim 4, characterized in that: The step of calculating the estimated moving trajectory of the obstacle based on the trajectory prediction algorithm includes: If the obstacle is a motor vehicle, the motion constraint equation is constructed based on the road curvature of the target curve and the vehicle kinematic model. The radial velocity of the millimeter-wave radar and the position information of the lidar point cloud are fused through the extended Kalman filter to predict the future trajectory of the vehicle in the global coordinate system. The LSTM network is used to model the long-term dependency of driving behavior and correct the trajectory prediction result. If the obstacle is a non-motorized vehicle, the interaction between it and the surrounding environment and mobile targets is modeled through a social-aware multi-layer graph network. Combined with the probability distribution of pedestrian crossing hotspots, a multimodal trajectory prediction result is generated, and Monte Carlo sampling is used to screen out the top K candidate trajectories with the highest confidence. If the obstacle is a non-moving obstacle, its stationary state is confirmed by the time difference detection of the lidar point cloud, its current position is directly output as a fixed trajectory point, and whether it is a temporary stationary obstacle is determined based on historical data; The predicted trajectories of all the above obstacles are uniformly mapped to the spatiotemporal grid map in the global coordinate system, and the prediction results are synchronized to the visual twin image; The trajectory prediction result is updated once every preset time window, and the obstacle IDs of the previous and next frames are matched to ensure the timing consistency of the trajectory.
7. The method for curve warning based on target detection according to claim 6, characterized in that: The fusion of road contours, lane lines, obstacle positions and estimated movement trajectories into visual twin images includes: A virtual curved road 3D scene is established based on the global coordinate system. The road contour and lane line coordinates are mapped into a continuous spline curve with curvature constraints and rendered as a semi-transparent blue ribbon layer. The obstacle location information is converted into dynamic markers in the 3D scene, different geometric icons are used to distinguish obstacle categories, and their positions are updated in real time through centroid coordinates; The estimated moving trajectory is discretized into a trajectory point set according to the time series, and a smooth trajectory line is generated by Brazier curve fitting. The smooth trajectory line is superimposed in front of the moving direction of the corresponding obstacle, and the color of the trajectory line gradually changes from the first color to the second color along the prediction time window; The lighting rendering model is introduced to dynamically adjust the brightness and flickering frequency of the trajectory line according to the relative speed between the obstacle and the moving target. The greater the speed difference, the higher the flickering frequency. The fused three-dimensional scene is projected onto a two-dimensional plane in real time through a lightweight rendering engine to generate a visual twin image that matches the resolution of the display device, and the timing consistency between the image frame and the sensor data is ensured through a timestamp synchronization mechanism.
8. A curve warning system based on target detection, characterized in that: The system comprises: An image processing module, used to generate a visual twin image of the target curve, wherein the visual twin image reflects the real-time motion state of the moving target relative to the road within the target curve interval, and the visual twin image includes the road portion, lane lines, real-time position of obstacles, and estimated movement trajectory of obstacles; The image processing module is further used to display the visual twin image on display devices on both sides of the target curve; The logic storage and control module determines whether to issue a warning message based on the data sent by the image processing module and feeds back the determination result to the image processing module. The conditions for determining whether to issue a warning message are as follows: if the motion trajectory of the moving target overlaps with the position of the obstacle, or the moving trajectory of the moving target overlaps with the estimated moving trajectory of the obstacle, a warning message is issued. The warning message includes sound prompt information and image prompt information. The graphic prompt information includes: rendering a visual twin image corresponding to the moving target on a display device on the side of the road consistent with the moving direction of the moving target.
9. A curve warning terminal based on target detection, characterized in that: The terminal comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
Citation Information
Cited By
Road dynamic obstacle millimeter wave-visual fusion perception method and system
CN120491049A
Millimeter wave-vision fusion perception method and system for dynamic road obstacles
CN120491049B
Industrial mobile robot path planning method based on artificial potential field method and dynamic prediction
CN120871887A
Artificial Potential Field Method and Dynamic Prediction for Path Planning of Industrial Mobile Robots
CN120871887B
Outdoor operation electric machine positioning system based on Beidou technology
CN121634174A
