360-degree panoramic look-around driving assistance system and method for AI intelligent early warning

Abstract

The invention relates to the technical field of AI intelligent early warning, in particular to an AI intelligent early warning 360-degree panoramic looking-around driving assistance system and method, and the system comprises a target detection and tracking module, a behavior prediction module, a trajectory mapping module and a collision risk assessment module. The target detection and tracking module, the behavior prediction module, the trajectory mapping module and the collision risk assessment module are connected in sequence; the target detection and tracking module is used for detecting and tracking the targets to obtain an independent historical track of each target; the behavior prediction module is used for predicting a future moving track of the target based on the independent historical track of each target to obtain a predicted track; therefore, analysis and prediction are performed after the target is detected, the follow-up collision risk is judged according to the predicted trajectory, and the information is displayed on the vehicle-mounted device in time or early warning is performed through the loudspeaker, so that a driver is assisted to effectively avoid the risk, and the driving safety is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AI intelligent early warning, and particularly relates to a 360 panoramic surround driving assistance system and method with AI intelligent early warning. Background Art

[0002] At present, with the rapid development of automotive intelligent technologies, the Advanced Driving Assistance System (ADAS) has played an important role in improving driving safety and convenience. Among them, the 360 panoramic surround system provides a panoramic image of the vehicle's surrounding environment for the driver through multi-camera and sensor fusion technology, effectively eliminating visual blind spots and being widely used in scenarios such as parking assistance and low-speed driving.

[0003] However, in the aforementioned prior art, the driving assistance system usually only has the function of photographing surrounding targets and then displaying them on the in-vehicle screen, but does not have the ability to predict the behaviors of surrounding targets (such as pedestrians and vehicles), and cannot accurately predict their future movement trajectories. Therefore, the driver cannot effectively avoid risks. Summary of the Invention

[0004] The purpose of the present invention is to provide a 360 panoramic surround driving assistance system and method with AI intelligent early warning, which solves the problem that in the prior art, the driving assistance system usually only has the function of photographing surrounding targets and then displaying them on the in-vehicle screen, but does not have the ability to predict the behaviors of surrounding targets, and cannot accurately predict their future movement trajectories. Therefore, the driver cannot effectively avoid risks.

[0005] To achieve the above purpose, the present invention provides a 360 panoramic surround driving assistance system with AI intelligent early warning, including a target detection and tracking module, a behavior prediction module, a trajectory mapping module, and a collision risk assessment module. The target detection and tracking module, the behavior prediction module, the trajectory mapping module, and the collision risk assessment module are connected in sequence;

[0006] The target detection and tracking module is used to detect and track targets to obtain the individual historical trajectories of each target;

[0007] The behavior prediction module is used to predict the future movement trajectories of targets based on the individual historical trajectories of each target to obtain predicted trajectories;

[0008] The trajectory mapping module is used to map the predicted trajectories into the vehicle's built-in 360 panoramic surround or in-vehicle screen, and mark the predicted trajectories on the screen with lines for users to observe;

[0009] The collision risk assessment module is used to calculate the predicted trajectories based on the state of its own vehicle, and then assess the collision risk and take corresponding early warning measures.

[0010] Among them, the target detection and tracking module includes a data acquisition unit, a target detection unit, and a target tracking unit, and the data acquisition unit, the target detection unit, and the target tracking unit are connected in sequence;

[0011] The data acquisition unit is used to rely on multiple cameras and radars to detect the image data and physical data of the target in real time, and obtain image information and radar information;

[0012] The target detection unit is used to analyze the image information and radar information using a deep learning model;

[0013] The target tracking unit uses a multi-target tracking algorithm to track the detected target and achieve continuous tracking of the target.

[0014] Among them, the target detection unit includes an image input subunit, an anchor box prediction subunit, and an overlap removal subunit, and the image input subunit, the anchor box prediction subunit, and the overlap removal subunit are connected in sequence;

[0015] The image input subunit is used to input the real-time image frames of multiple cameras into the YOLOv8 model, and the model extracts features through a convolutional neural network;

[0016] The anchor box prediction subunit is used to use the anchor box to predict the bounding box and category of the target;

[0017] The overlap removal subunit is used to remove overlapping bounding boxes through non-maximum suppression, retain the detection result with the highest confidence, and obtain a target list, where the target list is the bounding box, category, and confidence of each target.

[0018] Among them, the target tracking unit includes a target prediction subunit, a target matching subunit, and a re-matching subunit, and the target prediction subunit, the target matching subunit, and the re-matching subunit are connected in sequence;

[0019] The target prediction subunit is used to input the target list into the Kalman filter algorithm to predict the position of the target in the next frame and obtain the predicted position;

[0020] The target matching subunit is used to use the Hungarian algorithm to match the detection result of the current frame with the predicted position;

[0021] The re-matching subunit is used to initialize a new tracker for the unmatched target; for the lost target, try to re-match in subsequent frames, and finally obtain the unique ID of each target and its trajectory in consecutive frames, which is the historical trajectory.

[0022] Among them, the behavior prediction module includes a trajectory encoding unit, a social pooling unit, and a trajectory decoding unit, which are connected in sequence;

[0023] The trajectory encoding unit is used to encode the historical trajectory of the target using LSTM and extract time series features;

[0024] The social pooling unit is used to fuse the state information of surrounding targets into the features of the current target through a pooling layer based on the interaction relationship between multiple targets;

[0025] The trajectory decoding unit is used to predict the future trajectory of the target using another LSTM decoder and output a position sequence within the next five seconds, which is the predicted trajectory.

[0026] Among them, the collision risk assessment module includes a relative distance calculation unit, a relative speed calculation unit, a collision time calculation unit, and a risk classification unit, which are connected in sequence;

[0027] The relative distance calculation unit is used to calculate the Euclidean distance d between the host vehicle and the target corresponding to the predicted trajectory;

[0028] The relative speed calculation unit is used to calculate the relative speed Urel between the host vehicle and the target corresponding to the predicted trajectory;

[0029] The collision time calculation unit is used to use the formula to calculate the collision time and obtain the TTC value;

[0030] The risk classification unit is used to classify the risk level according to the TTC value.

[0031] Among them, the risk classification unit includes a high-risk classification sub-unit, a medium-risk classification sub-unit, and a low-risk classification sub-unit, which are connected in sequence;

[0032] When TTC < 2 seconds, the high-risk classification sub-unit is used to evaluate the collision risk as high risk, turn on the vehicle's built-in 360 panoramic view function, trigger an audible alarm and a visual prompt, and at the same time display the visual prompt in the corresponding area of the 360 panoramic view for the user to observe;

[0033] When 2 seconds ≤ TTC < 5 seconds, the medium-risk classification sub-unit is used to evaluate the collision risk as medium risk, turn on the vehicle's built-in 360 panoramic view function, and perform a visual prompt in the corresponding area of the 360 panoramic view;

[0034] The low risk classification subunit is used to assess the collision risk as low risk and not trigger an early warning when TTC is ≥ 5 seconds.

[0035] The present invention also provides a 360-degree panoramic view driving assistance method with AI intelligent warning, which adopts the above-mentioned 360-degree panoramic view driving assistance system with AI intelligent warning, and comprises the following steps:

[0036] Detecting objects around the car by using the data acquisition unit;

[0037] Perform location analysis and tracking based on the information data obtained from the detection;

[0038] Generate a predicted movement trajectory within the next five seconds based on the analyzed location information and tracked information;

[0039] Calculate the predicted movement trajectory and conduct collision risk assessment;

[0040] Take corresponding warning actions based on different risks to ensure driving safety.

[0041] The present invention provides an AI intelligent warning 360-degree panoramic view driving assistance system and method, wherein the target detection and tracking module is used to detect and track targets to obtain a separate historical trajectory for each target; the behavior prediction module is used to predict the future movement trajectory of the target based on the separate historical trajectory of each target to obtain a predicted trajectory; the trajectory mapping module is used to map the predicted trajectory to the vehicle's built-in 360-degree panoramic view or vehicle screen, and mark the predicted trajectory on the screen with lines for the user to observe; the collision risk assessment module is used to calculate the predicted trajectory based on the vehicle's own vehicle state, and then assess the collision risk and take corresponding warning measures;

[0042] Therefore, by analyzing and predicting the target after detection, the subsequent collision risk is judged according to the predicted trajectory, and this information is promptly displayed on the vehicle computer or warned through the speaker, thereby assisting the driver to effectively avoid risks and greatly improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0044] Figure 1 It is a schematic diagram of the 360-degree panoramic view driving assistance system with AI intelligent warning of the present invention.

[0045] Figure 2 It is a schematic diagram of the target detection and tracking module of the present invention.

[0046] Figure 3 It is a schematic diagram of the target detection unit of the present invention.

[0047] Figure 4 It is a schematic diagram of the target tracking unit of the present invention.

[0048] Figure 5 It is a schematic diagram of the behavior prediction module of the present invention.

[0049] Figure 6 It is a schematic diagram of the risk division unit of the present invention.

[0050] Figure 7 It is a flowchart of the steps of the 360-degree panoramic view driving assistance method of the AI ​​intelligent warning of the present invention.

[0051] 1-target detection and tracking module, 101-data acquisition unit, 102-target detection unit, 1021-image input subunit, 1022-anchor box prediction subunit, 1023-overlap removal subunit, 103-target tracking unit, 1031-target prediction subunit, 1032-target matching subunit, 1033-rematching subunit, 2-behavior prediction module, 201-trajectory encoding unit, 202-social pooling unit, 203-trajectory decoding unit, 3-trajectory mapping module, 4-collision risk assessment module, 401-relative distance calculation unit, 402-relative speed calculation unit, 403-collision time calculation unit, 404-risk division unit, 4041-high risk division subunit, 4042-medium risk division subunit, 4043-low risk division subunit. DETAILED DESCRIPTION

[0052] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0053] See also Figures 1 to 6 The present invention provides a 360-degree panoramic view driving assistance system with AI intelligent warning, which specifically includes:

[0054] The target detection and tracking module 1 is used to detect and track targets and obtain a separate historical trajectory for each target;

[0055] Specifically include:

[0056] The data acquisition unit 101 is used to detect the image data and physical data of the target in real time by using multiple cameras and radars to obtain image information and radar information;

[0057] By installing multiple cameras and lidar on the vehicle, it is possible to comprehensively scan and photograph the surrounding things, preparing for subsequent target analysis.

[0058] The target detection unit 102 is used to analyze the image information and radar information using a deep learning model;

[0059] Specifically, it includes:

[0060] The image input subunit 1021 is used to input real-time image frames of multiple cameras into the YOLOv8 model, and the model extracts features through a convolutional neural network;

[0061] Multiple cameras will capture real-time images simultaneously and input these images as image frames into the YOLOv8 model. These image frames will undergo preprocessing (resizing, normalization) to meet the requirements of model input; when extracting features, the CNN automatically extracts different levels of features from the original image through the combination of convolutional layers and pooling layers. The convolutional layer uses filters to perform convolutional operations on the image to extract local features; the pooling layer downsamples the feature map to reduce the size of the feature map and retain important feature information; then in YOLOv8, a feature pyramid network (such as PAN-FPN) is used to fuse feature maps of different scales to enhance the feature expression ability of the model. At the same time, structures such as cross-layer connections are also used to efficiently transfer feature information and prevent features from being lost in the deep network; thus, through the features extracted by the convolutional neural network, the YOLOv8 model can more accurately identify and locate targets in the image. These features contain key information such as the texture, color, and shape of the target, helping the model make more accurate judgments; and the CNN can automatically learn feature representations from a large number of training samples, which makes the YOLOv8 model have strong generalization ability. Even when facing unseen scenarios or targets, the model can accurately detect based on the extracted features.

[0062] The anchor box prediction subunit 1022 is used to predict the bounding box and category of the target using anchor boxes;

[0063] Through predefined anchor boxes, the system can quickly locate and classify targets in the image, thus realizing intelligent warning of the driving environment. Specifically, the anchor boxes provide prior information of different lengths and aspect ratios for the model, enabling the model to adapt to targets of different sizes and shapes. In this way, when the system detects potential dangers, it can more accurately identify the category and location of the target, providing more timely warning information for the driver and enhancing driving safety;

[0064] In the input image, the system generates multiple anchor boxes of different sizes and ratios centered on each pixel or each position of the feature map. These anchor boxes cover various shapes and sizes that the target may appear, providing rich prior information for the model. The system evaluates the matching degree by calculating the intersection over union (IoU) between each anchor box and the ground truth target box. The higher the IoU, the greater the overlap between the anchor box and the ground truth target box, and the higher the matching degree.

[0065] The overlapping removal subunit 1023 is used to remove overlapping bounding boxes through non-maximum suppression, retain the detection result with the highest confidence, and obtain a target list, where the target list is the bounding box, category, and confidence of each target.

[0066] During the prediction process, multiple overlapping prediction boxes may be generated. To simplify the output and improve accuracy, the system uses NMS to remove similar prediction boxes. NMS selects the best prediction box as the final result based on the confidence and overlap degree of the prediction boxes.

[0067] The target tracking unit 103 uses a multi-object tracking algorithm to track the detected targets and achieve continuous tracking of the targets.

[0068] Specifically, it includes:

[0069] The target prediction subunit 1031 is used to input the target list into the Kalman filter algorithm to predict the position of the target in the next frame and obtain the predicted position.

[0070] Based on the target state (such as position, speed, etc.) at the current moment and the control input (such as the steering and acceleration of the vehicle), use the system dynamics model to predict the target state at the next moment. At the same time, predict the estimated uncertainty (i.e., the error covariance matrix) of the current state to reflect the reliability of the prediction result. When new observation data (i.e., the position information of the target in the next frame) is available, calculate the Kalman gain, which determines which part of the predicted value and the observed value is more reliable. Use the Kalman gain and the observation data to adjust the predicted target state to obtain an updated target state estimate, and then update the error covariance matrix to reflect the uncertainty of the updated state estimate. Thus, through the Kalman filter algorithm, the historical position and speed information of the target, as well as the observation data of the current frame, can be combined to optimally estimate the position of the target in the next frame, thereby improving the accuracy of target tracking.

[0071] The target matching subunit 1032 is used to match the detection result of the current frame with the predicted position using the Hungarian algorithm.

[0072] By effectively correlating the detection data and the prediction data, an accurate data basis is provided for subsequent operations such as target state update and trajectory prediction, enabling the entire target tracking system to operate more smoothly and accurately. At the same time, by matching the detection results of the current frame with the predicted positions obtained based on information such as the previous frame, the corresponding relationship of the targets between different frames can be determined, that is, to judge which target in the previous frame the target in the current frame is a continuation of, so as to achieve continuous tracking of the target.

[0073] First, construct a bipartite graph:

[0074] Determine the nodes: Take the detection results and predicted positions of the current frame as the two vertex sets of the bipartite graph respectively. For example, assume that 3 targets are detected in the current frame, denoted as D1, D2, and D3 respectively, and at the same time there are 3 positions predicted based on information such as the previous frame, denoted as P1, P2, and P3. Then these 6 elements are the nodes in the two vertex sets of the bipartite graph respectively.

[0075] Define the edges and weights: Calculate a certain similarity measure or distance measure between the detection results and the predicted positions, and use it as the weight of the edges in the bipartite graph. For example, the Euclidean distance can be used to measure the distance between the center of the detection box and the predicted position. The smaller the distance, the more matching they are, and the higher the weight of the edge. Assume that the Euclidean distance between the detection target D1 and the predicted position P2 is 5, then the weight of the edge connecting D1 and P2 can be set to a value related to 5, such as 1 / 5 (the smaller the distance, the greater the weight).

[0076] Secondly, find the maximum matching:

[0077] Initialize the matching: Start from an empty matching, that is, initially no detection results are matched with the predicted positions.

[0078] Greedy selection: Try to match the detection results with the predicted positions in the order of the weights of the edges from largest to smallest. For example, among all the edges connecting the detection results and the predicted positions, first select the detection result and the predicted position corresponding to the edge with the largest weight for matching. Assume that the edge connecting D1 and P2 has the largest weight, then first try to match D1 with P2.

[0079] Conflict handling: If a conflict occurs during the matching process, that is, a certain detection result has been matched with a predicted position, and another predicted position also needs to be matched with this detection result, then adjustment is required. The Hungarian algorithm solves the conflict through mechanisms such as backtracking, and tries to find other feasible matching schemes to ensure that the final obtained matching is the maximum matching. For example, when trying to match D2 with P2 as well, it is found that P2 has already been matched with D1. At this time, the algorithm will check whether D1 can be matched with other predicted positions. If it can, then adjust the matching object of D1 so that D2 can be matched with P2.

[0080] Final matching result: After a series of matching and adjustment operations, the final matching result is the best matching solution between the current frame detection result and the predicted position determined by the Hungarian algorithm, so that the overall matching weight is the largest, that is, the matching degree between the detection result and the predicted position is the highest.

[0081] The rematching subunit 1033 is used to initialize a new tracker for unmatched targets; for lost targets, try to rematch in subsequent frames, and finally obtain the unique ID of each target and its trajectory in consecutive frames, which is the historical trajectory.

[0082] After the current frame is matched with the predicted position, there will be some detection results that are not successfully matched with any predicted position. These unmatched detection results are likely to represent newly appeared targets. New trackers are created for unmatched targets, and the tracker's state information is usually initialized, such as the target's position, size, speed, etc. For example, a newly appeared pedestrian target will be initialized based on its detection box position and size in the current frame. A tracker specifically used to track the pedestrian will be assigned a unique ID, and a separate track can be generated for each target.

[0083] The behavior prediction module 2 is used to predict the future movement trajectory of the target based on the historical trajectory of each target to obtain a predicted trajectory;

[0084] Specifically include:

[0085] The trajectory encoding unit 201 is used to use LSTM to encode the historical trajectory of the target and extract time series features;

[0086] By processing the historical trajectory data of the target through LSTM, it is possible to automatically capture the time series characteristics in the trajectory, such as the changes in the speed of the target's movement, the laws of direction changes, etc., and convert the original trajectory data into a more representative feature vector for subsequent analysis and processing; the memory characteristics of LSTM enable it to effectively mine the long-term dependencies between different time points in the historical trajectory, remember the target's movement pattern over a longer time range, such as whether the target has periodic movement patterns, or specific movement preferences in a specific environment, etc., to provide more comprehensive information support for subsequent predictions; the complex historical trajectory data is encoded into a low-dimensional feature vector, while retaining key information, redundant information in the data is removed, the amount of data is reduced, the operating efficiency and processing speed of the model are improved, and subsequent units are facilitated to perform rapid calculations and analysis.

[0087] The social pooling unit 202 is used to fuse the state information of the surrounding targets into the features of the current target through a pooling layer based on the interactive relationship between the multiple targets;

[0088] In an actual scenario, the movement of a target is often not isolated and is affected by other surrounding targets. This unit can capture such interactions based on the interaction relationships between multiple targets, such as behaviors like avoidance between vehicles, following or avoidance between pedestrians, etc., making the model's understanding of target movement more comprehensive and accurate; through the pooling layer, it fuses the state information of surrounding targets, such as position, speed, direction, etc., into the features of the current target, adding more context information to the feature representation of the current target, enriching the feature dimension, helping the model better understand the environment where the current target is located and the possible impacts it may receive, and thus more accurately predicting its future trajectory; considering the interaction relationships between targets and the surrounding environment information enables the model to better adapt to various complex scenarios, improving the model's generalization ability in different environments, and being able to more accurately predict the trajectories for different numbers and distributions of target groups.

[0089] The trajectory decoding unit 203 is used to predict the future trajectory of the target using another LSTM decoder and output a position sequence within the next five seconds, which is the predicted trajectory.

[0090] Using another LSTM decoder, based on the features extracted by the trajectory encoding unit 201 and the features fused by the social pooling unit 202, and according to the existing historical trajectory features and the influence information of surrounding targets, it predicts the future trajectory of the target, generates a position sequence within the next five seconds, and directly provides specific prediction results for intelligent early warning; it can output a continuous position sequence, that is, the specific position of the predicted target at each future time point, forming a complete predicted trajectory instead of isolated several prediction points, which can more intuitively display the future movement trend of the target, provide more detailed and valuable information for the early warning system, and facilitate making more accurate decisions; during the prediction process, the LSTM decoder can continuously adjust and optimize the prediction results according to the input features, consider the current state of the target and environmental changes in real time, and dynamically correct the predicted trajectory, making the prediction results more accurate and reliable, and being able to better handle various complex and changeable situations.

[0091] The trajectory mapping module 3 is used to map the predicted trajectory to the vehicle's built-in 360 panoramic view or the car machine screen, and mark the predicted trajectory on the screen with a line for the user to observe;

[0092] The collision risk assessment module 4 is used to calculate the predicted trajectory based on the state of its own vehicle, and then conduct a collision risk assessment and take corresponding early warning measures.

[0093] Specifically, it includes:

[0094] The relative distance calculation unit 401 is configured to calculate the Euclidean distance d between the host vehicle and the target corresponding to the predicted trajectory;

[0095] The relative speed calculation unit 402 is configured to calculate the relative speed Urel between the host vehicle and the target corresponding to the predicted trajectory;

[0096] The time to collision calculation unit 403 is configured to use the formula to calculate the time to collision and obtain the TTC value;

[0097] The obtained TTC value represents the time of the impending collision in the current vehicle body state, and thus reflects the subsequent risk level.

[0098] The risk classification unit 404 is configured to classify the risk level according to the TTC value.

[0099] Specifically, it includes:

[0100] The high-risk classification sub-unit 4041 is configured to, when TTC < 2 seconds, evaluate the collision risk as high risk, turn on the vehicle's built-in 360 panoramic surround view function, and trigger an audible alarm and a visual prompt, and at the same time display the visual prompt in the corresponding area of the 360 panoramic surround view for the user to observe;

[0101] After determining it as high risk, an audible alarm and a visual prompt are triggered. If the user triggers the warning and the time has exceeded 50% of the TTC, the system will forcibly take braking measures to ensure driving safety to the greatest extent.

[0102] The medium-risk classification sub-unit 4042 is configured to, when 2 seconds ≤ TTC < 5 seconds, evaluate the collision risk as medium risk, turn on the vehicle's built-in 360 panoramic surround view function, and perform a visual prompt in the corresponding area of the 360 panoramic surround view;

[0103] After triggering the medium risk, it indicates that the relative speed is relatively fast or the distance from the target is relatively close, but it is still within the controllable range. Therefore, only a visual prompt is performed and no coercive measures are taken.

[0104] The low-risk classification sub-unit 4043 is configured to, when TTC ≥ 5 seconds, evaluate the collision risk as low risk and not trigger a warning.

[0105] Low risk means that the relative speed is relatively slow or the distance from the target is relatively far, so no warning is triggered.

[0106] Please refer to Figure 7 , the present invention also provides a 360 panoramic surround view driving assistance method with AI intelligent warning, including the following steps:

[0107] S1: Detect the targets around the vehicle through the data acquisition unit 101;

[0108] S2: Perform position analysis and tracking based on the information data obtained from detection;

[0109] S3: Generate a predicted movement trajectory within the next five seconds based on the analyzed position information and the tracked information;

[0110] S4: Calculate for the predicted movement trajectory to conduct a collision risk assessment;

[0111] S5: Make corresponding warning actions based on different risks to ensure driving safety.

[0112] Among them, the vehicle surrounding targets are detected by the data acquisition unit 101; position analysis and tracking are performed based on the information data obtained from detection; a predicted movement trajectory within the next five seconds is generated based on the analyzed position information and the tracked information; calculations are made for the predicted movement trajectory to conduct a collision risk assessment; corresponding warning actions are made based on different risks to ensure driving safety.

[0113] What is disclosed above is only one or more preferred embodiments of the present application, and the scope of rights of the present application cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A 360-degree panoramic view driving assistance system with AI intelligent warning, characterized in that: It includes a target detection and tracking module, a behavior prediction module, a trajectory mapping module and a collision risk assessment module, wherein the target detection and tracking module, the behavior prediction module, the trajectory mapping module and the collision risk assessment module are connected in sequence; The target detection and tracking module is used to detect and track targets and obtain a separate historical trajectory for each target; The behavior prediction module is used to predict the future movement trajectory of the target based on the historical trajectory of each target to obtain a predicted trajectory; The trajectory mapping module is used to map the predicted trajectory to the vehicle's built-in 360-degree panoramic view or vehicle computer screen, and mark the predicted trajectory on the screen with lines for the user to observe; The collision risk assessment module is used to calculate the predicted trajectory based on the vehicle status, and then assess the collision risk and take corresponding early warning measures.

2. The 360-degree panoramic view driving assistance system with AI intelligent warning as claimed in claim 1, characterized in that: The target detection and tracking module comprises a data acquisition unit, a target detection unit and a target tracking unit, wherein the data acquisition unit, the target detection unit and the target tracking unit are connected in sequence; The data acquisition unit is used to detect the image data and physical data of the target in real time by using multiple cameras and radars to obtain image information and radar information; The target detection unit is used to analyze the image information and radar information using a deep learning model; The target tracking unit uses a multi-target tracking algorithm to track the detected targets to achieve continuous tracking of the targets.

3. The 360-degree panoramic view driving assistance system with AI intelligent warning as claimed in claim 2, characterized in that: The object detection unit comprises an image input subunit, an anchor box prediction subunit and an overlap removal subunit, wherein the image input subunit, the anchor box prediction subunit and the overlap removal subunit are connected in sequence; The image input subunit is used to input real-time image frames of multiple cameras into a YOLOv8 model, and the model extracts features through a convolutional neural network; The anchor box prediction subunit is used to predict the bounding box and category of the target using the anchor box; The overlap removal subunit is used to remove overlapping bounding boxes by non-maximum suppression, retain the detection result with the highest confidence, and obtain a target list, wherein the target list includes the bounding box, category and confidence of each target.

4. The 360-degree panoramic view driving assistance system with AI intelligent warning as claimed in claim 3, characterized in that: The target tracking unit comprises a target prediction subunit, a target matching subunit and a rematching subunit, wherein the target prediction subunit, the target matching subunit and the rematching subunit are connected in sequence; The target prediction subunit is used to input the target list into the Kalman filter algorithm to predict the position of the target in the next frame to obtain the predicted position; The target matching subunit is used to match the detection result of the current frame with the predicted position using the Hungarian algorithm; The rematching subunit is used to initialize a new tracker for an unmatched target; For lost targets, we try to rematch them in subsequent frames, and finally get the unique ID of each target and its trajectory in consecutive frames, which is the historical trajectory.

5. The 360-degree panoramic view driving assistance system with AI intelligent warning as claimed in claim 4, characterized in that: The behavior prediction module includes a trajectory encoding unit, a social pooling unit and a trajectory decoding unit, and the trajectory encoding unit, the social pooling unit and the trajectory decoding unit are connected in sequence; The trajectory encoding unit is used to use LSTM to encode the historical trajectory of the target and extract time series features; The social pooling unit is used to fuse the state information of surrounding targets into the features of the current target through a pooling layer based on the interactive relationship between multiple targets; The trajectory decoding unit is used to predict the future trajectory of the target using another LSTM decoder, and output a position sequence within the next five seconds, which is the predicted trajectory.

6. The 360-degree panoramic view driving assistance system with AI intelligent warning as claimed in claim 5, characterized in that: The collision risk assessment module comprises a relative distance calculation unit, a relative speed calculation unit, a collision time calculation unit and a risk division unit, wherein the relative distance calculation unit, the relative speed calculation unit, the collision time calculation unit and the risk division unit are connected in sequence; The relative distance calculation unit is used to calculate the Euclidean distance d between the vehicle and the target corresponding to the predicted trajectory; The relative speed calculation unit is used to calculate the relative speed Urel between the vehicle and the target corresponding to the predicted trajectory; The collision time calculation unit is used to use the formula Calculate the collision time and obtain the TTC value; The risk classification unit is used to classify risk levels according to the TTC value.

7. The 360-degree panoramic view driving assistance system with AI intelligent warning as claimed in claim 6, characterized in that: The risk division unit comprises a high-risk division subunit, a medium-risk division subunit and a low-risk division subunit, and the high-risk division subunit, the medium-risk division subunit and the low-risk division subunit are connected in sequence; The high-risk classification subunit is used to assess the collision risk as high risk when TTC is less than 2 seconds, turn on the vehicle's built-in 360-degree panoramic view function, trigger a sound alarm and a visual prompt, and display the visual prompt in the corresponding area of ​​the 360-degree panoramic view for the user to observe; The medium risk classification sub-unit is used to assess the collision risk as medium risk when 2 seconds ≤ TTC < 5 seconds, turn on the vehicle's built-in 360-degree panoramic view function, and provide visual prompts in the corresponding area of ​​the 360-degree panoramic view; The low risk classification subunit is used to assess the collision risk as low risk and not trigger an early warning when TTC is ≥ 5 seconds.

8. An AI intelligent warning 360-degree panoramic view driving assistance method, using the AI ​​intelligent warning 360-degree panoramic view driving assistance system as claimed in claim 7, characterized in that: The steps include: Detecting objects around the car by using the data acquisition unit; Perform location analysis and tracking based on the information data obtained from the detection; Generate a predicted movement trajectory within the next five seconds based on the analyzed location information and tracked information; Calculate the predicted movement trajectory and conduct collision risk assessment; Take corresponding warning actions based on different risks to ensure driving safety.

Images