A machine learning-based multi-modal fusion obstacle intelligent early warning method

Abstract

The application provides a kind of multi-modal fusion obstacle intelligent early warning method based on machine learning, it is related to machine learning intelligent early warning technical field, including steps: the driving sight range of vehicle is zoned, and the horizontal boundary azimuth range and vertical boundary pitch angle range of each partition are calculated respectively;Real-time acquisition of vehicle's eye tracker data and radar point cloud data and extract obstacles;Calculate the azimuth and pitch angle of the obstacle, map the obstacle to the partition of the driving sight range, and get the obstacle area;Calculate the spatial deviation angle of the driving sight fixation point and the obstacle, the time decay attention intensity and the obstacle mutation intensity, calculate the risk value of the obstacle area, if it is within the safety threshold, the obstacle area is in a safe state and no warning is given, otherwise, input the obstacle area and obstacle related information into the knowledge enhanced neural network, and carry out intelligent multi-level response warning.The application improves the real-time and accuracy of early warning in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of machine learning intelligent early warning technology, specifically to a multimodal fusion obstacle intelligent early warning method based on machine learning. Background Technology

[0002] With the rapid development of industrial automation and intelligence, the number of vehicles in factory settings such as production, logistics, and metal processing is increasing daily. The complex environment created by stacked materials, the mixed human-vehicle work patterns, and adverse factors such as light and dust pose serious threats to driving safety. Traditional obstacle warning systems have many limitations in assisted driving. They cannot dynamically adapt to the complex road scenarios and high obstacle density in factory areas, and are prone to low-quality warnings such as false alarms or alarm overload. False alarms can interfere with the driver's judgment of road conditions and operational decisions, even reducing driving efficiency; while alarm overload can cause driver fatigue with warning signals, gradually ignoring or even blocking warning prompts, potentially leading to missed real dangers and increasing the risk of accidents.

[0003] Therefore, providing a reliable intelligent obstacle warning method for vehicles traveling within the factory area, especially those operating in complex environments such as material yards, is of paramount importance. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention aims to provide a multimodal fusion-based intelligent obstacle warning method based on machine learning. Through the collaborative processing of eye-tracking data and radar point cloud data, the radar point cloud data is recursively divided to efficiently query obstacles. By dynamically adjusting the spatial radius and minimum point threshold, spatial clustering is performed on the obstacle point cloud to filter out noise points and obtain the obstacles. The attention state of the obstacle's partition is updated based on real-time gaze points. By calculating the obstacle's azimuth and pitch angles, the obstacle is mapped to the partition to determine the partition's state. Finally, the partition state and attention state are matched to trigger multi-level visual and auditory alarms, making this method suitable for complex scenarios requiring high-precision obstacle recognition, dynamic attention perception, and natural interaction.

[0005] Specifically, the present invention provides a multimodal fusion-based intelligent obstacle warning method based on machine learning, which includes the following steps:

[0006] S1. Divide the vehicle's driving line of sight into zones, and calculate the horizontal boundary azimuth angle range and vertical boundary pitch angle range of each zone.

[0007] S2. Real-time acquisition of vehicle eye tracker data and radar point cloud data; the eye tracker data is used to provide driver gaze attention information, and the radar point cloud data is used to provide spatial information about the vehicle's surroundings.

[0008] S3. Extract obstacles and their distances using real-time acquired radar point cloud data:

[0009] S4. Calculate the azimuth and pitch angles of the obstacles based on the obstacle information, and map the obstacles to the corresponding zones within the driver's line of sight by using the horizontal boundary azimuth angle range and the vertical boundary pitch angle range of each zone, thus obtaining the obstacle zone.

[0010] S5. Calculate the spatial deviation angle between the driver's gaze point and the obstacle, as well as the time-decayed attention intensity.

[0011] The spatial deviation angle between the driver's gaze point and the obstacle is: :

[0012] ;

[0013] in, Number the driving line of sight zones. For timestamps, This represents the unit direction vector of the gaze point in the current frame of the eye tracker data. This refers to the unit direction vector of obstacles in radar data;

[0014] S6. Evaluate the mutation intensity of the obstacle. If the mutation intensity is greater than the mutation threshold, proceed to S9; otherwise, proceed to S7.

[0015] S7. Calculate the obstacle zone risk value using the obtained spatial deviation angle, time-decayed attention intensity, and obstacle abrupt change intensity:

[0016] S8. Determine whether the risk value of the obstacle area is within the safety threshold. If yes, the obstacle area is in a safe state and no warning is issued. Otherwise, the obstacle area is in a dangerous state and S9 is executed.

[0017] S9. Input the obstacle area and related obstacle information into the knowledge enhancement neural network to perform intelligent multi-level response early warning for the obstacle area.

[0018] Furthermore, the S1 divides the vehicle's driving line of sight into coarse-grained and fine-grained zones according to different driving scenarios.

[0019] Furthermore, S3 utilizes real-time acquired radar point cloud data to extract obstacles, specifically including:

[0020] The obtained radar point cloud data is recursively divided into sub-nodes using a spatial segmentation algorithm until the point cloud density of each sub-node meets the threshold for the clustering algorithm to identify obstacles. The sub-nodes containing obstacle information are denoised by dynamically adjusting the spatial radius and minimum point number threshold in the clustering algorithm, and the obstacles are extracted.

[0021] Further, S4, calculates the azimuth and pitch angles of the obstacles based on the obstacle information, and maps the obstacles to their respective zones within the driver's line of sight using the horizontal boundary azimuth angle range and the vertical boundary pitch angle range of each zone, thus obtaining the obstacle zone, specifically including:

[0022] S41. Calculate the obstacle azimuth angles based on the extracted obstacle information. With pitch angle ;

[0023] S42. Based on the obstacle's azimuth angle The difference between the obstacle and the midpoint of the horizontal boundary azimuth range is used to map the obstacle to its corresponding horizontal partition using the horizontal mapping partitioning rules;

[0024] S43. Based on the obstacle's pitch angle The difference between the obstacle and the midpoint of the vertical boundary pitch angle range is used to map the obstacle to its corresponding vertical partition using the vertical mapping partitioning rules;

[0025] S44. Merge the horizontal and vertical zones to which the obstacle belongs, and map the extracted obstacle to the zone to which it belongs within the driver's line of sight to obtain the obstacle zone.

[0026] Furthermore, the horizontal mapping partitioning rules in S42 include:

[0027] If the azimuth angle of the obstacle The difference between the midpoint of the azimuth range of the horizontal boundary When this happens, the obstacle is determined to belong to the corresponding zone;

[0028] like If the partition is not found, then exclude that partition and continue matching other partitions;

[0029] like If a node is located at the boundary of two partitions, the parent node containing the point cloud of the obstacle is located. The parent node is recursively split, and all child nodes containing the obstacle are extracted. The proportion of the point cloud of each child node is calculated, and the node with the largest proportion of valid point cloud is assigned to that partition.

[0030] Furthermore, the vertical mapping partitioning rules in S43 include:

[0031] If the obstacle's pitch angle The difference between the midpoint of the pitch angle range and the vertical boundary When this happens, the obstacle is determined to belong to the corresponding zone;

[0032] like If the partition is not found, then exclude that partition and continue matching other partitions;

[0033] like When the obstacle is located at the boundary of multiple partitions, the parent node containing the point cloud of the obstacle is located, and the parent node is recursively split. All child nodes containing the obstacle are extracted, and the proportion of the point cloud of each child node is counted. The node with the largest proportion of valid point cloud is assigned to that partition.

[0034] Furthermore, S5 calculates the spatial deviation angle between the driver's gaze point and the obstacle, as well as the time-decrease in attention intensity, specifically including:

[0035] S51. Calculate the spatial deviation angle between the driver's gaze point and the obstacle. ;

[0036] ;

[0037] in, Number the driving line of sight zones. For timestamps, This represents the unit direction vector of the gaze point in the current frame of the eye tracker data. This refers to the unit direction vector of obstacles in radar data;

[0038] S52. Calculate the instant matching degree ;

[0039] ;

[0040] in, , is the angle sensitivity coefficient;

[0041] S52, Calculating Time-Decrease Attention Intensity :

[0042] ;

[0043] in, The time decay coefficient, It reflects the driver's level of continuous attention to the obstacle area at any given moment.

[0044] Furthermore, the intensity of obstacle mutations in S6 for:

[0045] ;

[0046] in, This represents the Euclidean distance between the obstacle and the vehicle in the current frame. This represents the number of point clouds representing obstacle clusters in the current frame. Let be the standard deviation of the distance between the obstacle and the vehicle within the sliding time window. Let be the standard deviation of the number of obstacle point clouds within the sliding time window. To prevent constants with a denominator of 0.

[0047] Furthermore, the obstacle zone risk value in S8 for:

[0048] ;

[0049] in, This is the normalized mutation intensity score.

[0050] Furthermore, S9 inputs the obstacle area and related obstacle information into the knowledge-enhancing neural network to perform intelligent multi-level response warnings for the obstacle area, specifically including:

[0051] S91. Perform high-order feature extraction and sliding window statistics on obstacle areas and related obstacle information;

[0052] S92. Outputting a comprehensive risk score via the network. :

[0053] ;

[0054] in: t The feature vector is the result of modulation by the Stat-FiLM layer. For model network parameters, It is a multi-layer nonlinear mapping function;

[0055] S93. Develop intelligent multi-level response early warning for obstacle areas based on comprehensive risk scores.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] 1. The present invention divides the driving field of vision of the vehicle into zones and sets photoelectric prompting devices on the visible outlines. Taking the front window as an example, photoelectric prompting devices (such as RGB smart light strips) that do not obstruct the driver's field of vision are set on the four edges of the front window inside the vehicle. When a warning is needed, the outline of each zone is marked with colored photoelectric prompts. Compared with the reminders of the traditional instrument panel or central control screen, it is more conducive to the driver to directly locate the position of obstacles and take effective obstacle avoidance measures.

[0058] 2. This invention uses the horizontal boundary azimuth angle and vertical boundary pitch angle under the driver's line of sight coordinate system to realize the mapping of obstacles to each zone, which makes the display of obstacle position more "human-centered". Combined with eye tracker, it can realize the leap from static model to dynamic model. The calculated angle can be accurately aligned with what the driver sees.

[0059] 3. This invention features a highly efficient data processing chain: First, the original point cloud is spatially indexed using an octree to quickly filter out "regional point clouds that may contain obstacles" (such as the nearby area). Then, DBSCAN performs density clustering on these points to obtain structured obstacle clusters. This process significantly reduces the amount of data processed by subsequent algorithms (avoiding full point cloud computing) while ensuring that the clustered objects are "meaningful obstacles" rather than noise.

[0060] 4. This invention assesses the intensity of obstacle mutations and directly triggers warnings for "highly mutated obstacles that users are not paying attention to". It can obtain the dynamic attributes of highly mutated obstacles before entering the risk assessment stage, thus achieving earlier risk identification and response triggering.

[0061] 5. This invention calculates the risk value of the obstacle zone by judging the thresholds of spatial deviation angle, time decay attention intensity, and sudden change intensity. It combines the obstacle with the attention state (noticed / not noticed) to divide the obstacle zone into safe and dangerous states. The obstacle zone in the safe state that the driver has noticed is excluded during the warning, and the warning is only given to the dangerous state obstacle zone that the driver has not noticed. This effectively avoids invalid alarm interference and improves the significance of the warning and driving safety.

[0062] 6. This invention performs high-order feature extraction and sliding window statistics on obstacle areas and obstacle-related information based on machine learning. It outputs a comprehensive risk score through a knowledge-enhanced neural network, and conducts intelligent multi-level response early warning for obstacle areas. It constructs a closed-loop mechanism of "precise perception - intelligent clustering - dynamic early warning", thereby improving the real-time performance, accuracy and natural interaction of early warning in complex scenarios. Attached Figure Description

[0063] Figure 1 This is a flowchart of a multimodal fusion-based intelligent obstacle early warning method based on machine learning according to the present invention;

[0064] Figure 2 This is a comparison chart of the number of alarms in an early warning method according to an embodiment of the present invention compared to traditional methods. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present invention, the appendices in the embodiments of the present invention will be described below. Figure 1-2 The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0066] Taking the driving of vehicles in the complex environment of a steel coil yard within a factory area as an example, the coexistence of people, vehicles, and materials in a steel coil yard presents numerous safety hazards. This invention utilizes eye trackers to acquire visual information and lidar to acquire environmental information of the steel coil yard, performing multi-vision identification and positioning of personnel and obstacles, and conducting multi-modal obstacle registration, detection, early warning, and hazard avoidance. Furthermore, the method of this invention can also be extended to solve the problem of safe driving of vehicles on open roads.

[0067] As one embodiment of the present invention, as shown in the appendix Figure 1 As shown, this invention provides a multimodal fusion-based intelligent obstacle early warning method based on machine learning, the specific implementation of which is as follows:

[0068] S1. Divide the driving line of sight range of vehicles within the factory area into zones, and calculate the horizontal boundary azimuth angle range and vertical boundary pitch angle range of each zone. Specific steps include:

[0069] S11. Divide the driving visibility range of vehicles within the factory area into zones.

[0070] The driver's field of vision refers to the range of road and surrounding environment that a driver can clearly see from the driver's seat. This is a key factor in ensuring safe driving, helping drivers to promptly detect potential hazards such as pedestrians, vehicles, and obstacles on the road. Dividing the driver's field of vision into sections (m) allows for precise identification of the section's location, facilitating quick positioning and adjustment of driving actions by using photoelectric indicators on the edges of these sections.

[0071] The zoning of the field of vision needs to be determined based on different driving scenarios and traffic conditions. For example, taking driving in a complex steel coil yard environment within a factory area as an example, in such complex road scenarios with complicated surroundings, heavy traffic, or limited visibility, the field of vision zoning is fine-grained, meaning the number of zoning can be m=6 or m=9, facilitating attention to the top, bottom, left, center, and right, suitable for situations such as falling steel coils, limited visibility, and emergencies in complex steel coil yard environments. In simple road scenarios with open visibility, flat roads, and relatively stable traffic flow within a factory area, the forward field of vision zoning logic is coarse-grained, meaning the number of zoning is m=3.

[0072] S12. Calculate the azimuth range of the horizontal boundary and the elevation range of the vertical boundary for each partition.

[0073] One implementation method involves capturing the driver's eye coordinates and gaze vectors using an eye tracker to determine the horizontal azimuth and vertical pitch ranges of the entire driving field of vision. Then, based on the number of zones, the horizontal azimuth and vertical pitch ranges of each zone are calculated proportionally. This can include an initialization step: after the driver is seated, a voice prompt guides them to look at the highlighted areas of the light strip around the vehicle window (left / right boundaries: the inner edges where the left and right window pillars connect to the front window; upper / lower boundary points: the midpoint of the top edge of the windshield and the midpoint of the bottom edge of the windshield). The recorded four gaze vector units are converted into azimuth and pitch angles, thus obtaining the driver's actual perceived field of vision, which is then divided into zones proportionally.

[0074] Taking the aforementioned nine partitions as an example, they include the top-left partition, left-middle partition, bottom-left partition, top-middle partition, center partition, bottom-middle partition, top-right partition, right-middle partition, and bottom-right partition. For instance, the horizontal boundary azimuth range of each partition can be as follows: Left partition... Middle section Right partition The vertical boundary pitch angle range for each partition can be as follows: Lower partition , middle partition Upper partition After being arranged and combined, the horizontal boundary azimuth angle range and the vertical boundary elevation angle range of each partition are formed. For example, the horizontal boundary azimuth angle range of the upper left partition is the range of the left partition. The vertical boundary pitch angle range is the upper partition. And so on. The driving visibility zone is numbered as follows: ,at this time It is a positive integer from 1 to 9.

[0075] Taking the three partitions mentioned above as an example, they include a left partition, a middle partition, and a right partition. (For example, the azimuth range of the horizontal boundary in the left partition is...) The range of azimuth angles of the horizontal boundary in the middle section is: The range of azimuth angles of the horizontal boundary in the right partition is The pitch angle range of the vertical boundary is all The driving visibility zone is numbered as follows: ,at this time It is a positive integer between 1 and 3.

[0076] In addition, photoelectric warning devices, such as RGB smart light strips, are installed around the perimeter of the windshield inside the vehicle to ensure unobstructed visibility for the driver. These devices work in conjunction with sound prompts to achieve precise, zoned photoelectric warnings as described in this invention.

[0077] Furthermore, the embodiments of the present invention are illustrated by taking the partitioning of the forward line of sight and obstacle judgment as an example. By applying the method of the present invention, the partitioning of the side line of sight and the rearview mirror line of sight and obstacle mapping judgment can also be set.

[0078] S2. Real-time acquisition of vehicle eye tracker data and radar point cloud data; the eye tracker data is used to provide driver gaze attention information, and the radar point cloud data is used to provide spatial information about the vehicle's surroundings.

[0079] An eye tracker is a device used to track and record eye movements. It can accurately measure and analyze information such as the eye's movement trajectory, fixation point, and the speed and frequency of eye movements, and calculate the fixation point through algorithms, thereby providing information on the driver's gaze (i.e., where the eyes are looking).

[0080] Radar systems measure the distance, speed, and direction of objects by emitting radio waves and receiving the waves reflected back from them. Simultaneously, radar can generate three-dimensional point cloud data of objects, obtaining their position and shape in space through the reflected signals, thus achieving precise perception of surrounding objects.

[0081] This invention utilizes a high-precision eye tracker deployed in the center of the vehicle's dashboard that does not obstruct the driver's core field of vision to acquire real-time information about the driver's gaze. It also relies on a 360° surround-view radar system with multiple sensors at the front, rear, and sides of the vehicle to acquire real-time information about the surrounding space. Specifically, in this invention's method, eye-tracking data, including the three-dimensional coordinates of the eyeball and the normalized coordinates of the gaze point, is stored in a thread-safe buffer to ensure real-time performance. Radar point cloud data, including the three-dimensional position of obstacles and obstacle reflection intensity information, is parsed and loaded in a time sequence, serving as the core basis for environmental perception. Radar data is loaded in timestamp order and synchronized with the eye tracker time to ensure consistent data timing across different sensors. A unified clock synchronization mechanism ensures precise alignment of the driver's gaze with the environmental point cloud in both time and space.

[0082] S3. Extract obstacles and their distances using real-time acquired radar point cloud data:

[0083] To accurately identify obstacles in various scenarios, this invention employs a two-stage detection algorithm combining octree segmentation and DBSCAN clustering. First, the octree algorithm recursively divides the radar point cloud data obtained in step S2 into sub-regions for spatial segmentation. Then, the density-based spatial clustering algorithm DBSCAN is used to extract obstacles. Specifically, the octree algorithm recursively divides the radar point cloud data obtained in step S2 into child nodes until the point cloud density of each child node meets the DBSCAN threshold for obstacle detection. Noise is denoised in the child nodes containing obstacle information by dynamically adjusting the spatial radius and minimum point count threshold in the DBSCAN algorithm, thereby extracting obstacles in different road scenarios. The distance between obstacles and vehicles can be directly obtained from radar data.

[0084] The octree algorithm incorporates a scene feature library and establishes a "feature-termination threshold" mapping relationship based on real-time acquired scene dynamic features. It can also dynamically adjust according to scene complexity and automatically retrieve the corresponding termination threshold. When the density of child nodes reaches the dynamic threshold of the scene, the recursion terminates autonomously without the need for manual switching of scene parameters. The DBSCAN parameters are initially matched to the range of the spatial radius and minimum point number threshold through the scene feature library. Then, the spatial radius and minimum point number threshold are fine-tuned by combining the K-distance map and neighborhood density of the real-time point cloud, ensuring accurate extraction of obstacles in different scenes and laying the foundation for subsequent fusion with eye-tracking data.

[0085] The DBSCAN algorithm uses a parameter space radius ε to define the range of a "neighborhood" in the point cloud space. Points within a distance of ε around a given point are considered its neighborhood. The minimum number of points threshold MinPts is used to determine dense clustering. When the number of points in a point's neighborhood is greater than or equal to MinPts, the point is considered a core point, and its region forms a dense cluster.

[0086] As an example, for simple road scenarios: the DBSCAN algorithm has a spatial radius ε of 0.8-1.2m and a minimum point threshold MinPts of 8-15, which is suitable for recognizing large obstacles such as vehicles; for complex road scenarios: the DBSCAN algorithm has a spatial radius ε of 0.5-0.8m and a minimum point threshold MinPts of 15-30, which is suitable for small / dense obstacles such as steel coils and temporary materials.

[0087] Furthermore, the recursive splitting of the octree algorithm only recursively partitions the child nodes containing obstacles. The specific process is as follows:

[0088] Based on the real-time scene characteristics, the expression for determining the maximum allowed edge length of the current child node is:

[0089] ;

[0090] If the edge length of the current child node If the edge length of the current child node is... Then, the recursive splitting continues until... .

[0091] The specific process of adaptive point cloud density determination is as follows: if the point cloud density of the kth child node is... less than the lower density limit for the DBSCAN algorithm When the node reaches a certain value, the recursion terminates and it is marked as a "non-obstacle child node"; if the point cloud density of the k-th child node is... The density lower bound for the DBSCAN algorithm is greater than or equal to that of the algorithm. And the point cloud density of the k-th child node The density limit is less than or equal to that of the DBSCAN algorithm. If the point cloud density of the k-th child node is [value missing], then the validity of the DBSCAN clustering algorithm for that child node is determined; if the point cloud density of the k-th child node is [value missing], then [value missing]. Greater than the density limit of the DBSCAN algorithm If so, the recursive splitting continues.

[0092] The specific process for determining the validity of the DBSCAN clustering algorithm for the child node is as follows: If the proportion of randomly selected sample points within child node k is... If the recursion terminates, the child node is output, and the obstacle is extracted using the DBSCAN algorithm; if the proportion of randomly selected sample points within child node k is... If the condition is met, the recursion terminates and the node is marked as a "non-obstacle child node".

[0093] S4. Calculate the azimuth and pitch angles of obstacles based on obstacle information, and map the obstacles to their respective zones within the driver's line of sight using the horizontal boundary azimuth angle range and the vertical boundary pitch angle range of each zone, thus obtaining the obstacle zone. Specific steps include:

[0094] S41. Calculate the obstacle azimuth angles based on the extracted obstacle information. With pitch angle .

[0095] In reality, drivers are constantly moving: they adjust their seating position, leaning forward or back; they turn their heads, changing the position of their eyes relative to the interior of the car. Drivers of different statures have completely different eye positions while driving.

[0096] A fixed eye-tracking model can cause dynamic parallax in the positioning markers. When you lean forward, the markers that were originally aligned will "misalign." The purpose of an eye tracker is to capture the precise three-dimensional spatial coordinates of the driver's eyes in real time and dynamically, thereby completely solving this problem. In other words, it calculates the azimuth angle of obstacles using the driver's line-of-sight coordinate system. With pitch angle .

[0097] One possible implementation method includes:

[0098] S411. Establish the vehicle coordinate system, eye tracker coordinate system, and driver line of sight coordinate system.

[0099] The first step is to establish all the necessary coordinate systems, including:

[0100] Vehicle coordinate system: Origin: Vehicle center of mass. Axes: X-axis forward, Y-axis to the right, Z-axis upward. Function: A unified reference for data from all sensors (radar, eye tracker).

[0101] Eye tracker coordinate system: Origin: The optical center of the eye tracker camera. Function: The data directly output by the eye tracker is based on its own coordinate system.

[0102] Driving line-of-sight coordinate system: Origin: Midpoint of the driver's eyes (or center of a single eye). Function: This is the target coordinate system of this invention; all information ultimately presented to the driver must be transformed into this coordinate system.

[0103] S412. Calculate coordinate system transformation, including:

[0104] Suppose that the radar has detected an object and obtained its coordinates in the vehicle coordinate system, P_vehicle = (x_v, y_v, z_v).

[0105] Coordinate system transformation 1: From eye tracker data to driver's eye coordinates, this is the starting point and foundation of the entire process.

[0106] Eye tracker output: Modern eye trackers not only output the driver's gaze point on the screen, but also calculate the three-dimensional coordinates of the eyes in the eye tracker coordinate system, P_eye_tracker = (x_et, y_et, z_et), using binocular parallax. The static transformation matrix T_et_to_v from the eye tracker coordinate system to the vehicle coordinate system is calculated. This matrix contains the position and orientation of the eye tracker within the vehicle. The vehicle coordinates of the eyes are calculated: P_eye_vehicle = T_et_to_v * P_eye_tracker. The real-time, dynamic coordinates of the driver's eyes in the vehicle coordinate system are obtained, P_eye_vehicle = (x_eye, y_eye, z_eye).

[0107] Coordinate system transformation 2: Relative coordinates from the object to the driver's eye.

[0108] The object's position in the vehicle coordinate system: P_vehicle = (x_v, y_v, z_v). The driver's eye position in the vehicle coordinate system: P_eye_vehicle = (x_eye, y_eye, z_eye). The object's position vector relative to the driver's eye, P_relative, is: P_relative = P_vehicle - P_eye_vehicle = (x_v - x_eye, y_v - y_eye, z_v - z_eye). This P_relative vector represents the object's position in a coordinate system with the driver's eye as the origin, but whose axes remain parallel to the vehicle coordinate system.

[0109] Coordinate system transformation 3: From relative coordinates to azimuth and elevation angles.

[0110] Angles can be directly calculated using P_relative = (x_rel, y_rel, z_rel). Here, we need to clarify the coordinate axis definitions: Assume y_rel is the longitudinal distance (front-to-back) of the object relative to the driver. Assume x_rel is the lateral distance (left-to-right) of the object relative to the driver. Assume z_rel is the vertical distance (up-down) of the object relative to the driver. Calculation formulas: Azimuth: Azimuth = arctan(x_rel / y_rel). Pitch: Elevation = arctan(z_rel / y_rel).

[0111] Furthermore, coordinate system transformation 4: consider the driver's head posture.

[0112] Eye trackers can not only track eye position, but high-end models can also track head posture. If the driver's head turns 20 degrees to the left, then his "directly in front" is no longer the X-axis direction of the vehicle.

[0113] At this point, the P_relative vector needs to be rotated in reverse before calculating the angles. The eye tracker outputs the head rotation angles (e.g., yaw, pitch, roll). Construct a reverse rotation matrix R_head_inverse. Rotate the relative coordinates: P_relative_aligned = R_head_inverse * P_relative, and use the new coordinates (x', y', z') of P_relative_aligned to calculate the final azimuth and pitch angles. This step makes the display of obstacle positions more "human-centered," allowing the system to understand the driver's line of sight even when looking in the rearview mirror.

[0114] Eye trackers enable a leap from static to dynamic models. The angles calculated in this way can be precisely aligned with what the driver sees.

[0115] S42. Based on the obstacle's azimuth angle The difference between the obstacle and the midpoint of the horizontal boundary azimuth range is used to map the obstacle to its corresponding horizontal partition using the horizontal mapping partitioning rules.

[0116] Calculate the azimuth angle of the obstacle Midpoint of the horizontal boundary of the target partition The difference This allows us to determine the horizontal zone to which the obstacle belongs. The specific calculation process is as follows:

[0117] The azimuth angle reflects the position of the obstacle cluster center in the horizontal direction (left and right). It is calculated based on the horizontal component in the obstacle's three-dimensional coordinates and is used to determine whether it belongs to the left, center, or right horizontal zone in the field of view.

[0118] Furthermore, the specific process for defining the horizontal partition where the obstacle is located is as follows:

[0119] Horizontal mapping partitioning rules: Calculate the azimuth angle of obstacles Midpoint of the horizontal boundary of the target partition The difference ,like If so, then determine the corresponding zone to which the obstacle belongs; if If α is at the boundary between two partitions, then the parent node containing the obstacle point cloud is located, and recursive splitting is performed on the parent node to extract all child nodes containing the obstacle. The proportion of point cloud in each child node is counted, and the partition with the largest proportion of valid point cloud is assigned to that partition.

[0120] S43. Based on the obstacle's pitch angle The difference between the obstacle and the midpoint of the vertical boundary pitch angle range is used to map the obstacle to its corresponding vertical partition using the vertical mapping partitioning rules.

[0121] Pitch angle of the obstacle Midpoint of the vertical boundary of the target partition The difference This allows us to obtain the vertical partition to which the obstacle belongs. The specific calculation process is as follows:

[0122] The pitch angle reflects the position of the obstacle cluster center in the vertical direction (up and down). It is calculated based on the vertical component in three-dimensional coordinates and is used to determine whether it belongs to the upper, middle, or lower vertical partition in the field of view.

[0123] Vertical mapping partitioning rules: Calculate the pitch angle of obstacles Midpoint of the vertical boundary of the target partition The difference ,like At that time, and If so, then the obstacle is determined to be the corresponding zone; if Sometimes If the partition is excluded, the matching continues to other partitions; if it is at the intersection of multiple partitions, the parent node containing the point cloud of the obstacle is located, the parent node is recursively split, all child nodes containing the obstacle are extracted, the point cloud quantity ratio of each child node is counted, and the partition with the largest effective point cloud ratio is assigned to that partition.

[0124] S44. Merge the horizontal and vertical zones to which the obstacle belongs, and map the extracted obstacle to the zone to which it belongs within the driver's line of sight to obtain the obstacle zone.

[0125] Based on the horizontal and vertical zones to which the obstacle belongs, such as the horizontal zone being the left zone and the vertical zone being the middle zone, the extracted obstacle is mapped to the left-middle zone within the driver's line of sight. The resulting obstacle zone is the left-middle zone, and so on.

[0126] S5. Calculate the spatial deviation angle between the driver's gaze point and the obstacle, as well as the time-decayed attention intensity, specifically including:

[0127] S51. Calculate the spatial deviation angle between the driver's gaze point and the obstacle.

[0128] The unit direction vector of the gaze point in the current frame extracted from the eye tracker data. and the unit direction vector of obstacles in radar data Calculate the spatial deviation angle between the gaze point and the obstacle's normalized coordinates in the partition. The expression is as follows:

[0129] ;

[0130] in, Number the driving line of sight zones. For timestamps.

[0131] S52. Calculate the instant matching degree.

[0132] Then, the matching degree is calculated instantly: the deviation angle is converted into the matching degree through an exponential mapping function.

[0133] ;

[0134] in, (Generally taken as 0.05–0.2) is the angle sensitivity coefficient, which controls the degree of influence of angle changes on the matching degree.

[0135] Objective: Since "angle" is a geometric quantity, it is not convenient to use it directly as an input feature of a neural network. Therefore, the system uses an exponential function to convert the angle into a "matching degree" value between 0 and 1.

[0136] S52. Calculate the time-decayed attention intensity:

[0137] Driver attention intensity is calculated using a recursive time decay equation:

[0138] ;

[0139] in, Let be the time decay coefficient, and let the initial attention value be . , This reflects the driver's level of continuous attention to the obstacle area at any given moment.

[0140] Calculated The value, as one of the higher-order features, is passed along with features such as obstacle distance, point cloud density, and mutation intensity to the Knowledge Augmentation Neural Network (KAN) for joint risk modeling and comprehensive assessment.

[0141] This design enables neural networks to adaptively learn the nonlinear relationship between attention and risk, avoiding robustness issues caused by manually set thresholds.

[0142] S6. Evaluate the mutation intensity of the obstacle. If the mutation intensity is greater than the mutation threshold, proceed to S9; otherwise, proceed to S7.

[0143] To characterize the dynamic changes of obstacles, this invention calculates the abrupt change intensity of radar point cloud data at adjacent time points after obstacle extraction. This index reflects abnormal fluctuations in the distance or shape of obstacles relative to vehicles and is an important measure of environmental dynamics.

[0144] set up This represents the Euclidean distance between the obstacle and the vehicle in the current frame. This represents the number of point clouds representing obstacle clusters in the current frame. Let be the standard deviation of the distance between the obstacle and the vehicle within the sliding time window. Let be the standard deviation of the number of obstacle point clouds within the sliding time window. To prevent constants with a denominator of 0.

[0145] Mutation strength for: ;

[0146] when When the threshold is 2.0, it indicates that the obstacle has changed significantly or is approaching within a short period of time. The system marks the obstacle and its associated zone as a "dynamic high-risk object," and then executes S9. As input to the higher-order feature model, this is used in subsequent fusion models. That is, when When the condition is high, it indicates a high-risk state, and there is no need to wait for subsequent attention feature judgment; the process can directly proceed to the KAN risk assessment module.

[0147] By calculating mutation intensity indices in advance, the dynamic properties of obstacles can be obtained before entering the risk assessment stage, enabling earlier risk identification and response triggering.

[0148] S7. Calculate the obstacle zone risk value using the judgment thresholds of spatial deviation angle, time decay attention intensity, and abrupt change intensity:

[0149] Calculate the normalized mutation strength score , , This is the score for the maximum possible mutation intensity in the scene. Normalized As a risk amplification factor, it matches the deviation angle. and time decay of attention intensity Jointly calculate partition risk values:

[0150] ;

[0151] when When the threshold is typically set to 0.2-0.3, the partition is considered a dangerous area; otherwise, the partition is considered safe and the corresponding point cloud is not entered into the subsequent neural network.

[0152] S8. Determine whether the risk value of the obstacle area is less than the safety threshold. If yes, the obstacle area is in a safe state and no warning is issued. Otherwise, the obstacle area is in a dangerous state and S9 is executed.

[0153] S9. Input the obstacle area and related obstacle information into the knowledge enhancement neural network to perform intelligent multi-level response early warning for the obstacle area.

[0154] To further improve the system's risk identification accuracy and response capability in complex road and factory environments, this invention extends the traditional alarm mechanism into a comprehensive risk assessment and response control process that integrates high-order feature extraction, risk distribution modeling, and intelligent neural network decision-making, including:

[0155] S91. Perform high-level feature extraction and sliding window statistics on obstacle areas and related obstacle information.

[0156] Higher-order feature extraction: Within the time window, feature vectors are formed.

[0157]

[0158] in: Distance between obstacles and vehicles; Point cloud density; Obstacle azimuth; Obstacle pitch angle; Spatial deviation angle; Time decays attention intensity; Mutation intensity score (calculated from distance and point cloud change rate).

[0159] The KAN network introduces a Statistical Feature Modulation (Stat-FiLM) layer in the feature input layer, which maps the distribution parameters to channel modulation coefficients and adaptively weights the input features.

[0160] ;

[0161] ;

[0162] in: For the first The original features of each input channel , This represents the modulation coefficient of the channel. , For input features Calculate statistics (such as the mean) ,variance ).

[0163] Pay attention to class features ( During modulation, these regions are given higher weights to enhance risk response in areas of low driver attention, achieving adaptive identification of "low attention-high risk" regions. This involves applying an additional weight amplification factor during modulation. This enhances the response under low attention conditions.

[0164] ;

[0165] in, Used to enhance risk perception in areas not noticed by the driver.

[0166] The fusion layer can be viewed as a multi-layer nonlinear mapping function. :

[0167] S92. Outputting a comprehensive risk score via the network. :

[0168] ;

[0169] in: The feature vector is modulated by the Stat-FiLM layer; These are the parameters of the KAN network.

[0170] Output Comprehensive risk score. The network outputs a comprehensive risk score. , range [0,1].

[0171] S93. Develop intelligent multi-level response early warning for obstacle areas based on comprehensive risk scores.

[0172] Based on risk score The numerical range is used to implement an adaptive alarm strategy, as shown in Table 1 below:

[0173] Table 1: Comparison Table of Risk Scoring and Alarm Strategies

[0174]

[0175] The alarms should be classified according to their urgency level, following the principle of "from weak to strong," to avoid excessively disturbing the driver when the danger is not significant.

[0176] Alarms can be divided into visual alarms and auditory alarms, among which:

[0177] Visual Alarm: Photoelectric warning devices are installed around the perimeter of the vehicle's interior windshields (the left, right, and rearview mirrors can also be used as separate zones). These photoelectric warning devices, such as RGB smart light strips, intelligently control and alert the obstruction zone using different colored light strips along its outline. For example, if the obstruction zone is the upper left zone, the light strips on the left and top edges of the upper left zone's outline will flash; if it's the middle left zone, the light strip on the left side of the zone will flash; if it's the lower left zone, the light strips on the left and bottom edges of the zone will flash, and so on. This is used in conjunction with audio prompts to achieve precise zoned photoelectric warnings in this invention. On the dashboard or central control screen, a static, soft icon (such as a gray or white vehicle / radar icon) displays the approximate location and distance of the obstacle.

[0178] Low risk level: Simply alert the driver to the presence of a potential but non-emergency obstacle.

[0179] Visual warning: A yellow photoelectric outline is used to warn of obstacles on the edge of the corresponding zone inside the vehicle's windshield. Alternatively, a static, soft icon (such as a gray or white vehicle / radar icon) can be used on the instrument panel or center console to display the approximate location and distance of the obstacle. Auditory warning: A very faint beeping sound.

[0180] Medium risk level: Clearly inform the driver that the risk is escalating and that they need to pay immediate attention and prepare to take action.

[0181] Visual warning: An orange photoelectric outline is used to warn of obstacles on the edge of the corresponding zone inside the vehicle's windshield. Alternatively, text such as "Obstacle" or "Caution" can be displayed on the instrument panel or center console screen. Auditory warning: An intermittent, medium-volume "beep" alarm is emitted. The frequency of the alarm may increase as the distance decreases.

[0182] High-risk level: Issues the strongest warning, prompting the driver to take immediate evasive action and making final preparations for the intervention of Automatic Emergency Braking (AEB).

[0183] Visual alarm: A highly visible red photoelectric outline is used to warn of obstacles. The background of the entire dashboard or center console screen may turn red momentarily. Auditory alarm: A continuous, rapid, high-volume alarm sound is emitted.

[0184] Furthermore, drivers can customize tactile alerts: Steering wheel vibration: A vibration motor within the steering wheel alerts the driver. Seat vibration: Vibration on the left or right side of the seat indicates the direction from which an obstacle is coming, providing a very intuitive feel. Brake pedal pulse / vibration: The system gently vibrates the brake pedal, simulating a "foot feel," strongly implying "Step on the brakes!"

[0185] Furthermore, for high-risk situations, the system can employ intelligent automatic braking or steering avoidance to prevent accidents caused by the driver's delayed reaction and ensure safety.

[0186] Through this mechanism, the system can respond to potential dangers in real time and in a tiered manner, taking into account both safety and driving experience.

[0187] Based on the same obstacle warning scenario, the comparison chart of the number of alarms using the warning method of this invention compared to the traditional method using only radar point cloud data is shown below. Figure 2 As shown, Figure 2 This is a comparison image of results taken every 5 frames within frames 0-232. Figure 2 As can be seen, the solid line represents the number of alarms from the traditional radar point cloud algorithm, while the dashed line represents the number of alarms from the radar fusion eye-tracking data algorithm. Within the frame sequence range covered by the experiment, the number of radar point cloud alarms fluctuated significantly overall, reaching 18 at point a. In contrast, the number of alarms from the radar fusion eye-tracking data algorithm was relatively more stable and lower overall. Furthermore, the radar fusion eye-tracking data algorithm achieved redundancy-free alarms; at point b, the radar point cloud alarms occurred 12 times, while the radar fusion eye-tracking data algorithm triggered 0 alarms. Compared to traditional radar point cloud alarms, it provides more accurate obstacle warnings, reduces unnecessary alarm interference, and improves the accuracy and practicality of the warnings.

[0188] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal fusion-based intelligent obstacle early warning method based on machine learning, characterized in that, It includes the following steps: S1. Divide the vehicle's driving line of sight into zones, and calculate the horizontal boundary azimuth angle range and vertical boundary pitch angle range of each zone. S2. Real-time acquisition of vehicle eye tracker data and radar point cloud data; the eye tracker data is used to provide driver gaze attention information, and the radar point cloud data is used to provide spatial information about the vehicle's surroundings. S3. Extract obstacles and their distances using real-time acquired radar point cloud data: S4. Calculate the azimuth and pitch angles of the obstacles based on the obstacle information, and map the obstacles to the corresponding zones within the driver's line of sight by using the horizontal boundary azimuth angle range and the vertical boundary pitch angle range of each zone, thus obtaining the obstacle zone. S5. Calculate the spatial deviation angle between the driver's gaze point and the obstacle, as well as the time-decayed attention intensity. The spatial deviation angle between the driver's gaze point and the obstacle is: : ； in, Number the driving line of sight zones. For timestamps, This represents the unit direction vector of the gaze point in the current frame of the eye tracker data. This refers to the unit direction vector of obstacles in radar data; S6. Evaluate the mutation intensity of the obstacle. If the mutation intensity is greater than the mutation threshold, proceed to S9; otherwise, proceed to S7. S7. Calculate the obstacle zone risk value using the obtained spatial deviation angle, time-decayed attention intensity, and obstacle abrupt change intensity: S8. Determine whether the risk value of the obstacle area is within the safety threshold. If yes, the obstacle area is in a safe state and no warning is issued. Otherwise, the obstacle area is in a dangerous state and S9 is executed. S9. Input the obstacle area and related obstacle information into the knowledge enhancement neural network to perform intelligent multi-level response early warning for the obstacle area.

2. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 1, characterized in that, The S1 classifies the vehicle's driving line of sight into coarse-grained and fine-grained zones based on different driving scenarios.

3. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 1, characterized in that, S3 uses real-time acquired radar point cloud data to extract obstacles, specifically including: The obtained radar point cloud data is recursively divided into sub-nodes using a spatial segmentation algorithm until the point cloud density of each sub-node meets the threshold for the clustering algorithm to identify obstacles. The sub-nodes containing obstacle information are denoised by dynamically adjusting the spatial radius and minimum point number threshold in the clustering algorithm, and the obstacles are extracted.

4. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 2, characterized in that, S4. Calculate the azimuth and pitch angles of obstacles based on obstacle information, and map the obstacles to their respective zones within the driver's line of sight using the horizontal boundary azimuth angle range and the vertical boundary pitch angle range of each zone, thus obtaining the obstacle zones, specifically including: S41. Calculate the obstacle azimuth angles based on the extracted obstacle information. With pitch angle ; S42. Based on the obstacle's azimuth angle The difference between the obstacle and the midpoint of the horizontal boundary azimuth range is used to map the obstacle to its corresponding horizontal partition using the horizontal mapping partitioning rules; S43. Based on the obstacle's pitch angle The difference between the obstacle and the midpoint of the vertical boundary pitch angle range is used to map the obstacle to its corresponding vertical partition using the vertical mapping partitioning rules; S44. Merge the horizontal and vertical zones to which the obstacle belongs, and map the extracted obstacle to the zone to which it belongs within the driver's line of sight to obtain the obstacle zone.

5. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 4, characterized in that, The horizontal mapping partitioning rules in S42 include: If the azimuth angle of the obstacle The difference between the midpoint of the azimuth range of the horizontal boundary When this happens, the obstacle is determined to belong to the corresponding zone; like If the partition is not found, then exclude that partition and continue matching other partitions; like If a node is located at the boundary of two partitions, the parent node containing the point cloud of the obstacle is located. The parent node is recursively split, and all child nodes containing the obstacle are extracted. The proportion of the point cloud of each child node is calculated, and the node with the largest proportion of valid point cloud is assigned to that partition.

6. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 5, characterized in that, The vertical mapping partitioning rules in S43 include: If the obstacle's pitch angle The difference between the midpoint of the pitch angle range and the vertical boundary When this happens, the obstacle is determined to belong to the corresponding zone; like If the partition is not found, then exclude that partition and continue matching other partitions; like When the obstacle is located at the boundary of multiple partitions, the parent node containing the point cloud of the obstacle is located, and the parent node is recursively split. All child nodes containing the obstacle are extracted, and the proportion of the point cloud of each child node is counted. The node with the largest proportion of valid point cloud is assigned to that partition.

7. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 1, characterized in that, S5. Calculate the spatial deviation angle between the driver's gaze point and the obstacle, as well as the time-decayed attention intensity, specifically including: S51. Calculate the spatial deviation angle between the driver's gaze point and the obstacle. ; ； in, Number the driving line of sight zones. For timestamps, This represents the unit direction vector of the gaze point in the current frame of the eye tracker data. This refers to the unit direction vector of obstacles in radar data; S52. Calculate the instant matching degree ; ； in, , is the angle sensitivity coefficient; S52, Calculate the time-decayed attention intensity : ； in, The time decay coefficient, It reflects the driver's level of continuous attention to the obstacle area at any given moment.

8. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 1, characterized in that, S6 obstacle mutation intensity for: ； in, This represents the Euclidean distance between the obstacle and the vehicle in the current frame. This represents the number of point clouds representing obstacle clusters in the current frame. Let be the standard deviation of the distance between the obstacle and the vehicle within the sliding time window. Let be the standard deviation of the number of obstacle point clouds within the sliding time window. To prevent constants with a denominator of 0.

9. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 1, characterized in that, Risk value of obstacle zone in S8 for: ； in, This is the normalized mutation intensity score.

10. The multimodal fusion obstacle intelligent early warning method based on machine learning according to claim 1, characterized in that, S9. Input the obstacle area and related obstacle information into the knowledge-enhancing neural network to perform intelligent multi-level response early warning for the obstacle area, specifically including: S91. Perform high-order feature extraction and sliding window statistics on obstacle areas and related obstacle information; S92. Outputting a comprehensive risk score via the network. : ； in: t The feature vector is the result of modulation by the Stat-FiLM layer. For model network parameters, It is a multi-layer nonlinear mapping function; S93. Develop intelligent multi-level response early warning for obstacle areas based on comprehensive risk scores.

Images