A real-time monitoring method and system based on layer and operator layer fusion

Abstract

The present invention discloses a real-time monitoring method and system based on the fusion of layers and operator layers, which belongs to the field of autonomous driving and aims to solve the problem of how to achieve accurate environmental perception, real-time monitoring and safe automatic parking by efficiently integrating vehicle-mounted sensor data. The original data of the vehicle's surrounding environment is collected by vehicle-mounted sensors, and the original data is preprocessed to establish three-dimensional point cloud data, feature recognition is performed based on the three-dimensional point cloud data, a feature layer is constructed based on the result of feature recognition, and the operator layer is applied to extract and integrate the layer information to construct a feature set, divide the target planning area according to the feature set, plan a preset path in the target planning area, determine the parking path through user interaction, generate a parking instruction based on the parking path, control the vehicle's movement, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and more specifically to a real-time monitoring method and system based on the fusion of layers and operator layers. Background Art

[0002] With the continuous development of autonomous driving technology, automatic parking, as an important component, has received widespread attention. The automatic parking system can assist or completely replace the driver to complete the parking operation of the vehicle, improve parking efficiency, and reduce traffic accidents caused by improper parking. It also provides convenience for drivers who are not good at parking or have physical limitations. However, in actual applications, the automatic parking system faces complex environmental challenges, such as changing road conditions, different types of parking spaces, and the presence of dynamic obstacles around. Traditional automatic parking systems usually rely on on-board sensors (such as ultrasonic radar, cameras, etc.) to perceive the surrounding environment and achieve precise control of the vehicle through control algorithms. However, these systems often have some limitations, such as insufficient perception capabilities in complex environments, poor adaptability to different types of parking spaces, and slow processing speeds.

[0003] The invention patent with announcement number CN112180373B discloses a multi-sensor fusion intelligent parking system, including: a sensor subsystem and a signal processing subsystem; the sensor subsystem is arranged around the vehicle and is used to collect environmental information around the vehicle; the signal processing subsystem performs information fusion based on the environmental information provided by different sensor subsystems, thereby determining the spatial information and position information of parking spaces of different shapes, and determining autonomous parking path planning.

[0004] Although the existing technology provides a safer guarantee information for the automatic driving function in the parking scene, it still fails to solve the problem of how to achieve accurate environmental perception, real-time monitoring and safe automatic parking by efficiently integrating the on-board sensor data. Therefore, in order to overcome these limitations, the present invention proposes a real-time monitoring method and system based on the fusion of layer and operator layer. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a real-time monitoring method and system based on the fusion of layers and operator layers, which solves the problem of how to achieve accurate environmental perception, real-time monitoring and safe automatic parking by efficiently integrating vehicle-mounted sensor data.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A real-time monitoring system based on the fusion of graph and operator layers, including an environment perception module, a graph and operator fusion module, a decision-making and planning module, and a control execution module;

[0008] Specifically, the environment perception module is used to collect raw data of the vehicle's surrounding environment through on-board sensors, and pre-process the raw data to establish three-dimensional point cloud data;

[0009] Specifically, the image-computation fusion module is used to perform feature recognition based on the three-dimensional point cloud data, construct a feature layer based on the result of feature recognition, and apply the operator layer to extract and integrate the layer information to construct a feature set;

[0010] Specifically, the decision planning module is used to divide the target planning area according to the feature set, plan a preset path in the target planning area, and determine the parking path through user interaction;

[0011] Specifically, the execution control module is used to generate a parking instruction according to the parking path, control the vehicle's movement, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking.

[0012] Specifically, the specific steps of constructing the point cloud data include:

[0013] Preprocess the collected raw data, including denoising, standardization and enhancement;

[0014] Perform data registration on the vehicle-mounted sensor and align it in the same coordinate system;

[0015] Extract features from the registered data points and perform feature data association;

[0016] The data fusion algorithm is used to integrate the data from different sensors, and point cloud data is generated based on the fused data.

[0017] Specifically, the specific steps of the data registration include:

[0018] Establish the objective function and set it as the sum of the squares of the point-to-point distances and the sum of the regularization terms, that is:

[0019] ,in, is the objective function, which represents the sum of the distances between the source data points and the target data points. is the rotation matrix, is the translation vector, and are the corresponding point pairs in the source data points and the target data points, respectively. is the regularization parameter, It is about the rotation matrix The regularization term of ;

[0020] The objective function is iteratively optimized, and the rotation matrix and translation vector are updated according to the gradient of the objective function. The update formula is as follows:

[0021] , ,in, and is the learning rate, and They are The rotation matrix and translation vector of the iteration, and are the updated rotation matrix and translation vector respectively.

[0022] Configuring the Convergence Threshold When the change of the objective function is less than the convergence threshold, the optimization is stopped and the optimal rotation matrix and translation vector, as well as the registered data points, are output. The convergence judgment formula is: ,in, is a small positive threshold, is the objective function of the kth iteration, is the objective function of the k+1th iteration.

[0023] Specifically, the specific steps of constructing the feature set include:

[0024] The layers are divided into static layers and dynamic layers; the features of the static layers and dynamic layers are extracted through the operator layer, including: boundary feature recognition of the static layer; trajectory analysis of the dynamic layer; and feature sets are constructed based on the information extracted from the operator layer features.

[0025] Specifically, the specific steps of the boundary feature recognition include:

[0026] For each point cloud, configure the neighborhood range , calculate the number of point clouds in the neighborhood, and then calculate the local probability density function of the point cloud. , the probability density function is calculated as follows:

[0027] ,in, is the kernel function, are other points in the neighborhood, is the number of points in the neighborhood, is the smoothing parameter;

[0028] Derivate the probability density function of the point cloud, obtain the density gradient, and configure the gradient threshold , when the modulus of the density gradient is greater than When , mark the point as an edge point;

[0029] Boundary features are extracted from edge points, including coordinates, normal vectors, and curvature. The curvature calculation formula is:

[0030] ,in, and are the coordinates of the edge points, and yes and The first derivative of and yes and The second derivative of .

[0031] Specifically, the specific steps of the boundary feature recognition also include:

[0032] Based on the edge point coordinates, the noise points are identified and the local density of each edge point is calculated by the sum of the inverse distances. The local density is calculated as follows:

[0033] ,in, It is the calculation point cloud The number of edge points of the local density, It is a point cloud No. The nearest edge point, is the distance calculation formula;

[0034] The local outlier factor of each edge point is calculated based on the local density, that is:

[0035] ,in, yes The neighbor set of yes The local density of Representing a collection The modulus of , which is the number of elements in the set;

[0036] Configuring the Noise Threshold , the local outlier factor value is greater than the noise threshold The edge points are marked as noise points and deleted.

[0037] Specifically, the specific steps of the trajectory analysis include:

[0038] Extract dynamic objects from dynamic layers and initialize dynamic object collections ,in, It is The state of a dynamic object, including the state information of the dynamic object, is the corresponding initial weight, is the number of initialized dynamic objects;

[0039] For each dynamic object, a tracking window is configured to record its position in consecutive frames and predict the position of the dynamic object in the next frame, that is:

[0040] ,in, is the state transition function, is the control input, is the process noise, and They are The first dynamic object Frame and The state of the frame;

[0041] Update weights and calculate the weight of each dynamic object:

[0042] ,in, is the observation model, which is used to express the probability of observation in a given state;

[0043] Normalize the weights, that is , resamples the dynamic objects according to the normalized weights.

[0044] Specifically, the specific steps of the trajectory analysis also include:

[0045] A real-time estimate of the environment is performed using a weighted set of dynamic objects, namely:

[0046] ,in, is the state estimate of the set of dynamic objects;

[0047] Compute the error covariance of the estimated state , which is used to quantify the accuracy of the estimate, is calculated as follows:

[0048]

[0049] Configuring Error Threshold , used to determine the risk level of the estimated state, when If the error is greater than the error threshold, the estimate is determined to be at the first risk level, otherwise it is determined to be at the second risk level.

[0050] Specifically, the specific steps of the preset path planning include:

[0051] Plan the drivable area of ​​the preset path, extract the boundary features of the boundary points from the static layer, and divide the drivable area according to the boundary features;

[0052] Configuring Adjustment Thresholds , used to adjust the width of the dynamic object's motion trajectory. When the risk level of the dynamic object's motion trajectory is level 1, the width of the dynamic object's motion trajectory is , when the risk level of the motion trajectory of the dynamic object is the second risk level, the width of the motion trajectory of the dynamic object is C;

[0053] Plan the prohibited driving area of ​​the preset path, obtain the motion trajectory of the dynamic object in the real-time dynamic layer, and divide the prohibited driving area according to the motion trajectory of the dynamic object and its width;

[0054] The non-prohibited driving area in the drivable area is obtained as the target planning area, and a preset path is planned in the target planning area by using a path planning algorithm.

[0055] Specifically, the specific steps of generating the parking instruction and controlling the vehicle movement include:

[0056] receiving a parking path, serializing coordinate points in the parking path, and converting them into an instruction set for the vehicle to move;

[0057] Generate vehicle control instructions, including accelerator, brake, and steering wheel angle, based on the sequence of coordinate points in the parking path;

[0058] Send the parking command to the actuator, including the electric power steering system and the braking system, to control the vehicle to drive according to the command;

[0059] When the vehicle is moving, the surrounding environment is monitored based on real-time vehicle sensor data. When an abnormality is detected at a coordinate point in the parking path, an emergency braking command is issued.

[0060] A real-time monitoring method based on layer and operator layer fusion includes the following steps:

[0061] Step S1: collecting raw data of the vehicle's surrounding environment through on-board sensors, and preprocessing the raw data to create three-dimensional point cloud data;

[0062] Step S2: performing feature recognition based on the three-dimensional point cloud data, constructing a feature layer based on the result of feature recognition, and applying an operator layer to extract and integrate the layer information to construct a feature set;

[0063] Step S3: dividing the target planning area according to the feature set, planning a preset path in the target planning area, and determining the parking path through user interaction;

[0064] Step S4: Generate a parking instruction according to the parking path, control the vehicle's movement, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking.

[0065] Beneficial effects of the present invention:

[0066] 1. By dividing the environmental information into layers, it is possible to more accurately capture the details around the vehicle, such as obstacles, pedestrians, etc., thus providing a reliable data basis for automatic parking. The distinction between static layers and dynamic layers enables the processing of information on stationary objects and moving objects separately, improving the speed and accuracy of data processing.

[0067] 2. Use the operator layer to extract and integrate the layer information and build a comprehensive set of environmental features, so that the decision-making module can make more reasonable and accurate plans in complex environments. It not only improves the accuracy of environmental perception, but also provides rich and structured information for the decision-making planning module.

[0068] 3. The execution control module generates specific driving instructions based on the planned parking path, monitors the actual movement of the vehicle, and immediately takes braking measures in case of abnormalities, thus ensuring the safety of the automatic parking process to the greatest extent. At the same time, users can confirm or adjust the parking path through the interactive interface, meeting personalized needs and enhancing user participation and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a structural diagram of a real-time monitoring system based on the fusion of layer and operator layer;

[0070] Figure 2 Flowchart of the specific steps for feature set construction;

[0071] Figure 3 A flowchart of the specific steps for boundary feature identification;

[0072] Figure 4 A flowchart of the specific steps of trajectory analysis;

[0073] Figure 5 It is a flow chart of a real-time monitoring method based on the fusion of layer and operator layer. DETAILED DESCRIPTION

[0074] Example 1

[0075] See also Figure 1 ,This embodiment introduces a real-time monitoring system based on the fusion of graph and operator layers, including an environment perception module, a graph and operator fusion module, a decision-making and planning module, and a control execution module;

[0076] Specifically, the environment perception module is used to collect raw data of the vehicle's surrounding environment through on-board sensors, and pre-process the raw data to establish three-dimensional point cloud data;

[0077] Specifically, the image-computation fusion module is used to perform feature recognition based on 3D point cloud data, construct feature layers based on the results of feature recognition, and apply operator layers to extract and integrate layer information to construct feature sets;

[0078] Specifically, the decision planning module is used to divide the target planning area according to the feature set, plan a preset path in the target planning area, and determine the parking path through user interaction;

[0079] Specifically, the execution control module is used to generate a parking instruction according to the parking path, control the vehicle's movement, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking.

[0080] In this embodiment, the vehicle collects raw data of the surrounding environment through on-board sensors, including cameras, radars, and lidars. The raw data includes other parked vehicles, pedestrians, road markings, obstacles, etc. The environment perception module preprocesses the raw data, including denoising, standardization, and enhancement, in preparation for subsequent processing. The preprocessed data is input into the image calculation fusion module, and multiple layers are constructed according to the extracted environmental features. For example, one layer specifically displays other objects around the vehicle, that is, the obstacle layer, and another layer displays the ground markings, that is, the road layer. Then, the corresponding operator layer is applied to each layer. In this way, the information of different layers is integrated to construct a comprehensive set of environmental features. Using the feature set obtained from the image calculation fusion module, the decision planning module considers the current position of the vehicle, the position of the target parking space, and the surrounding environmental factors, divides the target planning area, and plans a parking path in the target planning area. In addition, the user interaction function allows the driver to confirm or adjust the recommended parking path through the vehicle interface. Once the path is determined, the decision planning module generates a series of detailed parking instructions. After receiving the parking instruction, the execution control module is responsible for accurately controlling the vehicle to move along the planned path. The system adjusts the response of the throttle, brake and steering wheel to ensure that the vehicle is smoothly driven into the parking space. At the same time, the safety guarantee mechanism continuously monitors the vehicle status and surrounding environment so that it can take immediate action, such as emergency braking, when encountering any potential danger, thus ensuring the safety of the entire automatic parking process.

[0081] Specifically, the specific steps of constructing point cloud data include:

[0082] The collected raw data is preprocessed, including denoising, standardization and enhancement; the data from the on-board sensors is registered and aligned in the same coordinate system; features are extracted from the registered data points and feature data is associated; data fusion algorithms are used to integrate data from different sensors, and point cloud data is generated based on the fused data.

[0083] In this embodiment, during the driving process of the vehicle, the on-board sensor collects environmental data in real time. All raw data are timestamped, and the collected raw data are preprocessed, including using statistical filtering methods to remove outliers for laser radar data; median filtering for camera images to reduce noise; and signal processing technology to eliminate interference for millimeter-wave radar data. Contrast enhancement and edge sharpening are performed on camera images to improve image quality; interpolation processing is performed on laser radar data to increase point cloud density. Data from different on-board sensors are registered and aligned in the same coordinate system. Laser radar is used as the main sensor, and the data of the camera and millimeter-wave radar are registered to the coordinate system of the laser radar. Features are extracted from the registered data points, and feature data association is performed. For example, the three-dimensional coordinates of the points are extracted from the laser radar data; color information is extracted from the camera image; and speed and distance information are extracted from the millimeter-wave radar. The feature data of different sensors are matched according to the timestamp to ensure that each point cloud data point contains complete feature information with the same timestamp. The data fusion algorithm is used to integrate the data from different sensors, and the Kalman filter is used to fuse the data from different sensors to obtain the optimal estimate of each point. The fused data is converted into a point cloud format, such as PCD or PLY format supported by PCL. Each point in the point cloud contains its coordinates, color, intensity and other information in three-dimensional space.

[0084] Specifically, the specific steps of data registration include:

[0085] Establish an objective function to measure the degree of registration between different sensor data. Set the objective function to the sum of the squares of the sum of the point-to-point distances and the sum of the regularization terms to prevent overfitting of the function, that is:

[0086] ,in, is the objective function, which represents the sum of the distances between the source data points and the target data points. is the rotation matrix, which is used to describe the rotation transformation from the source data point to the target data point. is a translation vector, which is used to describe the translation transformation from the source data point to the target data point. and are the corresponding point pairs in the source data points and the target data points, respectively. is the regularization parameter, which is used to control the weight of the regularization term; It is about the rotation matrix The regularization term of ;

[0087] The objective function is iteratively optimized. In each iteration, the rotation matrix and translation vector are updated according to the gradient of the objective function. The update formula is as follows:

[0088] , ,in, and is the learning rate, and They are The rotation matrix and translation vector of the iteration, and are the updated rotation matrix and translation vector respectively.

[0089] Configuring the Convergence Threshold When the change of the objective function is less than the convergence threshold, the optimization is stopped and the optimal rotation matrix and translation vector, as well as the registered data points, are output. The convergence judgment formula is:

[0090] ,in, is a small positive threshold, is the objective function of the k+1th iteration.

[0091] Specifically, the specific steps of layer construction include:

[0092] Downsample the point cloud data to reduce the point cloud density; perform spatial clustering on the point cloud data to form point groups, and use deep learning to identify targets on the point groups to classify environmental object types, which include obstacles, pedestrians, road markings, parking spaces, and traffic lights; divide the layers according to the environmental object types.

[0093] In this embodiment, the point cloud data is downsampled to reduce the point cloud density. The voxel filtering method is used to downsample the original point cloud data to reduce the resolution and density of the point cloud. The size of the voxel grid is set to 0.1m×0.1m×0.1m, and the point cloud data in each voxel is averaged or a representative point cloud data point is retained. Downsampling can effectively reduce the amount of point cloud data, thereby reducing computational complexity and processing time. At the same time, this also helps to eliminate noise and redundant information, and improve the accuracy and stability of subsequent feature extraction. The point cloud data is spatially clustered, and points belonging to the same object are clustered together to form multiple point groups. Through clustering, the points in the point cloud are divided into different groups according to their spatial distribution, and each group represents a potential object or subject. Select a clustering algorithm, and set appropriate parameters for the selected clustering algorithm. The downsampled point cloud data is input into the clustering algorithm to perform a clustering operation. Combined with the application scenarios and prior knowledge of the vehicle automatic parking system, a pre-trained deep learning model, exemplarily PointNet++ or SPD, is used to extract features from the point cloud and classify them to identify environmental objects such as obstacles, pedestrians, road markings, parking spaces, and traffic lights. Through deep learning, various objects in the environment are automatically identified and understood, providing important decision support for autonomous driving. A separate layer is created for each type of environmental object. Exemplarily, all cluster points marked as obstacles constitute an obstacle layer, all cluster points marked as pedestrians constitute a pedestrian layer, and so on; by dividing the layers, complex environmental information is organized hierarchically to more intuitively understand and process environmental information and make more accurate decisions.

[0094] See also Figure 2 ,Specifically, the specific steps of constructing the feature set include:

[0095] Divide the layers into static layers and dynamic layers; extract features of the static layers and dynamic layers through the operator layer, including: identifying boundary features of the static layers; performing trajectory analysis on the dynamic layers; constructing feature sets based on information extracted from the operator layer features;

[0096] In this embodiment, the layers are divided into static layers and dynamic layers according to the characteristics of the objects in the environment, such as whether they are moving. This helps to distinguish static objects that will not change their positions, such as buildings, road markings, etc., from dynamic objects that may move, such as pedestrians, vehicles, etc. Simplify the subsequent processing flow, making the analysis and understanding of different types of objects more efficient and accurate. For static layers, the focus is on identifying the boundary features therein. Identify edge points to provide a stable reference frame for the automatic parking system. This helps to accurately locate parking spaces and identify potential parking obstacles. For dynamic layers, the motion trajectory of moving objects is obtained to track the position changes of moving objects. The processing of dynamic layers enables the automatic parking system to accurately predict and respond to changes in the environment, enhancing adaptability and responsiveness. This is crucial to avoiding potential collisions with moving objects. By integrating the feature information of static layers and dynamic layers, a unified feature set is constructed. The feature set contains the boundary information of the static layer and the motion trajectory of the dynamic layer, which is used to plan the parking path; the construction of the feature set provides a comprehensive environmental model for the automatic parking system, so that it can make accurate decisions under complex and changing road conditions. This includes determining the optimal parking path, evaluating available space and performing precise vehicle control maneuvers.

[0097] See also Figure 3 Specifically, the specific steps of boundary feature recognition include:

[0098] For each point cloud, configure the neighborhood range , calculate the number of point clouds in the neighborhood. By setting the neighborhood range, the local changes of the point cloud data can be effectively captured, the accuracy and robustness of edge detection can be improved, and then the local probability density function of the point cloud can be calculated. , the probability density function is calculated as follows:

[0099] ,in, is the kernel function, are other points in the neighborhood, is the number of points in the neighborhood, is a smoothing parameter; by calculating the local probability density function, the high-density area and the low-density area in the point cloud data can be effectively distinguished, so as to accurately identify the potential edge points.

[0100] Derivate the probability density function of the point cloud, obtain the density gradient, and configure the gradient threshold , when the modulus of the density gradient is greater than When , the point is marked as an edge point; in order to accurately identify the edge points in the point cloud data, it provides key information for the subsequent boundary feature extraction. Through the change of density gradient, the edge information in the point cloud data can be effectively captured to achieve accurate positioning of the boundary.

[0101] Boundary features are extracted from edge points, including coordinates, normal vectors, and curvature. The curvature calculation formula is:

[0102] ,in, and are the coordinates of the edge points, and yes and The first derivative of and yes and By extracting boundary features, we can deeply analyze the geometric shape and topological structure in point cloud data, providing strong support for scene understanding and target recognition.

[0103] Based on the edge point coordinates, the noise points are identified and the local density of each edge point is calculated by the sum of the inverse distances. The local density is calculated as follows:

[0104] ,in, It is the calculation point cloud The number of edge points of the local density, It is a point cloud No. The nearest edge point, is the distance calculation formula; it can effectively identify isolated points that are far away from other edge points, so that they can be regarded as noise and removed.

[0105] The local outlier factor of each edge point is calculated based on the local density, that is:

[0106] ,in, yes The neighbor set of yes The local density of Representing a collection The modulus is the number of elements in the set. The larger the local outlier factor, the more likely the point is to be a noise or outlier.

[0107] Configuring the Noise Threshold , the local outlier factor value is greater than the noise threshold The edge points are marked as noise points and deleted. The noise interference in the point cloud data is removed to improve the accuracy and reliability of the boundary features.

[0108] See also Figure 4 ,Specifically, the specific steps of trajectory analysis include:

[0109] Extract dynamic objects from dynamic layers and initialize dynamic object collections ,in, It is The state of a dynamic object, including the state information of the dynamic object: position, velocity, acceleration, is the corresponding initial weight, usually set to , is the number of initialized dynamic objects;

[0110] For each dynamic object, a tracking window is configured to record its position in consecutive frames and predict the position of the dynamic object in the next frame, that is:

[0111] ,in, is the state transfer function, including the update of state information, is the control input, is the process noise, which is used to simulate the uncertainty factors in the environment. and They are The first dynamic object Frame and The status of the frame.

[0112] Specifically, the state transfer function includes:

[0113] Assume that the state vector of the dynamic object is ,in, They are the dynamic objects in the layer space coordinate system. Moment Axis direction and The coordinate positions of the dynamic objects in the Moment and The speed of the dynamic object in the direction Moment and Acceleration in direction;

[0114] For two-dimensional motion, the state transfer matrix is ​​expressed as:

[0115] ,in, is the time interval, The control input can be a change in acceleration, i.e. , The process noise can be a zero-mean Gaussian noise, whose covariance matrix is ​​expressed as:

[0116]

[0117] Combining the above elements, the state transition function is expressed as:

[0118] ,in is a matrix that maps control input to state space and is expressed as:

[0119]

[0120] The state transfer function combines the current state, control input and process noise to predict the state of the dynamic object at the next moment. The model is adjusted and expanded according to actual conditions to adapt to different dynamic environments and application scenarios.

[0121] The weight is updated to reflect the degree of matching with the latest observation data. The weight of the dynamic object is calculated as follows:

[0122] ,in, It is an observation model, which is used to express the probability of observation under a given state. The use of multimodal vehicle sensor data can improve the accuracy of observation;

[0123] Normalize the weights, that is , resample the dynamic objects according to the normalized weights to solve the problem that the weights of dynamic objects gradually decrease over time, which reduces the true degree of reflection of dynamic objects, resulting in only a few dynamic objects having significant weights, leading to the loss of dynamic object diversity;

[0124] Use a weighted set of dynamic objects to estimate the environment in real time, and combine multiple dynamic objects to provide a more comprehensive state estimate, namely:

[0125] ,in, is the state estimate of the set of dynamic objects;

[0126] Compute the error covariance of the estimated state , which is used to quantify the accuracy of the estimate, is calculated as follows:

[0127]

[0128] Configuring Error Threshold , used to determine the risk level of the estimated state, when If it is greater than the error threshold, the estimate is judged to be at the first risk level, otherwise it is judged to be at the second risk level;

[0129] Specifically, the specific steps of preset path planning include:

[0130] Plan the drivable area of ​​the preset path, extract the boundary features of the boundary points from the static layer, and divide the drivable area according to the boundary features;

[0131] Configuring Adjustment Thresholds , used to adjust the width of the dynamic object's motion trajectory. The width of the motion trajectory is used to limit the area range of the dynamic object's motion trajectory. When the risk level of the dynamic object's motion trajectory is the first risk level, the width of the dynamic object's motion trajectory is , when the risk level of the motion trajectory of the dynamic object is the second risk level, the width of the motion trajectory of the dynamic object is C;

[0132] Plan the prohibited driving area of ​​the preset path, obtain the motion trajectory of the dynamic object in the real-time dynamic layer, and divide the prohibited driving area according to the motion trajectory of the dynamic object and its width;

[0133] The non-prohibited driving area in the drivable area is obtained as the target planning area, and a preset path is planned in the target planning area by using a path planning algorithm.

[0134] In this embodiment, boundary points are extracted from the static layer, and the coordinates, normal vectors and curvature of the boundary points are obtained. Based on the extracted boundary features, the areas where the vehicle can drive safely are clearly marked. Through fine boundary feature extraction and area division, all safe driving areas can be accurately identified, thereby effectively preventing the vehicle from entering any possible dangerous areas. According to the risk level of the motion trajectory of the dynamic object, the width of the motion trajectory is flexibly configured. Specifically, the width of the motion trajectory of the first risk level is set to , and the width of the trajectory of the second-level risk level is set to C, where Greater than C, by adjusting the threshold, the width of the motion trajectory of dynamic objects with different risk levels can be flexibly adjusted, thereby improving the accuracy and safety of path planning, real-time monitoring of dynamic objects in the dynamic layer, and recording their motion trajectory and width. According to the motion trajectory and width of the dynamic object, the prohibited driving area is marked to ensure the safety of the vehicle. By real-time monitoring and dynamic division of prohibited driving areas, it is possible to respond quickly to changes in the environment and ensure that the vehicle does not enter the area where it may collide with moving obstacles. Determine the target planning area, remove the prohibited driving area from the drivable area, and the target planning area obtained is the area where the vehicle needs to plan a path. Apply the path planning algorithm, and use the path planning algorithm, such as the A* algorithm, the RRT algorithm, etc., within the target planning area to plan an optimal path from the starting point to the end point. To ensure that the vehicle can park automatically safely and efficiently.

[0135] Specifically, the specific steps of generating a parking instruction and controlling the vehicle movement include:

[0136] receiving a parking path, serializing coordinate points in the parking path, and converting them into an instruction set for the vehicle to move;

[0137] Generate vehicle control instructions, including accelerator, brake, and steering wheel angle, based on the sequence of coordinate points in the parking path;

[0138] Send the parking command to the actuator, including the electric power steering system and the braking system, to control the vehicle to drive according to the command;

[0139] When the vehicle is moving, the surrounding environment is monitored based on real-time vehicle sensor data. When an abnormality is detected at a coordinate point in the parking path, an emergency braking command is issued to ensure that the vehicle stops in time.

[0140] In this embodiment, the execution control module first receives the preset parking path generated by the path planning module. This path consists of a series of coordinate points, which define the optimal driving route of the vehicle from the current position to the target parking space. The coordinate points in the preset path are serialized and converted into a set of instructions that the vehicle can follow step by step. Each coordinate point represents an intermediate position or final target position that the vehicle needs to reach. According to the sequence of path points, vehicle control instructions are generated, including throttle, brake, steering wheel angle, etc. These instructions ensure that the vehicle can accurately drive along the preset path. The parking instruction is sent to the corresponding actuator, such as the electric power steering system and the braking system, through the vehicle's electronic control unit to control the vehicle to drive according to the instruction. The speed, direction, position and other information of the vehicle are continuously monitored to ensure that the vehicle strictly follows the preset path. If a potential collision risk or other abnormal situation is detected during the parking process, the execution control module will immediately issue an emergency braking instruction to ensure that the vehicle stops in time to avoid accidents.

[0141] Example 2

[0142] See also Figure 1 This embodiment introduces a real-time monitoring method based on the fusion of layer and operator layer, including the following steps:

[0143] Step S1: Collect raw data of the vehicle's surrounding environment through on-board sensors, pre-process the raw data, and establish three-dimensional point cloud data; in order to achieve high-precision perception of the vehicle's surrounding environment and provide a reliable data basis for subsequent feature recognition and path planning.

[0144] Step S2: Perform feature recognition based on the three-dimensional point cloud data, construct a feature layer based on the result of feature recognition, and apply the operator layer to extract and integrate the layer information to construct a feature set; convert complex environmental information into a layer structure that is easy to understand and process, thereby improving the efficiency and accuracy of data processing.

[0145] Step S3: Divide the target planning area according to the feature set, plan a preset path in the target planning area, and determine the parking path through user interaction; to ensure the safety and feasibility of the vehicle's travel path while meeting the user's personalized needs.

[0146] Step S4: Generate parking instructions based on the parking path, control the vehicle's movement, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking. Through real-time monitoring and dynamic adjustment, the vehicle can be safely parked automatically in complex environments, improving driving experience and safety.

[0147] Specifically, the specific steps of constructing the feature set include:

[0148] Divide the layers into static layers and dynamic layers; extract features of the static layers and dynamic layers through the operator layer, including: identifying boundary features of the static layers; performing trajectory analysis on the dynamic layers; constructing feature sets based on information extracted from the operator layer features;

[0149] Specifically, the specific steps of boundary feature recognition include:

[0150] For each point cloud, configure the neighborhood range , calculate the number of point clouds in the neighborhood, and then calculate the local probability density function of the point cloud. , the probability density function is calculated as follows:

[0151] ,in, is the kernel function, are other points in the neighborhood, is the number of points in the neighborhood, is the smoothing parameter;

[0152] Derivate the probability density function of the point cloud, obtain the density gradient, and configure the gradient threshold , when the modulus of the density gradient is greater than When , mark the point as an edge point;

[0153] Boundary features are extracted from edge points, including coordinates, normal vectors, and curvature. The curvature calculation formula is:

[0154] ,in, and are the coordinates of the edge points, and yes and The first derivative of and yes and The second derivative of

[0155] Based on the edge point coordinates, the noise points are identified and the local density of each edge point is calculated by the sum of the inverse distances. The local density is calculated as follows:

[0156] ,in, It is calculated The number of edge points of the local density, Yes No. The nearest edge point, is the distance calculation formula;

[0157] The local outlier factor of each edge point is calculated based on the local density, that is:

[0158] ,in, yes The neighbor set of yes The local density of Representing a collection The modulus is the number of elements in the set. The larger the local outlier factor, the more likely the point is to be a noise or outlier.

[0159] Configuring the Noise Threshold , the local outlier factor value is greater than the noise threshold The edge points are marked as noise points and deleted.

[0160] Specifically, the specific steps of trajectory analysis include:

[0161] Extract dynamic objects from dynamic layers and initialize dynamic object collections ,in, It is The state of a dynamic object, including the state information of the dynamic object: position, velocity, acceleration, is the corresponding initial weight, usually set to , is the number of initialized dynamic objects;

[0162] For each dynamic object, a tracking window is configured, and its position in consecutive frames is recorded, and the position of the dynamic object in the next frame is predicted, that is:

[0163] ,in, is the state transfer function, including the update of state information, is the control input, is the process noise, which is used to simulate the uncertainty factors in the environment. and They are The first dynamic object Frame and The status of the frame.

[0164] The weight is updated to reflect the degree of matching with the latest observation data. The weight of the dynamic object is calculated as follows:

[0165] ,in, It is an observation model, which is used to express the probability of observation under a given state. The use of multimodal vehicle sensor data can improve the accuracy of observation;

[0166] Normalize the weights, that is , resample the dynamic objects according to the normalized weights to solve the problem that the weights of dynamic objects gradually decrease over time, which reduces the true degree of reflection of dynamic objects, resulting in only a few dynamic objects having significant weights, leading to the loss of dynamic object diversity;

[0167] Use a weighted set of dynamic objects to estimate the environment in real time, and combine multiple dynamic objects to provide a more comprehensive state estimate, namely:

[0168] ,in, is the state estimate of the set of dynamic objects;

[0169] Compute the error covariance of the estimated state , which is used to quantify the accuracy of the estimate, is calculated as follows:

[0170]

[0171] Configuring Error Threshold , used to determine the risk level of the estimated state, when If it is greater than the error threshold, the estimate is judged to be at the first risk level, otherwise it is judged to be at the second risk level;

[0172] Specifically, the specific steps of preset path planning include:

[0173] Plan the drivable area of ​​the preset path, extract the boundary features of the boundary points from the static layer, and divide the drivable area according to the boundary features;

[0174] Configuring Adjustment Thresholds , used to adjust the width of the dynamic object's motion trajectory. The width of the motion trajectory is used to limit the area range of the dynamic object's motion trajectory. When the risk level of the dynamic object's motion trajectory is the first risk level, the width of the dynamic object's motion trajectory is , when the risk level of the motion trajectory of the dynamic object is the second risk level, the width of the motion trajectory of the dynamic object is C;

[0175] Plan the prohibited driving area of ​​the preset path, obtain the motion trajectory of the dynamic object in the real-time dynamic layer, and divide the prohibited driving area according to the motion trajectory of the dynamic object and its width;

[0176] The non-prohibited driving area in the drivable area is taken as the target planning area, and a preset path is planned in the target planning area using the path planning algorithm.

[0177] Working principle and its effect:

[0178] A real-time monitoring method and system based on the fusion of layers and operator layers, which collects data on the surrounding environment through on-board sensors and can identify and process various features in the environment, such as obstacles, pedestrians and road markings, thereby providing accurate information for vehicle autonomous driving.

[0179] The environment perception module collects raw data of the vehicle's surroundings through on-board sensors. The raw data is preprocessed, such as denoising, standardization and enhancement, to establish a three-dimensional point cloud data model, to achieve high-precision perception of the vehicle's surroundings, and to provide a reliable data basis for subsequent feature recognition and path planning. The image-computing fusion module divides the layers into static layers and dynamic layers. The static layer mainly displays objects that do not move in the environment, such as buildings and road boundaries; the dynamic layer focuses on objects that may move, such as pedestrians and other vehicles. Each layer is feature extracted and integrated through the corresponding operator layer. The static layer extracts edge information through boundary feature recognition, while the dynamic layer tracks the position changes of moving objects through trajectory analysis. These feature information are used to construct a comprehensive feature set to provide necessary input for the decision-making and planning module. Based on the feature set obtained from the image-computing fusion module, the decision-making and planning module is responsible for dividing the target planning area, planning a preset path in the target planning area, and determining the final parking path through user interaction. This ensures the safety and feasibility of the vehicle's travel path while meeting the personalized needs of users. The execution control module receives the parking path instructions from the decision-making planning module, serializes the coordinate points on the path and converts them into a specific set of instructions for the vehicle to move, such as accelerator, brake, steering wheel angle, etc. Then, these instructions are sent to the actuator to control the vehicle to move according to the instructions. During the vehicle's movement, the surrounding environment is continuously monitored, and emergency braking measures are immediately taken when abnormal conditions are detected to ensure safe automatic parking.

[0180] Through the above steps, a real-time monitoring method and system based on the fusion of layers and operator layers provides an innovative solution. It can not only improve the efficiency and accuracy of data processing, but also meet the personalized needs of users while ensuring driving safety. Combining advanced sensor technology, data processing algorithms and user interaction design, it provides a powerful and reliable support platform for autonomous driving vehicles.

[0181] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A real-time monitoring system based on the fusion of graph and operator layers, including an environment perception module, a graph and operator fusion module, a decision-making and planning module, and a control execution module; The environment perception module is used to collect raw data of the vehicle's surrounding environment through on-board sensors, and pre-process the raw data to establish three-dimensional point cloud data; The image-calculation fusion module is used to perform feature recognition based on the three-dimensional point cloud data, construct a feature layer based on the result of feature recognition, and apply the operator layer to extract and integrate the layer information to construct a feature set; The decision planning module is used to divide the target planning area according to the feature set, plan a preset path in the target planning area, and determine the parking path through user interaction; The control execution module is used to generate a parking instruction according to the parking path, control the vehicle to move, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking; The specific steps of constructing the point cloud data include: Preprocess the collected raw data, including denoising, standardization and enhancement; perform data registration on the on-board sensor data and align them in the same coordinate system; extract features from the registered data points and associate the feature data; use data fusion algorithms to integrate data from different sensors and generate point cloud data based on the fused data; The specific steps of the data registration include: Establish the objective function and set it as the sum of the squares of the sum of the point-to-point distances and the sum of the regularization terms to prevent overfitting, that is: ,in, is the objective function, which represents the sum of the distances between the source data points and the target data points. is the rotation matrix, is the translation vector, and are the corresponding point pairs in the source data points and the target data points, respectively. is the regularization parameter, It is about the rotation matrix The regularization term of ; The objective function is iteratively optimized, and the rotation matrix and translation vector are updated according to the gradient of the objective function. The update formula is as follows: , ,in, and yes learning rate, and They are The rotation matrix and translation vector of the iteration, and They are the updated rotation matrix and translation vector respectively; Configuring the Convergence Threshold When the change of the objective function is less than the convergence threshold, the optimization is stopped and the optimal rotation matrix and translation vector, as well as the registered data points, are output. The convergence judgment formula is: ,in, is a small positive threshold, is the objective function of the kth iteration, is the objective function of the k+1th iteration; The specific steps of constructing the feature set include: Divide the layers into static layers and dynamic layers; extract features of the static layers and dynamic layers through the operator layer, including: identifying boundary features of the static layers; performing trajectory analysis on the dynamic layers; constructing feature sets based on information extracted from the operator layer features; The specific steps of the boundary feature recognition include: For each point cloud, configure the neighborhood range , calculate the number of point clouds in the neighborhood, and then calculate the local probability density function of the point cloud. , the probability density function is calculated as follows: ,in, is the kernel function, are other points in the neighborhood, is the number of points in the neighborhood, is the smoothing parameter; Derivate the probability density function of the point cloud, obtain the density gradient, and configure the gradient threshold , when the modulus of the density gradient is greater than When , mark the point as an edge point; Boundary features are extracted from edge points, including coordinates, normal vectors, and curvature. The curvature calculation formula is: ,in, and are the coordinates of the edge points, and yes and The first derivative of and yes and The second derivative of The specific steps of the boundary feature recognition also include: Based on the edge point coordinates, noise points are identified and the local density of each edge point is calculated by the sum of inverse distances. The local density is calculated as follows: ,in, It is the calculation point cloud The number of edge points of the local density, It is a point cloud No. The nearest edge point, is the distance calculation formula; The local outlier factor of each edge point is calculated based on the local density, that is: ,in, It is a point cloud The neighbor set of yes The local density of Representing a collection The modulus of , which is the number of elements in the set; Configuring the Noise Threshold , the local outlier factor value is greater than the noise threshold The edge points are marked as noise points and deleted.

2. A real-time monitoring system based on the fusion of layer and operator layer as claimed in claim 1, characterized in that: The specific steps of the trajectory analysis include: Extract dynamic objects from dynamic layers and initialize dynamic object collections ,in, Yes, yes The state of a dynamic object, including the state information of the dynamic object, is the corresponding initial weight, is the number of initialized dynamic objects; For each dynamic object, a tracking window is configured to record its position in consecutive frames and predict the position of the dynamic object in the next frame, that is: ,in, is the state transition function, is the control input, is the process noise, and They are The first dynamic object and The state of the frame; Update weights and calculate the weight of each dynamic object: ,in, is the observation model, which is used to express the probability of observation in a given state; Normalize the weights, that is , resamples the dynamic objects according to the normalized weights.

3. A real-time monitoring system based on the fusion of layer and operator layer as claimed in claim 2, characterized in that: The specific steps of the trajectory analysis also include: A real-time estimate of the environment is performed using a weighted set of dynamic objects, namely: ,in, is the state estimate of the set of dynamic objects; Compute the error covariance of the estimated state , which is used to quantify the accuracy of the estimate, is calculated as follows: , configure the error threshold , used to determine the risk level of the estimated state, when If the error is greater than the error threshold, the estimate is determined to be at the first risk level, otherwise it is determined to be at the second risk level.

4. A real-time monitoring system based on the fusion of layer and operator layer as claimed in claim 1, characterized in that: The specific steps of the preset path planning include: Plan the drivable area of ​​the preset path, extract the boundary features of the boundary points from the static layer, and divide the drivable area according to the boundary features; Configuring Adjustment Thresholds , used to adjust the width of the dynamic object's motion trajectory. When the risk level of the dynamic object's motion trajectory is level 1, the width of the dynamic object's motion trajectory is C, when the risk level of the motion trajectory of the dynamic object is the second risk level, the width of the motion trajectory of the dynamic object is C; Plan the prohibited driving area of ​​the preset path, obtain the motion trajectory of the dynamic object in the real-time dynamic layer, and divide the prohibited driving area according to the motion trajectory of the dynamic object and its width; The non-prohibited driving area in the formable area is obtained as the target planning area, and a preset path is planned in the target planning area by using the path planning algorithm.

5. A real-time monitoring method based on the fusion of a layer and an operator layer, which is executed based on a real-time monitoring system based on the fusion of a layer and an operator layer as claimed in any one of claims 1 to 4, characterized in that: The following steps are involved: Step S1: collecting raw data of the vehicle's surrounding environment through on-board sensors, and preprocessing the raw data to create three-dimensional point cloud data; Step S2: performing feature recognition based on the three-dimensional point cloud data, constructing a feature layer based on the result of feature recognition, and applying an operator layer to extract and integrate the layer information to construct a feature set; Step S3: dividing the target planning area according to the feature set, planning a preset path in the target planning area, and determining the parking path through user interaction; Step S4: Generate a parking instruction according to the parking path, control the vehicle's movement, and monitor abnormal conditions in real time to control the vehicle to complete automatic parking.

Images