360-degree surround image-assisted reversing radar dynamic obstacle recognition method and system

Abstract

The present invention provides a 360-degree surround image-assisted reversing radar dynamic obstacle recognition method and system, which relates to the field of automobile communication transmission technology, including fusing the first feature, the second feature and the third feature to construct a global pose graph; obtaining a preliminary recognition result of the dynamic obstacle through DBSCAN clustering algorithm processing, and obtaining the enclosing rectangular frame of the obstacle through the minimum convex hull method combined with the fuzzy line segment method; inputting the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle into the deep reinforcement learning model for calculation, and obtaining the final optimized motion trajectory and behavior intention prediction, so as to achieve active obstacle avoidance and safe reversing. The beneficial effects of the present invention are to significantly improve the accuracy and robustness of dynamic obstacle recognition, enhance the environmental perception ability and intelligent decision-making ability of the vehicle, and provide important technical support for the assisted driving system.

Description

Technical Field

[0001] The present invention relates to the field of automobile communication transmission technology, and in particular to a 360-degree surround image-assisted reversing radar dynamic obstacle recognition method and system. Background Art

[0002] With the accelerated advancement of smart mine construction, the transformation of the coal mining industry towards digitalization and intelligence has become an inevitable trend. Underground mines have narrow roads, rugged terrain, many obstacles, and insufficient light, which poses a great challenge to mine car drivers in driving safely. Existing technologies mainly solve recognition problems in complex environments by introducing more sensors, enhancing the generalization ability of deep learning models, or combining multiple data sources (such as images and lidar point clouds). However, these technologies still face many problems when applied in mines, such as the reliability of sensors in harsh environments, the complexity of multi-sensor data fusion, and high real-time requirements. Summary of the invention

[0003] The purpose of the present invention is to provide a 360-degree surround image-assisted reversing radar dynamic obstacle recognition method and system to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0004] In a first aspect, the present application provides a 360-degree surround image-assisted reversing radar dynamic obstacle recognition method, comprising:

[0005] The polarized light data collected by the bionic polarized skylight sensor on the top of the mining vehicle is denoised and normalized to extract the first feature, wherein the first feature includes the intensity and angle features of the polarized light in the sky; the point cloud data collected by the lidar is filtered and downsampled to remove noise points and outliers, and the second feature is extracted, wherein the second feature includes edges, planes and corner points; the distance and speed data measured by the odometer are calibrated to obtain the calibrated third feature, wherein the third feature includes the mileage and speed data of the odometer; the first feature, the second feature and the third feature are fused to construct a global pose graph, wherein the global pose graph includes environmental information, pose nodes, pose edges, global constraints and framework information;

[0006] According to the point cloud data of the laser radar, the DBSCAN clustering algorithm is used to cluster points with close distances in space to obtain the preliminary recognition results of dynamic obstacles. Based on the preliminary recognition results, the attention mechanism is used to extract the geometric features of the obstacles, where the geometric features include position and size features, shape and texture features, and motion features; the Zernike invariant moment feature extraction method is used to extract the rotation, translation, and magnification invariant historical contour features of the obstacles from the multi-frame data of the global pose graph, and the minimum convex hull method is combined with the fuzzy line segment method to obtain the enclosing rectangular frame of the obstacle, and the boundary information of the enclosing rectangular frame is extracted to obtain the geometric boundary information of the obstacle, where the geometric boundary information includes the boundary points, edge line end points, and inflection points of the obstacle;

[0007] The extracted geometric features of the obstacle are compared with the static features in the pose graph to confirm the identity and motion state of the obstacle, where the static features include fixed structures in the environment; the environmental perception algorithm is used to analyze the impact of air pressure changes on air flow in the mine, and then the environmental factors affecting the movement of the obstacle are obtained, where the environmental factors affecting the movement of dynamic obstacles include airflow resistance, airflow disturbance, and airflow pressure difference; and the thermal imaging data of the infrared sensor is used to identify the thermal characteristics of the obstacle, where the thermal characteristics include temperature distribution and thermal radiation intensity;

[0008] The geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle are input into the deep reinforcement learning model for calculation, and the future motion trajectory and behavioral intention of the obstacle are predicted to obtain preliminary prediction results, where the preliminary prediction results include motion direction, speed change and behavior mode; the preliminary prediction results and the actually observed obstacle state data are input into the Kalman filter, the state estimation of the obstacle is updated, and the final optimized motion trajectory and behavior intention prediction are obtained;

[0009] By adopting a multi-agent collaborative control algorithm and combining the final optimized motion trajectory and behavior intention prediction, the vehicle's driving speed, direction and obstacle avoidance strategy are automatically adjusted to achieve active obstacle avoidance and safe reversing.

[0010] Preferably, the first feature, the second feature and the third feature are fused to construct a global pose graph, which includes:

[0011] The first feature and the second feature are subjected to weighted fusion processing, and different weights are assigned to the first feature and the second feature respectively based on the importance of the polarized light data in the first feature in determining the heading angle of the vehicle and the accuracy of the point cloud data in the second feature in acquiring the geometric structure of the environment, and a preliminary fusion feature is obtained by a weighted summation method, wherein the preliminary fusion feature includes information and data of the initial position and heading of the vehicle;

[0012] The Kalman filter method is used to fuse the preliminary fusion features and the third feature. The vehicle's motion model is combined with the sensor observation data through a prediction-update cycle. The high-precision geometric information of the lidar in the second feature and the continuity of the odometer in the third feature are used to optimize the pose nodes and pose edges in the pose graph through a graph optimization processing algorithm. The position and posture of the pose nodes are adjusted through iterative optimization to obtain an optimized pose graph framework, where the goal of graph optimization is to minimize the relative pose measurement error between pose nodes and the constraint error of pose edges.

[0013] Extract environmental features from point cloud data, use feature matching algorithm to match pose nodes in pose graph framework with environmental features, establish association between pose nodes and environmental features, fuse pose nodes, pose edges, environmental features and global constraint information to construct global pose graph.

[0014] Preferably, the point cloud data of the laser radar is processed by the DBSCAN clustering algorithm to cluster points close to each other in space to obtain a preliminary recognition result of the dynamic obstacle. Based on the preliminary recognition result, the geometric features of the obstacle are extracted using the attention mechanism, which includes:

[0015] Using laser radar to collect point cloud data containing three-dimensional spatial information of the vehicle's surrounding environment, where each point contains its coordinate position in space and reflection intensity;

[0016] Set the neighborhood radius and minimum number of points, traverse each point in the point cloud, and determine whether the number of points in the neighborhood is greater than or equal to MinPts, If so, the point in the point cloud data currently being traversed and evaluated by the DBSCAN algorithm is set as the core point. Starting from the core point, the points in its neighborhood are expanded to form a cluster, and it is checked whether the newly added point is a core point. If so, the expansion continues until no new points can be added; if not, the point cloud data is continued to be traversed, and the clustering result is recorded as the preliminary recognition result of the dynamic obstacle, where each cluster contains the point cloud data belonging to the same obstacle;

[0017] Based on the preliminary recognition results of dynamic obstacles, the similarity between the feature vector of each point and the overall feature vector of the obstacle is calculated, and the softmax function is used for normalization to obtain the importance weight of each point;

[0018] According to the importance weight of each point, the point cloud data is classified and weighted to obtain the geometric characteristics of the obstacle.

[0019] Preferably, the Zernike invariant moment feature extraction method is used to extract the rotational translational magnification invariant historical contour features of the obstacle from the multi-frame data of the global pose graph, and the minimum convex hull method is combined with the fuzzy line segment method to obtain the enclosing rectangular frame of the obstacle, which includes:

[0020] Collect the contour information of obstacles at different time points, where the contour information includes the shape change and movement trajectory of the obstacles;

[0021] Normalize the contour information, map the normalized result to the unit circle, sum the values ​​of all points, and then calculate the Zernike moment feature;

[0022] The obtained Zernike moment features are combined to form a feature sequence, where the feature sequence includes the contour changes of the obstacle in the time series, which is recorded as the historical contour feature;

[0023] The convex hull of the contour point set of the historical contour feature is calculated using the Quickhull algorithm to obtain the convex hull contour. The specific steps are to select the point at the lower left corner of the point set as the starting point, and search for the vertices of the convex hull in a counterclockwise direction until returning to the starting point; each edge of the convex hull contour is recorded as a fuzzy line segment, and the fuzzy boundary of the boundary point is determined according to the membership function of the fuzzy line segment;

[0024] Combined with the fuzzy boundary processed by the fuzzy line segment method, the minimum bounding rectangle of the obstacle is calculated.

[0025] Preferably, the environmental perception algorithm is used to analyze the influence of air pressure changes on the air flow in the mine, thereby obtaining environmental factors affecting the movement of obstacles, wherein the environmental factors affecting the movement of dynamic obstacles include airflow resistance, airflow disturbance and airflow pressure difference, including:

[0026] According to the air pressure sensors installed in the mine, the air pressure data is collected in real time, and the computational fluid dynamics model is used to simulate the changes in air flow caused by the changes in air pressure, so as to predict the speed, direction and pressure field of the air flow;

[0027] By analyzing the speed, direction and pressure field of air flow, airflow resistance, airflow disturbance and airflow pressure difference are extracted.

[0028] Preferably, the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle are input into the deep reinforcement learning model for calculation, and the future motion trajectory and behavior intention of the obstacle are predicted to obtain preliminary prediction results, including:

[0029] Normalize the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle to obtain a preprocessed feature vector;

[0030] The preprocessed feature vector is input into the deep reinforcement learning model, the strategy is optimized by maximizing the Q value of the Critic network, and the parameters of the Actor network are updated using the gradient ascent method;

[0031] Minimize the following loss function to evaluate the difference between the predicted Q value and the target Q value;

[0032] Using the soft update method, the parameters of the target network are brought closer to the parameters of the current network, and through the forward propagation of the model, the preliminary prediction results of the future motion trajectory and behavioral intention of the obstacle are obtained. The preliminary prediction results include the position information, speed information, motion direction information and behavioral intention information of the obstacle at the future time point.

[0033] Preferably, the multi-agent collaborative control algorithm is used to automatically adjust the vehicle's speed, direction and obstacle avoidance strategy in combination with the final optimized motion trajectory and behavior intention prediction, including:

[0034] Using the Markov game algorithm, a collaborative control model is constructed to obtain the control strategy of each agent. The model regards vehicles and obstacles as multiple agents, each of which has a corresponding state, action, and reward function. The state of the vehicle includes the current position, speed, and direction, and the actions include acceleration, deceleration, and steering. The reward function is designed to encourage the vehicle to drive safely and avoid collisions with obstacles. The control strategy is the best action taken by the agent in different states to achieve the global optimal driving path and obstacle avoidance strategy.

[0035] At each time step, the control strategy is used to make real-time decisions based on the current position, speed, and direction of the vehicle and obstacles. The real-time action instructions of the vehicle at the current time step are obtained through the results of the real-time decision, and then the corresponding actions are performed to adjust the vehicle's driving speed and direction.

[0036] In the second aspect, the present application also provides a 360-degree surround image-assisted reversing radar dynamic obstacle recognition system, including:

[0037] Processing module: used to denoise and normalize the polarized light data collected by the bionic polarized skylight sensor on the top of the mining vehicle, and extract the first feature, wherein the first feature includes the intensity and angle characteristics of the polarized light in the sky; filter and downsample the point cloud data collected by the lidar, remove noise points and outliers, and extract the second feature, wherein the second feature includes edges, planes and corner points; calibrate the distance and speed data measured by the odometer to obtain the calibrated third feature, wherein the third feature includes the mileage and speed data of the odometer; fuse the first feature, the second feature and the third feature to construct a global pose graph, wherein the global pose graph includes environmental information, pose nodes, pose edges, global constraints and framework information;

[0038] Extraction module: It is used to process the point cloud data of the laser radar through the DBSCAN clustering algorithm, gather the points with close distance in space together, and obtain the preliminary recognition results of dynamic obstacles. Based on the preliminary recognition results, the geometric features of the obstacles are extracted using the attention mechanism, where the geometric features include position and size features, shape and texture features, and motion features; the Zernike invariant moment feature extraction method is used to extract the rotation, translation and magnification invariant historical contour features of the obstacles from the multi-frame data of the global pose graph, and the minimum convex hull method is combined with the fuzzy line segment method to obtain the enclosing rectangular frame of the obstacle, and the boundary information of the enclosing rectangular frame is extracted to obtain the geometric boundary information of the obstacle, where the geometric boundary information includes the boundary points, edge line end points and inflection points of the obstacle;

[0039] Identification module: used to compare the extracted geometric features of obstacles with the static features in the pose graph to confirm the identity and motion state of the obstacles, where the static features include fixed structures in the environment; use environmental perception algorithms to analyze the impact of air pressure changes on air flow in the mine, and then obtain environmental factors that affect the movement of obstacles, where environmental factors that affect the movement of dynamic obstacles include airflow resistance, airflow disturbance, and airflow pressure difference; and use thermal imaging data from infrared sensors to identify the thermal features of obstacles, where thermal features include temperature distribution and thermal radiation intensity;

[0040] Computation module: used to input the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle into the deep reinforcement learning model for calculation, predict the future motion trajectory and behavioral intention of the obstacle, and obtain preliminary prediction results, where the preliminary prediction results include motion direction, speed change and behavior mode; input the preliminary prediction results and the actually observed obstacle state data into the Kalman filter, update the state estimation of the obstacle, and obtain the final optimized motion trajectory and behavior intention prediction;

[0041] Adjustment module: It is used to adopt the multi-agent collaborative control algorithm, combined with the final optimized motion trajectory and behavior intention prediction, to automatically adjust the vehicle's driving speed, direction and obstacle avoidance strategy to achieve active obstacle avoidance and safe reversing.

[0042] In a third aspect, the present application also provides a 360-degree surround image-assisted reversing radar dynamic obstacle recognition device, including:

[0043] Memory for storing computer programs;

[0044] A processor is used to implement the steps of the 360-degree surround image-assisted reversing radar dynamic obstacle recognition method when executing the computer program.

[0045] In a fourth aspect, the present application also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned reversing radar dynamic obstacle recognition method based on 360-degree surround image assistance are implemented.

[0046] The beneficial effects of the present invention are:

[0047] The present invention obtains obstacle information from multiple angles and sensor dimensions by combining 360-degree surround view images and reversing radar. The 360-degree surround view images provide a panoramic view of the vehicle's surroundings, while the reversing radar provides precise distance information. This multimodal data fusion method significantly improves the accuracy of obstacle recognition, especially in complex environments and low visibility conditions.

[0048] The present invention utilizes polarized light data collected by the bionic polarized skylight sensor to extract the intensity and angle characteristics of the polarized light in the sky. These characteristics help determine the vehicle's heading angle, and combine the point cloud data of the lidar and the mileage and speed data of the odometer to construct a global pose graph, further enhancing the vehicle's perception of the surrounding environment and providing more comprehensive background information for the detection and identification of dynamic obstacles.

[0049] The present invention processes the point cloud data of the lidar through the DBSCAN clustering algorithm, which can cluster points with close distances in space together to form a preliminary recognition result of dynamic obstacles. It also combines the Zernike invariant moment feature extraction method to extract the historical contour features of the obstacles that are invariant to rotation, translation and amplification, further improving the robustness of dynamic obstacle detection, so that it can still be accurately recognized under different postures and size changes.

[0050] The present invention utilizes a deep reinforcement learning model and a Kalman filter to predict the future motion trajectory and behavioral intention of obstacles. The deep reinforcement learning model generates preliminary prediction results by learning the historical behavior patterns of obstacles, while the Kalman filter optimizes the prediction results based on actual observation data, thereby improving the accuracy and real-time performance of the prediction.

[0051] The present invention adopts a multi-agent collaborative control algorithm, combined with optimized motion trajectory and behavior intention prediction, which not only improves the vehicle's autonomous obstacle avoidance capability, but also optimizes the driving path to ensure the vehicle's safe and efficient driving in complex environments; through the environmental perception algorithm, the impact of air pressure changes on air flow in the mine is analyzed, and environmental factors that affect the movement of obstacles, such as airflow resistance, airflow disturbance and airflow pressure difference, can be identified, further enhancing the adaptability and flexibility of the system, enabling it to operate stably under different environmental conditions.

[0052] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 It is a schematic flow chart of a 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the structure of a 360-degree surround view image-assisted reversing radar dynamic obstacle recognition system according to an embodiment of the present invention;

[0056] Figure 3 It is a schematic diagram of the structure of the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device described in an embodiment of the present invention.

[0057] In the figure: 701, processing module; 702, extraction module; 703, recognition module; 704, calculation module; 705, adjustment module; 800, 360-degree surround image-assisted reversing radar dynamic obstacle recognition device; 801, processor; 802, memory; 803, multimedia component; 804, I / O interface; 805, communication component. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0060] Embodiment 1:

[0061] This embodiment provides a 360-degree surround image-assisted reversing radar dynamic obstacle recognition method.

[0062] See also Figure 1 , the figure shows that the method includes step S100, step S200, step S300, step S400 and step S500.

[0063] Step S100, denoising and normalizing the polarized light data collected by the bionic polarized skylight sensor on the top of the mining vehicle, extracting the first feature, wherein the first feature includes the intensity and angle characteristics of the polarized light of the sky; filtering and downsampling the point cloud data collected by the lidar, removing noise points and outliers, and extracting the second feature in the environment, wherein the second feature includes edges, planes and corner points; calibrating the distance and speed data measured by the odometer to obtain the calibrated third feature, wherein the third feature includes the mileage and speed data of the odometer; fusing the first feature, the second feature and the third feature to construct a global pose graph, wherein the pose graph includes environmental information, pose nodes, pose edges, global constraints and framework information.

[0064] It can be understood that step S100 includes step S101, step S102 and step S103, wherein:

[0065] Step S101, performing weighted fusion processing on the first feature and the second feature, and assigning different weights based on the importance of polarized light data in the first feature in determining the vehicle heading angle and the accuracy of point cloud data in the second feature in acquiring the geometric structure of the environment, and obtaining a preliminary fusion feature through a weighted summation method, wherein the preliminary fusion feature includes information and data of the vehicle's initial posture and heading;

[0066] It should be noted that polarization data plays an important role in determining the vehicle's heading angle. The sky polarization pattern provides a stable reference frame, and the vehicle can determine its heading angle by detecting the direction of polarized light in the sky. In clear weather, the polarization navigation system can achieve a high accuracy, such as ±0.2°. Therefore, in weighted fusion, the weight of polarization data can be set to a higher value, such as w 1 =0.6 to fully utilize its advantages in heading angle estimation. Point cloud data provides detailed 3D geometric information of the vehicle's surroundings, which is essential for determining the vehicle's initial pose. In addition, point cloud data can identify features such as edges, planes, and corners in the environment. This information helps to accurately locate the vehicle in the environment. Therefore, the weight of point cloud data in pose estimation can be set to w 2 = 0.4 to ensure its contribution to the surrounding geometry during the fusion process.

[0067] Therefore, the polarized light data and point cloud data are weighted and summed according to the determined weights to obtain the preliminary fusion features. The specific calculation formula is:

[0068]

[0069] in, F represents the initial fusion feature, w 1=0.6 and w 2=0.4 are the weights of polarized light data and point cloud data respectively.

[0070] In summary, by weighted fusion of polarized light data and point cloud data and comprehensive utilization of the advantages of the two types of data, the accuracy of the vehicle's initial position and heading estimation is improved. It is understandable that in a complex and changeable mine environment, the data of a single sensor may be affected by interference or errors. Weighted fusion processing can effectively reduce the uncertainty of a single data source and enhance the robustness of environmental perception.

[0071] Step S102, using the Kalman filter method to fuse the preliminary fusion feature and the third feature, combining the vehicle's motion model with the sensor observation data through a prediction-update cycle, using the high-precision geometric information of the laser radar in the second feature and the degree of continuity of the odometer in the third feature, and optimizing the pose nodes and pose edges in the pose graph through a graph optimization processing algorithm; and adjusting the position and posture of the pose nodes through iterative optimization to obtain an optimized pose graph framework, wherein the goal of the graph optimization is to minimize the relative pose measurement error between pose nodes and the constraint error of the pose edges;

[0072] It should be noted that in this step, based on the preliminary fusion features and the third feature, the third feature is the calibrated odometer data, including mileage and speed data, which reflect the vehicle's own motion state. After Kalman filtering, the preliminary fusion features and odometer data are fused using the Kalman filter. Through the prediction-update cycle, the vehicle's motion model is combined with the sensor observation data, and the vehicle's posture change is predicted based on the odometer data. Then, the prediction result is corrected with polarized light data and point cloud data to obtain posture nodes and posture edges. The posture nodes represent the posture state of the vehicle at different time points, and the posture edges represent the posture state of the vehicle at different time points. The relative posture relationship between the posture nodes is shown, including rotation and translation information. These posture nodes and posture edges constitute the basic framework of the posture graph. Then, according to the posture nodes and posture edges, the posture nodes and posture edges obtained from the Kalman filter processing may have certain errors and uncertainties. After graph optimization processing, a graph optimization algorithm is used, such as nonlinear least squares optimization based on the Gauss-Newton method, to optimize the posture nodes and posture edges in the posture graph, adjust the position and posture of the posture nodes, and obtain the optimized posture graph framework. This framework has higher accuracy and robustness, and can better reflect the motion trajectory and posture changes of the vehicle in the environment.

[0073] Step S103: extract environmental features from the point cloud data, use a feature matching algorithm to match the pose nodes in the pose graph framework with the environmental features, establish an association between the pose nodes and the environmental features, fuse the pose nodes, pose edges, environmental features, and global constraint information, and construct a global pose graph.

[0074] It should be noted that the feature matching algorithm uses algorithms such as point feature histogram (PFH) or view feature histogram (VFH) to extract environmental features from point cloud data. For example, the PFH algorithm calculates the geometric relationship between each point in the point cloud and its neighboring points to generate a feature histogram to describe the local geometric characteristics of the point cloud. The environmental features extracted include edge features, plane features, and corner features, which can accurately describe the geometric structure of the vehicle's surroundings.

[0075] It should be noted that the pose graph framework contains pose nodes and pose edges. Environmental features are edges, planes, corners, etc. extracted from point cloud data. Using feature matching algorithms such as random sampling consensus (RANSAC) or iterative closest point (ICP) algorithm, pose nodes in the pose graph framework are matched with environmental features to establish the association between pose nodes and environmental features. Then, pose nodes, pose edges, environmental features, and global constraint information are fused to construct a global pose graph, where global constraint information can include vehicle motion model constraints, environmental topological structure constraints, etc.

[0076] Step S200: According to the point cloud data of the laser radar, after being processed by the DBSCAN clustering algorithm, points with a relatively close distance in space are clustered together to obtain a preliminary recognition result of the dynamic obstacle. Based on the preliminary recognition result, the geometric features of the obstacle are extracted using the attention mechanism, wherein the geometric features include position and size features, shape and texture features, and motion features. The Zernike invariant moment feature extraction method is used to extract the historical contour features of the obstacle after rotation, translation, and magnification remain unchanged from the multi-frame data of the full pose graph, and the minimum convex hull method is combined with the fuzzy line segment method to obtain the enclosing rectangular frame of the obstacle, and the boundary information of the enclosing rectangular frame is extracted to obtain the geometric boundary information of the obstacle, wherein the geometric boundary information includes the boundary points, edge end points, and inflection points of the obstacle; the extracted geometric features of the obstacle are compared with the static features in the pose graph to confirm the identity and motion state of the obstacle, wherein the static features include fixed structures in the environment.

[0077] It can be understood that in this step, the position and size features include the position, length and width of the tracking point: the position of the tracking point represents the reference point coordinates of the obstacle in space, and the center of mass of the obstacle or a certain feature point is usually selected as the tracking point to determine the specific position of the obstacle in the environment; the length and width represent the maximum extension length and width of the obstacle in a certain direction, respectively, reflecting the size of the obstacle. These size information helps to determine the type of obstacle and the possible movement space requirements. Among them, the shape and texture features describe the overall shape of the obstacle, such as rectangle, circle, irregular shape, etc. Among them, the shape feature can be extracted through the boundary points, edge curvature and other information of the obstacle, which helps to identify the category and structural characteristics of the obstacle; the texture reflects the detailed features of the obstacle surface, such as smoothness, roughness, color distribution, etc. Texture features can be extracted through image processing technology, such as gray level co-occurrence matrix (GLCM), local binary pattern (LBP) and other methods, to provide richer information for the identification and classification of obstacles. Among them, the motion feature includes the deviation angle between the motion direction and the x-axis θ: It indicates the angle between the obstacle's moving direction and a reference coordinate axis (such as the x-axis), reflecting the obstacle's moving direction. This feature helps predict the obstacle's moving trajectory and possible collision risks; Movement speed and direction: Movement speed indicates the distance the obstacle moves per unit time, and movement direction indicates the direction in which the obstacle moves. These movement features can be calculated by the obstacle position change between consecutive frames, which is crucial for tracking and behavior prediction of dynamic obstacles.

[0078] Among them, in this step, these features include fixed structures in the environment, such as the edges of walls, planes on the ground, and corner points. These static features are extracted from the point cloud data of the lidar and represent the fixed structures in the environment around the vehicle. By comparing the geometric features of the obstacle with the static features in the environment, the identity and motion state of dynamic obstacles can be more accurately confirmed. For example, if the geometric features of an obstacle are similar to a static feature (such as the edge of a wall), but its motion state indicates that it is moving, it can be confirmed that the obstacle is a dynamic obstacle.

[0079] The pose node described in this step represents the pose of the vehicle at a specific point in time, that is, a combination of position and orientation. Each node contains the spatial coordinates (translation) and rotation parameters (rotation) of the vehicle at that moment, which can be expressed in 2D or 3D coordinate systems; the edges in the pose edge represent the relationship or constraints between nodes, which can come from a variety of sources, such as sensor observations, motion models, and loop detection. For example, sensor observation data provides information about the pose of feature points or other landmarks in the environment relative to the vehicle, while the motion model describes the relative pose changes of the vehicle between adjacent moments.

[0080] Among them, the constraints of the global constraints are provided by global sensors (such as GPS, magnetometer, barometer, etc.), which are used to align the local pose estimation to the global coordinates, eliminating the cumulative error of the local estimation. For example, the GPS factor can convert the longitude, latitude and altitude into position information in the local coordinate system; Optimization framework: The pose graph is optimized through an optimization framework (such as the Gauss-Newton method, the Levenberg-Marquardt method, etc.) to obtain more accurate vehicle trajectory and map information. During the optimization process, the poses of all nodes will be adjusted to meet the constraints of the edges, among which these contents together constitute the global pose graph, which enables it to provide accurate position and orientation information of the vehicle in the environment, as well as the relative relationship between the vehicle and the environment.

[0081] The step S200 includes step S201, step S202 and step S203, wherein:

[0082] Step S201, using a laser radar to collect point cloud data containing three-dimensional spatial information of the vehicle's surrounding environment, wherein each point contains its coordinate position (X, Y, Z) in space and reflection intensity;

[0083] It should be noted that the reflection intensity is related to the surface material and roughness of the target object, the laser incident angle, the laser wavelength and the energy density of the lidar.

[0084] Set the neighborhood radius and minimum number of points, traverse each point in the point cloud, and determine whether the number of points in the neighborhood is greater than or equal to MinPts, If so, the point in the point cloud data currently being traversed and evaluated by the DBSCAN algorithm is set as the core point. Starting from the core point, the points in its neighborhood are expanded to form a cluster, and it is checked whether the newly added point is a core point. If so, the expansion continues until no new points can be added; if not, the point cloud data is continued to be traversed, and the clustering result is recorded as the preliminary recognition result of the dynamic obstacle, where each cluster contains the point cloud data belonging to the same obstacle;

[0085] Step S202: Based on the preliminary recognition result of the dynamic obstacle, the similarity between the feature vector of each point and the overall feature vector of the obstacle is calculated, and the softmax function is used for normalization to obtain the importance weight of each point. The calculation formula is as follows:

[0086]

[0087] In the formula, Indicates The attention weights of the points, the softmax function is used to normalize the weights, For the The feature vector of a point, F is the overall feature vector of the obstacle;

[0088] It should be noted that the attention mechanism can focus the model on the key feature areas of the obstacle, and for the point cloud clustering data of each obstacle, the model will calculate the importance weight of each point, which reflects the importance of the point in the obstacle feature extraction. For example, for vehicle obstacles, the model may give higher weights to the points on the wheels and the edges of the vehicle body, because the geometric features of these areas are more critical for the recognition and classification of vehicles, where the weight calculation can be achieved through a differentiable attention function, which calculates the weight based on the similarity between the feature vector of the point and the overall feature vector of the obstacle. Therefore, through the calculation of the above attention mechanism, the importance weight of each point is obtained, and these weights provide a basis for subsequent feature classification and weighted processing.

[0089] Step S203: Classify and weight the point cloud data according to the importance weight of each point to obtain the geometric features of the obstacle. Classify the point cloud data according to the importance weight of the point. For example, the points with weights higher than a certain threshold are classified as key points, which contain the main geometric features of the obstacle. Then, weighted processing is performed on the key points to extract the geometric features of the obstacle. Specifically, the coordinate position, normal direction and other features of the key points can be weighted summed to obtain the comprehensive geometric features of the obstacle. For example, the position feature of the obstacle can be calculated by weighted averaging the coordinate positions of the key points:

[0090]

[0091] In the formula, Indicates The coordinate positions of the key points, Indicates The attention weight of each point, n It is expressed as the number of points in the point cloud clustering data belonging to the same dynamic obstacle; specifically, the coordinate position, normal direction and other features of the key points can be weighted and summed to obtain the comprehensive geometric features of the obstacle. Through the above classification and weighted processing, the geometric features of the obstacle are finally extracted, including position and size features (such as the center of mass position of the obstacle, the length, width and height of the bounding box), shape and texture features (such as the outline shape of the obstacle, surface texture features) and motion features (such as the speed and acceleration of the obstacle).

[0092] The step S200 further includes steps S204, S205, S206, S207 and S208, wherein:

[0093] Step S204: Collecting the contour information of the obstacle at different time points, wherein the contour information includes the shape change and movement trajectory of the obstacle;

[0094] Step S205: normalize the contour information, map the normalized result to the unit circle, sum the values ​​of all points, and then calculate the Zernike moment feature. The calculation formula is as follows:

[0095]

[0096] In the formula, is a radial polynomial, a function of the radial distance r The polynomial of is the expression of the Zernike moment, is the angle θ The sine function of mis an integer representing the rotational symmetry order of the Zernike moment;

[0097] First, the obstacle contour area in each frame of the image is normalized and mapped to the unit circle. Then, the Zernike moment is calculated, where the Zernike moment is a feature extraction method based on orthogonal polynomials, which can describe the changes of the image in different directions, different scales and rotation angles. Specifically, the calculated Zernike moment is a set of complex numbers, whose real and imaginary parts contain the shape information of the image. These moment values ​​are invariant moment features, which can describe the contour features of the obstacle and are invariant to rotation, translation and scale changes.

[0098] In this step, for the multi-frame data in the global pose graph, each frame contains the contour information of the obstacle at different time points, and the Zernike moment is calculated for each frame to obtain the invariant moment feature of each frame. Then the invariant moment features of the multi-frame data are combined to form a feature sequence, where this feature sequence describes the contour changes of the obstacle in the time series, that is, the historical contour feature. By analyzing the feature sequence, the movement mode and shape change of the obstacle can be identified. Because the Zernike moment is invariant, even if the obstacle is rotated, translated or enlarged, the extracted features can still accurately describe the contour of the obstacle.

[0099] Step S206, combining the obtained Zernike moment features to form a feature sequence, wherein the feature sequence includes the contour changes of the obstacle in the time series, which is recorded as the historical contour feature;

[0100] Step S207, using the Quickhull algorithm to calculate the convex hull of the contour point set of the historical contour feature to obtain the convex hull contour, wherein the specific steps are to select the point at the lower left corner of the point set as the starting point, and search for the vertices of the convex hull in a counterclockwise direction until returning to the starting point;

[0101] It should be noted that the minimum convex hull method can be used to obtain the convex hull contour of the obstacle. This contour is an approximate representation of the outer boundary of the obstacle. The convex hull contour is a convex polygon that encloses all contour points of the obstacle, providing a basis for subsequent fuzzy line segment method processing.

[0102] Each edge of the convex hull contour is recorded as a fuzzy line segment, and the fuzzy boundary of the boundary point is determined according to the membership function of the fuzzy line segment. The calculation formula of the membership function is:

[0103]

[0104] In the formula, is the coordinate of the boundary point, is the center point of the fuzzy line segment, is the width of the blurred line segment, is the membership function in the fuzzy line segment method;

[0105] It should be noted that in this step, the formula The role is to calculate the point Belongs to the membership of the fuzzy line segment. Specifically, it first calculates the point With center point The distance between , then divide this distance by the width , and get a normalized distance value. If this normalized distance value is less than 1, then the membership is ; If it is greater than or equal to 1, then the membership is 0. In this way, the membership function can describe the fuzziness of the boundary points and provide a basis for the subsequent calculation of the enclosing rectangle.

[0106] Step S208: Calculate the minimum bounding rectangle of the obstacle by combining the fuzzy boundary processed by the fuzzy line segment method.

[0107] It can be understood that this rectangular frame can completely contain the fuzzy boundary of the obstacle, providing accurate geometric information for the location and identification of the obstacle.

[0108] Step S300: Analyze the impact of air pressure changes on air flow in the mine using an environmental perception algorithm, and then obtain environmental factors that affect the movement of obstacles. Use thermal imaging data from infrared sensors to identify the thermal characteristics of obstacles. Environmental factors that affect the movement of dynamic obstacles include airflow resistance, airflow disturbance, and airflow pressure difference, and the special characteristics include temperature distribution and thermal radiation intensity.

[0109] It can be understood that in this step, the environmental information in the pose graph can provide a reference for analyzing the impact of air pressure changes on air flow in the mine, thereby obtaining more accurate environmental factors that affect the movement of obstacles. For example, the terrain undulations and spatial layout information in the pose graph may affect the flow path and speed of the airflow, thereby affecting environmental factors such as the airflow resistance of the obstacle.

[0110] Wherein, this step S300 includes step S301 and step S302:

[0111] Step S301, according to the air pressure sensor installed in the mine, the air pressure data is collected in real time, and the computational fluid dynamics model is used to simulate the air flow changes caused by the air pressure changes, so as to predict the speed, direction and pressure field of the air flow; in this step, the fluid dynamics model is based on the geometric structure of the mine, the layout of the ventilation system and the temporal and spatial distribution of the air pressure changes. It can be understood that the air pressure sensor installed in the mine is used to collect air pressure data in real time to establish a mathematical and physical model of the air flow in the mine. Generally, the air flow in the mine can be regarded as the viscous fluid flow of an incompressible fluid, and the control equation is the Navier-Stokes equation. Since the fluid flow in the mine is mostly turbulent flow, it is necessary to construct a turbulence model based on the actual situation, and then obtain the control equation of the air flow, where the control equation includes the mass conservation equation, the momentum conservation equation and the energy conservation equation. The specific form is as follows:

[0112]

[0113] In the formula, is the air density, is the velocity vector, It's pressure. is the identity matrix, is the stress tensor, E is the total energy, is the heat flux, t is the time, which is used to describe the change of fluid state over time. is the momentum density of the fluid, which is a vector field, is the momentum flux of the fluid;

[0114] In this step, according to the control equation of air flow, that is, the established mathematical and physical model of air flow in the mine, computational fluid dynamics (CFD) software, such as ANSYS Fluent or OpenFOAM, is used to mesh the fluid domain in the mine, where the accuracy and resolution of the meshing will directly affect the accuracy of the simulation results; then, an appropriate numerical solution method, such as the finite volume method (FVM) or the finite difference method (FDM), is selected to discretize and solve the control equation, thereby obtaining the velocity field, pressure field and temperature field of the air in the mine, so that these field data can be used to analyze the impact of air pressure changes on air flow.

[0115] Step S302: extract airflow resistance, airflow disturbance and airflow pressure difference by analyzing the air flow speed, direction and pressure field, wherein the airflow resistance is calculated as follows:

[0116]

[0117] In the formula, is the airflow resistance, is the air density, is the obstacle speed, is the drag coefficient, A is the frontal area of ​​the obstacle;

[0118] The calculation formula of air flow disturbance is as follows:

[0119]

[0120] In the formula, is the standard deviation of velocity, It is The speed of a point, is the average speed, N is the total number of points;

[0121] The calculation formula of air flow pressure difference is as follows:

[0122]

[0123] In the formula, is the air flow pressure difference, is the upstream pressure, is the downstream pressure.

[0124] Finally, the recognition of obstacles is enhanced by combining infrared sensor data, including: according to the environmental factors affecting the movement of obstacles: the influence of airflow factors on the movement of obstacles obtained from the dynamic model; then, after infrared sensor data fusion processing: combined with the thermal imaging data of infrared sensors, the recognition accuracy of obstacles is further enhanced. Among them, infrared sensors can detect the thermal characteristics of obstacles, such as temperature distribution and thermal radiation intensity, and then by fusing airflow factors with thermal feature data, the state of obstacles can be more comprehensively described; finally, the comprehensive characteristics of obstacles are obtained: including the comprehensive characteristics of obstacles obtained by fusion processing, including motion characteristics and thermal characteristics under the influence of airflow factors, and these comprehensive characteristics provide richer information for subsequent obstacle behavior prediction and trajectory estimation. Through the above steps, the environmental perception algorithm can be used to analyze the impact of air pressure changes on air flow in the mine, and then the environmental factors affecting the movement of obstacles can be obtained, and the recognition accuracy of obstacles can be enhanced by combining infrared sensor data.

[0125] Through the above analysis, airflow resistance, airflow disturbance and airflow pressure difference are extracted. These factors reflect the specific impact of air pressure changes on air flow in the mine, providing a basis for subsequent obstacle motion analysis and important information for subsequent obstacle behavior prediction and trajectory estimation.

[0126] Step S400: Input the geometric boundary information of the obstacle, the identity and motion state of the dynamic obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle into the deep reinforcement learning model for calculation, predict the future motion trajectory and behavioral intention of the obstacle, and obtain preliminary prediction results, wherein the preliminary prediction results include motion direction, speed change, and behavior pattern; input the preliminary prediction results output by the deep reinforcement learning model and the actually observed obstacle state data into the Kalman filter, update the state estimation of the obstacle, and obtain the final optimized motion trajectory and behavioral intention prediction.

[0127] It can be understood that in this step, the input features: the input received by the model includes the geometric boundary information of the obstacle (such as the size and position of the enclosing rectangular box), the identity and motion state of the dynamic obstacle (such as speed, direction, etc.), the environmental factors that affect the movement of the obstacle (such as airflow changes, etc.), and the thermal characteristics of the obstacle (such as temperature distribution, etc.). The calculation process is: the deep reinforcement learning model predicts the future motion trajectory and behavioral intention of the obstacle by learning the correlation between these input features, performing complex calculations and reasoning. For example, the model may consider the current speed and direction of the obstacle, combined with environmental factors (such as airflow resistance) and thermal characteristics (such as temperature changes may affect the performance of the obstacle) to predict its future motion path. The final output result is the prediction result of the future motion trajectory and behavioral intention of the obstacle. These results provide a basis for subsequent intelligent decision-making and obstacle avoidance strategy formulation. The pose nodes and pose edge information in the pose graph can provide the Kalman filter with the initial state estimation of the obstacle, including parameters such as position and speed, which helps the Kalman filter to more accurately update the state estimation of the obstacle and obtain the final optimized motion trajectory and behavioral intention prediction.

[0128] Specifically, step S400 includes step S401, step S402, step S403 and step S404, wherein:

[0129] Step S401: normalize the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the motion of the obstacle, and the thermal characteristics of the obstacle to obtain a preprocessed feature vector;

[0130] The preprocessed feature vectors are input into the deep reinforcement learning model. These feature vectors include the geometric boundary information, identity and motion state of the obstacle, environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle.

[0131] Step S402: Input the preprocessed feature vector into the deep reinforcement learning model, optimize the strategy by maximizing the Q value of the Critic network, and use the gradient ascent method to update the parameters of the Actor network;

[0132] In step S402, the purpose of the Actor network is to generate the future motion trajectory and behavioral intention of the obstacle. The specific steps are as follows: Input layer: The feature vector is input to the input layer of the Actor network. Hidden layer: Forward propagation is performed through the neurons of multiple hidden layers. Each hidden layer usually uses the ReLU activation function to increase the nonlinear ability of the model. Output layer: The output layer generates the future motion trajectory and behavioral intention of the obstacle. The activation function of the output layer usually selects tanh or sigmoid to normalize the output value to a suitable range. For example, the speed and direction can be normalized to the interval [-1, 1], and the position can be normalized to the interval [0, 1]. Among them, the purpose of the Critic network is to evaluate the value of the action generated by the Actor network. The specific steps are as follows: Input layer: The state vector and the action vector generated by the Actor network are input to the input layer of the Critic network. Hidden layer: Forward propagation is performed through the neurons of multiple hidden layers. Each hidden layer usually uses the ReLU activation function to increase the nonlinear ability of the model. Output layer: The output layer generates a Q value, which represents the expected reward under the current state and action. The output layer usually uses a linear activation function and outputs a Q value.

[0133] Step S403: Minimize the following loss function to evaluate the difference between the predicted Q value and the target Q value. The calculation formula is as follows:

[0134]

[0135] In the formula, represents the mean square error between the predicted Q value and the target Q value, represents the model parameters. In this formula represents the parameters of the Q network, N represents the number of samples, Represents the state of the environment at a certain point in time. Indicates the actions taken in a specific state. Indicates in status Take action Expected return, represents the target Q value;

[0136] Step S404: Use a soft update method to bring the parameters of the target network closer to the parameters of the current network, and obtain preliminary prediction results of the future motion trajectory and behavior intention of the obstacle through forward propagation of the model, where the preliminary prediction results include the position information, speed information, motion direction information and behavior intention information of the obstacle at the future time point.

[0137] In this step, in the dynamic obstacle recognition scenario of mining vehicles, the state It can include the geometric boundary information, identity and motion state of the obstacle, environmental factors affecting the movement of the obstacle, and thermal characteristics of the obstacle. This may include the vehicle's speed, direction, and obstacle avoidance strategy. In this step, the loss function of the Critic network is It is used to evaluate the accuracy of network parameters in predicting the future motion trajectory and behavior intention of obstacles. By minimizing this loss function, the parameters of the Critic network can be optimized to improve the accuracy of the prediction.

[0138] Through the above steps, the geometric boundary information, identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle can be input into the deep reinforcement learning model for calculation, and the future movement trajectory and behavioral intention of the obstacle can be predicted to obtain preliminary prediction results. These prediction results provide an important basis for subsequent intelligent decision-making and obstacle avoidance strategies.

[0139] Step S500: adopt a multi-agent collaborative control algorithm, combined with the final optimized motion trajectory and behavior intention prediction, to automatically adjust the vehicle's driving speed, direction and obstacle avoidance strategy to achieve active obstacle avoidance and safe reversing.

[0140] It can be understood that in this step, the global constraints and framework information in the pose graph can provide the multi-agent collaborative control algorithm with the constraints and reference framework of the global environment, which helps the algorithm to consider the limitations and goals of the global environment when formulating the vehicle's driving speed, direction and obstacle avoidance strategy, and achieve better intelligent decision-making and collaborative control to ensure active obstacle avoidance and safe reversing.

[0141] Specifically, step S500 includes step S501 and step S502, wherein:

[0142] Step S501: Use the Markov game algorithm to construct a collaborative control model and obtain the control strategy of each agent; wherein the model regards the vehicle and the obstacle as multiple agents, and each agent has a corresponding state, action and reward function; wherein the state of the vehicle includes the current position, speed and direction, and the action includes acceleration, deceleration and steering, and the reward function is designed to encourage the vehicle to drive safely and avoid collision with obstacles; wherein the control strategy is the best action taken by the agent in different states to achieve the global optimal driving path and obstacle avoidance strategy;

[0143] Step S502: At each time step, a control strategy is used to make real-time decisions based on the current position, speed, and direction of the vehicle and obstacles. The real-time action instructions of the vehicle at the current time step are obtained through the results of the real-time decisions, and then corresponding actions are performed to adjust the vehicle's driving speed and direction.

[0144] It should be noted that at each time step, a collaborative control strategy is used to make real-time decisions based on the current state (including the position, speed and direction of the vehicle and obstacles). Specifically, the vehicle controller will select the best action, such as acceleration, deceleration or steering, according to the strategy to avoid collision with obstacles while maintaining an efficient driving path. Through real-time decision-making, the real-time action instructions of the vehicle at the current time step, such as acceleration instructions, steering angles, etc., are obtained. After receiving the real-time action instructions, the vehicle control system performs corresponding actions, such as adjusting the engine power, changing the steering angle, etc., to adjust the vehicle's driving speed and direction, and feedback the vehicle's actual driving state, including position, speed and direction, in real time through the vehicle's sensors (such as IMU, GPS, wheel speed sensors, etc.). Finally, this feedback information will be used for the next real-time decision, forming a closed-loop control process to ensure that the vehicle can respond to environmental changes in real time and drive safely and efficiently.

[0145] Through the above steps, the application of multi-agent collaborative control algorithm in dynamic obstacle recognition and avoidance of mining vehicles can be realized to ensure the safe driving of vehicles in complex environments. These steps reflect the creativity of multi-agent collaborative control algorithm, combining deep reinforcement learning and real-time feedback adjustment to improve the robustness and adaptability of the system.

[0146] Embodiment 2:

[0147] like Figure 2 As shown, this embodiment provides a 360-degree surround image-assisted reversing radar dynamic obstacle recognition system, see Figure 2 The system comprises:

[0148] Processing module 701: used to denoise and normalize the polarized light data collected by the bionic polarized skylight sensor on the top of the mining vehicle, and extract the first feature, wherein the first feature includes the intensity and angle features of the polarized light of the sky; filter and downsample the point cloud data collected by the laser radar, remove noise points and outliers, and extract the second feature, wherein the second feature includes edges, planes and corner points; calibrate the distance and speed data measured by the odometer to obtain the calibrated third feature, wherein the third feature includes the mileage and speed data of the odometer; fuse the first feature, the second feature and the third feature to construct a global pose graph, wherein the global pose graph includes environmental information, pose nodes, pose edges, global constraints and framework information;

[0149] Extraction module 702: used to process the point cloud data of the laser radar through the DBSCAN clustering algorithm, gather points close to each other in space, obtain preliminary recognition results of dynamic obstacles, and extract geometric features of the obstacles using the attention mechanism based on the preliminary recognition results, where the geometric features include position and size features, shape and texture features, and motion features; use the Zernike invariant moment feature extraction method to extract the rotation, translation, and magnification invariant historical contour features of the obstacles from multi-frame data of the global pose graph, and use the minimum convex hull method combined with the fuzzy line segment method to obtain the enclosing rectangular frame of the obstacles, extract boundary information of the enclosing rectangular frame, and obtain the geometric boundary information of the obstacles, where the geometric boundary information includes boundary points, edge line end points, and inflection points of the obstacles;

[0150] Identification module 703: used to compare the extracted geometric features of the obstacle with the static features in the pose graph to confirm the identity and motion state of the obstacle, where the static features include fixed structures in the environment; use the environmental perception algorithm to analyze the impact of air pressure changes on the air flow in the mine, and then obtain the environmental factors that affect the movement of the obstacle, where the environmental factors that affect the movement of dynamic obstacles include airflow resistance, airflow disturbance and airflow pressure difference; and use the thermal imaging data of the infrared sensor to identify the thermal features of the obstacle, where the thermal features include temperature distribution and thermal radiation intensity;

[0151] Calculation module 704: used to input the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the motion of the obstacle, and the thermal characteristics of the obstacle into the deep reinforcement learning model for calculation, predict the future motion trajectory and behavior intention of the obstacle, and obtain a preliminary prediction result, wherein the preliminary prediction result includes the motion direction, speed change and behavior mode; input the preliminary prediction result and the actually observed obstacle state data into the Kalman filter, update the state estimation of the obstacle, and obtain the final optimized motion trajectory and behavior intention prediction;

[0152] Adjustment module 705: used to adopt a multi-agent collaborative control algorithm, combined with the final optimized motion trajectory and behavior intention prediction, to automatically adjust the vehicle's driving speed, direction and obstacle avoidance strategy to achieve active obstacle avoidance and safe reversing.

[0153] Specifically, the processing module 701 includes:

[0154] A first calculation unit is used to perform weighted fusion processing on the first feature and the second feature, and assign different weights based on the importance of polarized light data in the first feature in determining the vehicle heading angle and the accuracy of point cloud data in the second feature in acquiring the geometric structure of the environment, and obtain a preliminary fusion feature through a weighted summation method, wherein the preliminary fusion feature includes information and data of the initial position and heading of the vehicle;

[0155] Fusion optimization unit: used to fuse the preliminary fusion features and the third feature using the Kalman filter method, combine the vehicle's motion model with the sensor observation data through a prediction-update cycle, and optimize the pose nodes and pose edges in the pose graph using the high-precision geometric information of the lidar in the second feature and the continuity of the odometer in the third feature through a graph optimization processing algorithm; and adjust the position and posture of the pose nodes through iterative optimization to obtain an optimized pose graph framework, where the goal of graph optimization is to minimize the relative pose measurement error between pose nodes and the constraint error of the pose edges;

[0156] Construction unit: used to extract environmental features from point cloud data, use feature matching algorithm to match pose nodes in the pose graph framework with environmental features, establish the association between pose nodes and environmental features, fuse pose nodes, pose edges, environmental features and global constraint information, and construct a global pose graph.

[0157] Specifically, the extraction module 702 includes:

[0158] Acquisition unit: used to collect point cloud data containing three-dimensional spatial information of the vehicle's surrounding environment using laser radar, where each point contains its coordinate position and reflection intensity in space;

[0159] Clustering unit: used to set the neighborhood radius and minimum number of points, traverse each point in the point cloud, and determine whether the number of points in the neighborhood is greater than or equal to MinPts, If so, the point in the point cloud data currently being traversed and evaluated by the DBSCAN algorithm is set as the core point. Starting from the core point, the points in its neighborhood are expanded to form a cluster, and it is checked whether the newly added point is a core point. If so, the expansion continues until no new points can be added; if not, the point cloud data is continued to be traversed, and the clustering result is recorded as the preliminary recognition result of the dynamic obstacle, where each cluster contains the point cloud data belonging to the same obstacle;

[0160] The second calculation unit is used to calculate the similarity between the feature vector of each point and the overall feature vector of the obstacle based on the preliminary recognition result of the dynamic obstacle, and normalize it using the softmax function to obtain the importance weight of each point. The calculation formula is as follows:

[0161]

[0162] In the formula, Indicates The attention weights of the points, the softmax function is used to normalize the weights, For the The feature vector of a point, F is the overall feature vector of the obstacle;

[0163] The first processing unit is used to classify and weight the point cloud data according to the importance weight of each point to obtain the geometric features of the obstacle.

[0164] Specifically, the extraction module 702 includes:

[0165] The second acquisition unit is used to collect the contour information of the obstacle at different time points, wherein the contour information includes the shape change and movement trajectory of the obstacle;

[0166] The second processing unit is used to normalize the contour information, map the normalized result to the unit circle, sum the values ​​of all points, and then calculate the Zernike moment feature. The calculation formula is as follows:

[0167]

[0168] In the formula, is a radial polynomial, a function of the radial distance r The polynomial of is the expression of the Zernike moment, is the angle θ The sine function of m is an integer representing the rotational symmetry order of the Zernike moment;

[0169] Combination unit: used to combine the obtained Zernike moment features to form a feature sequence, where the feature sequence includes the contour changes of the obstacle in the time series, recorded as the historical contour feature;

[0170] The third calculation unit is used to calculate the convex hull of the contour point set of the historical contour feature using the Quickhull algorithm to obtain the convex hull contour. The specific steps are: select the point in the lower left corner of the point set as the starting point, and search for the vertices of the convex hull in a counterclockwise direction until returning to the starting point; each edge of the convex hull contour is recorded as a fuzzy line segment, and the fuzzy boundary of the boundary point is determined according to the membership function of the fuzzy line segment. The calculation formula of the membership function is:

[0171]

[0172] In the formula, is the coordinate of the boundary point, is the center point of the fuzzy line segment, is the width of the blurred line segment, is the membership function in the fuzzy line segment method;

[0173] Combined with the fuzzy boundary processed by the fuzzy line segment method, the minimum bounding rectangle of the obstacle is calculated.

[0174] Specifically, the identification module 703 includes:

[0175] Prediction unit: used to collect air pressure data in real time based on the air pressure sensor installed in the mine, and use the computational fluid dynamics model to simulate the changes in air flow caused by changes in air pressure, so as to predict the speed, direction and pressure field of air flow;

[0176] The fourth calculation unit is used to extract airflow resistance, airflow disturbance and airflow pressure difference by analyzing the speed, direction and pressure field of air flow, wherein the calculation formula of airflow resistance is as follows:

[0177]

[0178] In the formula, is the airflow resistance, is the air density, is the obstacle speed, is the drag coefficient, A is the frontal area of ​​the obstacle;

[0179] The calculation formula of air flow disturbance is as follows:

[0180]

[0181] In the formula, is the standard deviation of velocity, It is The speed of a point, is the average speed, N is the total number of points;

[0182] The calculation formula of air flow pressure difference is as follows:

[0183]

[0184] In the formula, is the air flow pressure difference, is the upstream pressure, is the downstream pressure.

[0185] Specifically, the calculation module 704 includes:

[0186] The third processing unit is used to normalize the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle to obtain a pre-processed feature vector;

[0187] Input unit: used to input the preprocessed feature vector into the deep reinforcement learning model, optimize the strategy by maximizing the Q value of the Critic network, and update the parameters of the Actor network using the gradient ascent method;

[0188] Evaluation unit: used to minimize the following loss function and evaluate the difference between the predicted Q value and the target Q value. The calculation formula is as follows:

[0189]

[0190] In the formula, represents the mean square error between the predicted Q value and the target Q value, represents the model parameters. In this formula represents the parameters of the Q network, N represents the number of samples, Represents the state of the environment at a certain point in time. Indicates the actions taken in a specific state. Indicates in status Take action Expected return, represents the target Q value;

[0191] Prediction result unit: used to use soft update method to bring the parameters of the target network closer to the parameters of the current network, and obtain preliminary prediction results of the future motion trajectory and behavior intention of the obstacle through forward propagation of the model, where the preliminary prediction results include the position information, speed information, motion direction information and behavior intention information of the obstacle at the future time point.

[0192] Specifically, the adjustment module 705 includes:

[0193] Control strategy unit: used to build a collaborative control model using the Markov game algorithm to obtain the control strategy of each agent; the model regards vehicles and obstacles as multiple agents, each of which has a corresponding state, action and reward function; the vehicle state includes the current position, speed and direction, the actions include acceleration, deceleration and steering, and the reward function is designed to encourage the vehicle to drive safely and avoid collisions with obstacles; the control strategy is the best action taken by the agent in different states to achieve the global optimal driving path and obstacle avoidance strategy;

[0194] Adjustment unit: It is used to make real-time decisions using control strategies at each time step based on the current position, speed and direction of the vehicle and obstacles, and obtain the real-time action instructions of the vehicle at the current time step through the results of the real-time decision, and then perform corresponding actions to adjust the vehicle's driving speed and direction.

[0195] In summary, the present invention is specifically designed for the complex environment in the mine. The method significantly improves the accuracy and robustness of obstacle identification by combining 360-degree surround images and reversing radars and using multimodal data fusion technology. Specifically, the present invention first processes the point cloud data of the lidar through the DBSCAN clustering algorithm, gathers points that are close in space together, and forms a preliminary recognition result of dynamic obstacles. Then, the Zernike invariant moment feature extraction method is used to extract the historical contour features of the obstacles that are rotationally and translationally amplified and unchanged, further improving the robustness of dynamic obstacle detection. In addition, the present invention also combines a deep reinforcement learning model and a Kalman filter to predict the future motion trajectory and behavioral intention of the obstacle, and optimizes the accuracy and real-time performance of the prediction results. Finally, a multi-agent collaborative control algorithm is used, combined with the optimized motion trajectory and behavioral intention prediction, to automatically adjust the vehicle's driving speed, direction, and obstacle avoidance strategy to ensure the vehicle's safe and efficient driving in complex environments.

[0196] It should be noted that, regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] Embodiment 3:

[0198] Corresponding to the above method embodiment, a 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device is also provided in this embodiment. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device described below and the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method described above can be referenced to each other.

[0199] Figure 3 FIG. 8 is a block diagram of a 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device 800 according to an exemplary embodiment. Figure 3 As shown, the 360-degree surround image assisted reversing radar dynamic obstacle recognition device 800 includes: a processor 801 and a memory 802. The 360-degree surround image assisted reversing radar dynamic obstacle recognition device 800 also includes one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0200] The processor 801 is used to control the overall operation of the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device 800 to complete all or part of the steps in the above-mentioned 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method. The memory 802 is used to store various types of data to support the operation of the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device 800, which may include, for example, instructions for any application or method operating on the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device 800, as well as application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 802 or sent via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules can be a keyboard, a mouse or buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the 360-degree surround image-assisted reversing radar dynamic obstacle recognition device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include: a Wi-Fi module, a Bluetooth module or an NFC module.

[0201] In an exemplary embodiment, the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device 800 can be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method.

[0202] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions may be executed by the processor 801 of the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition device 800 to complete the above-mentioned 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method.

[0203] Embodiment 4:

[0204] Corresponding to the above method embodiment, a readable storage medium is also provided in this embodiment. The readable storage medium described below and the 360-degree surround image-assisted reversing radar dynamic obstacle recognition method described above can be referred to each other.

[0205] A computer program is stored on the readable storage medium, and when the computer program is executed by the processor, the steps of the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method of the above method embodiment are implemented.

[0206] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0208] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A 360-degree surround image-assisted reversing radar dynamic obstacle recognition method, characterized in that: include: De-noising and normalizing the polarized light data collected by the bionic polarized skylight sensor on the top of the mining vehicle to extract the first feature, wherein the first feature includes the intensity and angle features of the polarized light of the sky; The point cloud data collected by the laser radar is filtered and downsampled to remove noise points and outliers, and the second feature is extracted, where the second feature includes edges, planes and corner points; the distance and speed data measured by the odometer are calibrated to obtain the calibrated third feature, where the third feature includes the mileage and speed data of the odometer; the first feature, the second feature and the third feature are fused to construct a global pose graph, where the global pose graph includes environmental information, pose nodes, pose edges, global constraints and framework information; According to the point cloud data of the lidar, the DBSCAN clustering algorithm is used to cluster points close to each other in space to obtain the preliminary recognition results of dynamic obstacles. Based on the preliminary recognition results, the attention mechanism is used to extract the geometric features of the obstacles, including position and size features, shape and texture features, and motion features. The Zernike invariant moment feature extraction method is used to extract the rotation, translation and magnification invariant historical contour features of the obstacle from the multi-frame data of the global pose graph, and the minimum convex hull method is combined with the fuzzy line segment method to obtain the obstacle's enclosing rectangular frame. The boundary information of the enclosing rectangular frame is extracted to obtain the geometric boundary information of the obstacle, where the geometric boundary information includes the boundary points, edge line end points and inflection points of the obstacle. The extracted geometric features of the obstacle are compared with the static features in the pose graph to confirm the identity and motion state of the obstacle, where the static features include fixed structures in the environment; the environmental perception algorithm is used to analyze the impact of air pressure changes on air flow in the mine, and then the environmental factors affecting the movement of the obstacle are obtained, where the environmental factors affecting the movement of dynamic obstacles include airflow resistance, airflow disturbance, and airflow pressure difference; and the thermal imaging data of the infrared sensor is used to identify the thermal characteristics of the obstacle, where the thermal characteristics include temperature distribution and thermal radiation intensity; The geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle are input into the deep reinforcement learning model for calculation, and the future motion trajectory and behavioral intention of the obstacle are predicted to obtain preliminary prediction results, where the preliminary prediction results include motion direction, speed change and behavior pattern; The preliminary prediction results and the actually observed obstacle state data are input into the Kalman filter to update the obstacle state estimation and obtain the final optimized motion trajectory and behavior intention prediction; By adopting a multi-agent collaborative control algorithm and combining the final optimized motion trajectory and behavior intention prediction, the vehicle's driving speed, direction and obstacle avoidance strategy are automatically adjusted to achieve active obstacle avoidance and safe reversing.

2. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: The first feature, the second feature and the third feature are fused to construct a global pose graph, which includes: The first feature and the second feature are subjected to weighted fusion processing, and different weights are assigned to the first feature and the second feature respectively based on the importance of the polarized light data in the first feature in determining the heading angle of the vehicle and the accuracy of the point cloud data in the second feature in acquiring the geometric structure of the environment, and a preliminary fusion feature is obtained by a weighted summation method, wherein the preliminary fusion feature includes information and data of the initial position and heading of the vehicle; The Kalman filter method is used to fuse the preliminary fusion features and the third feature. The vehicle's motion model is combined with the sensor observation data through a prediction-update cycle. The high-precision geometric information of the lidar in the second feature and the continuity of the odometer in the third feature are used to optimize the pose nodes and pose edges in the pose graph through a graph optimization processing algorithm. The position and posture of the pose nodes are adjusted through iterative optimization to obtain an optimized pose graph framework, where the goal of graph optimization is to minimize the relative pose measurement error between pose nodes and the constraint error of pose edges. Extract environmental features from point cloud data, use feature matching algorithm to match pose nodes in pose graph framework with environmental features, establish association between pose nodes and environmental features, fuse pose nodes, pose edges, environmental features and global constraint information to construct global pose graph.

3. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: According to the point cloud data of the laser radar, the DBSCAN clustering algorithm is used to cluster points close to each other in space to obtain the preliminary recognition results of dynamic obstacles. Based on the preliminary recognition results, the geometric features of the obstacles are extracted using the attention mechanism, including: Using laser radar to collect point cloud data containing three-dimensional spatial information of the vehicle's surrounding environment, where each point contains its coordinate position in space and reflection intensity; Set the neighborhood radius and minimum number of points, traverse each point in the point cloud, and determine whether the number of points in the neighborhood is greater than or equal to MinPts, If so, the point in the point cloud data currently being traversed and evaluated by the DBSCAN algorithm is set as the core point. Starting from the core point, the points in its neighborhood are expanded to form a cluster, and it is checked whether the newly added point is a core point. If so, the expansion continues until no new points can be added; if not, the point cloud data is continued to be traversed, and the clustering result is recorded as the preliminary recognition result of the dynamic obstacle, where each cluster contains the point cloud data belonging to the same obstacle; Based on the preliminary recognition results of dynamic obstacles, the similarity between the feature vector of each point and the overall feature vector of the obstacle is calculated, and the softmax function is used for normalization to obtain the importance weight of each point. The calculation formula is as follows: In the formula, Indicates The attention weights of the points, the softmax function is used to normalize the weights, For the The feature vector of a point, F is the overall feature vector of the obstacle; According to the importance weight of each point, the point cloud data is classified and weighted to obtain the geometric characteristics of the obstacle.

4. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: The Zernike invariant moment feature extraction method is used to extract the rotation, translation and magnification invariant historical contour features of the obstacle from the multi-frame data of the global pose graph, and the obstacle enclosing rectangular frame is obtained by combining the minimum convex hull method with the fuzzy line segment method, which includes: Collect the contour information of obstacles at different time points, where the contour information includes the shape change and movement trajectory of the obstacles; Normalize the contour information, map the normalized result to the unit circle, sum the values ​​of all points, and then calculate the Zernike moment feature. The calculation formula is as follows: In the formula, is a radial polynomial, a function of the radial distance r The polynomial of is the expression of the Zernike moment, is the angle θ The sine function of m is an integer representing the rotational symmetry order of the Zernike moment; The obtained Zernike moment features are combined to form a feature sequence, where the feature sequence includes the contour changes of the obstacle in the time series, which is recorded as the historical contour feature; The Quickhull algorithm is used to calculate the convex hull of the contour point set of the historical contour feature to obtain the convex hull contour. The specific steps are to select the point in the lower left corner of the point set as the starting point, and search for the vertices of the convex hull in a counterclockwise direction until returning to the starting point; each edge of the convex hull contour is recorded as a fuzzy line segment, and the fuzzy boundary of the boundary point is determined according to the membership function of the fuzzy line segment. The calculation formula of the membership function is: In the formula, is the coordinate of the boundary point, is the center point of the fuzzy line segment, is the width of the blurred line segment, is the membership function in the fuzzy line segment method; Combined with the fuzzy boundary processed by the fuzzy line segment method, the minimum bounding rectangle of the obstacle is calculated.

5. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: The environmental perception algorithm is used to analyze the impact of air pressure changes on air flow in the mine, and then obtain environmental factors that affect the movement of obstacles. The environmental factors that affect the movement of dynamic obstacles include airflow resistance, airflow disturbance, and airflow pressure difference, including: According to the air pressure sensors installed in the mine, the air pressure data is collected in real time, and the computational fluid dynamics model is used to simulate the changes in air flow caused by the changes in air pressure, so as to predict the speed, direction and pressure field of the air flow; By analyzing the speed, direction and pressure field of air flow, the airflow resistance, airflow disturbance and airflow pressure difference are extracted. The calculation formula of airflow resistance is as follows: In the formula, is the airflow resistance, is the air density, is the obstacle speed, is the drag coefficient, A is the frontal area of ​​the obstacle; In the formula, is the standard deviation of velocity, It is The speed of a point, is the average speed, N is the total number of points; The calculation formula of air flow pressure difference is as follows: In the formula, is the air flow pressure difference, is the upstream pressure, is the downstream pressure.

6. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: The geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle are input into the deep reinforcement learning model for calculation, and the future motion trajectory and behavior intention of the obstacle are predicted to obtain preliminary prediction results, including: Normalize the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle to obtain a preprocessed feature vector; The preprocessed feature vector is input into the deep reinforcement learning model, the strategy is optimized by maximizing the Q value of the Critic network, and the parameters of the Actor network are updated using the gradient ascent method; Minimize the following loss function to evaluate the difference between the predicted Q value and the target Q value, which is calculated as follows: In the formula, represents the mean square error between the predicted Q value and the target Q value, Represents the model parameters. In this formula represents the parameters of the Q network, N represents the number of samples, Represents the state of the environment at a certain point in time. Indicates the actions taken in a specific state. Indicates in status Take action Expected return, represents the target Q value; Using the soft update method, the parameters of the target network are brought closer to the parameters of the current network, and through the forward propagation of the model, the preliminary prediction results of the future motion trajectory and behavioral intention of the obstacle are obtained. The preliminary prediction results include the position information, speed information, motion direction information and behavioral intention information of the obstacle at the future time point.

7. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: The multi-agent collaborative control algorithm is used to automatically adjust the vehicle's speed, direction and obstacle avoidance strategy in combination with the final optimized motion trajectory and behavior intention prediction, including: Using the Markov game algorithm, a collaborative control model is constructed to obtain the control strategy of each agent. The model regards vehicles and obstacles as multiple agents, each of which has a corresponding state, action, and reward function. The state of the vehicle includes the current position, speed, and direction, and the actions include acceleration, deceleration, and steering. The reward function is designed to encourage the vehicle to drive safely and avoid collisions with obstacles. The control strategy is the best action taken by the agent in different states to achieve the global optimal driving path and obstacle avoidance strategy. At each time step, the control strategy is used to make real-time decisions based on the current position, speed, and direction of the vehicle and obstacles. The real-time action instructions of the vehicle at the current time step are obtained through the results of the real-time decision, and then the corresponding actions are performed to adjust the vehicle's driving speed and direction.

8. A 360-degree surround view image-assisted reversing radar dynamic obstacle recognition system, based on the 360-degree surround view image-assisted reversing radar dynamic obstacle recognition method according to claim 1, characterized in that: include: Processing module: used for denoising and normalizing the polarized light data collected by the bionic polarized skylight sensor on the top of the mining vehicle, and extracting the first feature, wherein the first feature includes the intensity and angle features of the polarized light of the sky; The point cloud data collected by the laser radar is filtered and downsampled to remove noise points and outliers, and the second feature is extracted, where the second feature includes edges, planes and corner points; the distance and speed data measured by the odometer are calibrated to obtain the calibrated third feature, where the third feature includes the mileage and speed data of the odometer; the first feature, the second feature and the third feature are fused to construct a global pose graph, where the global pose graph includes environmental information, pose nodes, pose edges, global constraints and framework information; Extraction module: It is used to process the point cloud data of the lidar through the DBSCAN clustering algorithm, cluster the points close to each other in space, and obtain the preliminary recognition results of dynamic obstacles. Based on the preliminary recognition results, the attention mechanism is used to extract the geometric features of the obstacles, where the geometric features include position and size features, shape and texture features, and motion features; The Zernike invariant moment feature extraction method is used to extract the rotation, translation and magnification invariant historical contour features of the obstacle from the multi-frame data of the global pose graph, and the minimum convex hull method is combined with the fuzzy line segment method to obtain the obstacle's enclosing rectangular frame. The boundary information of the enclosing rectangular frame is extracted to obtain the geometric boundary information of the obstacle, where the geometric boundary information includes the boundary points, edge line end points and inflection points of the obstacle. Identification module: used to compare the extracted geometric features of obstacles with the static features in the pose graph to confirm the identity and motion state of the obstacles, where the static features include fixed structures in the environment; use environmental perception algorithms to analyze the impact of air pressure changes on air flow in the mine, and then obtain environmental factors that affect the movement of obstacles, where environmental factors that affect the movement of dynamic obstacles include airflow resistance, airflow disturbance, and airflow pressure difference; and use thermal imaging data from infrared sensors to identify the thermal features of obstacles, where thermal features include temperature distribution and thermal radiation intensity; Computation module: used to input the geometric boundary information of the obstacle, the identity and motion state of the obstacle, the environmental factors affecting the movement of the obstacle, and the thermal characteristics of the obstacle into the deep reinforcement learning model for calculation, predict the future movement trajectory and behavior intention of the obstacle, and obtain preliminary prediction results, where the preliminary prediction results include movement direction, speed change and behavior pattern; The preliminary prediction results and the actually observed obstacle state data are input into the Kalman filter to update the obstacle state estimation and obtain the final optimized motion trajectory and behavior intention prediction; Adjustment module: It is used to adopt the multi-agent collaborative control algorithm, combined with the final optimized motion trajectory and behavior intention prediction, to automatically adjust the vehicle's driving speed, direction and obstacle avoidance strategy to achieve active obstacle avoidance and safe reversing.

9. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition system according to claim 8, characterized in that: The processing module includes: A first calculation unit is used to perform weighted fusion processing on the first feature and the second feature, and assign different weights based on the importance of polarized light data in the first feature in determining the vehicle heading angle and the accuracy of point cloud data in the second feature in acquiring the geometric structure of the environment, and obtain a preliminary fusion feature through a weighted summation method, wherein the preliminary fusion feature includes information and data of the initial position and heading of the vehicle; Fusion optimization unit: used to fuse the preliminary fusion features and the third feature using the Kalman filter method, combine the vehicle's motion model with the sensor observation data through a prediction-update cycle, and optimize the pose nodes and pose edges in the pose graph using the high-precision geometric information of the lidar in the second feature and the continuity of the odometer in the third feature through a graph optimization processing algorithm; and adjust the position and posture of the pose nodes through iterative optimization to obtain an optimized pose graph framework, where the goal of graph optimization is to minimize the relative pose measurement error between pose nodes and the constraint error of the pose edges; Construction unit: used to extract environmental features from point cloud data, use feature matching algorithm to match pose nodes in the pose graph framework with environmental features, establish the association between pose nodes and environmental features, fuse pose nodes, pose edges, environmental features and global constraint information, and construct a global pose graph.

10. The 360-degree surround view image-assisted reversing radar dynamic obstacle recognition system according to claim 8, characterized in that: The extraction module includes: The first acquisition unit is used to acquire point cloud data containing three-dimensional spatial information of the vehicle's surrounding environment using a laser radar, wherein each point contains its coordinate position and reflection intensity in space; Clustering unit: used to set the neighborhood radius and minimum number of points, traverse each point in the point cloud, and determine whether the number of points in the neighborhood is greater than or equal to MinPts, If so, the point in the point cloud data currently being traversed and evaluated by the DBSCAN algorithm is set as the core point. Starting from the core point, the points in its neighborhood are expanded to form a cluster, and it is checked whether the newly added point is a core point. If so, the expansion continues until no new points can be added; if not, the point cloud data is continued to be traversed, and the clustering result is recorded as the preliminary recognition result of the dynamic obstacle, where each cluster contains the point cloud data belonging to the same obstacle; The second calculation unit is used to calculate the similarity between the feature vector of each point and the overall feature vector of the obstacle based on the preliminary recognition result of the dynamic obstacle, and normalize it using the softmax function to obtain the importance weight of each point. The calculation formula is as follows: In the formula, Indicates The attention weights of the points, the softmax function is used to normalize the weights, For the The feature vector of a point, F is the overall feature vector of the obstacle; The first processing unit is used to classify and weight the point cloud data according to the importance weight of each point to obtain the geometric features of the obstacle.

Images