Intelligent obstacle detection system for driverless subway based on multimodal AI sensors

Abstract

The present invention relates to the field of obstacle detection technology, and discloses an unmanned subway obstacle intelligent detection system based on a multimodal AI sensor, including a multimodal AI sensor, a preprocessing unit, a data processing unit, and a communication unit; in the data preprocessing of the present invention, point cloud filtering denoising, downsampling to reduce the load, image graying to highlight the contour, and Gaussian filtering to remove noise all improve the processing efficiency and detection accuracy. Efficient data fusion establishes multi-coordinate system conversion relationships to avoid blind spot misjudgment and enhance adaptability to complex environments. A comprehensive obstacle detection strategy covers the multi-link processing of laser radar and the application of the Canny algorithm of visual images, which verify each other to improve reliability. When an obstacle is detected, it can be displayed in real time and alarmed in stages, and early warning and dispatching control can be achieved by means of network transmission of information. It can also brake or avoid according to the situation to ensure operational safety, and the system architecture is reasonable, adaptable and scalable, which helps the development of technology.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle detection, and in particular to an intelligent obstacle detection system for unmanned subways based on a multimodal AI sensor. Background Art

[0002] Traditional subway obstacle detection methods often rely on a single type of sensor, such as using only a LiDAR sensor or a visual sensor. When only a LiDAR sensor is used, although it can accurately obtain the distance information and three-dimensional spatial position information of the target object, construct point cloud data, and effectively detect the position and general shape of the obstacle, the specific feature recognition ability of the obstacle is relatively weak, and it is difficult to distinguish different types of obstacles, such as pedestrians, vehicles or other objects.

[0003] Relying solely on visual sensors can obtain rich image information, and image recognition technology can be used to identify the category, texture, color and other characteristics of obstacles, which helps to understand the properties of obstacles in more detail. However, the accuracy of visual sensors in distance measurement is relatively low, especially in severe weather conditions. The image quality will be seriously affected, resulting in a significant decrease in detection accuracy, and may even lead to misjudgment or missed judgment.

[0004] In addition, both single LiDAR sensors and visual sensors have certain limitations when facing the complex and ever-changing subway operating environment. For example, in scenes with drastic changes in light, visual sensors may not be able to capture images clearly due to excessive light or darkness; in environments with a large number of reflective surfaces or interference objects, the point cloud data of LiDAR sensors may be disturbed, affecting the reliability of obstacle detection. Summary of the invention

[0005] The purpose of the present invention is to provide an unmanned subway obstacle intelligent detection system based on a multimodal AI sensor, which solves the technical problems raised in the background technology.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] The driverless subway obstacle intelligent detection system based on multimodal AI sensors includes:

[0008] Multimodal AI sensor, which uses lidar sensor and visual sensor as multimodal AI sensor and is used to collect target data, where the target data includes: using lidar sensor to emit laser beam and receive reflected light, and obtaining point cloud data of target object by measuring the laser round trip time; obtaining image information in front of the train through visual sensor;

[0009] A preprocessing unit, connected to the multimodal AI sensor, for preprocessing the point cloud data acquired by the lidar sensor and the image information acquired by the visual sensor;

[0010] A data processing unit is used to receive the target data pre-processed by the multimodal AI sensor, and perform detection and analysis on the target data, wherein the detection and analysis include data fusion and obstacle detection, and determine obstacle information based on the detection and analysis results;

[0011] The communication unit is used to exchange information between the detection system and the train control system. It is to upload the obstacle information detected by the data processing unit to the train control system in a timely manner to remind the train operator to take corresponding control measures.

[0012] As a further solution of the present invention: the preprocessing method includes point cloud data preprocessing and image information preprocessing.

[0013] As a further solution of the present invention: the point cloud data is preprocessed as follows:

[0014] First, filter the point cloud data to remove noise points;

[0015] Specifically:

[0016] Step A1.01: Mark a single data point in the point cloud data obtained by laser radar scanning as (x i ,y i ,z i ), and mark other points adjacent to a single data point as (x j ,y j ,z j );

[0017] Where i represents the data point number in the point cloud data, j represents the data point number adjacent to the i-th data point, i=1, 2, ...u, u represents the number of all data points in the point cloud data, j=1, 2, ...N, N is the number of other points adjacent to the i-th data point in the point cloud data;

[0018] Step A1.02, then pass:

[0019]

[0020] Calculate the distance d between each data point in the point cloud data and each of its adjacent points ij ;

[0021] Step A1.03, then:

[0022]

[0023] Calculate the average distance d between each data point in the point cloud data and all its adjacent points avg，i ;

[0024] Where N is the number of corresponding data points and other adjacent points in the point cloud data;

[0025] Step A1.04: calculate the average distance d between each data point in the point cloud data and all its adjacent points avg，i Compare with the preset distance threshold Td:

[0026] If d avg，i >Td, it means that the i-th data point in the point cloud data is a noise point, and it will be removed;

[0027] If d avg，i ≤Td, it means that the i-th data point in the point cloud data is not a noise point, and it is retained;

[0028] Then, down-sampling of point cloud data is performed to reduce the amount of data;

[0029] The voxel grid downsampling algorithm is used to divide the three-dimensional space into voxel grids of fixed size. For each point in the voxel grid, its centroid point is taken as a representative point to be retained;

[0030] Specifically:

[0031] Step A1.11. Extract the coordinates of all points in the voxel grid and label them as (x k ,y k ,z k ), k = 1, 2, ... M, M represents the number of all points in the voxel grid;

[0032] Step A1.12, by:

[0033]

[0034] Calculate the coordinates of the centroid (x c ,y c ,z c );

[0035] As a further solution of the present invention: wherein, the determination of other points adjacent to a single data point is achieved through a pre-set specified distance range, and the other adjacent points refer to other single data points whose distances to the single data point are within the set distance range.

[0036] As a further solution of the present invention: the image information preprocessing method is as follows:

[0037] First, the image information obtained by the visual sensor is grayed out, which is to convert the color image in the image information into a gray image to reduce the amount of data and highlight the contour features of the image;

[0038] The grayscale processing method is as follows:

[0039] Step A2.01, extract the pixel points of the color image in the image information, and mark the pixel point coordinates as (p t ,q t ), where t=1, 2, ...e, e represents the number of pixel coordinates;

[0040] Step A2.02: As the coordinates of each pixel point are obtained (p t ,q t ) in the red, green, and blue channels, respectively, and are recorded as R (p t ,q t )、G(p t ,q t )、B(p t ,q t );

[0041] Step A2.03, then by:

[0042]

[0043] Calculate the grayscale value H (x, y) corresponding to the coordinates of each pixel point;

[0044] Then, the image information acquired by the visual sensor is filtered to remove image noise;

[0045] It uses Gaussian filtering algorithm;

[0046] For each pixel in the image, the gray value calculation formula after filtering is:

[0047]

[0048] in, is the Gaussian kernel function, for The standard deviation of , w is the filter window size, is the normalization coefficient.

[0049] As a further solution of the present invention: the data fusion method is as follows:

[0050] Realize the fusion of lidar point cloud data and visual image data, and establish the coordinate transformation relationship between the radar coordinate system, visual sensor coordinate system, and pixel coordinate system;

[0051] The specific method is as follows:

[0052] According to the installation position, angle and calibration parameters of the laser radar and visual sensor, the laser radar coordinate system (x s1 ,y s1 ,z s1 )、Visual sensor coordinate system(x s2 ,y s2 ,z s2 ), pixel coordinate system (x s3 ,y s3 );

[0053] The laser radar coordinate system is a three-dimensional coordinate system established with the laser radar itself as a reference, with the center position of the laser radar as the origin. The direction of the coordinate axis is determined by the installation method and design of the laser radar. It is used to describe the position of the target point detected by the laser radar in its own reference system.

[0054] The vision sensor coordinate system is a three-dimensional coordinate system built with the vision sensor as the center. The origin is located at the optical center of the camera. The direction of the coordinate axis is related to the imaging geometry of the camera. It is used to describe the position of the target object relative to the camera within the field of view of the vision sensor.

[0055] The pixel coordinate system is based on the coordinate system of the imaging plane of the visual sensor, but is measured in pixels. The origin is determined by the corners of the image, and the coordinate axes are parallel to the horizontal and vertical directions of the image.

[0056] The points in the laser radar coordinate system are transformed into the visual sensor coordinate system through the rotation matrix and translation vector;

[0057] The conversion formula is:

[0058]

[0059] Among them, R s1s2 and T s1s2 are the rotation matrix and translation vector between the laser radar coordinate system and the visual sensor coordinate system respectively;

[0060] The visual sensor coordinate system is converted to the pixel coordinate system with the help of the camera intrinsic parameter matrix;

[0061] The conversion formula is:

[0062]

[0063] Among them, K s2s3 is the camera intrinsic parameter matrix, and , fx, fy are the components of the camera focal length in the x and y directions, cx, cy are the coordinates of the image center;

[0064] The image coordinate system is converted to the pixel coordinate system through scaling and offset transformation.

[0065] As a further solution of the present invention: the obstacle detection method includes obstacle detection based on laser radar point cloud data and obstacle detection based on visual image data.

[0066] As a further solution of the present invention: obstacle detection based on laser radar point cloud data is as follows:

[0067] Step B2.01: Extraction of region of interest

[0068] First, retrieve the drivable area corresponding to the road surface and intersection from the map, and remove the background objects corresponding to the roadside buildings and trees;

[0069] Then, according to the pre-set regional boundary conditions, it is determined whether the points in the point cloud data are within the region of interest, and if not, they are removed;

[0070] Step B2.02: Ground segmentation

[0071] Detect road edges and fit cubic curve equation:

[0072] Y=β 0 +β 1 X+β 2 X 2 +β 3 X 3

[0073] where β 0 , β 1 , β 2 , β 3 is the fitting coefficient;

[0074] Then, determine whether the point is a ground point based on the distance from the point to the curve;

[0075] If the distance is less than the corresponding preset judgment threshold, the corresponding point is considered to be a ground point and is removed;

[0076] If the distance is greater than the corresponding preset judgment threshold, the corresponding point is considered not to be a ground point and is judged as an obstacle point on the ground;

[0077] Step B2.03, using the Euclidean distance clustering algorithm to form multiple clusters of obstacle points on the ground, and then segmenting the obstacles;

[0078] Extract two obstacle points from it, and then calculate the Euclidean distance between the two obstacle points. Only then compare the obtained distance value with the preset clustering threshold:

[0079] If the obtained distance value is less than the clustering threshold, the two obstacle points are classified into the same cluster;

[0080] Step B2.04, fitting a bounding box for each cluster to determine a three-dimensional bounding box for enclosing the point cloud cluster;

[0081] Then, the coordinate values ​​of all points in the three-dimensional bounding box are obtained, and the average value is taken as the center point of the three-dimensional bounding box;

[0082] Step B2.05, obtaining point cloud data corresponding to two consecutive time nodes, and then determining the center point of the corresponding three-dimensional bounding box through steps B2.01 to B2.04;

[0083] Then the center points of the three-dimensional bounding boxes corresponding to the two time nodes are used to calculate the displacement values ​​of the obstacles at the two time nodes through the Euclidean distance formula;

[0084] Then the moving speed of the obstacle is calculated by using the speed formula by combining the position value and the time difference between the two time nodes;

[0085] Then, the motion state of the obstacle is determined based on whether the moving speed value is 0. If the moving speed value is 0, it means that the obstacle is a dynamic object; if the moving speed value is not 0, it means that the obstacle is a static object.

[0086] As a further solution of the present invention: obstacle detection based on visual image data is as follows:

[0087] The Canny edge detection algorithm is used to perform edge detection on the preprocessed grayscale image;

[0088] Step B2.11, obtain the gradient of the image in the horizontal and vertical directions, and record it as G x (p, q) and G y (p, q);

[0089] Among them, the Sobel operator of the image gradient in the horizontal and vertical directions is calculated:

[0090] The specific method is as follows:

[0091] First, obtain the predefined gradient Sobel operator template in the horizontal direction:

[0092]

[0093] The template in the vertical direction is:

[0094]

[0095] Convolve the template with the image to calculate the gradient;

[0096] The formula is as follows:

[0097]

[0098] In the formula, is the pixel value of the image at the pixel point (p+b1, q+b2);

[0099] Among them, b2 represents the offset of the template in the vertical direction, and b1 represents the offset of the template in the horizontal direction;

[0100] And the value range of b1 and b2 is from -1 to 1;

[0101] Step B2.12, then pass:

[0102]

[0103] Calculate the gradient magnitude G(p, q) and gradient direction θ(p, q) of the image;

[0104] Step B2.13, then obtain the edge image of the obstacle through non-maximum suppression and double threshold detection.

[0105] As a further solution of the present invention: when an obstacle is detected, it is displayed in real-time image form and an alarm is sounded through an audible and visual alarm;

[0106] The sound and light alarm has two levels of alarm modes. When the first level sound and light alarm is on, the indicator light is red and there is a voice prompt; when the second level sound and light alarm is on, the indicator light is orange and there is a voice prompt.

[0107] At the same time, the onboard control host sends the alarm information to the train monitoring system through the PIS network, so that the train monitoring system can transmit the alarm information to the control center for early warning, and implement dispatching control for serious faults to ensure the safety of train operation;

[0108] And according to the location of the obstacle and the running status of the train, the train is controlled to brake or take other avoidance measures.

[0109] Beneficial effects of the present invention:

[0110] High-precision detection: Through the multi-modal data collection of LiDAR sensors and visual sensors, combined with their respective advantages in distance measurement and image recognition, obstacles in front of the train can be detected more comprehensively and accurately. The point cloud data obtained by LiDAR provides accurate three-dimensional spatial position information, which can accurately determine the position and shape of obstacles; the image information of the visual sensor helps to identify the type, texture and other characteristics of the obstacle. The two complement each other and effectively improve the accuracy of obstacle detection.

[0111] Data preprocessing optimization: The filtering process in point cloud data preprocessing can effectively remove noise points, reduce data interference, and improve the quality and reliability of subsequent processing data; downsampling processing can significantly reduce the amount of data without losing key information, reduce the system's computational burden, and improve processing efficiency, ensuring that the system can process a large amount of point cloud data in real time and meet the strict real-time requirements of unmanned subways; the grayscale processing in image information preprocessing converts color images into grayscale images, which not only reduces the amount of data, but also highlights the contour features of the image, which is conducive to subsequent image processing operations such as edge detection; the Gaussian filtering algorithm effectively removes image noise, makes the image clearer, and further improves the obstacle detection accuracy based on visual image data.

[0112] Efficient data fusion: The precise coordinate conversion relationship between the laser radar coordinate system, the visual sensor coordinate system and the pixel coordinate system is established, realizing the seamless fusion of the laser radar point cloud data and the visual image data. This fusion method can fully integrate the data of the two sensors, allowing the system to perceive and analyze obstacles from multiple dimensions, avoiding the blind spots or misjudgments that may exist in a single sensor, thereby providing a richer and more accurate information basis for obstacle detection and identification, and enhancing the system's ability to adapt to complex environments.

[0113] Comprehensive obstacle detection strategy: The obstacle detection process based on LiDAR point cloud data covers multiple links such as region of interest extraction, ground segmentation, clustering algorithm, bounding box fitting, and motion state judgment. Through region of interest extraction, we can quickly focus on areas where obstacles may exist, reducing unnecessary calculations; ground segmentation can accurately distinguish between the ground and obstacles, improving the pertinence of obstacle detection; Euclidean distance clustering algorithm and bounding box fitting can effectively identify and locate multiple obstacles, and the processing of continuous time node data can determine the motion state of obstacles, providing an important basis for the dynamic decision-making of trains, helping trains to plan their routes in advance or take appropriate avoidance measures; obstacle detection based on visual image data uses the Canny edge detection algorithm, which can accurately extract the edge image of obstacles by calculating the gradient amplitude and direction, and combining non-maximum suppression and dual threshold detection, and further identify the outline and features of obstacles, which can be mutually verified with the LiDAR detection results to improve the reliability and accuracy of the overall detection.

[0114] Integration of safety warning and control: When an obstacle is detected, the system can intuitively display it in real-time images and issue graded alarms through sound and light alarms. The first-level sound and light alarm can promptly attract the attention of train operators, and the second-level sound and light alarm provides further warning information. At the same time, the on-board control host uses the PIS network to quickly send the alarm information to the train monitoring system, and then transmit it to the control center for early warning, so as to facilitate effective dispatching control in the event of serious failures and ensure the safety of the entire train operation network. In addition, according to the location of the obstacle and the operating status of the train, the system can intelligently control the train to brake or take other avoidance measures, such as triggering emergency braking when the distance between the obstacle and the train is less than the safety threshold, to avoid collision accidents to the greatest extent, and ensure the operation safety of the unmanned subway and the safety of passengers' lives and property.

[0115] System adaptability and scalability: The detection system architecture of the present invention is reasonably designed, and the functions of each module are clear and closely coordinated with each other, which has strong adaptability and scalability. It can be flexibly adjusted and optimized according to different driverless subway line environments, operation requirements and the development of sensor technology. For example, it is convenient to update sensor parameters, improve data processing algorithms or add new functional modules to adapt to increasingly complex traffic scenarios and continuously improving safety standards, providing strong support for the continuous development and improvement of driverless subway technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] The present invention will be further described below in conjunction with the accompanying drawings.

[0117] Figure 1 It is a system block diagram of the unmanned subway obstacle intelligent detection system based on multimodal AI sensors of the present invention. DETAILED DESCRIPTION

[0118] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0119] Embodiment 1

[0120] See also Figure 1 As shown, the present invention is an unmanned subway obstacle intelligent detection system based on a multimodal AI sensor, comprising:

[0121] Multimodal AI sensor, which uses lidar sensor and visual sensor as multimodal AI sensor and is used to collect target data;

[0122] The target data include:

[0123] The laser radar sensor emits a laser beam and receives the reflected light, and obtains the point cloud data of the target object by measuring the laser round-trip time;

[0124] The lidar sensor emits a laser beam at a specified sampling period to scan the front of the train and obtain point cloud data;

[0125] In this embodiment, the sampling period is 50ms and the sampling frame rate is 20 frames / second;

[0126] Point cloud data contains the three-dimensional spatial position information of the target object. The coordinates of each point can be expressed as (x, y, z), where x and y represent the horizontal and vertical positions, and z represents the distance information.

[0127] Acquire image information in front of the train through visual sensors;

[0128] The visual sensor collects image data in front of the train at a specified sampling frame rate. The image data is represented as a two-dimensional matrix. Each element in the matrix corresponds to a pixel in the image, and the pixel value represents the information corresponding to the color and brightness.

[0129] In this embodiment, the sampling frame rate is 25 frames / second;

[0130] A data processing unit, used to receive target data from a multimodal AI sensor and perform detection, analysis and processing on the target data, wherein the detection, analysis and processing includes data fusion and obstacle detection;

[0131] Step B1: Data fusion

[0132] Realize the fusion of lidar point cloud data and visual image data, and establish the coordinate transformation relationship between the radar coordinate system, visual sensor coordinate system, and pixel coordinate system;

[0133] The specific method is as follows:

[0134] According to the installation position, angle and calibration parameters of the laser radar and visual sensor, the laser radar coordinate system (x s1 ,y s1 ,z s1 )、Visual sensor coordinate system(x s2 ,y s2 ,z s2 ), pixel coordinate system (x s3 ,y s3 );

[0135] The laser radar coordinate system is a three-dimensional coordinate system established with the laser radar itself as a reference, with the center position of the laser radar as the origin. The direction of the coordinate axis is determined by the installation method and design of the laser radar. It is used to describe the position of the target point detected by the laser radar in its own reference system.

[0136] The vision sensor coordinate system is a three-dimensional coordinate system built with the vision sensor as the center. The origin is located at the optical center of the camera. The direction of the coordinate axis is related to the imaging geometry of the camera. It is used to describe the position of the target object relative to the camera within the field of view of the vision sensor.

[0137] The pixel coordinate system is based on the coordinate system of the imaging plane of the visual sensor, but is measured in pixels. The origin is determined by the corners of the image, and the coordinate axes are parallel to the horizontal and vertical directions of the image.

[0138] The points in the laser radar coordinate system are transformed into the visual sensor coordinate system through the rotation matrix and translation vector;

[0139] The conversion formula is:

[0140]

[0141] Among them, R s1s2 and T s1s2 are the rotation matrix and translation vector between the laser radar coordinate system and the visual sensor coordinate system respectively;

[0142] The visual sensor coordinate system is converted to the pixel coordinate system with the help of the camera intrinsic parameter matrix;

[0143] The conversion formula is:

[0144]

[0145] Among them, K s2s3 is the camera intrinsic parameter matrix, and , fx, fy are the components of the camera focal length in the x and y directions, cx, cy are the coordinates of the image center;

[0146] The image coordinate system is converted to the pixel coordinate system through scaling and offset transformation;

[0147] In this embodiment, the camera intrinsic parameter matrix directly converts points in the visual sensor coordinate system to the image coordinate system with pixels as units, so that the pixel coordinates can be conveniently used to locate and analyze objects in the image in subsequent digital image processing;

[0148] Step B2: Obstacle Detection

[0149] Step B2.0: Obstacle detection based on LiDAR point cloud data

[0150] Step B2.01: Extraction of region of interest

[0151] First, retrieve the drivable area corresponding to the road surface and intersection from the map, and remove the background objects corresponding to the roadside buildings and trees;

[0152] Then, according to the pre-set regional boundary conditions, it is determined whether the points in the point cloud data are within the region of interest, and if not, they are removed;

[0153] Step B2.02: Ground segmentation

[0154] Detect road edges and fit cubic curve equation:

[0155] Y=β 0 +β 1 X+β 2 X 2 +β 3 X 3

[0156] where β 0 , β 1 , β 2 , β 3 is the fitting coefficient;

[0157] Then, determine whether the point is a ground point based on the distance from the point to the curve;

[0158] If the distance is less than the corresponding preset judgment threshold, the corresponding point is considered to be a ground point and is removed;

[0159] If the distance is greater than the corresponding preset judgment threshold, the corresponding point is considered not to be a ground point and is judged as an obstacle point on the ground;

[0160] Step B2.03, using the Euclidean distance clustering algorithm to form multiple clusters of obstacle points on the ground, and then segmenting the obstacles;

[0161] Extract two obstacle points from it, and then calculate the Euclidean distance between the two obstacle points. Only then compare the obtained distance value with the preset clustering threshold:

[0162] If the obtained distance value is less than the clustering threshold, the two obstacle points are classified into the same cluster;

[0163] Step B2.04, fitting a bounding box for each cluster to determine a three-dimensional bounding box for enclosing the point cloud cluster;

[0164] Then, the coordinate values ​​of all points in the three-dimensional bounding box are obtained, and the average value is taken as the center point of the three-dimensional bounding box;

[0165] Step B2.05, obtaining point cloud data corresponding to two consecutive time nodes, and then determining the center point of the corresponding three-dimensional bounding box through steps B2.01 to B2.04;

[0166] Then the center points of the three-dimensional bounding boxes corresponding to the two time nodes are used to calculate the displacement values ​​of the obstacles at the two time nodes through the Euclidean distance formula;

[0167] Then the moving speed of the obstacle is calculated by using the speed formula by combining the position value and the time difference between the two time nodes;

[0168] Then, the motion state of the obstacle is determined based on whether the moving speed value is 0. If the moving speed value is 0, it means that the obstacle is a dynamic object; if the moving speed value is not 0, it means that the obstacle is a static object.

[0169] Step B2.1: Obstacle detection based on visual image data

[0170] The Canny edge detection algorithm is used to perform edge detection on the preprocessed grayscale image;

[0171] Step B2.11, obtain the gradient of the image in the horizontal and vertical directions, and record it as G x (p, q) and G y (p, q);

[0172] In this embodiment, the Sobel operator is used to calculate the gradient of the image in the horizontal and vertical directions:

[0173] The specific method is as follows:

[0174] First, obtain the predefined gradient Sobel operator template in the horizontal direction:

[0175]

[0176] The template in the vertical direction is:

[0177]

[0178] Convolve the template with the image to calculate the gradient;

[0179] The formula is as follows:

[0180]

[0181] In the formula, is the pixel value of the image at the pixel point (p+b1, q+b2);

[0182] Among them, b2 represents the offset of the template in the vertical direction, and b1 represents the offset of the template in the horizontal direction;

[0183] And the value range of b1 and b2 is from -1 to 1;

[0184] Step B2.12, then pass:

[0185]

[0186] Calculate the gradient magnitude G(p, q) and gradient direction θ(p, q) of the image;

[0187] Step B2.13, then obtain the edge image of the obstacle through non-maximum suppression and double threshold detection;

[0188] Among them, non-maximum suppression and dual threshold detection are existing technologies, so they are not described in detail here.

[0189] This embodiment collects target data through laser radar sensors and visual sensors, performs data fusion, and establishes a conversion relationship between multiple coordinate systems. This fusion method enables the system to integrate the precise distance information of the laser radar and the rich image information of the visual sensor, perceive and analyze the environment in front of the train from different dimensions, effectively avoid the limitations of single sensor detection, and improve the comprehensiveness and accuracy of obstacle detection. In the obstacle detection process based on laser radar point cloud data, the extraction of regions of interest can focus on key areas, reduce the amount of calculation and improve detection efficiency; ground segmentation uses fitting curves to accurately distinguish between the ground and obstacles; Euclidean distance clustering algorithm and bounding box fitting can accurately identify and locate multiple obstacles; the obstacle movement speed and state are calculated through continuous time node data processing, which provides an important basis for the dynamic decision-making of the train, and helps the train to plan the driving path in advance or take avoidance measures to ensure driving safety. The Canny edge detection algorithm is used to process the visual image. The gradient of the image in the horizontal and vertical directions is obtained through a specific gradient calculation method, and then the gradient amplitude and direction are calculated. The edge image of the obstacle is obtained by combining non-maximum suppression and double threshold detection. This can effectively extract the contour information of the obstacle, assist in identifying the position and shape of the obstacle in the image, complement and verify the lidar detection results, and further improve the overall detection accuracy.

[0190] Embodiment 2

[0191] As the second embodiment of the present invention, when the present application is implemented, compared with the first embodiment, the technical solution of the present embodiment is different from that of the first embodiment only in that the present embodiment further includes:

[0192] A preprocessing unit, used for preprocessing the point cloud data acquired by the laser radar sensor and the image information acquired by the visual sensor;

[0193] The preprocessing method is as follows:

[0194] Step A1: Point cloud data preprocessing

[0195] First, filter the point cloud data to remove noise points;

[0196] Specifically:

[0197] Step A1.01: Mark a single data point in the point cloud data obtained by laser radar scanning as (x i ,y i ,z i ), and mark other points adjacent to a single data point as (x j ,y j ,z j );

[0198] Among them, other adjacent points refer to other data points that are spatially close to the current single data point within the laser radar scanning range. A reasonable distance range is determined in advance according to the scanning resolution and accuracy of the laser radar, and other data points within the distance range are defined as other adjacent points;

[0199] In this embodiment, the distance range corresponding to other adjacent points is set within a radius of 0.1 meters, and other data points within a distance of less than 0.1 meters from a single data point are defined as other adjacent points;

[0200] Where i represents the data point number in the point cloud data, j represents the data point number adjacent to the i-th data point, i=1, 2, ...u, u represents the number of all data points in the point cloud data, j=1, 2, ...N, N is the number of other points adjacent to the i-th data point in the point cloud data;

[0201] Step A1.02, then pass:

[0202]

[0203] Calculate the distance d between each data point in the point cloud data and each of its adjacent points ij ;

[0204] Step A1.03, then:

[0205]

[0206] Calculate the average distance d between each data point in the point cloud data and all its adjacent points avg，i ;

[0207] Where N is the number of corresponding data points and other adjacent points in the point cloud data;

[0208] Step A1.04: calculate the average distance d between each data point in the point cloud data and all its adjacent points avg，i Compare with the preset distance threshold Td:

[0209] If d avg，i >Td, it means that the i-th data point in the point cloud data is a noise point, and it will be removed;

[0210] If d avg，i ≤Td, it means that the i-th data point in the point cloud data is not a noise point, and it is retained;

[0211] Then, down-sampling of point cloud data is performed to reduce the amount of data;

[0212] The voxel grid downsampling algorithm is used to divide the three-dimensional space into voxel grids of fixed size. For each point in the voxel grid, its centroid point is taken as a representative point to be retained;

[0213] Specifically:

[0214] Step A1.11. Extract the coordinates of all points in the voxel grid and label them as (x k ,y k ,z k ), k = 1, 2, ... M, M represents the number of all points in the voxel grid;

[0215] Step A1.12, by:

[0216]

[0217] Calculate the coordinates of the centroid (x c ,y c ,z c );

[0218] Step A2: Image information preprocessing

[0219] First, the image information obtained by the visual sensor is grayed out, which is to convert the color image in the image information into a gray image to reduce the amount of data and highlight the contour features of the image;

[0220] The grayscale processing method is as follows:

[0221] Step A2.01, extract the pixel points of the color image in the image information, and mark the pixel point coordinates as (p t ,q t ), where t=1, 2, ...e, e represents the number of pixel coordinates;

[0222] Step A2.02: As the coordinates of each pixel point are obtained (p t ,q t ) in the red, green, and blue channels, respectively, and are recorded as R (p t ,q t )、G(p t ,q t )、B(p t ,q t );

[0223] Step A2.03, then by:

[0224]

[0225] Calculate the grayscale value H (x, y) corresponding to the coordinates of each pixel point;

[0226] Then, the image information acquired by the visual sensor is filtered to remove image noise;

[0227] It uses Gaussian filtering algorithm;

[0228] For each pixel in the image, the gray value calculation formula after filtering is:

[0229]

[0230] in, is the Gaussian kernel function, for The standard deviation of , w is the filter window size, is the normalization coefficient.

[0231] This embodiment adds a preprocessing unit on the basis of the first embodiment. The filtering process in the point cloud data preprocessing can effectively remove noise points, reduce data interference, and improve data quality, making the subsequent obstacle detection based on point cloud data more reliable; the downsampling process greatly reduces the amount of data without losing key information, reduces the computational complexity of data processing, and improves the processing efficiency of the system, ensuring that the system can meet the strict real-time requirements of the driverless subway and respond to and process obstacle information in a timely manner. The grayscale processing in the image information preprocessing converts the color image into a grayscale image, reducing the amount of data while highlighting the contour features of the image, which is more conducive to subsequent image processing operations such as edge detection; the Gaussian filtering algorithm effectively removes image noise, making the image clearer, enhancing the accuracy and stability of obstacle detection based on visual image data, and further improving the detection performance of the entire system in complex environments.

[0232] Embodiment 3

[0233] As the third embodiment of the present invention, when the present application is implemented, compared with the first and second embodiments, the technical solution of this embodiment is to combine the solutions of the first and second embodiments, and the difference between the technical solution of this embodiment and the first and second embodiments is that this embodiment also includes:

[0234] The communication unit is used to exchange information between the detection system and the train control system, and to upload the obstacle information detected by the data processing unit to the train control system in a timely manner;

[0235] When an obstacle is detected, it will be displayed in real-time image and an alarm will be sounded through the sound and light alarm;

[0236] The sound and light alarm has two levels of alarm modes. When the first level sound and light alarm is on, the indicator light is red and there is a voice prompt; when the second level sound and light alarm is on, the indicator light is orange and there is a voice prompt.

[0237] At the same time, the onboard control host sends the alarm information to the train monitoring system through the PIS network, so that the train monitoring system can transmit the alarm information to the control center for early warning, and implement dispatching control for serious faults to ensure the safety of train operation;

[0238] And according to the location of the obstacle and the running status of the train, the train is controlled to brake or take other avoidance measures. For example, when the distance between the obstacle and the train is less than the safety threshold, emergency braking is triggered.

[0239] This embodiment adds a communication unit on the basis of the first and second embodiments. The communication unit realizes the information interaction between the detection system and the train control system. When an obstacle is detected, the obstacle information can be uploaded to the train control system in time. At the same time, the obstacle is displayed in real-time image mode, and a two-level alarm is performed through the sound and light alarm, which can quickly attract the attention of the train operator and prompt him to take corresponding measures. The on-board control host sends the alarm information to the train monitoring system through the PIS network, and then transmits it to the control center for early warning, which is convenient for implementing effective dispatching control in the case of serious faults and ensuring the safety of the entire train operation network. In addition, according to the location of the obstacle and the running status of the train, the system can intelligently control the train to brake or take other avoidance measures, such as triggering emergency braking when the distance between the obstacle and the train is less than the safety threshold, avoiding the occurrence of collision accidents to the greatest extent, ensuring the operation safety of the unmanned subway and the safety of passengers' lives and property, and realizing the close linkage and collaborative work between the detection system and the train operation control.

[0240] Embodiment 4

[0241] As the fourth embodiment of the present invention, when the present application is specifically implemented, compared with the first, second and third embodiments, the technical solution of this embodiment is to combine and implement the solutions of the above-mentioned first, second and third embodiments.

[0242] This embodiment integrates all the technical solutions of the first, second and third embodiments, and therefore has the beneficial effects of all the above embodiments. That is, it realizes the collection and fusion of multimodal data, optimized data preprocessing, accurate obstacle detection, and a complete safety warning and train control linkage mechanism, comprehensively improving the performance of the unmanned subway obstacle intelligent detection system, and performs well in ensuring the safe operation of trains, adapting to complex environments, improving system efficiency, and realizing overall system coordination, providing strong technical support for the stable and safe operation of unmanned subways.

[0243] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0244] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. Unmanned subway obstacle intelligent detection system based on multimodal AI sensor, characterized by: include: Multimodal AI sensor, which uses lidar sensor and visual sensor as multimodal AI sensor and is used to collect target data, where the target data includes: using lidar sensor to emit laser beam and receive reflected light, and obtaining point cloud data of target object by measuring the laser round trip time; obtaining image information in front of the train through visual sensor; A preprocessing unit connected to the multimodal AI sensor, which is used to preprocess the point cloud data acquired by the laser radar sensor and the image information acquired by the visual sensor, wherein the preprocessing method includes point cloud data preprocessing and image information preprocessing; A data processing unit is used to receive target data preprocessed by the multimodal AI sensor, and perform detection, analysis and processing on the target data, wherein the detection, analysis and processing include data fusion and obstacle detection, and the obstacle detection method includes obstacle detection based on lidar point cloud data and obstacle detection based on visual image data, and obstacle information is determined based on the detection, analysis and processing results; Obstacle detection based on LiDAR point cloud data is as follows: Step B2.01, first retrieve the drivable area corresponding to the road surface and intersection from the map, and remove the background objects corresponding to the roadside buildings and trees; Then, according to the pre-set regional boundary conditions, it is determined whether the points in the point cloud data are within the region of interest, and if not, they are removed; Step B2.02: Detect the road edge and fit the cubic curve equation: Y=β0+β1X+β2X 2 +β3X 3 Among them, β0, β1, β2, and β3 are fitting coefficients; Then, determine whether the point is a ground point based on the distance from the point to the curve; If the distance is less than the corresponding preset judgment threshold, the corresponding point is considered to be a ground point and is removed; If the distance is greater than the corresponding preset judgment threshold, the corresponding point is considered not to be a ground point and is judged as an obstacle point on the ground; Step B2.03, using the Euclidean distance clustering algorithm to form multiple clusters of obstacle points on the ground, and then segmenting the obstacles; Extract two obstacle points from it, and then calculate the Euclidean distance between the two obstacle points. Only then compare the obtained distance value with the preset clustering threshold: If the obtained distance value is less than the clustering threshold, the two obstacle points are classified into the same cluster; Step B2.04, fitting a bounding box for each cluster to determine a three-dimensional bounding box for enclosing the point cloud cluster; Then, the coordinate values ​​of all points in the three-dimensional bounding box are obtained, and the average value is taken as the center point of the three-dimensional bounding box; Step B2.05, obtaining point cloud data corresponding to two consecutive time nodes, and then determining the center point of the corresponding three-dimensional bounding box through steps B2.01 to B2.04; Then the center points of the three-dimensional bounding boxes corresponding to the two time nodes are used to calculate the displacement values ​​of the obstacles at the two time nodes through the Euclidean distance formula; Then the moving speed of the obstacle is calculated by using the speed formula using the position value and the time difference between the two time nodes; Then, the motion state of the obstacle is determined based on whether the moving speed value is 0. If the moving speed value is 0, it means that the obstacle is a dynamic object; if the moving speed value is not 0, it means that the obstacle is a static object. Obstacle detection based on visual image data is as follows: Step B2.11: Get the gradient of the image in the horizontal and vertical directions and record it as G x (p, q) and G y (p, q); Step B2.12, then pass: ； Calculate the gradient magnitude G(p, q) and gradient direction θ(p, q) of the image; Step B2.13, then obtain the edge image of the obstacle through non-maximum suppression and double threshold detection; The communication unit is used to exchange information between the detection system and the train control system, and upload the obstacle information detected by the data processing unit to the train control system.

2. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 1 is characterized in that: Point cloud data preprocessing is as follows: First, filter the point cloud data to remove noise points; Then, down-sampling of point cloud data is performed to reduce the amount of data; The downsampling process uses a voxel grid downsampling algorithm to divide the three-dimensional space into voxel grids of fixed size. For each point in the voxel grid, its centroid point is taken as a representative point to be retained.

3. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 2 is characterized in that: The filtering process of point cloud data is as follows: Step A1.01: Mark a single data point in the point cloud data obtained by laser radar scanning as (x i ,y i ,z i ), and mark other points adjacent to a single data point as (x j ,y j ,z j ); Among them, other adjacent points refer to other single data points whose distance from a single data point is within a preset distance range, i represents the sequence number of the data point in the point cloud data, and j represents the sequence number of the data point adjacent to the i-th data point; Step A1.02, then pass: ； Calculate the distance d between each data point in the point cloud data and each of its adjacent points ij ; Step A1.03, then: ； Calculate the average distance d between each data point in the point cloud data and all its adjacent points avg，i ; Where N is the number of corresponding data points and other adjacent points in the point cloud data; Step A1.04: calculate the average distance d between each data point in the point cloud data and all its adjacent points avg，i Compare with the preset distance threshold Td: If d avg，i >Td, it means that the i-th data point in the point cloud data is a noise point, and it will be removed; If d avg，i ≤Td, it means that the i-th data point in the point cloud data is not a noise point, and it is retained.

4. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 2 is characterized in that: The downsampling process is as follows: Step A1.

11. Extract the coordinates of all points in the voxel grid and label them as (x k ,y k ,z k ), k = 1, 2, ... M, M represents the number of all points in the voxel grid; Step A1.12, by: ； Calculate the coordinates of the centroid (x c ,y c ,z c ).

5. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 1 is characterized in that: The image information preprocessing method is as follows: First, the image information obtained by the visual sensor is grayed out, which is to convert the color image in the image information into a gray image to reduce the amount of data and highlight the contour features of the image; Then, the image information obtained by the visual sensor is filtered to remove image noise.

6. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 5 is characterized in that: The grayscale processing method is as follows: Step A2.01, extract the pixel points of the color image in the image information, and mark the pixel point coordinates as (p t ,q t ), where t=1, 2, ...e, e represents the number of pixel coordinates; Step A2.02: As the coordinates of each pixel point are obtained (p t ,q t ) in the red, green, and blue channels, respectively, and are recorded as R (p t ,q t )、G(p t ,q t )、B(p t ,q t ); Step A2.03, then by: ； Calculate the grayscale value H (x, y) corresponding to the coordinates of each pixel point.

7. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 5 is characterized in that: Image filtering uses Gaussian filtering algorithm, which is as follows; For each pixel in the image, the gray value calculation formula after filtering is: ； in, is the Gaussian kernel function, for The standard deviation of , w is the filter window size, is the normalization coefficient.

8. The unmanned subway obstacle intelligent detection system based on multimodal AI sensor according to claim 1 is characterized in that: The data fusion method is as follows: According to the installation position, angle and calibration parameters of the laser radar and visual sensor, the laser radar coordinate system (x s1 ,y s1 ,z s1 )、Visual sensor coordinate system(x s2 ,y s2 ,z s2 ), pixel coordinate system (x s3 ,y s3 ); The points in the laser radar coordinate system are transformed into the visual sensor coordinate system through the rotation matrix and translation vector; The conversion formula is: ； Among them, R s1s2 and T s1s2 are the rotation matrix and translation vector between the laser radar coordinate system and the visual sensor coordinate system respectively; The visual sensor coordinate system is converted to the pixel coordinate system with the help of the camera intrinsic parameter matrix; The conversion formula is: ； Among them, K s2s3 is the camera intrinsic parameter matrix, and , fx, fy are the components of the camera focal length in the x and y directions, cx, cy are the coordinates of the image center; The image coordinate system is converted to the pixel coordinate system through scaling and offset transformation.

Images