Accurate recognition system and method for road surface obstacles of environmental sanitation unmanned vehicle

Abstract

The invention relates to an accurate recognition system and method for road obstacles of an environmental sanitation unmanned vehicle, and the system comprises a multi-modal data collection module which is used for collecting the road environment data in the driving process of the environmental sanitation unmanned vehicle; the data preprocessing module is used for carrying out cleaning, alignment and format conversion processing on the collected data; the improved SBL algorithm operation module is used for receiving the standardized data, dynamically optimizing feature extraction weights according to current pavement environment features, calling an improved EM iterative algorithm, fusing visual image features after weight optimization with laser radar point cloud features, and constructing a dynamically updated sparse dictionary; sparse reconstruction and probability estimation are carried out on the fused feature data through an improved SBL algorithm, and a preliminary recognition result of the obstacle is obtained; and the obstacle information output module is used for analyzing and formatting the preliminary identification result and outputting the obstacle information. The obstacle recognition precision and response speed of the unmanned vehicle can be improved.

Description

Technical Field

[0001] This invention belongs to the field of unmanned sanitation vehicle technology, specifically relating to a system and method for accurate road obstacle identification of unmanned sanitation vehicles. Background Technology

[0002] Current obstacle recognition technology for unmanned sanitation vehicles primarily relies on mainstream convolutional neural network (CNN) algorithms to construct a visual recognition system. This system acquires road environment images via an onboard camera, inputs these images into a pre-trained CNN model, and extracts image features through layered operations such as convolution, pooling, and fully connected layers. Then, based on the label mapping relationship of training samples, it achieves obstacle classification and localization. Some technical solutions may integrate LiDAR equipment, but only use the point cloud data acquired by LiDAR as an independent distance detection signal to compensate for the lack of depth information in visual images, without performing deep fusion processing of the two types of data. Furthermore, at the algorithm level, traditional technologies have not been adapted and optimized for the unstructured features of complex urban road conditions, still using fixed-parameter feature extraction modes and directly applying training models from general scenarios for obstacle recognition.

[0003] The existing technology has the following problems: (1) Poor adaptability to complex scenes: Traditional CNN algorithms rely on a large number of labeled samples for training. In unstructured scenes such as leaf cover, water reflection, and shadow occlusion, the scene features are very different from the training samples, which can easily lead to feature extraction deviations and a significant decrease in recognition accuracy.

[0004] (2) Low utilization of multimodal data: The point cloud data of LiDAR and the visual image data have not been deeply integrated and are used only as independent signals. The complementary advantages of point cloud data in three-dimensional spatial information and image data in texture features cannot be fully utilized, and the ability to identify small-volume, low-contrast obstacles (such as gravel and discarded plastic bags) is weak.

[0005] (3) High overfitting and misidentification rate: CNN models trained in general scenarios are prone to overfitting in specific sanitation scenarios. They are prone to misidentifying non-obstacle targets on the road surface (such as irregular piles of fallen leaves and reflective areas of water accumulation) as obstacles, leading to errors in obstacle avoidance decisions by unmanned vehicles.

[0006] (4) Insufficient response speed: Traditional algorithms require a large number of hierarchical operations when processing complex scene data, which takes a long time and is difficult to meet the decision-making needs of sanitation unmanned vehicles for real-time obstacle avoidance.

[0007] Therefore, it is necessary to develop a system and method for accurate road obstacle recognition of unmanned sanitation vehicles. Summary of the Invention

[0008] The purpose of this invention is to provide a system and method for accurate road obstacle recognition of unmanned sanitation vehicles, so as to improve the accuracy and response speed of unmanned vehicles in obstacle recognition.

[0009] In a first aspect, the present invention provides a road obstacle precision identification system for unmanned sanitation vehicles, comprising a multimodal data acquisition module, a data preprocessing module, an improved SBL algorithm calculation module, and an obstacle information output module; The multimodal data acquisition module includes a visual image acquisition unit and a lidar data acquisition unit, which are used to collect road environment data during the operation of the sanitation unmanned vehicle. The two units collect data synchronously and transmit it to the data preprocessing module. The data preprocessing module is used to clean, align and convert the format of the collected data, and transmit the processed standardized data to the improved SBL algorithm calculation module. The improved SBL algorithm operation module is used to receive the standardized data, dynamically optimize the feature extraction weights according to the current road environment characteristics, call the improved EM iterative algorithm, fuse the optimized visual image features with the lidar point cloud features, construct a dynamically updated sparse dictionary, and perform sparse reconstruction and probability estimation on the fused feature data through the improved SBL algorithm to obtain the preliminary identification results of obstacles, and transmit the results to the obstacle information output module. The obstacle information output module is used to parse and format the preliminary identification results and output the obstacle information to the unmanned vehicle decision control unit.

[0010] Optionally, the visual image acquisition unit captures two-dimensional image information of the road scene in real time, and the lidar data acquisition unit acquires three-dimensional point cloud information of the road scene in real time. During the acquisition process, the acquisition timestamp and device attitude information are recorded simultaneously. The device attitude information includes the camera angle and the lidar scanning angle.

[0011] Optionally, the data preprocessing module includes: The data cleaning unit is used to remove isolated noise points in visual images using a median filtering algorithm and to remove distant stray points and invalid point clusters with low density from lidar point cloud data that are outside the effective detection range using a statistical filtering algorithm. The spatiotemporal alignment unit achieves temporal dimension matching between visual images and LiDAR point cloud data based on timestamps, and spatial dimension alignment based on vehicle equipment calibration parameters through coordinate transformation algorithms. The vehicle equipment calibration parameters include the relative position and angle deviation of the camera and LiDAR. The format conversion unit converts visual images into grayscale feature matrices and LiDAR point cloud data into a set of three-dimensional coordinate vectors.

[0012] Optionally, the improved SBL algorithm operation module includes: An adaptive kernel function adjustment unit is used to extract key scene features from the output data of the data preprocessing module. These key scene features include the grayscale contrast of the visual image and the density variation of the LiDAR point cloud. The grayscale contrast is the average grayscale difference between adjacent pixels in the image, and the point cloud density variation is the difference in the number of point clouds in different regions. Based on these key scene features, the current road surface environment type is determined. A radial basis function kernel is used as the basic kernel function, and the kernel function width parameter is dynamically adjusted according to the road surface environment type. The optimized kernel function is then applied to the feature extraction process, dynamically adjusting the extraction weights of different feature dimensions to obtain a weighted feature matrix. A multimodal data fusion sparse dictionary update unit is used to construct a dynamic sparse dictionary using the weighted feature matrix as input through an improved EM algorithm. The improved EM algorithm includes an E-step and an M-step, wherein the E-step is an expectation calculation, which includes calculating the posterior mean and posterior covariance of the sparse coefficient matrix; the M-step is a maximization calculation, which updates the sparse dictionary, noise variance, and multimodal data weight factors based on the posterior probability distribution, wherein the multimodal data weight factors include image data weight factors and point cloud data weight factors, and the sum of the image data weight factors and point cloud data weight factors is equal to one.

[0013] Optionally, the road surface environmental characteristics include a leaf-covered environment, a water-reflective environment, and a shaded environment.

[0014] Secondly, the method for accurate road obstacle identification of unmanned sanitation vehicles according to the present invention employs the road obstacle identification system for unmanned sanitation vehicles as described in the present invention, and includes the following steps: Step 1: Synchronously acquire 2D images and 3D point cloud data of the road surface, and record the timestamp of the acquisition and the device attitude information; Step 2: Clean the collected data, perform spatiotemporal alignment and format conversion to obtain standardized data; Step 3: Determine the road surface environment type based on the standardized data, use the radial basis function as the basic kernel function, optimize the kernel function parameters based on the road surface environment type, apply the optimized kernel function to the feature extraction process of the standardized data, calculate the similarity between the input features and the sample features through the kernel function, dynamically adjust the extraction weights of different feature dimensions, and obtain a weighted feature matrix; Step 4: Construct a multimodal data fusion sparse dictionary based on the improved EM algorithm; Step 5: Perform sparse reconstruction and probability estimation based on the improved SBL algorithm to obtain obstacle recognition results; Step 6: Extract the three-dimensional coordinate information and size parameters of the obstacle, and analyze the motion state of the obstacle; Step 7: Convert the extracted obstacle information into a preset data format and transmit it to the unmanned vehicle decision control unit.

[0015] Optionally, step 3 specifically includes: Key scene features are extracted from standardized data. These key scene features include the gray-level contrast of visual images and the density variation of LiDAR point clouds. The gray-level contrast is the average gray-level difference between adjacent pixels in the image, and the density variation of point clouds is the difference in the number of point clouds in different regions. The current road environment type is determined based on these key scene features. The radial basis function is used as the basic kernel function, and the kernel function width parameter is dynamically adjusted according to the road surface environment type. The optimized kernel function is applied to the feature extraction process of standardized data. The similarity between the input features and the sample features is calculated through the kernel function. The extraction weights of different feature dimensions are dynamically adjusted to obtain a weighted feature matrix, including a visual image feature matrix and a LiDAR point cloud feature matrix.

[0016] Optionally, step 4 specifically includes: Initialization steps: Fuse the visual image feature matrix and the lidar point cloud feature matrix to obtain the fused feature matrix, and initialize the sparse dictionary, sparse coefficient matrix and noise variance; Expected calculation steps: Based on the Bayesian probability model, a multimodal data weighting factor is introduced to correct the likelihood function of the fused features. The multimodal data weighting factor includes image data weighting factor and point cloud data weighting factor, and the sum of the image data weighting factor and point cloud data weighting factor is equal to one. Combining the indicator matrices of image data and point cloud data, the posterior mean and posterior covariance of the sparse coefficients are calculated using the Bayesian formula. Maximizing the computational steps: Based on the posterior probability distribution obtained from the expected computational steps, update the sparse dictionary, noise variance, and multimodal data weighting factors; when updating the sparse dictionary, iteratively update each dictionary atom by minimizing the reconstruction error; when updating the noise variance, calculate based on the mean of the reconstruction error; when updating the weighting factors, adjust according to the proportion of reconstruction error between the two modalities. Iterative convergence judgment steps: Repeat the expected calculation step and the maximization calculation step. After each iteration, calculate the update error of the sparse dictionary. When the update error is less than the set convergence threshold, stop the iteration and obtain the final multimodal fused sparse dictionary.

[0017] Optionally, step 5 specifically includes: Obtain the fused sparse dictionary and weighted feature matrix; Based on the fused sparse dictionary, the weighted feature matrix is ​​sparsely reconstructed, and the sparse coefficient matrix is ​​solved by the L1 regularization optimization algorithm. Retain the non-zero elements in the sparse coefficient matrix whose absolute values ​​are greater than a set threshold. The dictionary atoms corresponding to the non-zero elements are key atoms that match the obstacle features. Based on the distribution of non-zero elements in the sparse coefficient matrix, the posterior probability of obstacle recognition is calculated. For each sample, the probability that the region to which the sample belongs belongs to the obstacle category is calculated.

[0018] Optionally, in step 6, The three-dimensional coordinate information of the obstacle is taken as the average of the three-dimensional coordinates of all point clouds within the obstacle area; The length and width of the dimensions are calculated using the bounding box of the obstacle region in the image, and the height is calculated using the z-axis coordinate range of the LiDAR point cloud. The motion trend analysis calculates the instantaneous velocity of the obstacle using data from multiple consecutive frames, and uses a linear fitting algorithm to predict the direction of motion and the position in the next frame.

[0019] The beneficial effects of this invention are: (1) The present invention introduces an improved SBL algorithm with an adaptive kernel function adjustment mechanism, which can extract weights based on the dynamic optimization features of the road surface environment, thus solving the problem of poor adaptability to unstructured scenarios.

[0020] (2) The present invention is based on the multimodal data fusion sparse dictionary construction method of the improved EM algorithm to realize the deep fusion of lidar point cloud and visual image data, and improve the sparse representation capability of obstacle edge features.

[0021] (3) The present invention combines sparse reconstruction and probability estimation to reduce the false recognition rate in complex scenarios, and especially optimizes the recognition effect of small-volume, low-contrast obstacles. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the road obstacle accuracy recognition system for unmanned sanitation vehicles described in this application embodiment; Figure 2 This is a flowchart of the method for accurate road obstacle identification of unmanned sanitation vehicles described in the embodiments of this application; Figure 3 This is a flowchart of step 2 in the method for accurate identification of road obstacles by unmanned sanitation vehicles described in the embodiments of this application; Figure 4 This is a flowchart of step 3 in the method for accurate identification of road obstacles by unmanned sanitation vehicles described in the embodiments of this application; Figure 5 This is a flowchart of step 4 in the method for accurate identification of road obstacles by unmanned sanitation vehicles described in the embodiments of this application; Figure 6This is a flowchart of step 5 in the method for accurate identification of road obstacles by unmanned sanitation vehicles described in the embodiments of this application. Detailed Implementation

[0023] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0024] like Figure 1 As shown in the embodiments of this application, a road obstacle accuracy recognition system for unmanned sanitation vehicles includes a multimodal data acquisition module, a data preprocessing module, an improved SBL algorithm (SBL stands for Sparse Bayesian Learning) operation module, and an obstacle information output module. The modules work collaboratively through a data interaction interface, as detailed below: (1) Multimodal data acquisition module This multimodal data acquisition module includes a visual image acquisition unit and a lidar data acquisition unit, responsible for collecting road environment data during the operation of the unmanned sanitation vehicle. The visual image acquisition unit captures two-dimensional image information of the road scene in real time, while the lidar data acquisition unit acquires three-dimensional point cloud information of the road scene in real time. During the acquisition process, the acquisition timestamp and device attitude information are recorded simultaneously, including the camera angle and lidar scanning angle. Both acquire data synchronously and transmit the raw data to the data preprocessing module in real time.

[0025] (2) Data preprocessing module The data preprocessing module receives raw data from the multimodal data acquisition module and performs cleaning, alignment, and format conversion on the data. First, it removes noise interference (such as light spots and blurred areas) from the visual image and invalid points (such as distant clutter) from the LiDAR point cloud. Second, based on the spatiotemporal synchronization calibration parameters of the vehicle-mounted equipment, it aligns the visual image and the LiDAR point cloud data spatiotemporally to ensure that they correspond in the same time and spatial dimensions. Finally, it converts the preprocessed image data into a feature matrix format that meets the algorithm input requirements, converts the point cloud data into a three-dimensional coordinate vector format, and transmits the processed data to the improved SBL algorithm computation module.

[0026] For example, the data preprocessing module includes a data cleaning unit, a spatiotemporal alignment unit, and a format conversion unit. The data cleaning unit uses a median filtering algorithm to remove isolated noise points from the visual image and a statistical filtering algorithm to remove distant stray points exceeding the effective detection range and invalid point clusters with excessively low density from the LiDAR point cloud data. The spatiotemporal alignment unit achieves temporal dimension matching between the visual image and the LiDAR point cloud data based on timestamps, and achieves spatial dimension alignment based on the vehicle-mounted device calibration parameters, including the relative position and angular deviation between the camera and the LiDAR. The format conversion unit converts the visual image into a grayscale feature matrix and the LiDAR point cloud data into a set of three-dimensional coordinate vectors.

[0027] (3) Improved SBL algorithm operation module The improved SBL algorithm computation module is the core computational unit of the system, achieving real-time computation based on the vehicle-mounted edge computing unit. First, it receives standardized data from the data preprocessing module and dynamically optimizes feature extraction weights based on current road environment characteristics (such as the presence of fallen leaves or water reflection). Then, it calls the improved EM (Expectation Maximization) iterative algorithm to fuse the optimized visual image features with LiDAR point cloud features, constructing a dynamically updated sparse dictionary. Finally, the improved SBL algorithm performs sparse reconstruction and probability estimation on the fused feature data to obtain preliminary obstacle identification results, which are then transmitted to the obstacle information output module.

[0028] For example, the improved SBL algorithm operation module includes an adaptive kernel function adjustment unit and a multimodal data fusion sparse dictionary update unit. The adaptive kernel function adjustment unit is used to extract key scene features from the output data of the data preprocessing module. The key scene features include the gray-level contrast of the visual image and the density change of the LiDAR point cloud. The gray-level contrast is the average gray-level difference between adjacent pixels in the image, and the point cloud density change is the difference in the number of point clouds in different regions. Based on the key scene features, the current road environment type is determined. The radial basis function kernel function is used as the basic kernel function, and the kernel function width parameter is dynamically adjusted according to the road environment type. The optimized kernel function is applied to the feature extraction process, and the extraction weights of different feature dimensions are dynamically adjusted to obtain a weighted feature matrix. The multimodal data fusion sparse dictionary update unit is used to construct a dynamic sparse dictionary using the weighted feature matrix as input through an improved EM algorithm. The improved EM algorithm includes an E-step and an M-step, wherein the E-step is an expectation calculation, which includes calculating the posterior mean and posterior covariance of the sparse coefficient matrix; the M-step is a maximization calculation, which updates the sparse dictionary, noise variance, and multimodal data weight factors based on the posterior probability distribution, wherein the multimodal data weight factors include image data weight factors and point cloud data weight factors, and the sum of the image data weight factors and point cloud data weight factors is equal to one.

[0029] (4) Obstacle information output module The obstacle information output module receives the preliminary recognition results transmitted by the improved SBL algorithm processing module and further parses and formats these results. Specifically, this includes extracting the obstacle's three-dimensional coordinates and dimensional parameters (length, width, height), and analyzing the obstacle's motion state (such as stationary, moving, and direction and speed of movement) based on the changing trends of continuous frame data. Subsequently, the parsed obstacle information is converted into a data format that meets the requirements of the sanitation vehicle's obstacle avoidance decision-making system and output in real time to the vehicle's decision control unit, providing data support for obstacle avoidance operations.

[0030] The coordination process of each module is as follows: The multimodal data acquisition module collects raw data in real time, the data preprocessing module standardizes the raw data, the improved SBL algorithm operation module performs operations on the standardized data based on the improved algorithm to obtain preliminary identification results, and the obstacle information output module analyzes the results and outputs them to the decision system, forming a closed-loop workflow of acquisition-preprocessing-operation-output.

[0031] In traditional technologies, the SBL algorithm cannot dynamically adjust feature extraction weights according to the actual situation when facing complex and ever-changing road environments, resulting in a significant reduction in recognition performance in unstructured scenes. This application introduces an adaptive kernel function adjustment mechanism, enabling the algorithm to optimize feature extraction weights in real time based on current road environment characteristics (including leaf-covered environments, water-reflective environments, and shadow-occluded environments), such as the presence of leaf cover or water reflection, thereby effectively solving the problem of poor adaptability to unstructured scenes. Furthermore, existing technologies have low fusion levels between LiDAR point cloud data and visual image data, failing to fully leverage the advantages of both, especially when recognizing small, low-contrast obstacles. This application employs a multimodal data fusion sparse dictionary construction method based on the improved expectation-maximization (EM) algorithm, deeply fusing visual image features with LiDAR point cloud features to construct a dynamically updated sparse dictionary, significantly improving the sparse representation capability of obstacle edge features and optimizing the recognition performance for small, low-contrast obstacles. Meanwhile, this application combines sparse reconstruction and probability estimation in obstacle recognition processes, which effectively reduces the false recognition rate in complex scenarios and further improves the accuracy of recognition.

[0032] Furthermore, this application constructs a closed-loop system architecture that integrates multimodal data acquisition, preprocessing, algorithm computation, and information output, ensuring that the entire system can run in real time on the vehicle-mounted edge computing unit, providing a rapid response for obstacle avoidance decisions of unmanned vehicles, and meeting the decision-making needs of sanitation unmanned vehicles for real-time obstacle avoidance in complex road conditions.

[0033] Through a series of innovative improvements and optimizations, this application aims to provide a more accurate, efficient, and reliable road obstacle recognition solution for unmanned sanitation vehicles, thereby effectively reducing the risk of obstacle avoidance decision errors caused by recognition errors, significantly improving the driving safety and operational efficiency of unmanned sanitation vehicles in complex road conditions, and promoting the development and application of autonomous driving technology for unmanned sanitation vehicles.

[0034] like Figure 2 As shown in the embodiments of this application, a method for accurate road obstacle recognition of unmanned sanitation vehicles employs the road obstacle recognition system for unmanned sanitation vehicles as described in the embodiments of this application. The method includes the following steps: Step 1: Synchronous Acquisition of Multimodal Data The multimodal data acquisition module of the unmanned sanitation vehicle is activated, and a synchronous acquisition trigger mechanism is set up for the visual image acquisition unit and the lidar data acquisition unit to ensure that both acquire road environment data at the same time interval. The visual image acquisition unit captures two-dimensional images of the road surface, while the lidar data acquisition unit scans three-dimensional point cloud data of the road surface. During the acquisition process, the acquisition timestamp and equipment attitude information (such as camera angle and lidar scanning angle) are recorded in real time. The raw image data, raw point cloud data, and corresponding spatiotemporal information are packaged and transmitted to the data preprocessing module.

[0035] like Figure 3 As shown, step 2: data preprocessing Step 2.1 Data Cleaning: The median filtering algorithm is used to remove noise from the original visual image and filter out isolated noise points in the image; the statistical filtering algorithm is used to process the original LiDAR point cloud data, setting distance thresholds and density thresholds to remove long-distance stray points that are outside the effective detection range and invalid point clusters with too low density.

[0036] Step 2.2 Spatiotemporal Alignment: Based on the timestamps recorded in Step 1, the visual image and the LiDAR point cloud data are matched in terms of time dimension; then, based on the calibration parameters of the vehicle equipment (such as the relative position and angle deviation between the camera and the LiDAR), the LiDAR point cloud data is mapped to the two-dimensional coordinate system of the visual image through a coordinate transformation algorithm to achieve spatial dimension alignment between the two.

[0037] Step 2.3 Format Conversion: Convert the cleaned and aligned visual image into a grayscale feature matrix (convert the RGB channel values ​​of the image into single-channel grayscale values, construct an M×N dimension matrix, where M and N are the height and width of the image, respectively); convert the LiDAR point cloud data into a set of three-dimensional coordinate vectors (each point cloud data is represented as a three-dimensional vector of (x, y, z), and all valid point clouds constitute a set of three-dimensional vectors), and transmit the converted and standardized data to the improved SBL algorithm calculation module.

[0038] like Figure 4 As shown, step 3: Adaptive kernel function adjustment After receiving standardized data, the improved SBL algorithm operation module first starts the adaptive kernel function adjustment unit, as follows: Step 3.1: Environmental feature judgment: Extract key scene features from standardized data, including the gray-scale contrast of visual images (calculate the average gray-scale difference between adjacent pixels in the image) and the density change of LiDAR point clouds (calculate the difference in the number of point clouds in different areas). Based on these features, determine the current road surface environment type (such as leaf-covered environment, water accumulation and reflective environment, shadow occlusion environment).

[0039] Step 3.2: Kernel Function Parameter Optimization: The Radial Basis Function (RBF) is used as the basic kernel function, and its original expression is: Where K(x) i x j ) represents the kernel function value (i.e., x) i With x j (similarity), x i x j γ is the input feature vector, and γ is the kernel function width parameter. x represents i With x j Euclidean distance, The square of the Euclidean distance. It is an exponential function.

[0040] Based on the environment type determined in step 3.1, dynamically adjust the γ value: If the environment is covered with fallen leaves, the fallen leaves will cause the feature edges to be blurred, so the value of γ needs to be reduced (to enhance the local capture ability of the kernel function). If the environment is a waterlogged and reflective environment, false features will be generated in the reflective areas. The γ value needs to be increased (to reduce the interference of false features). If the environment is occluded by shadows, the grayscale of the shadow area is low, and the γ value needs to be adjusted appropriately (to balance the sensitivity and anti-interference of feature extraction).

[0041] Step 3.3: Feature Extraction Weight Update: The optimized kernel function is applied to the feature extraction process of standardized data. The similarity between the input features and the sample features is calculated through the kernel function. The extraction weights of different feature dimensions are dynamically adjusted. Key features related to obstacles (such as edge contours and three-dimensional shapes) are given higher weights, while environmental interference features (such as leaf textures and water reflections) are given lower weights, resulting in a weighted feature matrix.

[0042] like Figure 5 As shown, step 4: Construction of a sparse dictionary for multimodal data fusion (based on the improved EM algorithm) The multimodal data fusion sparse dictionary update unit is started. Taking the weighted feature matrix obtained in step 3 as input, a dynamic sparse dictionary is constructed through the improved EM algorithm. The specific process is as follows: Step 4.1: Initialize parameters: Set the visual image feature matrix to Ximg (dimension D1×N, where D1 is the image feature dimension and N is the number of samples), and the LiDAR point cloud feature matrix to Xlidar (dimension D2×N, where D2 is the point cloud feature dimension). The fused feature matrix is ​​X=[Ximg;Xlidar] (dimension is (D1+D2)×N); Initialize the sparse dictionary A (dimension (D1+D2)×K, where K is the number of dictionary atoms), the sparse coefficient matrix α (dimension K×N), and the noise variance σ².

[0043] Step 4.2: Step E (Expected Calculation): Based on the Bayesian probability model, the posterior probability distribution of the sparse coefficient matrix α is calculated. In the improved SBL algorithm, a multimodal data weighting factor ω (ω1 is the weighting factor for image data, ω2 is the weighting factor for point cloud data, and ω1+ω2=1) is introduced to modify the likelihood function of the fused features. The modified likelihood function (used to represent the probability of observing the fused feature matrix X given a sparse dictionary A, a sparse coefficient matrix α, and a noise variance σ²) is: in, ∏ represents the conditional probability density function (likelihood function value); ∏ represents the product operation on N samples; n is the sample index (from 1 to N). As a normalization factor, it ensures that the sum of the probability integrals is 1; This represents the nth column of the fusion feature matrix X (i.e., the fusion feature vector of the nth sample). This represents the nth column of the sparse coefficient matrix α (i.e., the sparse coefficient vector of the nth sample). Represents the fused feature vector With dictionary A and coefficients The squared Euclidean distance between the reconstructed feature vectors (i.e., the squared reconstruction error). It is an exponential function used to convert the reconstruction error into a probability decay form; Indicator matrix representing image data, An indicator matrix representing point cloud data (1 for the corresponding dimension and 0 for others). Used to weight and correct the likelihood contribution of different modal data.

[0044] Calculate the posterior mean of α using Bayes' theorem. and posterior covariance .

[0045] Posterior mean The calculation formula is as follows: Used to calculate the sparsity coefficient of the nth sample. The posterior mean, i.e., given observed features back, The most likely value. Among them, Let α be the prior covariance matrix. Its inverse matrix; Let A be the transpose of the sparse dictionary A.

[0046] Posterior covariance The calculation formula is as follows: Used to calculate the sparsity coefficient of the nth sample. The posterior covariance matrix is ​​used to measure the posterior mean. The larger the diagonal element of the matrix, the higher the uncertainty of the corresponding coefficient.

[0047] Step 4.3: M-step (maximum calculation): Based on the posterior probability distribution obtained in the E-step, the sparse dictionary A, noise variance σ², and multimodal data weighting factor ω are updated. When updating dictionary A, the reconstruction error ||X-Aα||² is minimized for each dictionary atom. Perform iterative updates: in, This represents the dictionary matrix with the k-th column removed. This represents the coefficient matrix after removing the k-th row, where I is the identity matrix.

[0048] When updating σ², calculate the mean of the reconstruction error: This is used to calculate the updated noise variance, which estimates the noise intensity using the average reconstruction error of all samples. Here, σ² represents the updated noise variance. The normalization factor is (number of samples N multiplied by the fusion feature dimension D1+D2). The squared reconstruction error of the mean of the nth sample (using the posterior mean) The reconstruction error is calculated); tr(·) is the trace operation of the matrix (i.e., the sum of the diagonal elements of the matrix). The matrix representing the coefficient uncertainty of the nth sample that is propagated to the reconstruction error. This is a scalar measure of the uncertainty.

[0049] When updating the weight factor ω, the reconstruction error proportions of the two types of data are adjusted: ; ω2=1-ω1, Where Aimg is a submatrix of the dictionary corresponding to the image features. is a sub-vector of the corresponding image features in the coefficients.

[0050] Step 4.4: Iterative convergence judgment: Repeat the E-step and M-step. After each iteration, calculate the update error of the sparse dictionary A (the Frobenius norm of A in the current iteration and A in the previous iteration). When the update error is less than the set convergence threshold, stop the iteration and obtain the final multimodal fused sparse dictionary.

[0051] like Figure 5 As shown, step 5: Sparse reconstruction and probability estimation based on the improved SBL algorithm. Step 5.1: Sparse Reconstruction: Based on the fused sparse dictionary A obtained in Step 4, sparse reconstruction is performed on the weighted feature matrix X output in Step 3. The sparse coefficient matrix α is solved using the L1 regularization optimization algorithm, with the optimization objective being: in, This means finding the minimum value of the objective function with the sparse coefficient matrix α as the optimization variable, i.e., finding a set of optimal α values ​​that minimize the entire expression. λ is the regularization parameter (determined through cross-validation). It is the L1 norm of the sparse coefficient matrix α (i.e., the sum of the absolute values ​​of all elements in α).

[0052] After obtaining the sparse coefficient matrix α, the non-zero elements in α whose absolute values ​​are greater than a set threshold are retained. The dictionary atoms corresponding to these non-zero elements are the key atoms that match the obstacle features.

[0053] Step 5.2: Probability Estimation: Based on the distribution of non-zero elements in the sparse coefficient matrix, calculate the posterior probability of obstacle recognition. For each sample (i.e., each feature region in each frame of data), calculate the probability P that the region belongs to the obstacle category: Where S is the sum of the non-zero elements of the sparse coefficient matrix corresponding to the region, and β is the probability adjustment parameter (calibrated using the label data of the training samples).

[0054] When P is greater than the set probability threshold, the area is determined to be an obstacle area; otherwise, it is determined to be a non-obstacle area.

[0055] like Figure 6 As shown, step 6: obstacle information extraction and motion trend analysis Step 6.1: Obstacle Information Extraction: For the obstacle area identified in Step 5, extract the three-dimensional coordinate information by combining the LiDAR point cloud data (take the average of the three-dimensional coordinates of all point clouds in the obstacle area as the center coordinates of the obstacle), and extract the size parameters by combining the visual image data (calculate the length and width through the bounding box of the obstacle area in the image, and calculate the height through the z-axis coordinate range of the LiDAR point cloud).

[0056] Step 6.2: Movement Trend Analysis Obtain the center coordinates of obstacles from 10 consecutive frames of data (the number of frames can be adjusted according to actual needs), calculate the coordinate change between adjacent frames, and obtain the instantaneous velocity of the obstacle (coordinate change divided by frame interval time). By fitting the velocity data of consecutive frames using a linear fitting algorithm, the movement direction of obstacles and their position in the next frame are predicted, thus forming information on the movement trend of obstacles.

[0057] Step 7: Obstacle Information Output and Obstacle Avoidance Decision Support The obstacle's 3D coordinates, size parameters, and motion trend information extracted in step 6 are converted into a data format recognizable by the sanitation unmanned vehicle obstacle avoidance decision-making system (such as the CAN bus protocol format), and transmitted in real time to the unmanned vehicle's decision control unit via a data interface. Based on the received obstacle information, the decision control unit determines whether obstacle avoidance is required (e.g., triggering an obstacle avoidance command when the distance between the obstacle and the unmanned vehicle is less than a safety threshold), thus enabling the unmanned vehicle to achieve precise obstacle avoidance.

[0058] This application effectively addresses the problems of low obstacle recognition accuracy and high false recognition rate in existing sanitation unmanned vehicles under complex urban road conditions by improving the core of the sparse Bayesian learning algorithm and combining it with multimodal data deep fusion technology. Specifically, the adaptive kernel function adjustment mechanism enables dynamic adaptation to unstructured scenes, and the multimodal fusion sparse dictionary enhances the representation ability of obstacle features, significantly improving the recognition effect of small-volume, low-contrast obstacles such as gravel and discarded plastic bags. Simultaneously, the algorithm can be computed in real time on the onboard edge computing unit, and the output obstacle 3D coordinates, size, and motion trend information can provide fast and accurate data support for obstacle avoidance decisions by the unmanned vehicle, reducing the risk of obstacle avoidance decision errors caused by recognition errors and improving the driving safety and operational efficiency of sanitation unmanned vehicles under complex road conditions.

[0059] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A precise obstacle recognition system for unmanned sanitation vehicles, characterized in that, It includes a multimodal data acquisition module, a data preprocessing module, an improved SBL algorithm calculation module, and an obstacle information output module; The multimodal data acquisition module includes a visual image acquisition unit and a lidar data acquisition unit, which are used to collect road environment data during the operation of the sanitation unmanned vehicle. The two units collect data synchronously and transmit it to the data preprocessing module. The data preprocessing module is used to clean, align and convert the format of the collected data, and transmit the processed standardized data to the improved SBL algorithm calculation module. The improved SBL algorithm operation module is used to receive the standardized data, dynamically optimize the feature extraction weights according to the current road environment characteristics, call the improved EM iterative algorithm, and fuse the optimized visual image features with the lidar point cloud features to construct a dynamically updated sparse dictionary. The improved SBL algorithm is used to perform sparse reconstruction and probability estimation on the fused feature data to obtain the preliminary identification results of obstacles, and the results are transmitted to the obstacle information output module. The obstacle information output module is used to parse and format the preliminary identification results and output the obstacle information to the unmanned vehicle decision control unit.

2. The road obstacle precision identification system for unmanned sanitation vehicles according to claim 1, characterized in that, The visual image acquisition unit captures two-dimensional image information of the road scene in real time, and the lidar data acquisition unit acquires three-dimensional point cloud information of the road scene in real time. During the acquisition process, the acquisition timestamp and device attitude information are recorded simultaneously. The device attitude information includes the camera angle and the lidar scanning angle.

3. The road obstacle precision identification system for unmanned sanitation vehicles according to claim 1, characterized in that, The data preprocessing module includes: The data cleaning unit is used to remove isolated noise points in visual images using a median filtering algorithm and to remove distant stray points and invalid point clusters with low density from lidar point cloud data that are outside the effective detection range using a statistical filtering algorithm. The spatiotemporal alignment unit achieves temporal dimension matching between visual images and LiDAR point cloud data based on timestamps, and spatial dimension alignment based on vehicle equipment calibration parameters through coordinate transformation algorithms. The vehicle equipment calibration parameters include the relative position and angle deviation of the camera and LiDAR. The format conversion unit converts visual images into grayscale feature matrices and LiDAR point cloud data into a set of three-dimensional coordinate vectors.

4. The road obstacle precision identification system for unmanned sanitation vehicles according to claim 1, characterized in that, The improved SBL algorithm operation module includes: An adaptive kernel function adjustment unit is used to extract key scene features from the output data of the data preprocessing module. These key scene features include the grayscale contrast of the visual image and the density variation of the LiDAR point cloud. The grayscale contrast is the average grayscale difference between adjacent pixels in the image, and the point cloud density variation is the difference in the number of point clouds in different regions. Based on these key scene features, the current road surface environment type is determined. A radial basis function kernel is used as the basic kernel function, and the kernel function width parameter is dynamically adjusted according to the road surface environment type. The optimized kernel function is then applied to the feature extraction process, dynamically adjusting the extraction weights of different feature dimensions to obtain a weighted feature matrix. A multimodal data fusion sparse dictionary update unit is used to construct a dynamic sparse dictionary using the weighted feature matrix as input through an improved EM algorithm. The improved EM algorithm includes an E-step and an M-step, wherein the E-step is an expectation calculation, which includes calculating the posterior mean and posterior covariance of the sparse coefficient matrix; the M-step is a maximization calculation, which updates the sparse dictionary, noise variance, and multimodal data weight factors based on the posterior probability distribution, wherein the multimodal data weight factors include image data weight factors and point cloud data weight factors, and the sum of the image data weight factors and point cloud data weight factors is equal to one.

5. The road obstacle precision identification system for unmanned sanitation vehicles according to claim 1, characterized in that, The road surface environmental characteristics include environments covered by fallen leaves, environments with water accumulation and reflection, and environments with shadows.

6. A method for accurate road obstacle identification by unmanned sanitation vehicles, characterized in that, The road obstacle accuracy identification system for unmanned sanitation vehicles according to any one of claims 1-5 includes the following steps: Step 1: Synchronously acquire 2D images and 3D point cloud data of the road surface, and record the timestamp of the acquisition and the device attitude information; Step 2: Clean the collected data, perform spatiotemporal alignment and format conversion to obtain standardized data; Step 3: Determine the road surface environment type based on the standardized data, use the radial basis function as the basic kernel function, optimize the kernel function parameters based on the road surface environment type, apply the optimized kernel function to the feature extraction process of the standardized data, calculate the similarity between the input features and the sample features through the kernel function, dynamically adjust the extraction weights of different feature dimensions, and obtain a weighted feature matrix; Step 4: Construct a multimodal data fusion sparse dictionary based on the improved EM algorithm; Step 5: Perform sparse reconstruction and probability estimation based on the improved SBL algorithm to obtain obstacle recognition results; Step 6: Extract the three-dimensional coordinate information and size parameters of the obstacle, and analyze the motion state of the obstacle; Step 7: Convert the extracted obstacle information into a preset data format and transmit it to the unmanned vehicle decision control unit.

7. The method for accurate road obstacle identification of unmanned sanitation vehicles according to claim 6, characterized in that, Step 3 specifically includes: Key scene features are extracted from standardized data. These key scene features include the gray-level contrast of visual images and the density variation of LiDAR point clouds. The gray-level contrast is the average gray-level difference between adjacent pixels in the image, and the density variation of point clouds is the difference in the number of point clouds in different regions. The current road environment type is determined based on these key scene features. The radial basis function is used as the basic kernel function, and the kernel function width parameter is dynamically adjusted according to the road surface environment type. The optimized kernel function is applied to the feature extraction process of standardized data. The similarity between the input features and the sample features is calculated through the kernel function. The extraction weights of different feature dimensions are dynamically adjusted to obtain a weighted feature matrix, including a visual image feature matrix and a LiDAR point cloud feature matrix.

8. The method for accurate road obstacle identification of unmanned sanitation vehicles according to claim 6, characterized in that, Step 4 specifically includes: Initialization steps: Fuse the visual image feature matrix and the lidar point cloud feature matrix to obtain the fused feature matrix, and initialize the sparse dictionary, sparse coefficient matrix and noise variance; Expected calculation steps: Based on the Bayesian probability model, a multimodal data weighting factor is introduced to correct the likelihood function of the fused features. The multimodal data weighting factor includes image data weighting factor and point cloud data weighting factor, and the sum of the image data weighting factor and point cloud data weighting factor is equal to one. Combining the indicator matrices of image data and point cloud data, the posterior mean and posterior covariance of the sparse coefficients are calculated using the Bayesian formula. Maximizing the computational steps: Based on the posterior probability distribution obtained from the expected computational steps, update the sparse dictionary, noise variance, and multimodal data weighting factors; when updating the sparse dictionary, iteratively update each dictionary atom by minimizing the reconstruction error; when updating the noise variance, calculate based on the mean of the reconstruction error; when updating the weighting factors, adjust according to the proportion of reconstruction error between the two modalities. Iterative convergence judgment steps: Repeat the expected calculation step and the maximization calculation step. After each iteration, calculate the update error of the sparse dictionary. When the update error is less than the set convergence threshold, stop the iteration and obtain the final multimodal fused sparse dictionary.

9. The method for accurate road obstacle identification of unmanned sanitation vehicles according to claim 6, characterized in that, Step 5 specifically includes: Obtain the fused sparse dictionary and weighted feature matrix; Based on the fused sparse dictionary, the weighted feature matrix is ​​sparsely reconstructed, and the sparse coefficient matrix is ​​solved by the L1 regularization optimization algorithm. Retain the non-zero elements in the sparse coefficient matrix whose absolute values ​​are greater than a set threshold. The dictionary atoms corresponding to the non-zero elements are key atoms that match the obstacle features. Based on the distribution of non-zero elements in the sparse coefficient matrix, the posterior probability of obstacle recognition is calculated. For each sample, the probability that the region to which the sample belongs belongs to the obstacle category is calculated.

10. The method for accurate road obstacle identification of unmanned sanitation vehicles according to claim 6, characterized in that, In step 6 The three-dimensional coordinate information of the obstacle is taken as the average of the three-dimensional coordinates of all point clouds within the obstacle area; The length and width of the dimensions are calculated using the bounding box of the obstacle region in the image, and the height is calculated using the z-axis coordinate range of the LiDAR point cloud. The motion trend analysis calculates the instantaneous velocity of the obstacle using data from multiple consecutive frames, and uses a linear fitting algorithm to predict the direction of motion and the position in the next frame.

Images