Road obstacle recognition method based on machine vision

Abstract

The invention discloses a road obstacle recognition method based on machine vision, and the method comprises the steps: carrying out the precise positioning of the spatial position of an obstacle through a multi-sensor information fusion algorithm according to the azimuth and distance information of the obstacle estimated through a sound source positioning algorithm and the position information outputted by an obstacle detection model, and improving the positioning precision; the method comprises the following steps: acquiring guide mark information, such as road signs and lane lines, extracting guide mark features through a semantic segmentation algorithm, carrying out dynamic fusion on the guide mark features and an obstacle recognition result, and improving the accuracy and robustness of obstacle recognition by using prior information; in order to solve the problem of insufficient parallax analysis precision in long-distance, high-speed motion and other scenes, a monocular depth estimation algorithm based on deep learning is introduced, pixel-level depth information is predicted from a single image through a convolutional neural network, and the precision of obstacle distance estimation is improved.

Description

Technical Field

[0001] The invention belongs to the field of information technology, and in particular relates to a road obstacle recognition method based on machine vision. Background Art

[0002] Obstacle recognition faces many technical challenges in complex dynamic environments. First, the information obtained by a single perception modality has limitations, and it is difficult to fully and accurately perceive the position, size and motion state of obstacles. Secondly, visual information is easily affected by factors such as lighting and weather, resulting in insufficient robustness of obstacle recognition. Furthermore, although auditory information can provide information such as the direction and distance of obstacles, it lacks accurate spatial positioning capabilities. In addition, the real-time requirements are high, and obstacle recognition and response decisions need to be completed in a very short time, which places strict requirements on the time complexity of the algorithm. At the same time, the diversity and uncertainty of obstacles, such as differences in shape, material, and motion mode, increase the difficulty of recognition. Although guide annotations can provide prior information, dynamic fusion still faces challenges during the recognition process. Parallax analysis can obtain the depth information of obstacles, but the accuracy is insufficient in scenarios such as long distances and high-speed movements. In summary, how to integrate multi-source heterogeneous information and improve the accuracy, robustness and adaptability of obstacle recognition while meeting real-time requirements is a key technical problem that needs to be solved urgently. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for identifying road obstacles based on machine vision.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] A road obstacle recognition method based on machine vision, comprising:

[0006] Step S1, acquiring multi-source heterogeneous sensor data including visual information and auditory information;

[0007] Step S2: inputting multi-source heterogeneous sensor data into the obstacle detection model to obtain the obstacle area;

[0008] Step S3: Obstacle position and distance information estimated by the sound source localization algorithm;

[0009] Step S4: using an obstacle recognition algorithm based on transfer learning to obtain the obstacle type;

[0010] Step S5: Generate a behavior decision instruction based on the obstacle area, obstacle orientation and distance information, and obstacle type.

[0011] Preferably, in step S2, the obstacle detection model adopts a convolutional neural network architecture to identify obstacle areas in multi-source heterogeneous sensor data through feature extraction and target detection algorithms.

[0012] Preferably, in step S3, a monocular depth estimation algorithm based on deep learning is adopted to predict pixel-level depth information from a single image through a convolutional neural network to obtain an accurate obstacle distance.

[0013] Preferably, in step S2, guidance annotation information is obtained, features of the guidance annotation are extracted through a semantic segmentation algorithm, and dynamically fused with multi-source heterogeneous sensor data to accurately locate the spatial position of the obstacle.

[0014] The present invention integrates visual and auditory information, uses image enhancement and sound source localization algorithms for preprocessing, and then uses a deep learning model for obstacle detection. In order to cope with a variety of obstacles, transfer learning is introduced to improve the generalization ability of recognition. At the same time, combined with guidance annotation information and monocular depth estimation, the recognition accuracy in long-distance and high-speed scenarios is improved. Finally, behavioral instructions are generated through a real-time decision-making algorithm to achieve timely response to obstacles. The present invention effectively solves the accuracy and robustness problems of obstacle recognition in complex environments, improves the safety and reliability of the autonomous driving system, and provides an innovative solution for the field of intelligent transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flow chart of the method for identifying road obstacles based on machine vision of the present invention. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0017] Embodiment 1:

[0018] like Figure 1 As shown, this embodiment provides a method for identifying road obstacles based on machine vision, including:

[0019] S101. Acquire multi-source heterogeneous sensor data including at least visual information and auditory information. For the visual information, use an image enhancement algorithm to perform preprocessing to improve image quality and reduce the influence of factors such as lighting and weather. For the auditory information, use a sound source localization algorithm to estimate the direction and distance of obstacles.

[0020] Acquire multi-source heterogeneous sensor data containing visual information and auditory information, and preprocess the visual information and auditory information; for the visual information, use an image enhancement algorithm to preprocess the acquired image to improve image quality, reduce the influence of external factors, and obtain preprocessed visual information; for the auditory information, use a sound source localization algorithm to process the acquired audio data, estimate the direction and distance of the obstacle, and obtain preprocessed auditory information; based on the preprocessed visual information and auditory information, perform fusion processing through a data fusion algorithm to obtain fused obstacle information; based on the fused obstacle information, use a target detection algorithm to identify and locate the obstacle, and obtain the category and spatial position information of the obstacle; based on the category and spatial position information of the obstacle, use a trajectory prediction algorithm to predict the movement trajectory of the obstacle, and judge the collision risk between the obstacle and the vehicle; if there is a collision risk, plan the driving path of the vehicle according to the predicted obstacle trajectory to avoid collision with the obstacle.

[0021] Specifically, when acquiring multi-source heterogeneous sensor data containing visual information and auditory information, a high-resolution camera can be used to acquire image data, and a microphone array can be used to collect audio data. For the acquired image data, an adaptive histogram equalization algorithm is used to enhance the image. This algorithm adjusts the grayscale histogram distribution of the image to improve the contrast and clarity of the image, making the details in the image more obvious. At the same time, a median filter algorithm is used to denoise the image, effectively eliminating the influence of salt and pepper noise and Gaussian noise in the image. For audio data, a sound source localization algorithm based on the cross-correlation function is used to estimate the azimuth of the obstacle by analyzing the time difference of the sound signal received by the microphone array, with an accuracy of less than 5 degrees. At the same time, by analyzing the energy attenuation of the sound signal, the distance between the obstacle and the vehicle is estimated, with an error of less than 1 meter. After obtaining the preprocessed visual information and auditory information, the Kalman filter algorithm is used to fuse the multi-sensor data. By establishing a motion model of the obstacle, the position and speed of the obstacle are dynamically tracked to generate fused obstacle information. On this basis, the YOLO v3 target detection algorithm based on deep learning is used to identify and locate obstacles. By performing multi-scale downsampling on the image and extracting features at different levels, accurate classification and position regression of obstacles are achieved, with an average accuracy of more than 90%. Finally, according to the category and spatial position information of the obstacle, a trajectory prediction algorithm based on the long short-term memory network (LSTM) is used to analyze the historical movement trajectory of the obstacle, predict its movement trend in the future, and determine whether there is a risk of collision with the vehicle. If there is a risk of collision, the artificial potential field method is used for path planning based on the predicted obstacle trajectory to generate an optimal driving path to avoid obstacles and ensure the driving safety of the vehicle.

[0022] S102. Input the preprocessed visual information into an obstacle detection model based on deep learning. The model adopts a convolutional neural network architecture to identify obstacle areas in the image through feature extraction and target detection algorithms, and obtains attribute information such as its position and size.

[0023] The acquired visual information is preprocessed, including image denoising, contrast enhancement, etc., to improve the image quality and prepare for subsequent obstacle detection. The preprocessed visual information is input into the deep learning obstacle detection model based on convolutional neural network, which extracts multi-scale and multi-level features of the image through multi-layer convolution and pooling operations. On the basis of feature extraction, a region candidate network (such as RPN) is used to generate candidate regions, and then the candidate regions are classified and regressed to determine whether they are obstacles and obtain the location coordinates of the obstacles. For the detected obstacle regions, the precise outline and size information of the obstacles are further obtained through edge detection, contour extraction and other algorithms. According to the location, size and other attribute information of the obstacles, combined with the scene prior knowledge, the obstacles are classified and semantically understood to determine their specific categories, such as pedestrians, vehicles, traffic signs, etc. Different strategies are used to deal with the identified different categories of obstacles. For example, for pedestrians, tracking and trajectory prediction are required; for vehicles, their movement trends need to be analyzed; for traffic signs, semantic interpretation is required, etc. The location, size, category, semantic information and other attributes of obstacles are integrated to form a structured obstacle data representation, providing environmental perception information for upper-level decision-making planning and behavior control, and realizing the interaction and adaptation of intelligent systems and the environment.

[0024] Specifically, when preprocessing the acquired visual information, the median filtering algorithm is first used to denoise the image. A 5×5 filter window is used to process each pixel of the image and its neighboring pixels to remove Gaussian noise and salt and pepper noise in the image and improve the signal-to-noise ratio of the image. Then, the image contrast is enhanced by the histogram equalization algorithm, and the grayscale histogram of the image is stretched to the entire grayscale range, making the image contrast more vivid and the details clearer. The preprocessed visual information is input into the YOLOv3 obstacle detection model based on the convolutional neural network. The model contains 53 convolutional layers and 5 pooling layers. Through multi-scale feature fusion and upsampling operations, features are extracted at different scales of the input image to generate 13×13, 26×26 and 52×52 feature maps. On the feature map, the region candidate network is used to generate candidate regions. The softmax classifier is used to determine whether the candidate region contains obstacles. The position coordinates of the obstacles are obtained through bounding box regression. The average precision (AP) of obstacle detection is 83% when tested on the KITTI dataset. For the detected obstacle area, the Canny edge detection algorithm is used to extract the edge contour of the obstacle. The precise contour and size information of the obstacle are obtained through contour approximation and polygon fitting algorithms. According to the geometric features such as aspect ratio and area of ​​the obstacle, combined with the scene prior knowledge, the decision tree classifier is used to classify the obstacles, and the obstacles are divided into different categories such as pedestrians, vehicles, and traffic signs. The classification accuracy rate reaches more than 95%. For the identified pedestrian obstacles, the Kalman filter algorithm is used for tracking to predict the position of pedestrians at future moments. Combined with the movement information such as the speed and acceleration of pedestrians, a pedestrian trajectory prediction model is constructed to predict the movement trajectory of pedestrians in the next 1 second, with an average prediction error of less than 5 meters. Finally, the attribute information such as the position, size, category, and trajectory of the obstacle is integrated to generate structured obstacle data in JSON format and transmit it to the decision planning module to achieve comprehensive perception and understanding of obstacles in the environment and provide reliable environmental information for the safety decision of the unmanned driving system.

[0025] Based on feature extraction, a region candidate network (such as RPN) is used to generate candidate regions, and then the candidate regions are classified and regressed to determine whether they are obstacles and obtain the location coordinates of the obstacles.

[0026] According to the acquired image data, a convolutional neural network is used to extract features from the image to obtain a feature representation of the image; the extracted image features are input into a region candidate network, and multiple candidate regions are generated through the region candidate network; for each generated candidate region, a regional convolutional neural network is used to extract features to obtain a feature representation of the candidate region; the feature representation of the candidate region is input into a classification network and a regression network, and the classification network is used to determine whether the candidate region is an obstacle, and the position coordinates of the obstacle are obtained through the regression network; if the output probability of the classification network is greater than a preset threshold, the candidate region is determined to be an obstacle, and its position coordinates are obtained; according to the position coordinates of the obstacle, the position of the obstacle in the image is determined, and the position information of the obstacle is output; the position information of the obstacle is passed to a subsequent decision planning module for path planning and decision control of the autonomous driving vehicle.

[0027] Specifically, after acquiring the image data, the pre-trained ResNet-50 convolutional neural network is first used to extract features from the image. The network contains 50 convolutional layers, and the original image is converted into a 512-dimensional feature vector representation through layer-by-layer convolution and pooling operations. Then, the extracted image features are input into the region proposal network (RPN). RPN generates a series of anchor boxes by sliding a 3x3 convolution kernel on the feature map, and each anchor box corresponds to a candidate region. Then, for each candidate region, the ROI pooling layer is used to map it to a feature map of a fixed size (such as 7x7), and the regional convolutional neural network (R-CNN) is used to further extract regional features. After passing through the fully connected layer, the regional features are input into the classification network and regression network respectively. The classification network calculates the probability that the candidate region belongs to an obstacle through the Softmax function. If the probability is greater than 8, it is determined to be an obstacle. The regression network obtains the precise position coordinates of the obstacle through bounding box regression. Finally, according to the position coordinates of the obstacle, its pixel position in the original image is determined, and the position information is passed to the decision planning module. The decision-making planning module comprehensively considers factors such as obstacle location and vehicle status, generates the optimal collision-free driving path through motion planning algorithms (such as A* search), and controls the vehicle to drive along the planned path to achieve safe and efficient autonomous driving.

[0028] S103, based on the obstacle orientation and distance information estimated by the sound source localization algorithm, combined with the position information output by the obstacle detection model, the spatial position of the obstacle is accurately located through a multi-sensor information fusion algorithm to improve the positioning accuracy.

[0029] The initial position and distance information of the obstacle are estimated according to the sound source localization algorithm; the obstacle position information output by the obstacle detection model is obtained; if the output information of the sound source localization algorithm and the obstacle detection model is quite different, it is judged that there are multiple obstacles and the obstacles need to be classified and marked; the information of sound source localization and obstacle detection is integrated through the Kalman filter algorithm to obtain a more accurate spatial position of the obstacle; the movement trajectory of the obstacle is tracked and predicted by the particle filter algorithm, and the position information of the obstacle is updated in real time; according to the spatial position and movement trajectory of the obstacle, it is judged whether the obstacle poses a threat to navigation, and if so, an obstacle avoidance warning is issued; the above steps are continuously executed to obtain the position changes of the obstacle in real time to ensure the safety and positioning accuracy during navigation.

[0030] Specifically, the ambient sound signal is collected through the microphone array, and the azimuth of the sound source is calculated using the sound source localization algorithm such as GCC-PHAT. The distance of the obstacle is estimated according to the microphone spacing and sampling rate, and it is preliminarily determined that the obstacle is located about 5 meters ahead with an azimuth of 30 degrees. At the same time, the obstacle detection model outputs an obstacle 2 meters ahead with an azimuth of 35 degrees based on the visual image information. Comparing the results of the two methods, it is found that the position error exceeds 5 meters and the azimuth error exceeds 10 degrees. It is judged that there may be multiple obstacles, and the obstacles are classified and marked by the clustering algorithm. Then, the position information of the sound source localization and visual detection is input into the Kalman filter. Through the two steps of prediction and update, the obstacle position is dynamically tracked, and the weighted average method is used to fuse the two sensor data to obtain a more accurate spatial coordinate of the obstacle (8 meters, 32 degrees). On this basis, the particle filter algorithm is used to model the movement of the obstacle, and the movement trajectory of the obstacle in the future is predicted according to the trend of speed and direction changes. If the predicted trajectory intersects with the navigation path and the distance is less than the safety threshold of 1 meter, it is judged that there is a collision risk, and an obstacle avoidance warning is immediately issued to adjust the navigation strategy. The system executes the above process in a loop at a frequency of 10 Hz, monitors the position changes of obstacles in real time, makes timely decision responses, and ensures the safety and positioning accuracy of the navigation process to the greatest extent.

[0031] The Kalman filter algorithm is used to fuse the information of sound source localization and obstacle detection to obtain a more accurate spatial position of the obstacle.

[0032] Acquire sound source localization information and obstacle detection information, wherein the sound source localization information includes the spatial coordinates of the sound source, and the obstacle detection information includes the rough position of the obstacle;

[0033] Determine the distance from the sound source to the obstacle based on the sound source localization information, and preliminarily estimate the spatial position of the obstacle in combination with the obstacle detection information;

[0034] Establishing a Kalman filter model, taking the distance from the sound source to the obstacle as an observation quantity, and taking the spatial position of the obstacle as a state quantity;

[0035] In the Kalman filter model, the spatial position of the obstacle is predicted by a state transfer equation according to the sound source localization information and the obstacle detection information;

[0036] The difference between the distance from the sound source to the obstacle and the predicted distance is calculated by an observation equation to obtain an observation error;

[0037] The predicted spatial position of the obstacle is corrected using the observation error to obtain the optimal estimated position of the obstacle;

[0038] If the preset number of iterations is not reached, the state transfer equation is executed again to continue predicting the spatial position of the obstacle;

[0039] If the preset number of iterations is reached, the optimal estimated position is output as the precise spatial position information of the obstacle.

[0040] Specifically, after obtaining the sound source localization information and obstacle detection information, the sound source localization information includes the spatial coordinates (x, y, z) of the sound source, and the obstacle detection information includes the rough position of the obstacle (x0, y0, z0). According to the spatial coordinates of the sound source and the rough position of the obstacle, the Euclidean distance formula is used to calculate the distance d from the sound source to the obstacle = √((x-x0)2+(y-y0)2+(z-z0)2), and the spatial position of the obstacle (x1, y1, z1) is preliminarily estimated in combination with the obstacle detection information. Then, a Kalman filter model is established, and the distance d from the sound source to the obstacle is used as the observation quantity, and the spatial position of the obstacle (x1, y1, z1) is used as the state quantity. In the Kalman filter model, according to the sound source localization information and the obstacle detection information, the spatial position of the obstacle is predicted by the state transfer equation xt = Axt-1+But-1, where A is the state transfer matrix and B is the control input matrix. Then, the difference between the distance from the sound source to the obstacle and the predicted distance is calculated through the observation equation zt=Hxt+vt to obtain the observation error, where H is the observation matrix and vt is the observation noise. The predicted spatial position of the obstacle is corrected using the observation error, and the formula is xt=xt+Kt(zt-Hxt), where xt is the corrected obstacle position, xt is the predicted obstacle position, and Kt is the Kalman gain. If the preset number of iterations (such as 10 times) is not reached, return to execute the state transfer equation and continue to predict the spatial position of the obstacle; if the preset number of iterations is reached, the optimal estimated position is output. As the precise spatial position information of the obstacle. For example, if the sound source is located at (5,0,0) and the rough position of the obstacle is (3,4,0), the distance from the sound source to the obstacle is calculated to be 47, and the obstacle position is preliminarily estimated to be (3,3,0). After 10 iterations of Kalman filtering, the precise position of the obstacle is finally obtained as (2,8,0), and the positioning error is reduced to less than 2, achieving accurate estimation of the spatial position of the obstacle.

[0041] S104. In view of the diversity and uncertainty of obstacles, an obstacle recognition algorithm based on transfer learning is adopted. Through the pre-trained general feature extraction model and a small amount of scene-specific sample data, it can quickly adapt to new obstacle types and improve the generalization ability of recognition.

[0042] Obtain a large-scale dataset containing multiple obstacle types, and pre-train a deep learning model for feature extraction on the dataset to obtain a feature representation with generalization ability. For the target application scenario, collect a small amount of obstacle sample data in the scenario, and use the pre-trained model to extract the feature vectors of these samples. Input the extracted feature vector into a small fine-tuning network, and train it to quickly adapt to the obstacle classification task in the target scenario. In actual application, use the pre-trained model to extract features from obstacle images, and then input the features into the fine-tuned classifier for recognition and judgment. To improve the robustness of recognition, data enhancement can be performed on sample data during training to simulate the influence of diversity and uncertainty factors. During the recognition process, if the confidence of the classifier output is lower than the set threshold, it is judged as an unknown obstacle, indicating that there is a risk of recognition uncertainty. Continuously collect obstacle data in actual scenarios, and regularly perform incremental training on the fine-tuning network to enable the recognition algorithm to continuously adapt to environmental changes and maintain good generalization ability.

[0043] Specifically, when developing an obstacle recognition system, we first obtain large-scale image data containing multiple obstacle types from public image datasets such as ImageNet, and use deep convolutional neural networks (such as ResNet-50) to pre-train the feature extraction model to ensure that the model can capture the key visual features of various obstacles. After pre-training, the model can output a 256-dimensional feature vector, which represents the high-level semantic information of the input image. Subsequently, in a specific application scenario, such as an urban road environment, about 1,000 accurately labeled obstacle images are collected, and the feature vectors of these images are extracted using a pre-trained deep learning model. These feature vectors are then used as input to a fine-tuned network, which uses a smaller network structure, such as a fully connected network with two hidden layers and 128 neurons in each layer. Through a small number of iterative training (e.g., 500 iterations), the network is quickly adapted to the obstacle classification task in this scenario. In actual applications, the obstacle image is first processed by the pre-trained feature extraction model to extract a high-dimensional feature vector, which is then input into the fine-tuned classification network. The classification network outputs the probability distribution of each obstacle. If the highest probability value is less than 70%, the obstacle is judged as an unknown type, and the system will remind the driver to pay attention to the potential uncertainty risk. In order to improve the robustness of the system, data enhancement techniques such as random rotation, scaling, and color transformation are introduced during the training process to simulate the impact of different environmental changes on obstacle images. In addition, the system regularly collects new obstacle image data from actual applications, and performs incremental training on the classification network every time a certain number of images are accumulated (for example, every 1,000 images) to ensure that the system can adapt to the continuous changes in the environment, thereby maintaining a high recognition accuracy and good generalization ability.

[0044] S105, obtaining guidance annotation information, such as road signs, lane lines, etc., extracting the features of the guidance annotation through a semantic segmentation algorithm, and dynamically fusing them with the obstacle recognition results, using prior information to improve the accuracy and robustness of obstacle recognition.

[0045] Obtain a road image containing road signs and lane lines, preprocess the image, enhance the image quality and contrast, and prepare for subsequent processing. Use a semantic segmentation algorithm to segment the preprocessed road image, extract the features of guide annotations such as road signs and lane lines, and obtain the segmentation results of the guide annotations. Obtain the obstacle recognition results output by the obstacle recognition module, including information such as the type, position, and size of the obstacle. Dynamically fuse the guide annotation segmentation results with the obstacle recognition results, comprehensively consider the spatial position relationship between the two, and judge the correlation between the obstacle and the guide annotation. If the obstacle is located near the guide annotation, the prior information of the guide annotation is used to correct and optimize the obstacle recognition results to improve the accuracy of obstacle recognition. Through dynamic fusion, the obstacle recognition results are continuously updated, and the obstacle information feedback is also used to optimize the segmentation effect of the guide annotation, so as to achieve the complementarity and improvement of the two. Comprehensively use the recognition results of the guide annotation and the obstacle to output the final environmental perception results, provide reliable environmental information for autonomous driving decision-making and control, and improve the robustness and safety of the autonomous driving system.

[0046] Specifically, in order to obtain high-quality road images, a high-resolution camera, such as a camera with a resolution of 1920×1080, can be used. At the same time, image enhancement algorithms such as histogram equalization and contrast enhancement can be used to improve the contrast and clarity of the image. In terms of semantic segmentation algorithms, a fully convolutional neural network (FCN) based on deep learning can be used to divide road images into different semantic categories such as road signs and lane lines. Through training sample data, the FCN model can learn the feature representation of the guide annotation. The obstacle recognition module can output the types of obstacles such as pedestrians, vehicles, etc., as well as the position and size of their bounding boxes. When the semantic segmentation results are fused with the obstacle recognition results, the IoU (intersection over union) of the obstacle bounding box and the segmented area of ​​the guide annotation can be calculated. If the IoU is greater than a set threshold such as 5, it is considered that the two are associated. The obstacle recognition results are corrected using the prior information of the guide annotation, such as shape and color, for example, the road signs that are mistakenly identified as pedestrians are corrected. At the same time, obstacle information can also be fed back to optimize the semantic segmentation results, such as completing the lane line area covered by the vehicle. The final output of environmental perception results includes the location of road signs and lane lines, as well as the type and location of obstacles, which can serve as the decision-making basis for the autonomous driving system and improve its ability to cope with complex road environments.

[0047] S106. To address the problem of insufficient accuracy of disparity analysis in scenarios such as long distance and high-speed motion, a monocular depth estimation algorithm based on deep learning is introduced. The convolutional neural network is used to predict pixel-level depth information from a single image to improve the accuracy of obstacle distance estimation.

[0048] According to the acquired single image, the pre-trained convolutional neural network model is used to estimate the depth value of each pixel in the image to obtain a pixel-level depth map. According to the pixel-level depth map, the depth value of the obstacle pixel is calculated, and the pixel depth value is converted into the actual distance to obtain the distance estimation value of the obstacle. According to the distance estimation value of the obstacle, it is judged whether the obstacle is within the safe distance range. If the distance is less than the preset safety threshold, it is judged that there is a collision risk. According to the collision risk analysis results of the obstacle, it is determined whether obstacle avoidance measures need to be taken. If obstacle avoidance is required, the obstacle avoidance path is planned according to the position and speed information of the obstacle. The trajectory prediction algorithm is used to estimate the movement trajectory of the obstacle, and the obstacle avoidance strategy and path planning are dynamically adjusted according to the predicted obstacle trajectory. The online learning mechanism is used to fine-tune and update the depth estimation model in real time according to the new data collected during the obstacle avoidance process to improve the accuracy and adaptability of the depth estimation. The real-time speed information of the obstacle is obtained, and the perception range and obstacle avoidance strategy are dynamically adjusted according to the speed to ensure the obstacle avoidance effect and safety in high-speed motion scenes.

[0049] Specifically, in order to achieve the depth estimation and obstacle avoidance of obstacles, the pre-trained ResNet-50 convolutional neural network model is first used to process a single image, and the depth value of each pixel in the image is estimated by the deep learning algorithm to generate a pixel-level depth map with a resolution of 1024×768. Then, the depth value of the obstacle pixel in the depth map is used to convert the pixel depth value d into the actual distance D through the formula D=(B×f) / d, where B is the baseline length and f is the focal length of the camera, to obtain the actual distance estimate of the obstacle. Next, it is determined whether the distance of the obstacle is less than the preset safety threshold (such as 5 meters). If it is less than the safety threshold, it is determined that there is a collision risk and obstacle avoidance is required. When avoiding obstacles, the artificial potential field method is used to plan the obstacle avoidance path according to the position coordinates (x, y) and speed v of the obstacle to generate a smooth obstacle avoidance trajectory. At the same time, the Kalman filter algorithm is used to predict the movement trajectory of the obstacle to obtain the predicted position of the obstacle in the next 5 seconds, and the obstacle avoidance strategy is dynamically adjusted accordingly to improve the real-time and accuracy of obstacle avoidance. In the obstacle avoidance process, the depth estimation model is fine-tuned using the online learning mechanism based on the newly collected 1,000 image data. The Adam optimizer is used with the learning rate set to 0.01 and iterated 50 times to improve the accuracy and generalization ability of the model. Finally, the real-time speed information of the obstacle is obtained through the millimeter-wave radar. When the speed of the obstacle exceeds 60km / h, the perception range is dynamically expanded to 100 meters, and the obstacle avoidance strategy is adjusted accordingly to ensure the obstacle avoidance effect and safety in high-speed scenarios.

[0050] S107. Based on the obstacle recognition results of multi-source heterogeneous information fusion, a real-time decision algorithm is used to generate behavioral decision instructions such as obstacle avoidance and deceleration based on the position, speed, trajectory and other information of the obstacle, while meeting the real-time requirements, to ensure the system's timely response and processing of obstacles.

[0051] Obtain the raw data of obstacles through multi-source heterogeneous sensors, pre-process the raw data, and extract the characteristic information of obstacles. The extracted obstacle characteristic information is fused, and the properties of obstacles such as position, speed and trajectory are comprehensively analyzed to identify and classify obstacles. According to the obstacle identification results, the decision tree algorithm is used to make real-time decisions to judge the degree of impact and danger level of obstacles on the system. If the obstacle has a greater impact on the system and the danger level is high, an obstacle avoidance decision instruction is generated, and the control system performs obstacle avoidance operations to bypass the obstacle. If the obstacle has a smaller impact on the system and the danger level is low, a deceleration decision instruction is generated, and the control system reduces the speed to ensure safe passage through the obstacle. While performing obstacle avoidance or deceleration operations, the position, speed and trajectory changes of obstacles are continuously tracked, and the decision instructions are updated in real time to ensure the continuous response of the system to obstacles. In the whole process, the real-time performance of the system is guaranteed through optimization measures such as parallel computing and pipeline processing, so that the delay from obstacle identification to decision instruction generation meets the real-time requirements.

[0052] Specifically, the system obtains the original data of obstacles through multi-source heterogeneous sensors such as laser radar, millimeter wave radar, and ultrasonic sensors, performs pre-processing such as denoising and filtering on the original point cloud data, and extracts the characteristic information such as the edge, shape, and texture of the obstacle. The Kalman filter algorithm is used to estimate the position and speed of the obstacle, and the trajectory clustering algorithm is used to analyze the movement trajectory of the obstacle. The obstacle is identified by integrating multi-source information, and the support vector machine algorithm is used to classify the obstacles, with an identification accuracy of more than 95%. According to the type, position, speed and other attributes of the obstacle, the decision tree algorithm is used to make real-time decisions to judge the degree of impact and danger level of the obstacle on the system. When the distance between the obstacle and the system is less than the safety threshold (such as 5 meters) and the relative speed is greater than 10 meters / second, an obstacle avoidance decision instruction is generated, and the control system performs a detour to avoid the obstacle. When the distance between the obstacle and the system is greater than the safety threshold and the relative speed is less than 5 meters / second, a deceleration decision instruction is generated, and the control system reduces the speed to less than 2 meters / second to ensure safe passage. The system uses an extended Kalman filter algorithm to track the changes in the position and speed of obstacles in real time with a period of 20 milliseconds, and updates the decision instructions in real time according to the movement state of the obstacles. Through optimization measures such as parallel computing and pipeline processing, the delay from obstacle recognition to decision instruction generation is controlled within 50 milliseconds, meeting the system's real-time obstacle avoidance needs.

[0053] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for identifying road obstacles based on machine vision, characterized in that: include: Step S1, acquiring multi-source heterogeneous sensor data including visual information and auditory information; Step S2: inputting multi-source heterogeneous sensor data into the obstacle detection model to obtain the obstacle area; Step S3: Obstacle position and distance information estimated by the sound source localization algorithm; Step S4: using an obstacle recognition algorithm based on transfer learning to obtain the obstacle type; Step S5: Generate a behavior decision instruction based on the obstacle area, obstacle orientation and distance information, and obstacle type.

2. The method for identifying road obstacles based on machine vision according to claim 1, characterized in that: In step S2, the obstacle detection model adopts a convolutional neural network architecture to identify obstacle areas in multi-source heterogeneous sensor data through feature extraction and target detection algorithms.

3. The method for identifying road obstacles based on machine vision as claimed in claim 2, characterized in that: In step S3, a monocular depth estimation algorithm based on deep learning is used to predict pixel-level depth information from a single image through a convolutional neural network to obtain an accurate obstacle distance.

4. The method for identifying road obstacles based on machine vision as claimed in claim 3, characterized in that: In step S2, the guidance annotation information is obtained, the features of the guidance annotation are extracted through the semantic segmentation algorithm, and dynamically fused with multi-source heterogeneous sensor data to accurately locate the spatial position of the obstacle.

Images