An automatic driving obstacle avoidance system and method based on machine vision

Abstract

The application relates to the technical field of automatic driving obstacle avoidance systems, in particular to an automatic driving obstacle avoidance system and method based on machine vision, and the automatic driving obstacle avoidance system based on machine vision is composed of a data acquisition module, a data preprocessing module, a data fusion module, an image processing module, a decision module and a communication module.In the application, real-time semantic segmentation and image processing technology are used to provide more comprehensive scene understanding, the system can better identify road, pedestrian and vehicle targets, more accurate information can be provided for obstacle avoidance decision, a reinforcement learning and model predictive control method is used to optimize the decision strategy, the system can make efficient and safe obstacle avoidance decisions under complex scenes and moving obstacles, cross-modal data fusion and multi-modal perception improve the reliability of system perception, the limitations of a single sensor are overcome, eye movement tracking and emotional analysis optimize human-computer interaction, the attention and emotional state of a driver are better understood, and driving safety is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving obstacle avoidance systems, and in particular to an automatic driving obstacle avoidance system and method based on machine vision. BACKGROUND

[0002] An automatic driving obstacle avoidance system is a system that can help an autonomous vehicle to perceive and avoid obstacles. It uses various sensors, algorithms, and decision systems to monitor and analyze the environment around the vehicle and avoid potential obstacles by automatically controlling the steering, acceleration, and braking of the vehicle. The goal of this system is to ensure that the vehicle can safely avoid collisions with other vehicles, pedestrians, buildings, animals, or other obstacles during driving. Among them, the automatic driving obstacle avoidance system based on machine vision refers to a system that uses image recognition and analysis technology to perceive and avoid obstacles. It collects real-time images through the use of cameras or other visual sensors and processes and analyzes the images through computer vision algorithms to detect and identify obstacles around the vehicle.

[0003] In the actual use process of the existing automatic driving obstacle avoidance system, there are a large number of obstacles and traffic situations in complex scenes, and traditional methods may not be able to effectively process and analyze this information, resulting in performance degradation. Secondly, traditional methods may have accuracy and reliability problems in environmental perception and obstacle detection. The limitations of a single sensor and sensor noise may result in incomplete or inaccurate information, affecting the accuracy and safety of obstacle avoidance decisions. In addition, traditional methods may lack flexibility and intelligence in obstacle avoidance decisions. Traditional rule-based decision-making methods may not be able to adapt to complex and dynamic traffic environments, limiting the system's adaptive ability and decision-making effectiveness. Traditional methods may have limited understanding of the driver's emotions and attention state, and may not fully consider the driver's behavior and reaction. This may result in inaccurate or overly frequent warnings to the driver, affecting the driving experience and safety. Finally, traditional methods may have limitations in communication and cooperation in cooperative driving. Traditional methods may not be able to share and process information between vehicles in real time, limiting the system's collaborative decision-making and driving efficiency. SUMMARY

[0004] The purpose of the present application is to solve the shortcomings in the prior art and propose an automatic driving obstacle avoidance system and method based on machine vision.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solution: an automatic driving obstacle avoidance system based on machine vision is composed of a data acquisition module, a data preprocessing module, a data fusion module, an image processing module, a decision module, and a communication module.

[0006] The data acquisition module is connected with sensors of the vehicle, and the sensors include a camera, a laser radar, a radar, a sound sensor and an infrared sensor, and the sensors collect original data in real time and send the original data to the data preprocessing module;

[0007] The data preprocessing module receives the original data from the data acquisition module, pre-processes the original data, generates processed data after the pre-processing, and sends the processed data to the data fusion module and the image processing module;

[0008] The data fusion module receives the processed data, fuses the processed data together through a fusion algorithm, obtains fused data, and sends the fused data to the decision module;

[0009] The image processing module processes data in parallel with the data fusion module, receives image data in the processed data, performs deep learning processing including real-time semantic segmentation and target detection on the image data, divides the image data into a plurality of semantic regions, identifies key targets including vehicles, pedestrians and road signs in the image, integrates processing results, and sends the processing results to the decision module;

[0010] The decision module receives the fused data and the processing results, obtains an obstacle avoidance decision, optimizes the decision by using reinforcement learning and model predictive control methods in the decision process, obtains eye tracking and emotional analysis data to optimize the driving strategy, and generates a decision instruction;

[0011] The communication module receives the decision instruction and transmits the decision instruction to the control system of the vehicle, so as to realize obstacle avoidance, exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and feeds back the effective information to the decision module.

[0012] As a further scheme of the application, the data acquisition module includes a camera sub-module, a laser radar sub-module, a radar sub-module, a sound sensor sub-module and an infrared sensor sub-module;

[0013] The camera sub-module uses a camera sensor to collect image data;

[0014] The laser radar sub-module uses a laser radar sensor to collect laser point cloud data of the surrounding environment;

[0015] The radar sub-module uses a radar sensor to collect radar wave data in the environment;

[0016] The sound sensor sub-module uses a sound wave sensor to collect sound data of the surrounding environment;

[0017] The infrared sensor sub-module collects infrared signal data using an infrared sensor.

[0018] As a further scheme of the present application, the data preprocessing module comprises a data cleaning sub-module, a data formatting sub-module;

[0019] The data cleaning sub-module adopts a noise filtering algorithm, an outlier detection algorithm, and a missing value filling algorithm.

[0020] The noise filtering algorithm specifically adopts a filtering algorithm including mean filtering, median filtering, and Gaussian filtering to remove random noise in the original data.

[0021] The outlier detection algorithm includes a statistical-based outlier detection algorithm and a machine learning-based anomaly detection algorithm, which are used to identify and exclude abnormal points in the original data.

[0022] The missing value filling algorithm fills missing values using interpolation and regression methods when there are missing values in the original data.

[0023] The data formatting sub-module adopts a feature standardization algorithm, a feature encoding algorithm, and a data dimensionality reduction algorithm.

[0024] The feature standardization algorithm specifically scales and translates data by maximum and minimum scaling and mean and variance standardization, so that different features have the same scale.

[0025] The feature encoding algorithm converts non-numeric features into numeric features based on one-hot encoding and label encoding.

[0026] The data dimensionality reduction algorithm includes principal component analysis and linear discriminant analysis, which are used to reduce the number of features, reduce computational complexity, and reduce model training time.

[0027] As a further scheme of the present application, the data fusion module comprises a fusion algorithm sub-module and an environment perception sub-module.

[0028] The fusion algorithm sub-module includes a Kalman filter, a particle filter, a Gaussian mixture model, and a conditional random field.

[0029] The Kalman filter combines the processed data with a prediction model through recursive state estimation and observation update to obtain optimal state estimation and perform fusion tasks for linear systems and Gaussian noise.

[0030] The particle filter uses random particle resampling and particle weight update to approximately estimate the potential target state and complete the fusion task for non-linear and non-Gaussian distribution.

[0031] The Gaussian mixture model utilizes a set of Gaussian distributions to model the distribution of multi-sensor data, and performs data fusion tasks through maximum posterior probability estimation;

[0032] The conditional random field defines a probabilistic graphical model, models the relationship and constraint conditions of sensor data, and realizes data fusion and joint estimation;

[0033] The environment perception sub-module adopts object detection and recognition algorithm, map construction algorithm, motion estimation algorithm;

[0034] The object detection and recognition algorithm utilizes the fused data for target detection and recognition, including vehicles, pedestrians and road signs;

[0035] The map construction algorithm compares and matches the fused data with the existing map, and realizes accurate positioning of the vehicle position and route;

[0036] The motion estimation algorithm infers the motion information of objects in the environment, including speed, acceleration and motion trajectory, by analyzing the changes of sensor data.

[0037] As a further scheme of the application, the image processing module includes a real-time semantic segmentation sub-module and a target detection sub-module;

[0038] The real-time semantic segmentation sub-module includes an encoder, a decoder and a loss function;

[0039] The encoder uses convolution and pooling operations to gradually reduce the dimensionality of image features and extract feature information;

[0040] The decoder uses deconvolution and up-sampling operations to restore the feature map to the original image size, and combines the feature information for information fusion to generate a semantic segmentation result;

[0041] The loss function compares the difference between the semantic segmentation result and the true label, and optimizes the network parameters through backpropagation.

[0042] As a further scheme of the application, the function items of the target detection sub-module include candidate box generation, feature extraction, target classification and positioning;

[0043] The candidate box generation specifically generates candidate target boxes using a region proposal method;

[0044] The feature extraction specifically extracts feature representations within the candidate target boxes using convolutional neural networks including ResNet and VGGNet;

[0045] The target classification and positioning specifically comprises judging a target category using a classifier and performing position regression to obtain accurate target positioning based on the feature representation.

[0046] As a further scheme of the present application, the decision module comprises a decision algorithm submodule and a strategy optimization submodule.

[0047] The functional items of the decision algorithm submodule comprise state definition, action space definition, reward function definition, and reinforcement learning algorithm training.

[0048] The state definition specifically comprises converting the fused data and processing results into state representations required for driving decisions, including vehicle position, speed, and surrounding environment information.

[0049] The action space definition specifically comprises defining feasible driving actions based on a driving decision model, including acceleration, deceleration, and steering, to construct an action space.

[0050] The reward function definition specifically comprises designing a reward function to guide the optimization of driving decisions, including penalizing collisions and encouraging smooth driving.

[0051] The reinforcement learning algorithm training specifically comprises training using a reinforcement learning algorithm to optimize the driving decision model, so that the driving decision model can select the optimal driving action under different states.

[0052] As a further scheme of the present application, the functional items of the strategy optimization submodule comprise state perception, emotion analysis, and driving strategy optimization.

[0053] The state perception specifically comprises obtaining the focus points and visual behavior of the driver's gaze through eye tracking technology to understand the driver's attention and emotional state for the driving task.

[0054] The emotion analysis specifically comprises analyzing the emotional changes of the driver through emotion analysis technology, including anxiety and fatigue.

[0055] The driving strategy optimization specifically comprises optimizing the driving strategy using an MPC optimization algorithm to adjust the acceleration and steering parameters of the vehicle based on the results of state perception and emotion analysis.

[0056] As a further scheme of the present application, the communication module comprises an information sending submodule and an information receiving submodule.

[0057] The information sending submodule is equipped with a vehicle communication protocol, which specifically comprises a vehicle communication network CAN, FlexRay, and Ethernet.

[0058] Both the information sending submodule and the information receiving submodule adopt V2V communication technology and V2I communication technology.

[0059] The information sending sub-module fuses information from sensors of the vehicle and other vehicles and infrastructure based on sensor data fusion, provides accurate and comprehensive environmental perception.

[0060] An automatic driving obstacle avoidance method based on machine vision, comprising the following steps:

[0061] Using a data acquisition module, real-time acquisition of data around the vehicle as raw data;

[0062] The raw data is transmitted to the data preprocessing module for preprocessing work including data cleaning, formatting and dimensionality reduction to generate processed data;

[0063] The data fusion module receives the processed data and integrates the data collected by different sensors through a fusion algorithm to obtain fused data;

[0064] The image processing module processes the image data in the processed data through deep learning including real-time semantic segmentation and target detection to identify key targets in the image;

[0065] The decision module integrates the fused data and key targets to make an obstacle avoidance decision, optimizes the decision process through reinforcement learning and model predictive control method, obtains additional data through eye tracking and emotion analysis, optimizes the driving strategy, and generates a decision instruction;

[0066] The decision instruction is transmitted to the control system of the vehicle through the communication module to realize obstacle avoidance operation, the communication module also exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and feeds back the information to the decision module.

[0067] Compared with the prior art, the advantages and positive effects of the present application are:

[0068] In the present application, real-time semantic segmentation and image processing technology provides more comprehensive scene understanding, making the system better identify targets such as roads, pedestrians and vehicles, providing more accurate information for obstacle avoidance decision. Reinforcement learning and model predictive control method optimize the decision strategy, making the system make efficient and safe obstacle avoidance decisions in complex scenes and moving obstacles. Cross-modal data fusion and multi-modal perception improve the reliability of system perception, overcoming the limitations of single sensor. Eye tracking and emotion analysis optimize human-computer interaction, better understand the attention and emotional state of the driver, and improve driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A main system flowchart of an automatic driving obstacle avoidance system and method based on machine vision is proposed for the present application;

[0070] Figure 2 A data acquisition module flow chart of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application.

[0071] Figure 3 A data preprocessing module flow chart of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application.

[0072] Figure 4 A data fusion module flow chart of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application.

[0073] Figure 5 An image processing module flow chart of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application.

[0074] Figure 6 A decision module flow chart of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application.

[0075] Figure 7 A communication module flow chart of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application.

[0076] Figure 8 A working step schematic diagram of an automatic driving obstacle avoidance system and method based on machine vision is provided for the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0078] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0079] Example One

[0080] Please refer to Figure 1The application provides a technical scheme: an automatic driving obstacle avoidance system based on machine vision is composed of a data acquisition module, a data preprocessing module, a data fusion module, an image processing module, a decision module and a communication module;

[0081] The data acquisition module is connected with sensors of the vehicle, and the sensors include a camera, a laser radar, a radar, a sound sensor and an infrared sensor, and the sensors collect original data in real time and send the original data to the data preprocessing module;

[0082] The data preprocessing module receives the original data from the data acquisition module, performs preprocessing work on the original data, generates processed data after the preprocessing is completed, and sends the processed data to the data fusion module and the image processing module;

[0083] The data fusion module receives the processed data, fuses the processed data together through a fusion algorithm, obtains fused data, and sends the fused data to the decision module;

[0084] The image processing module processes data in parallel with the data fusion module, receives image data in the processed data, performs deep learning processing including real-time semantic segmentation and target detection on the image data, divides the image data into a plurality of semantic regions, identifies key targets including vehicles, pedestrians and road signs in the image, integrates processing results, and sends the processing results to the decision module;

[0085] The decision module receives the fused data and the processing results, obtains an obstacle avoidance decision, optimizes the decision by using reinforcement learning and model predictive control methods in the decision process, simultaneously obtains eye tracking and emotional analysis data to optimize a driving strategy, and generates a decision instruction;

[0086] The communication module receives the decision instruction and transmits the decision instruction to a control system of the vehicle, so that obstacle avoidance is realized, the communication module exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and feeds back the effective information to the decision module.

[0087] The automatic driving obstacle avoidance system based on machine vision is composed of a data acquisition module, a data preprocessing module, a data fusion module, an image processing module, a decision module, and a communication module. The data acquisition module is connected with the sensors of the vehicle, including a camera, a laser radar, a radar, a sound sensor, and an infrared sensor, which acquires raw data in real time and sends it to the data preprocessing module. The data preprocessing module processes the raw data to generate processed data, which is sent to the data fusion module and the image processing module. The data fusion module receives the processed data and fuses them together through a fusion algorithm to obtain fused data and send it to the decision module. The image processing module is parallel to the data fusion module and performs real-time semantic segmentation and target detection on the received image data through deep learning processing, and sends the processing results to the decision module. The decision module comprehensively fuses the processed data and processing results, makes an obstacle avoidance decision, optimizes the decision through reinforcement learning and model predictive control methods, and generates a decision instruction. The communication module receives the decision instruction and transmits it to the control system of the vehicle to realize obstacle avoidance operation. At the same time, the communication module exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and feeds it back to the decision module. This system implementation can bring high-precision perception ability, comprehensive information processing, powerful image processing and recognition ability, intelligent decision and obstacle avoidance ability, and real-time communication and information exchange advantages, improving the safety and driving efficiency of the vehicle.

[0088] Please refer to Figure 2 The data acquisition module includes a camera sub-module, a laser radar sub-module, a radar sub-module, a sound sensor sub-module, and an infrared sensor sub-module.

[0089] The camera sub-module uses a camera sensor to collect image data.

[0090] The laser radar sub-module uses a laser radar sensor to collect laser point cloud data of the surrounding environment.

[0091] The radar sub-module uses a radar sensor to collect radar wave data in the environment.

[0092] The sound sensor sub-module uses a sound sensor to collect sound data of the surrounding environment.

[0093] The infrared sensor sub-module uses an infrared sensor to collect infrared signal data.

[0094] The data acquisition module includes a camera sub-module, a laser radar sub-module, a radar sub-module, a sound sensor sub-module, and an infrared sensor sub-module. These sub-modules collectively realize multi-source and multi-dimensional data acquisition capabilities, bringing many beneficial effects to the automatic driving obstacle avoidance system. The camera sub-module provides visual perception capabilities for road markings, vehicles, and pedestrians, etc. by collecting image data. The laser radar sub-module provides accurate distance and position information for obstacle detection and positioning by collecting laser point cloud data. The radar sub-module can detect obstacles at a long distance by collecting radar wave data, enhancing the system's perception range. The sound sensor sub-module and the infrared sensor sub-module provide sound and infrared signal data in the environment, respectively, to supplement the information of other sensors. This multi-source and multi-dimensional data acquisition capability enables the system to have comprehensive and accurate perception of the surrounding environment, improving safety and the accuracy of driving decisions. In addition, the data acquisition module integrates data from different sensors through data fusion and comprehensive analysis, achieving a comprehensive understanding of the environment. This comprehensive analysis can improve the robustness and fault tolerance of the obstacle avoidance system, ensuring that the system can still function normally even if a sensor fails. In summary, the integration of the data acquisition module brings many advantages to the automatic driving obstacle avoidance system, including comprehensive perception capabilities, accurate obstacle detection, robust system stability, etc., promoting the development and application of automatic driving technology.

[0095] Please refer to Figure 3 The data preprocessing module includes a data cleaning sub-module and a data formatting sub-module.

[0096] The data cleaning sub-module adopts a noise filtering algorithm, an outlier detection algorithm, and a missing value filling algorithm.

[0097] The noise filtering algorithm specifically includes mean filtering, median filtering, and Gaussian filtering algorithms to remove random noise in the original data.

[0098] The outlier detection algorithm includes a statistical-based outlier detection algorithm and a machine learning-based anomaly detection algorithm to identify and exclude abnormal points in the original data.

[0099] The missing value filling algorithm uses interpolation and regression methods to fill in missing values when there are missing values in the original data.

[0100] The data formatting sub-module adopts a feature standardization algorithm, a feature encoding algorithm, and a data dimensionality reduction algorithm.

[0101] The feature standardization algorithm specifically includes maximum and minimum value scaling and mean and variance standardization to scale and translate data so that different features have the same scale.

[0102] Feature encoding algorithms, based on one-hot encoding and label encoding, convert non-numerical features into numerical features;

[0103] Data dimensionality reduction algorithms include principal component analysis and linear discriminant analysis. These algorithms are used to reduce the number of features, thereby reducing computational complexity and model training time.

[0104] The data preprocessing module is a crucial component of the autonomous driving obstacle avoidance system, comprising a data cleaning submodule and a data formatting submodule. The data cleaning submodule improves data quality and accuracy through noise filtering, outlier detection, and missing value imputation algorithms. Noise filtering removes random noise using filtering techniques, outlier detection identifies and eliminates anomalies, and missing value imputation handles missing data. The data formatting submodule employs feature standardization and feature encoding algorithms to standardize and unify the data. Feature standardization maps different feature values ​​to the same scale range, and feature encoding converts non-numerical features into numerical ones. Furthermore, dimensionality reduction algorithms reduce the number of features through methods such as principal component analysis and linear discriminant analysis, improving computational efficiency and model performance. By comprehensively applying these techniques, the data preprocessing module enhances data quality, standardizes data, and reduces dimensionality, thereby strengthening the data processing capabilities and computational efficiency of the autonomous driving obstacle avoidance system and supporting the realization of reliable and efficient autonomous driving.

[0105] Please see Figure 4 The data fusion module includes a fusion algorithm submodule and an environmental perception submodule;

[0106] The fusion algorithm submodule includes Kalman filter, particle filter, Gaussian mixture model, and conditional random field;

[0107] The Kalman filter combines the processed data with the prediction model through recursive state estimation and observation updates to obtain the optimal state estimate and perform the fusion task of linear system and Gaussian noise.

[0108] The particle filter employs random particle resampling and particle weight updates to approximately estimate the potential target state, thus completing the fusion task of nonlinear and non-Gaussian distributions.

[0109] Gaussian mixture models use a set of Gaussian distributions to model the distribution of multi-sensor data and perform data fusion tasks through maximum a posteriori probability estimation.

[0110] Conditional random fields define a probabilistic graphical model that models the relationships and constraints of sensor data to achieve data fusion and joint estimation.

[0111] The environmental perception submodule employs object detection and recognition algorithms, map building algorithms, and motion estimation algorithms.

[0112] Object detection and recognition algorithms utilize fused data to detect and recognize targets, including vehicles, pedestrians, and road signs;

[0113] The map building algorithm compares and matches the fused data with existing maps to achieve accurate positioning of vehicle location and route;

[0114] Motion estimation algorithms infer motion information of objects in the environment, including velocity, acceleration, and trajectory, by analyzing changes in sensor data.

[0115] The data fusion module is a crucial component of the autonomous driving obstacle avoidance system, comprising a fusion algorithm submodule and an environmental perception submodule. The fusion algorithm submodule employs algorithms such as Kalman filters, particle filters, Gaussian mixture models, and conditional random fields to fuse multi-source data and optimize state estimation. These algorithms can handle linear systems and data with Gaussian noise, nonlinearity, and non-Gaussian distributions, providing accurate state estimation and integrated data fusion capabilities. The environmental perception submodule enhances environmental perception capabilities using the fused data through algorithms such as object detection and recognition, map building, and motion estimation. Object detection and recognition algorithms accurately identify targets such as vehicles, pedestrians, and road signs; map building algorithms precisely locate vehicle positions and routes; and motion estimation algorithms infer object motion information. In summary, the integration and application of the data fusion module improves the data processing capabilities, environmental perception capabilities, and decision-making accuracy of the autonomous driving obstacle avoidance system, resulting in a safer and more reliable autonomous driving experience.

[0116] Please see Figure 5 The image processing module includes a real-time semantic segmentation submodule and an object detection submodule;

[0117] The real-time semantic segmentation submodule includes an encoder, a decoder, and a loss function;

[0118] The encoder uses convolution and pooling operations to gradually reduce the dimensionality of image features and extract feature information;

[0119] The decoder uses deconvolution and upsampling operations to restore the feature maps to the original image size, and combines the feature information to perform information fusion to generate semantic segmentation results;

[0120] The loss function compares the difference between the semantic segmentation results and the real labels, and optimizes the network parameters through backpropagation.

[0121] The object detection submodule includes functions such as candidate box generation, feature extraction, object classification, and localization.

[0122] Candidate box generation specifically involves using a region proposal method to generate candidate target boxes;

[0123] Feature extraction specifically involves using convolutional neural networks, including ResNet and VGGNet, to extract feature representations within candidate bounding boxes.

[0124] Specifically, target classification and localization involves using a classifier to determine the target category based on feature representation and performing location regression to obtain accurate target localization.

[0125] The image processing module is a crucial component of autonomous driving obstacle avoidance systems, comprising a real-time semantic segmentation submodule and an object detection submodule. The real-time semantic segmentation submodule utilizes an encoder, decoder, and loss function to perform real-time semantic segmentation of the input image, labeling each pixel in the image as belonging to a specific category and providing fine-grained semantic information. The object detection submodule, through candidate box generation, feature extraction, object classification, and localization, achieves efficient and accurate detection and localization of objects in the image, providing crucial information for subsequent decision-making and planning. The benefits of integrating these submodules include real-time performance, accuracy, and efficiency. Through real-time semantic segmentation and object detection, the image processing module can provide refined semantic information and object recognition results, enhancing the environmental perception capabilities and decision-making reliability of autonomous driving obstacle avoidance systems. This provides valuable support for achieving a safe and efficient autonomous driving experience.

[0126] Please see Figure 6 The decision-making module includes a decision algorithm submodule and a strategy optimization submodule;

[0127] The decision algorithm submodule includes functions for state definition, action space definition, reward function definition, and reinforcement learning algorithm training.

[0128] Specifically, the state definition involves transforming the fused data and processing results into a state representation required for driving decisions, including vehicle position, speed, and surrounding environment information.

[0129] The definition of action space specifically involves defining feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, to construct the action space.

[0130] The reward function is specifically defined as a function designed to guide the optimization of driving decisions, including penalizing collisions and encouraging smooth driving.

[0131] The reinforcement learning algorithm training specifically involves using a reinforcement learning algorithm to train and optimize the driving decision model, enabling the driving decision model to select the optimal driving action under different conditions.

[0132] The strategy optimization submodule includes functions such as state perception, sentiment analysis, and driving strategy optimization.

[0133] State perception specifically involves using eye-tracking technology to acquire the driver's gaze points and visual behavior, thereby understanding the driver's attention to the driving task and their emotional state.

[0134] Sentiment analysis specifically involves analyzing a driver's emotional changes, including anxiety and fatigue, using sentiment analysis techniques.

[0135] Specifically, driving strategy optimization involves using the MPC optimization algorithm to optimize the driving strategy based on the results of state perception and sentiment analysis, thereby adjusting the vehicle's acceleration and steering parameters.

[0136] The decision-making algorithm submodule, through functions such as state definition, action space definition, reward function definition, and reinforcement learning algorithm training, transforms the data fusion and processing results into the state representation required for driving decisions and defines the space of driving actions. By designing a suitable reward function and training it using reinforcement learning algorithms, the driving decision-making model can be optimized, enabling it to select the optimal driving action under different states. This optimization of the decision-making algorithm can improve the decision-making performance of the autonomous driving obstacle avoidance system, enabling it to make more appropriate and safer driving decisions.

[0137] The strategy optimization submodule includes functions for state perception, sentiment analysis, and driving strategy optimization. It uses eye-tracking technology to obtain the driver's gaze points and visual behavior, understanding the driver's focus on the driving task and their emotional state. Sentiment analysis technology analyzes changes in the driver's emotions, such as anxiety and fatigue. Based on the results of state perception and sentiment analysis, a model predictive control optimization algorithm is used to optimize the driving strategy, adjusting vehicle acceleration and steering parameters to achieve a safer and more comfortable driving experience. This strategy optimization improves the performance of the autonomous driving obstacle avoidance system and the driving experience.

[0138] Please see Figure 7 The communication module includes an information sending submodule and an information receiving submodule;

[0139] The information transmission submodule is equipped with vehicle communication protocols, specifically vehicle communication networks CAN, FlexRay, and Ethernet.

[0140] Both the information sending submodule and the information receiving submodule adopt V2V communication technology and V2I communication technology;

[0141] The information transmission submodule is based on sensor data fusion, which combines information from vehicle sensors and other vehicles and infrastructure to provide accurate and comprehensive environmental perception.

[0142] The communication module is a crucial component of an autonomous driving obstacle avoidance system, comprising an information sending submodule and an information receiving submodule. The information sending submodule utilizes vehicle communication protocols such as CAN, FlexRay, and Ethernet to ensure reliable data transmission and exchange. Employing V2V and V2I communication technologies, the information sending submodule can communicate with other vehicles and infrastructure in real time, providing comprehensive environmental perception. Through sensor data fusion, it integrates information from vehicle sensors and other vehicles and infrastructure, enabling accurate and comprehensive environmental perception and providing crucial information for decision-making. Integrating and applying these submodules achieves efficient, real-time, and reliable information exchange between vehicles and between vehicles and infrastructure, improving the autonomous driving obstacle avoidance system's understanding and perception of the surrounding environment, thereby enhancing the system's coordination, adaptability, and safety.

[0143] Please see Figure 8 A machine vision-based obstacle avoidance method for autonomous driving includes the following steps:

[0144] The data acquisition module is used to collect data around the vehicle in real time as raw data.

[0145] The raw data is transmitted to the data preprocessing module for preprocessing, including data cleaning, formatting, and dimensionality reduction, to generate processed data.

[0146] The data fusion module receives the processed data and integrates the data collected by different sensors through a fusion algorithm to obtain fused data.

[0147] The image processing module performs deep learning processing on the image data in the processed data, including real-time semantic segmentation and object detection, in order to identify key targets in the image;

[0148] The decision-making module integrates and merges post-data and key objectives to make obstacle avoidance decisions. It optimizes the decision-making process through reinforcement learning and model predictive control methods, uses eye tracking and sentiment analysis to obtain additional data, optimizes driving strategies, and generates decision instructions.

[0149] The decision-making instructions are transmitted to the vehicle's control system through the communication module to enable obstacle avoidance. The communication module also exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and transmits feedback information to the decision-making module.

[0150] The data acquisition module acquires real-time data about the vehicle's surroundings and preprocesses it to generate data suitable for subsequent processing. The data fusion module integrates data from multiple sensors, while the image processing module performs real-time semantic segmentation and target detection on the image data to achieve accurate environmental perception. The decision-making module integrates the fused data and target recognition results to make obstacle avoidance decisions and optimizes the decision-making process through reinforcement learning and model predictive control. The communication module transmits decision commands to the vehicle's control system and exchanges information with other vehicles and infrastructure. This integrated implementation provides high-quality data, accurate target recognition, and precise environmental perception, thereby enhancing driving decision-making and obstacle avoidance capabilities and improving the safety and adaptability of the autonomous driving obstacle avoidance system.

[0151] Working Principle: First, the data acquisition module acquires real-time data about the vehicle's surroundings as raw data. Next, the raw data is transmitted to the data preprocessing module for cleaning, formatting, and dimensionality reduction, generating processed data. Then, the data fusion module receives the processed data and integrates data from different sensors using a fusion algorithm, resulting in fused data. Next, the image processing module performs deep learning processing on the image data in the processed data, such as real-time semantic segmentation and object detection, to identify key targets. The decision-making module integrates the fused data and key targets to make obstacle avoidance decisions, using reinforcement learning and model predictive control methods to optimize the decision-making process. Simultaneously, eye tracking and sentiment analysis are used to acquire additional data, optimize driving strategies, and generate decision commands. Finally, the decision commands are transmitted to the vehicle's control system via the communication module to achieve obstacle avoidance. The communication module also exchanges information with other vehicles and intelligent transportation infrastructure, providing or acquiring effective information and transmitting feedback to the decision-making module. Integrating these steps provides accurate environmental perception, optimized decision-making, and efficient information exchange, thereby improving the safety and adaptability of the autonomous driving obstacle avoidance system and achieving a more reliable and intelligent autonomous driving experience.

[0152] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An autonomous driving obstacle avoidance system based on machine vision, characterized in that: The aforementioned machine vision-based autonomous driving obstacle avoidance system consists of a data acquisition module, a data preprocessing module, a data fusion module, an image processing module, a decision-making module, and a communication module. The data acquisition module is connected to the vehicle's sensors, including cameras, lidar, radar, sound sensors, and infrared sensors. The sensors collect raw data in real time and send it to the data preprocessing module. The data preprocessing module receives raw data from the data acquisition module, performs preprocessing on the raw data, generates processed data after preprocessing, and sends the processed data to the data fusion module and the image processing module. The data fusion module receives the processed data, merges the processed data together using a fusion algorithm, obtains the fused data, and sends it to the decision module. The image processing module processes data in parallel with the data fusion module. It receives image data from the processed data, performs deep learning processing on the image data including real-time semantic segmentation and target detection, divides the image data into multiple semantic regions, identifies key targets in the image including vehicles, pedestrians, and road signs, integrates the processing results, and sends them to the decision module. The decision module receives the fused data and processing results, derives an obstacle avoidance decision, and optimizes the decision using reinforcement learning and model predictive control methods during the decision-making process. At the same time, it acquires eye-tracking and sentiment analysis data to optimize the driving strategy and generate decision instructions. The communication module receives decision instructions and transmits them to the vehicle's control system to achieve obstacle avoidance. The communication module also exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and feeds back the effective information to the decision module. The decision-making module includes a decision algorithm submodule and a strategy optimization submodule; The decision algorithm submodule includes functions for state definition, action space definition, reward function definition, and reinforcement learning algorithm training. The state definition specifically refers to transforming the fused data and processing results into a state representation required for driving decisions, including vehicle position, speed, and surrounding environment information; The definition of the action space specifically refers to defining feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, to construct the action space; The reward function is specifically defined as a function designed to guide the optimization of driving decisions, including penalizing collisions and encouraging smooth driving. The reinforcement learning algorithm training specifically involves using a reinforcement learning algorithm to train and optimize the driving decision model, enabling the driving decision model to select the optimal driving action under different conditions. The functional items of the strategy optimization submodule include state perception, sentiment analysis, and driving strategy optimization. The state perception specifically refers to obtaining the driver's gaze points and visual behavior through eye-tracking technology to understand the driver's attention to the driving task and emotional state. The sentiment analysis specifically involves analyzing the driver's emotional changes, including anxiety and fatigue, using sentiment analysis technology. The driving strategy optimization specifically involves using the MPC optimization algorithm to optimize the driving strategy based on the results of state perception and sentiment analysis, thereby adjusting the vehicle's acceleration and steering parameters.

2. The machine vision-based obstacle avoidance system for autonomous driving according to claim 1, characterized in that: The data acquisition module includes a camera submodule, a lidar submodule, a radar submodule, a sound sensor submodule, and an infrared sensor submodule; The camera submodule uses a camera sensor to acquire image data; The lidar submodule uses lidar sensors to collect lidar point cloud data of the surrounding environment; The radar submodule uses radar sensors to collect radar wave data from the environment; The sound sensor submodule uses a sound wave sensor to collect sound data from the surrounding environment; The infrared sensor submodule uses an infrared sensor to collect infrared signal data.

3. The machine vision-based obstacle avoidance system for autonomous driving according to claim 1, characterized in that: The data preprocessing module includes a data cleaning submodule and a data formatting submodule; The data cleaning submodule employs noise filtering algorithm, outlier detection algorithm, and missing value filling algorithm. The noise filtering algorithm specifically employs filtering algorithms including mean filtering, median filtering, and Gaussian filtering to remove random noise from the original data. The outlier detection algorithm includes a statistical outlier detection algorithm and a machine learning-based outlier detection algorithm, which identifies and excludes outliers in the original data. The missing value imputation algorithm uses interpolation and regression methods to imput missing values ​​when there are missing values ​​in the original data. The data formatting submodule employs feature standardization algorithm, feature encoding algorithm, and data dimensionality reduction algorithm. The feature standardization algorithm specifically involves scaling and shifting the data through maximum and minimum value scaling and mean-variance standardization, so that different features have the same scale. The feature encoding algorithm is based on one-hot encoding and label encoding, which converts non-numerical features into numerical features; The data dimensionality reduction algorithm includes principal component analysis and linear discriminant analysis. The data dimensionality reduction algorithm is used to reduce the number of features, reduce computational complexity and model training time.

4. The machine vision-based obstacle avoidance system for autonomous driving according to claim 1, characterized in that: The data fusion module includes a fusion algorithm submodule and an environmental perception submodule; The fusion algorithm submodule includes Kalman filter, particle filter, Gaussian mixture model, and conditional random field; The Kalman filter combines the processed data with the prediction model through recursive state estimation and observation updates to obtain the optimal state estimate and perform the fusion task of linear system and Gaussian noise. The particle filter employs random particle resampling and particle weight updates to approximately estimate the potential target state, thus completing the fusion task of nonlinear and non-Gaussian distributions. The Gaussian mixture model uses a set of Gaussian distributions to model the distribution of multi-sensor data and performs data fusion tasks through maximum a posteriori probability estimation. The conditional random field defines a probabilistic graphical model to model the relationships and constraints of sensor data, thereby achieving data fusion and joint estimation. The environmental perception submodule employs object detection and recognition algorithms, map building algorithms, and motion estimation algorithms. The object detection and recognition algorithm uses the fused data to detect and recognize targets, including vehicles, pedestrians, and road signs; The map building algorithm compares and matches the fused data with existing maps to achieve accurate positioning of vehicle location and route; The motion estimation algorithm infers the motion information of objects in the environment, including speed, acceleration, and trajectory, by analyzing changes in sensor data.

5. The machine vision-based obstacle avoidance system for autonomous driving according to claim 1, characterized in that: The image processing module includes a real-time semantic segmentation submodule and an object detection submodule; The real-time semantic segmentation submodule includes an encoder, a decoder, and a loss function; The encoder uses convolution and pooling operations to gradually reduce the dimensionality of image features and extract feature information. The decoder uses deconvolution and upsampling operations to restore the feature map to the original image size, and combines the feature information to perform information fusion to generate semantic segmentation results; The loss function compares the semantic segmentation results with the real labels and optimizes the network parameters through backpropagation.

6. The machine vision-based obstacle avoidance system for autonomous driving according to claim 5, characterized in that: The target detection submodule includes candidate box generation, feature extraction, target classification, and localization. The candidate box generation specifically involves using a region proposal method to generate candidate target boxes. The feature extraction specifically involves using convolutional neural networks, including ResNet and VGGNet, to extract feature representations within the candidate target boxes. Specifically, the target classification and localization involves using a classifier to determine the target category based on the feature representation and performing location regression to obtain accurate target localization.

7. The machine vision-based obstacle avoidance system for autonomous driving according to claim 1, characterized in that: The communication module includes an information sending submodule and an information receiving submodule; The information transmission submodule is equipped with an in-vehicle communication protocol, which specifically includes in-vehicle communication networks such as CAN, FlexRay, and Ethernet. Both the information sending submodule and the information receiving submodule adopt V2V communication technology and V2I communication technology. The information transmission submodule is based on sensor data fusion, which integrates information from vehicle sensors and other vehicles and infrastructure to provide accurate and comprehensive environmental perception.

8. A machine vision-based obstacle avoidance method for autonomous driving, characterized in that, Includes the following steps: The data acquisition module is used to collect data around the vehicle in real time as raw data. The raw data is transmitted to the data preprocessing module for preprocessing, including data cleaning, formatting, and dimensionality reduction, to generate processed data. The data fusion module receives the processed data and integrates the data collected by different sensors through a fusion algorithm to obtain fused data; The image processing module performs deep learning processing, including real-time semantic segmentation and target detection, on the image data in the processed data to identify key targets in the image; The decision-making module integrates the fused data and key objectives to make obstacle avoidance decisions. It optimizes the decision-making process through reinforcement learning and model predictive control methods, uses eye tracking and sentiment analysis to obtain additional data, optimizes driving strategies, and generates decision instructions. The decision-making instructions are transmitted to the vehicle's control system through the communication module to achieve obstacle avoidance. The communication module also exchanges information with other vehicles and intelligent transportation infrastructure, provides or obtains effective information, and transmits feedback information to the decision-making module. The decision-making module includes a decision algorithm submodule and a strategy optimization submodule; The decision algorithm submodule includes functions for state definition, action space definition, reward function definition, and reinforcement learning algorithm training. The state definition specifically refers to transforming the fused data and processing results into a state representation required for driving decisions, including vehicle position, speed, and surrounding environment information; The definition of the action space specifically refers to defining feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, to construct the action space; The reward function is specifically defined as a function designed to guide the optimization of driving decisions, including penalizing collisions and encouraging smooth driving. The reinforcement learning algorithm training specifically involves using a reinforcement learning algorithm to train and optimize the driving decision model, enabling the driving decision model to select the optimal driving action under different conditions. The functional items of the strategy optimization submodule include state perception, sentiment analysis, and driving strategy optimization. The state perception specifically refers to obtaining the driver's gaze points and visual behavior through eye-tracking technology to understand the driver's attention to the driving task and emotional state. The sentiment analysis specifically involves analyzing the driver's emotional changes, including anxiety and fatigue, using sentiment analysis technology. The driving strategy optimization specifically involves using the MPC optimization algorithm to optimize the driving strategy based on the results of state perception and sentiment analysis, thereby adjusting the vehicle's acceleration and steering parameters.

Images