Intelligent driving assistance system based on machine vision and multi-sensor fusion

Abstract

According to the embodiment of the invention, the sensors are reasonably arranged at the first position and the second position of the automobile, various data such as distance, speed and road condition are acquired, and a comprehensive environment information source is provided for the system. Data collected by the sensors are transmitted to the central processing system through the data sending module, deep analysis is carried out through the neural network module, target detection is carried out through the target detection module, and deep analysis is carried out on the data processed by the central processing system through a neural network algorithm so as to identify complex road conditions or predict potential risks. And a set of complete data processing flow is formed. The image preprocessing module preprocesses an image acquired by the camera and then participates in overall data processing, so that the image data quality is ensured. The control system controls the driving state according to data of the central processing system and the target detection module, the power system responds to instructions of the control system to provide power, intelligent cooperation is achieved, and safe and efficient driving of the vehicle is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to an intelligent driving assistance system based on machine vision and multi-sensor fusion. Background Art

[0002] With the development of intelligent transportation, driverless technology has become a research hotspot. Vehicle behavior recognition and prediction is one of the key technologies, and machine vision technology has broad application prospects.

[0003] The existing technology includes image acquisition equipment (such as cameras), image processing algorithm modules and behavior analysis modules. The image acquisition equipment obtains vehicle images, the image processing algorithm module pre-processes the images and uses the target detection algorithm to identify the vehicle target, and the behavior analysis module performs simple behavior analysis based on this, such as determining whether the vehicle is driving in the lane.

[0004] The image acquisition device is directly connected to the image processing algorithm module to transmit image data, and the processing results are then passed to the behavior analysis module. During operation, the image acquisition device continuously acquires vehicle images and transmits them to the image processing algorithm module in real time, and processes the images according to the process, including preprocessing and target detection. The behavior analysis module analyzes the vehicle behavior after receiving the information. However, the existing technology lacks accuracy and reliability in complex environments, and its real-time performance is difficult to meet the needs. The behavior prediction capability is limited, and there is a lack of multi-sensor data fusion. The algorithm and model are not optimized enough. For example, the accuracy decreases in dim tunnels or rainy and snowy weather, the real-time performance is not enough when the vehicle is driving fast, and it is impossible to consider multiple factors affecting behavior prediction. It only relies on machine vision data, and the algorithm and model need to improve the recognition speed and accuracy.

[0005] At present, when faced with complex environments and the diversity of vehicles and roads, the recognition effect is poor, and special behaviors or complex scenes are difficult to judge accurately; the real-time performance is difficult to meet the needs, and when the vehicle is driving fast and needs to process a large amount of image data, it cannot be processed in time, resulting in delays in judgment and analysis; the vehicle behavior prediction ability is limited, and it is unable to fully consider factors such as driver intentions, changes in traffic rules and the complexity of road conditions, and the prediction results are inaccurate or incomplete; there is a lack of multi-sensor data fusion, and it only relies on machine vision data. There is a lack of effective integration and utilization of lidar, GPS and other data, which limits the system's perception and analysis capabilities; the algorithm and model are insufficiently optimized, and the recognition speed and accuracy need to be improved, which cannot meet the needs of efficient and accurate vehicle behavior recognition and prediction.

[0006] Secondly, the existing algorithms have poor adaptability and robustness in complex environments, and the extraction of vehicle behavior features is not comprehensive and accurate, which affects the accuracy of recognition and prediction; there are computing resource limitations, and processing large amounts of image data requires powerful computing resources. Existing technical hardware equipment may be insufficient to meet real-time processing requirements; there is a lack of models that comprehensively consider multiple factors. In terms of vehicle behavior prediction, there is a lack of effective models that comprehensively consider multiple factors, resulting in limited prediction capabilities; data fusion technology is imperfect, and the research and application of multi-sensor data fusion technology is not in-depth enough, and there is a lack of effective methods and algorithms to integrate and analyze data; there is insufficient investment in the research and development of algorithms and models. Due to limited research and development resources and unclear directions, the optimization of algorithms and models is not in-depth and comprehensive enough. Summary of the invention

[0007] In view of this, the purpose of the embodiment of the present invention is to provide an intelligent driving assistance system based on machine vision and multi-sensor fusion, which improves the adaptability and robustness of the algorithm to complex environments by learning and analyzing a large amount of different scene data, so as to more accurately identify vehicle behavior. The accuracy and reliability of vehicle behavior recognition are significantly improved, and vehicle behavior can be better identified in both special environments and complex scenes, providing more accurate information support for the intelligent driving system.

[0008] On the one hand, an embodiment of the present invention provides an intelligent driving assistance system based on machine vision and multi-sensor fusion, characterized in that it includes: A sensor, installed at a first position and a second position of the vehicle, for collecting information about the external environment of the vehicle; A data transmission module, connected to the sensor, for transmitting the external environment information of the vehicle collected by the sensor to a central processing system; A central processing system is connected to the data sending module and is used to receive the external environment information of the automobile and centrally process and analyze it; wherein the central processing system is connected to an external server through a wireless communication module to realize data uploading, remote monitoring or system updating.

[0009] Optionally, it also includes: The neural network module is connected to the central processing system and is used to use a neural network algorithm to perform in-depth analysis on the data processed by the central processing system to identify complex road conditions or predict potential dangers.

[0010] Optionally, it also includes: The target detection module is connected to the neural network module and is used to detect environmental information around the car based on data analyzed by the neural network module.

[0011] Optionally, it also includes: A camera, installed at the third position of the car, is used to collect image data in front of the car; The image preprocessing module is connected to the camera and is used to preprocess the collected image data.

[0012] Optionally, it also includes: The control system is connected to the central processing system and the target detection module, and is used to generate control instructions according to the processed data to adjust the driving state of the car.

[0013] Optionally, it also includes: The control system generates a safety brake signal in real time according to the position and motion state of the obstacle detected by the target detection module.

[0014] Optionally, it also includes: The power system is connected to the control system and is used to provide driving force according to the instructions of the control system to drive the vehicle.

[0015] Optionally, the power system further includes: Electric motor and energy management module to adjust power output to suit different driving conditions; Optionally, the target detection module further comprises the following steps: Load the pre-trained YOLOv8 model and initialize the processing pipeline for the input image; Get the detection results and extract the detection box coordinates and confidence; Among them, after filtering the detection boxes with confidence less than the threshold, the detection box of the vehicle and its confidence are output.

[0016] Optionally, the target detection module further comprises the following steps: Construct at least one data set and normalize it, based on a feed-forward neural network, with the input being the vehicle pixel coordinates and the output being the predicted distance; The neural network is trained by annotating data to optimize the mean square error of distance prediction, and the pixel coordinate center point of the detection box is input into the trained model to predict the actual distance.

[0017] The embodiment of the present invention reasonably arranges sensors at the first position and the second position of the car to collect various data such as distance, speed, and road conditions, thereby providing a comprehensive source of environmental information for the system. The data collected by the sensor is transmitted to the central processing system through the data sending module, and then undergoes deep analysis by the neural network module and target detection by the target detection module. The neural network algorithm is used to perform deep analysis on the data processed by the central processing system to identify complex road conditions or predict potential dangers. A complete set of data processing procedures is formed. The image preprocessing module preprocesses the images collected by the camera and then participates in the overall data processing to ensure the quality of the image data. The control system controls the driving state according to the data of the central processing system and the target detection module, and the power system responds to the control system command to provide power, realizing intelligent coordination and ensuring safe and efficient driving of the vehicle.

[0018] The present invention also has the following beneficial effects: The present invention designs a machine vision algorithm suitable for complex environments, taking into account factors such as lighting changes, weather conditions, road types, and the diversity of vehicle types. By learning and analyzing a large amount of data from different scenarios, the algorithm's adaptability and robustness to complex environments are improved, thereby more accurately identifying vehicle behavior. The accuracy and reliability of vehicle behavior recognition are significantly improved, and vehicle behavior can be better identified in both special environments and complex scenarios, providing more accurate information support for intelligent driving systems.

[0019] By using high-performance computing equipment and optimizing the algorithm structure and calculation process, unnecessary calculation steps are reduced and the execution efficiency of the algorithm is improved, so that the vehicle behavior can be judged and analyzed in a timely manner when the vehicle is driving fast and needs to process a large amount of image data. It meets the real-time requirements of vehicle behavior recognition and prediction, ensures that the system can respond quickly, and provides decision-making basis for the driver or the automatic driving system in a timely manner, thus improving traffic safety.

[0020] A behavior prediction model that comprehensively considers multiple factors such as driver intentions, traffic rules, and road conditions has been established. By analyzing and learning a large amount of actual data, the inherent relationship between these factors is explored to more accurately predict vehicle behavior. This greatly enhances the ability to predict vehicle behavior, making the prediction results more accurate and comprehensive, providing a more reliable basis for the decision-making of the intelligent driving system, and further improving traffic safety and intelligence.

[0021] By integrating data from multiple sensors such as LiDAR and GPS, an effective data fusion algorithm is designed to integrate and analyze information obtained by different sensors, making full use of the advantages of each sensor to improve the system's comprehensive perception and analysis capabilities of vehicle behavior. By achieving effective fusion of multi-sensor data, the system can obtain more comprehensive vehicle behavior information, improve the perception of complex environments and complex behaviors, and enhance the reliability and safety of the intelligent driving system.

[0022] We continuously optimize target detection, feature extraction, and classification algorithms, introduce more vehicle behavior features and environmental information, improve the accuracy and efficiency of algorithms and models, and conduct a large number of experiments and tests to continuously improve algorithms and models. This improves the recognition speed and accuracy of algorithms and models, meets the needs of efficient and accurate vehicle behavior recognition and prediction, and provides a guarantee for the efficient operation of intelligent driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 is a flow chart of a vehicle behavior prediction method based on machine vision and multi-sensor fusion provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of the overall process provided by an embodiment of the present invention; Figure 3 A schematic diagram of a target detection process provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0028] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0029] The block diagrams shown in the accompanying drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware charging modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0030] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0031] See also Figure 1 As shown, an embodiment of the present invention provides an intelligent driving assistance system based on machine vision and multi-sensor fusion, characterized in that it includes: A sensor, installed at a first position and a second position of the vehicle, for collecting information about the external environment of the vehicle; For example, the first position may be a sensor at the front of the car, and the second position may be a sensor at the rear of the car. The external environment information of the car may be the distance to the obstacle in front, the distance to the surrounding vehicles, the speed of the surrounding vehicles, the relative speed, the road type, the road surface condition, the slope information, the road friction coefficient, etc.

[0032] A data transmission module, connected to the sensor, for transmitting the external environment information of the vehicle collected by the sensor to a central processing system; A central processing system is connected to the data sending module and is used to receive the external environment information of the automobile and centrally process and analyze it; wherein the central processing system is connected to an external server through a wireless communication module to realize data uploading, remote monitoring or system updating.

[0033] Exemplarily, the central processing system is capable of simultaneously processing data from multiple sensors, such as radar, camera, and ultrasonic sensors.

[0034] Provides data fusion capabilities for environmental information to generate a complete perception map of the vehicle's surroundings. Wireless communication modules use technologies such as 5G, LTE or Wi-Fi to ensure high-speed, low-latency communication.

[0035] Specifically, it also includes: The neural network module is connected to the central processing system and is used to use a neural network algorithm to perform in-depth analysis on the data processed by the central processing system to identify complex road conditions or predict potential dangers.

[0036] For example, a BP neural network algorithm (Backpropagation Neural Network) may be used. The BP neural network algorithm is a type of artificial neural network used for supervised learning. The weights and biases are adjusted through backpropagation to minimize output errors.

[0037] Specifically, it also includes: The target detection module is connected to the neural network module and is used to detect environmental information around the car based on data analyzed by the neural network module.

[0038] Specifically, it also includes: A camera, installed at the third position of the car, is used to collect image data in front of the car; Exemplarily, the third location may be a sensor at the front of the vehicle.

[0039] The image preprocessing module is connected to the camera and is used to preprocess the collected image data.

[0040] Specifically, it also includes: The control system is connected to the central processing system and the target detection module, and is used to generate control instructions according to the processed data to adjust the driving state of the car.

[0041] Exemplarily, a PID controller is used to generate a control signal that can be used to adjust the behavior of a system (e.g., control the speed or position of a vehicle). The control signal is calculated based on the error between the set target distance ( desired_distance ) and the actual predicted distance ( actual_distance ). The calculation of the control signal takes into account three factors: proportional (Kp), integral (Ki), and differential (Kd).

[0042] For example, you can initialize components, detect data such as pixel coordinates, actual distance, etc., read images and perform target detection, and use a PID controller for speed control. The resulting control signal is an output for the control system, which can help achieve intelligent control of vehicles or other objects. By combining vehicle detection, distance prediction, and control algorithms, a more intelligent and automated system can be built.

[0043] See also Figure 2 The specific steps are as follows: S1, initialize components, initialize VehicleDetector (vehicle sensor); DistanceRegressor (distance regression factor); PIDController (PID controller); S2, build the model, use the Keras library to build a simple fully connected neural network, use the Adam optimizer to update the model's weights, and use the mean square error as the loss function to evaluate the model's prediction performance. Return the model; S3, train the model, randomly split the given data set into training set and test set for subsequent training and evaluation of the model; create a normalizer object to standardize the training set so that its mean is 0 and its standard deviation is 1. Use the statistics of the training set to standardize the test set to ensure that the standardization of the test set is consistent with that of the training set. Train the model on the training set for 100 iterations. Evaluate the trained model on the test set, calculate and return the test loss value to understand the performance of the model on unseen data. print(f"Test loss: {test loss}") returns the mean and standard deviation of the calculated training data; S4, data prediction, preprocesses (scaling) the input data and then uses the trained model to make predictions; S5, read the image, print("Unable to read the image, please check the path.") and return to the above operation to read again; if the image is not read, call the vehicle detection model to process the input image, traverse the detected vehicle information, so as to further process each detection result, convert the upper left corner coordinates of the detected vehicle into a two-dimensional NumPy array, predict the actual distance according to the pixel coordinates through the regression model, and define a target distance for subsequent control logic, such as using a PID controller to adjust the speed or position of the vehicle to maintain consistency with the target distance, and calculate the control output based on the error between the set value and the actual value to achieve stable control of the system. In this step, you can repeat the operation through print(f"Control signal: {control_signal}").

[0044] Specifically, it also includes: The control system generates a safety brake signal in real time according to the position and motion state of the obstacle detected by the target detection module.

[0045] For example, when a dangerous situation is detected, the system generates a braking signal according to a safety strategy (such as the shortest braking distance, vehicle dynamics constraints, etc.). The signal is transmitted to the actuator through the braking module to trigger emergency braking.

[0046] Specifically, it also includes: The power system is connected to the control system and is used to provide driving force according to the instructions of the control system to drive the vehicle.

[0047] Specifically, the power system further includes: Electric motor and energy management module to adjust power output to suit different driving conditions; Specifically, the target detection module further includes the following steps: Load the pre-trained YOLOv8 model and initialize the processing pipeline for the input image; Get the detection results and extract the detection box coordinates and confidence; Among them, after filtering the detection boxes with confidence less than the threshold, the detection box of the vehicle and its confidence are output.

[0048] For example, the YOLOv8 model is used to detect vehicles in the image and return the coordinates and confidence of the detection box. This step is to identify whether there are vehicles in the image and their locations.

[0049] It should be noted that the program for vehicle detection using the YOLO model first loads the pre-trained YOLO model, then opens the video file, and prompts the user how to exit the program if the video file cannot be opened; obtains the video frame rate, such as the actual length of the reference object, the pixel length of the reference object in the video, etc.; reads a frame from the video file, uses the YOLO model for target detection, and then traverses the detection results and obtains the detection box information, such as the detection box coordinates, confidence, category ID, etc. The category information is defined according to the YOLO model category.

[0050] See also Figure 3 As shown, the steps to detect cars using the YOLO model are as follows: S6, if a car model information failure is detected, the error is captured and processed, and if an error occurs when loading the model, the process returns to the previous step; S7, if the car model information is detected successfully, the YOLO model is trained, and the material is tried to be opened to determine whether it is opened successfully: if it is true, it will display that the video file cannot be opened and return to the previous step; if it is false, press the q key to exit the program, set the pixel length of the reference object to 100 pixels, read a frame of image from the video file, and determine whether there is an image displayed. If it is true, the YOLO model is used to detect the target of the current frame image, and if it is false, the resources related to the video file are released; S8, traverse all target results detected by the YOLO model, open a "VehicleDetection" window to extract the detection box information in the current detection result, traverse all detection boxes in the current detection result, if it is false, re-traverse all target results detected by the YOLO model, open a "VehicleDetection" window to extract the detection box information in the current detection result, and traverse all detection boxes in the current detection result; if it is true, extract the bounding box coordinates of the target from the detection box, and extract the category ID corresponding to the current detection box; S9, determine whether it is a car category ID, if it is false, repeat step S8, if it is true, draw a rectangular box on the current frame image, determine the center position of the currently detected target and detect whether there is a target center position, if it is false, repeat step S9, if it is true, calculate the pixel distance between the current target position and the previous target position, update the position of the previous frame, and display the braking force information on the current frame image so that the user can see the currently calculated braking force value in real time.

[0051] For example, the braking force is calculated using a fuzzy control system. The program detects the car from the video, calculates its speed and displays it on the video, and calculates the braking force based on the speed error.

[0052] For example, the category ID 2 represents cars. Cars with confidence greater than or equal to the threshold are displayed, geometric boxes such as rectangles are drawn, the center position of the front car, speed, and moving pixel distance are calculated and converted into actual distances and displayed; the position of the previous frame is updated, and the speed error is extracted, including the use of a fuzzy control system to calculate the braking force, and the braking force is displayed on the video frame; the detection results are displayed to detect whether the user presses the q key, etc., and finally the video file is released and all OpenCV windows are closed.

[0053] It can be understood that the steps of calculating the braking force by the fuzzy control system include: Define the fuzzy sets of input and output, the fuzzy sets of speed error, the fuzzy sets of braking force, fuzzy rules, etc.; create a fuzzy control system after accommodating the input speed error value to calculate the fuzzy control output. For example, the speed error is a fixed value, and the actual error should be calculated based on the test results; the speed error can also be randomly generated.

[0054] Specifically, the target detection module further includes the following steps: Construct at least one data set and normalize it, based on a feed-forward neural network, with the input being the vehicle pixel coordinates and the output being the predicted distance; The neural network is trained by annotating data to optimize the mean square error of distance prediction, and the pixel coordinate center point of the detection box is input into the trained model to predict the actual distance.

[0055] For example, a simple feedforward neural network (BP neural network) is used to predict the actual distance between the vehicle and the camera. The input is the pixel coordinates of the vehicle in the image, and the output is the predicted actual distance.

[0056] The embodiment of the present invention reasonably arranges sensors at the first position and the second position of the car to collect various data such as distance, speed, and road conditions, thereby providing a comprehensive source of environmental information for the system. The data collected by the sensor is transmitted to the central processing system through the data sending module, and then undergoes deep analysis by the neural network module and target detection by the target detection module. The neural network algorithm is used to perform deep analysis on the data processed by the central processing system to identify complex road conditions or predict potential dangers. A complete set of data processing procedures is formed. The image preprocessing module preprocesses the images collected by the camera and then participates in the overall data processing to ensure the quality of the image data. The control system controls the driving state according to the data of the central processing system and the target detection module, and the power system responds to the control system command to provide power, realizing intelligent coordination and ensuring safe and efficient driving of the vehicle.

[0057] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functions / modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0060] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0061] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0062] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0063] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0064] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An intelligent driving assistance system based on machine vision and multi-sensor fusion, characterized in that: include: A sensor, installed at a first position and a second position of the vehicle, for collecting information about the external environment of the vehicle; A data transmission module, connected to the sensor, for transmitting the external environment information of the vehicle collected by the sensor to a central processing system; A central processing system is connected to the data sending module and is used to receive the external environment information of the automobile and centrally process and analyze it; wherein the central processing system is connected to an external server through a wireless communication module to realize data uploading, remote monitoring or system updating.

2. The system according to claim 1, characterized in that Also includes: The neural network module is connected to the central processing system and is used to use a neural network algorithm to perform in-depth analysis on the data processed by the central processing system to identify complex road conditions or predict potential dangers.

3. The system according to claim 1, characterized in that Also includes: The target detection module is connected to the neural network module and is used to detect environmental information around the car based on data analyzed by the neural network module.

4. The system according to any one of claims 1 to 3, characterized in that Also includes: A camera, installed at the third position of the car, is used to collect image data in front of the car; The image preprocessing module is connected to the camera and is used to preprocess the collected image data.

5. The system according to claim 3, characterized in that Also includes: The control system is connected to the central processing system and the target detection module, and is used to generate control instructions according to the processed data to adjust the driving state of the car.

6. The system according to claim 5, characterized in that Also includes: The control system generates a safety brake signal in real time according to the position and motion state of the obstacle detected by the target detection module.

7. The system according to claim 6, characterized in that Also includes: The power system is connected to the control system and is used to provide driving force according to the instructions of the control system to drive the vehicle.

8. The system according to claim 7, characterized in that The power system further comprises: Electric motor and energy management module to adjust power output to suit different driving conditions; The system according to claim 7, characterized in that the target detection module further comprises the following steps: Load the pre-trained YOLOv8 model and initialize the processing pipeline for the input image; Get the detection results and extract the detection box coordinates and confidence; Among them, after filtering the detection boxes with confidence less than the threshold, the detection box of the vehicle and its confidence are output.

9. The system according to claim 3, characterized in that The target detection module also includes the following steps: Construct at least one data set and normalize it, based on a feed-forward neural network, with the input being the vehicle pixel coordinates and the output being the predicted distance; The neural network is trained by annotating data to optimize the mean square error of distance prediction, and the pixel coordinate center point of the detection box is input into the trained model to predict the actual distance.

Images