Intelligent automobile control system with license plate recognition and path tracking functions

Abstract

The invention discloses an intelligent automobile control system with license plate recognition and path tracking functions, particularly relates to the technical field of automobile control, and comprises a license plate recognition module, a path tracking module and a vehicle control module. The system assists a traffic management department to effectively supervise and optimize related operation processes no matter which complex condition the license plate is in, ensures that the vehicle runs according to an optimal path by means of multi-sensor fusion positioning and an intelligent algorithm, greatly improves the driving safety, realizes accurate operation control, combines a real-time feedback adjustment mechanism, and improves the driving safety of the vehicle. The intelligent degree of the vehicle is improved, driving is easier, and the riding experience of passengers is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile control, and more specifically to an intelligent automobile control system with license plate recognition and path tracking. Background Art

[0002] At present, with the rapid growth of car ownership in modern society, road traffic has become increasingly complex, bringing higher requirements to traffic management and drivers. For example, in terms of traffic management, manual supervision is inefficient and difficult to meet real-time monitoring needs. It is difficult for drivers to plan the optimal route under complex road conditions, which can easily lead to travel delays and increased accident risks.

[0003] With the development of science and technology, intelligent car control systems are becoming more and more perfect. However, the existing intelligent car control systems with license plate recognition and path tracking still have the following defects: 1. When recognizing license plates, once complex situations such as license plate damage, deformation, poor lighting, etc. occur, the recognition accuracy will drop significantly, and there is a lack of deep integration with other vehicle control systems, such as path tracking and vehicle control, limiting the application value; 2. When the vehicle positioning is performed solely by the GPS system, the GPS signal is easily blocked in urban areas with tall buildings and tunnels, resulting in a serious decrease in positioning accuracy, which in turn affects the accuracy of path tracking. In addition, the existing path planning algorithm is not perfect in comprehensively considering various actual traffic factors, such as real-time traffic congestion, road construction information, and speed limit requirements of different sections of the road. The planned path is often not optimal, which may cause the driver to encounter unnecessary congestion or violate traffic rules during driving; 3. In the automatic emergency braking system of traditional automobile control systems, due to the lack of precise braking control, excessive braking may sometimes occur, causing the vehicle to stop suddenly, causing discomfort or even danger to the passengers in the car. In the power steering system, there are also situations where the power assistance effect is poor or the steering response is not sensitive enough, affecting the driving experience. In addition, there is a lack of a close coordination mechanism between the existing vehicle control system and functions such as license plate recognition and path tracking. Each system operates relatively independently, and it is impossible to give full play to the overall advantages and realize intelligent linkage control.

[0004] In view of the above situation, the present invention provides an intelligent automobile control system with license plate recognition and path tracking. Summary of the invention

[0005] In order to overcome the above defects of the prior art, the present invention provides an intelligent automobile control system with license plate recognition and path tracking to solve the problems raised in the above background technology.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an intelligent automobile control system with license plate recognition and path tracking, comprising a license plate recognition module, a path tracking module and a vehicle control module; Among them, the license plate recognition module is used to identify the license plate information in front of the vehicle; The path tracking module plans and tracks the vehicle's driving path based on the recognized license plate information or other navigation information; The vehicle control module controls the steering, acceleration and braking of the vehicle according to the instructions of the path tracking module.

[0007] Preferably, the license plate recognition module includes an image acquisition unit, which uses a high-resolution and high-frame-rate camera, which has the functions of automatic focus, automatic aperture adjustment and exposure, and the camera is installed in a position to ensure that the best viewing angle image of the license plate in front of the vehicle and in the surrounding environment can be obtained; The camera captures images at a frame rate of 30fps and follows the following process: After the system is started, the camera is continuously turned on and captures an image sequence in front of the vehicle at a frame rate of 30fps. Each time the image data is captured, it is transmitted to the license plate recognition module through the vehicle network for subsequent processing.

[0008] Preferably, the license plate recognition module further includes an image processing and recognition unit, which is used to perform the following processing on the collected license plate image: S1, image preprocessing; S1.1. Use the weighted average method to convert the collected color license plate image into a grayscale image. The specific formula is: G=0.299R+0.587G+0.114B Where G represents the grayscale value of the pixel obtained after calculation, and the value range is usually 0-255, 0 represents black, and 255 represents white; R represents the color value of the red channel of the pixel in the color image; G represents the color value of the green channel of the pixel in the color image; B represents the color value of the blue channel of the pixel in the color image; S2.2, use Gaussian filtering to filter and denoise the grayscale image, by performing weighted average operation on each pixel in the image and its neighboring pixels, with the weight determined by the Gaussian function, to remove Gaussian noise; The formula for the Gaussian function is: e - in, Represents the coordinate offset relative to the current pixel on the image plane, such as, for the current pixel ( ), the coordinates of a point in the neighborhood are ( ), is the standard deviation of the Gaussian function; The specific steps of weighted average operation are: S2.21, determine the neighborhood range, select a square neighborhood of the image, such as 3×3, 5×5 or a larger area of ​​7×7 centered on the current pixel; S2.22. Calculate the weight of each pixel in the neighborhood. According to the Gaussian function, for each pixel in the neighborhood, the offset of its coordinates relative to the current pixel is substituted into the Gaussian function to calculate the weight value corresponding to the pixel. For example, for the current pixel point ( ) is the center of the 3×3 neighborhood, and the pixel coordinates in the neighborhood are , , , relative these coordinates to ( )’s offset is substituted into the Gaussian function to calculate their respective weights; S2.23, weighted average calculation, multiply the gray value of each pixel in the neighborhood by its corresponding weight, then add all the products, and finally divide by the sum of all weights. The result is the gray value of the current pixel after Gaussian filtering; For example, in a 3×3 neighborhood, the current pixel is ( ), the grayscale values ​​of its neighboring pixels are G-1, -1, G-1, 0, G-1, 1, and the corresponding weights are w-1, -1, w-1, 0, w-1, 1. Then after Gaussian filtering, the pixel ( ) gray value G filtered ( ) is calculated as follows: G filtered ( )= Median filtering is used to sort the pixel values ​​in the neighborhood of the pixel point, and the middle value is taken as the filtered value of the pixel point to remove salt and pepper noise. Specifically: S2.24, in the same manner as S2.21, select a square neighborhood of the image, such as 3×3, 5×5 centered on the current pixel; S2.25. Obtain and sort pixel values ​​in the neighborhood. For the current pixel, extract the grayscale values ​​of all pixels in its neighborhood to form a data sequence; For example, in a 3×3 neighborhood, the current pixel is ( ), collect all the grayscale values ​​of the 9 pixels in its neighborhood, g-1, -1, g-1,0, g-1,1, and then sort this data sequence from small to large or from large to small; S2.26. Take the middle value as the filtered value. After the sorting is completed, the middle value is determined according to whether the number of pixels in the neighborhood is an odd number or an even number. For example, if the neighborhood is an odd number of pixels such as 3×3 or 5×5, then the middle value is the value in the middle of the sorted data sequence. If the neighborhood is an even number of pixels such as 4×4, then the average of the two middle values ​​of the sorted data sequence is usually taken as the middle value, and this middle value is used as the gray value of the current pixel after median filtering. For example, for the current pixel ( ) is the 3×3 neighborhood centered on the g. The collected neighborhood pixel values ​​are sorted into g. sorted [1],g sorted [2]......,g sorted [9], then after median filtering, the pixel point ( ) gray value M filtered ( ) is: M filtered ( )=g sorted [5]; S2, license plate positioning; S2.1 uses the Sobel operator to perform edge detection on the preprocessed image. The Sobel operator determines the edge points by calculating the grayscale change rate of the image in the horizontal and vertical directions; S2.2, performing morphological operations on the detected edge image, including corrosion and expansion, wherein the corrosion operation removes small noise and isolated points in the edge image, and the expansion operation connects adjacent edge segments to make the edge of the license plate area more complete and clear; S3.3, based on the geometric features of the license plate, such as a rectangular shape and an aspect ratio between 3:1 and 5:1, search for possible license plate regions in the edge image after morphological processing, and screen out regions that meet the geometric features of the license plate as the initially located license plate region by traversing the connected regions in the edge image; S3, character segmentation and recognition; S3.1. For the located license plate area, its grayscale image is projected in the horizontal and vertical directions respectively. The horizontal projection can obtain the distribution of the characters in the vertical direction, and the vertical projection can obtain the distribution of the characters in the horizontal direction. The boundaries of the characters are determined according to the peaks and troughs of the projection curve. The area between two adjacent troughs is the area where a character is located, thereby segmenting the characters on the license plate; S3.2. Construct a character recognition model based on convolutional neural network. The model structure includes input layer, multiple convolutional layers, pooling layer, fully connected layer and output layer. Each segmented character image is normalized to make its size meet the input requirements of CNN model. The recognition accuracy is calculated by the formula Accuracy= Calculated; For example, the character images are uniformly adjusted to a size of 32×32 pixels, and the normalized character images are input into the CNN model for training and recognition. In the training stage, a large number of license plate character samples are used to train the model, including normal samples and samples with a certain degree of damage and deformation. The weight parameters of the model are continuously adjusted through the back propagation algorithm so that the model can accurately recognize different license plate characters. In the recognition stage, the character image to be recognized is input into the trained model, and the output layer of the model will give the recognition result of each character.

[0009] Preferably, the path tracking module includes a multi-sensor fusion positioning unit, which integrates multiple sensors of a global positioning system, an inertial measurement unit, a wheel speed sensor, and a vehicle-mounted camera, and realizes high-precision positioning of the vehicle in the following ways: S1, GPS data acquisition and processing, continuously receiving GPS signals from satellites through the vehicle-mounted GPS receiver to obtain the vehicle's latitude and longitude, altitude and timestamp information; Perform coordinate conversion on the received GPS data, converting it from the WGS84 coordinate system to a coordinate system suitable for local maps, such as the Beijing 50 coordinate system; Use differential GPS technology or correct GPS data to improve GPS positioning accuracy. The specific correction formula can be determined according to the correction method used; For example, if a simple linear correction is used, the corrected coordinates can be set to X' = X + ΔX, Y' = Y + ΔY, where X and Y are the original coordinates and ΔX and ΔY are the correction values; S2, IMU data acquisition and processing, the on-board IMU continuously collects the acceleration and angular velocity information of the vehicle, the acceleration includes linear acceleration and angular acceleration; The acceleration and angular velocity data collected by the IMU are fused using the quaternion method to accurately estimate the vehicle's attitude, such as the vehicle's roll angle, pitch angle, heading angle, etc. The quaternion value is continuously updated to reflect the vehicle's attitude changes in real time. The specific quaternion update formula can be determined according to the quaternion operation rules; For example, let the quaternion be q = [w, x, y, z], and the update formula at a certain time t is q (t) = q (t- 1) + Δq, where Δq is the quaternion increment calculated based on the acceleration and angular velocity data; S3, wheel speed sensor data acquisition and processing, the wheel speed sensor is installed on the wheel of the vehicle, collects the wheel speed data in real time, and calculates the vehicle's driving speed, driving distance and other driving status information based on the wheel speed and the vehicle's tire size parameters. The specific calculation formula is: Speed ​​= wheel speed × tire circumference; The driving distance is calculated by integrating the wheel speed. If the wheel speed is v (t), the driving distance S = ∫v(t)dt); S4, visual sensor data acquisition and processing, the on-board camera collects road environment images around the vehicle at a frame rate of 30fps, pre-processes the collected road environment images, uses Huffman transform to detect lane lines and deep learning-based traffic sign recognition to identify lane lines and road features of traffic signs on the road, and uses the detected lane line information as a reference for vehicle positioning and path tracking; S5. Multi-sensor data fusion, constructing a Kalman filter, taking information from GPS, IMU, wheel speed sensor and visual sensor as input. The Kalman filter is divided into a prediction stage and an update stage. In the prediction stage, the vehicle's state at the current moment, such as position, speed and posture, is predicted based on the vehicle's dynamic model and the state estimate at the previous moment. In the update stage, the data actually collected by each sensor at the current moment is compared with the result of the prediction stage. The prediction result is corrected through Kalman gain calculation to obtain the optimal state estimate of the vehicle at the current moment, thereby achieving high-precision positioning. The specific calculation steps and formulas of the Kalman filter can be determined according to the standard Kalman filter algorithm and the specific situation of the vehicle; For example, in the prediction phase: hatx k∣k−1 =Ax^ k−1∣k−1 +Bu k−1 P k∣k−1 =AP k−1∣k−1 A T +Q; Update phase: K k =P k∣k−1 H T (H Pk∣k−1 H T +R) −1 ;hatx k∣k =x^ k∣k−1 +K k (zk−Hx^ k∣k−1 );Pk∣k =(1−K k H)P k∣k−1 where x̂ represents the state estimate, P is the estimation error covariance, A and B are system matrices, H is the observation matrix, Q and R are the covariance matrices of process noise and observation noise respectively, and K k is the Kalman gain and zk is the observation value.

[0010] Preferably, the path tracking module further includes an intelligent path planning and tracking unit, and the intelligent path planning and tracking unit implements the path planning and tracking functions in the following manner: S1. Route planning: Obtain real-time traffic information through the Internet of Vehicles, including road congestion, construction sections, and road speed limits. Combined with road features identified by visual sensors, such as lane lines and traffic signs, the system can fully understand the road environment around the vehicle. Using the A algorithm for path planning, first build a search space that includes the vehicle's current location, destination, and all possible road nodes, and set an evaluation function for each road node. The evaluation function consists of two parts: one is the actual path cost from the starting node to the node, such as the driving distance, and the other is the estimated cost from the node to the destination, such as the estimated time to the destination based on real-time traffic information. Starting from the starting node, search for nodes in ascending order according to the value of the evaluation function until the destination node is found, thereby planning an optimal path; S2. Path tracking: construct a dynamic model of the vehicle, including the vehicle's mass, moment of inertia and tire parameters. The vehicle's position, speed and heading angle status information are used as input variables, and the vehicle's work, acceleration and deceleration control instructions are used as output variables. According to the vehicle's current state and the planned path, the MPC algorithm based on model predictive control is used to predict the vehicle's driving trajectory in the future, such as the next 5 seconds. The MPC algorithm sets an optimization objective function at each prediction moment, such as minimizing the deviation between the vehicle's actual driving trajectory and the planned path, while considering the vehicle's control constraints, such as the maximum steering angle, maximum acceleration and maximum deceleration. By solving the optimization objective function, the optimal control instructions at each prediction moment, namely steering, acceleration, deceleration and other instructions, are obtained. These instructions are sent to the vehicle control module to make the vehicle drive according to the planned path. At the same time, during the vehicle driving process, according to the real-time situation, such as the braking of the vehicle in front and the pedestrians crossing the road, the objective function and the predicted trajectory are continuously adjusted to ensure the safety of the vehicle. The specific calculation steps and formulas of the MPC algorithm can be determined according to the standard MPC algorithm and the specific situation of the vehicle; For example, let the vehicle's state vector be x, the control vector be u, the prediction domain be N, and the optimization objective function be J = ∑_{k = 1}^{N}(x_{k}^{} - x_{k})^{2} + ∑_{k = 1}^{N}(u_{k}^{} - u_{k})^{2}, where x_{k}^{} is the state vector under the planned path, x_{k} is the actual state vector, u_{k}^{} is the control vector under the planned path, and u_{k} is the actual control vector.

[0011] Preferably, the vehicle control module includes an actuator control unit, and the actuator control unit realizes precise control of the vehicle actuator in the following manner: S1, electric power steering or wire-controlled steering control. When the path tracking module issues a steering command, if an electric power steering system is used, the EPS electronic control unit ECU first calculates the required steering assist according to the command, and adjusts the output power of the motor through the PID control algorithm according to the vehicle's driving speed and steering angle parameters to achieve precise steering assist control; The basic formula of PID control algorithm is: u (t) = K_{p} e (t) + K_{i}\int_{0}^{t} e(s) ds + K_{d}\frac {de (t)}{dt}, where u (t) is the control output, K_{p}, K_{i}, K_{d} are the proportional, integral, and differential coefficients respectively, and e (t) is the error signal, that is, the difference between the target steering angle and the actual steering angle; If a steer-by-wire system is used, upon receiving the steering command from the path tracking module, the SBW ECU directly converts the command into an electronic signal and sends it to the steering actuator, such as the electric steering gear. Through precise signal conversion and control algorithms, it ensures that the steering actuator rotates at the required steering angle and speed, thus achieving full electronic control. The relationship between its control input and output can be expressed as: u=f control (e) Where u is the control input, e is the error, and f control is the control function, which is the PID control algorithm.

[0012] S2. Electronically controlled braking and acceleration system control. When the vehicle needs to brake, the ECU of the electronically controlled braking system will calculate the required braking pressure according to the set braking strategy based on the path tracking module and the vehicle's driving conditions, such as the distance to the vehicle in front and the vehicle speed. By adjusting the solenoid valve actuator in the braking system, the application of braking pressure is accurately controlled to achieve smooth braking operation and avoid sudden braking that may cause discomfort to passengers or affect driving safety.

[0013] Preferably, the vehicle control module further includes a real-time state feedback and adjustment unit, which implements real-time feedback and adjustment of the state of the vehicle actuator in the following manner: S1, state sensor data acquisition, the steering angle sensor is installed on the steering wheel or steering mechanism of the vehicle, collects the steering wheel rotation angle data in real time, and transmits the collected steering angle data to the vehicle control module and the path tracking module through the vehicle network; The brake pressure sensor is installed in the brake system to collect the pressure change data of the brake system in real time, and transmit the collected pressure change data to the vehicle control module and the path tracking module through the vehicle network; The throttle opening sensor is installed near the accelerator pedal to collect throttle opening data in real time, and transmits the collected throttle opening data to the vehicle control module and the path tracking module through the vehicle network; S2, Deviation analysis and adjustment. After the vehicle control module and the path tracking module receive the actual steering angle data transmitted by the steering angle sensor, they compare it with the steering angle required by the path tracking module and calculate the steering deviation. If the steering deviation exceeds the set threshold, such as ±5, the vehicle control module will recalculate the steering command through the PID control algorithm according to the deviation, adjust the steering assist or directly adjust the action of the steering actuator; After receiving the pressure change data transmitted by the brake pressure sensor, it is compared with the ideal brake pressure calculated according to the braking strategy to calculate the brake deviation. If the brake deviation exceeds the set threshold, such as ±10%, the electronically controlled brake system will adjust the operation of the solenoid valve actuator according to the deviation to ensure that the brake pressure meets the requirements and achieve smooth braking; After receiving the actual throttle opening data transmitted by the throttle opening sensor, and the acceleration deviation calculated according to the acceleration command, if the acceleration deviation exceeds the set threshold, such as ±10%, the electronic throttle control system will adjust the operation of the electronic control mechanism of the accelerator pedal according to the deviation; The adjustment process can be expressed as: = - = + = + + in, is the error at the current moment, and are the actual output and expected output at the current moment, respectively. is the control input at the next moment, and They are proportional and integral gains respectively.

[0014] Technical effects and advantages of the present invention: 1. The system of the present invention can accurately identify license plate information, even if the license plate is damaged, deformed or under different lighting conditions, it can also achieve high-precision recognition, which helps traffic management departments to monitor vehicles in real time and discover violations in a timely manner. It can also optimize the management process of parking lots, highway tolls and other scenes, and improve traffic operation efficiency; 2. The system of the present invention integrates functions such as license plate recognition, path tracking and vehicle control. The vehicle can automatically complete operations such as steering, acceleration and deceleration based on real-time information, without the need for frequent manual intervention by the driver, thereby improving the intelligence of the vehicle and reducing fatigue during long-distance driving. In addition, the vehicle control module can accurately control vehicle operations to avoid discomfort to passengers caused by inaccurate driver operations, and through the real-time feedback adjustment mechanism, deviations can be corrected in a timely manner to optimize the driving experience. 3. The present invention uses multi-sensor fusion positioning and intelligent path planning and tracking algorithms to ensure that the vehicle travels along the optimal path, avoids deviation from the safe route due to driver error or emergencies, reduces the probability of traffic accidents, and can adjust strategies in real time in emergency situations to ensure driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall system diagram of the present invention.

[0016] Figure 2 This is a system diagram of the path tracking module of the present invention.

[0017] Figure 3 This is a system diagram of the vehicle control module of the present invention.

[0018] Figure 4 This is a flowchart of the implementation of the image processing and recognition unit of the present invention.

[0019] Figure 5 This is a flow chart of the implementation of the path tracking module of the present invention.

[0020] Figure 6 This is a flow chart for implementing the intelligent path planning and tracking unit of the present invention.

[0021] Figure 7 This is a flow chart for implementing the actuator control unit of the present invention.

[0022] Figure 8 It is a sight line flow chart of the real-time status feedback and adjustment unit of the present invention.

[0023] The accompanying drawings are marked as follows: 1. License plate recognition module; 2. Path tracking module; 3. Vehicle control module; 4. Image acquisition unit; 5. Image processing and recognition unit; 6. Multi-sensor fusion positioning unit; 7. Global positioning system; 8. Inertial measurement unit; 9. Wheel speed sensor; 10. On-board camera; 11. Intelligent path planning and tracking unit; 12. Actuator control unit; 13. Real-time status feedback and adjustment unit. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] As attached Figure 1-8 As shown, the present invention provides an intelligent automobile control system with license plate recognition and path tracking, the system specifically comprises a license plate recognition module 1, a path tracking module 2 and a vehicle control module 3; Among them, the license plate recognition module 1 is used to recognize the license plate information in front of the vehicle; The path tracking module 2 plans and tracks the vehicle's driving path based on the identified license plate information or other navigation information; The vehicle control module 3 controls the steering, acceleration and braking of the vehicle according to the instructions of the path tracking module 2.

[0026] Specifically, when the smart car starts, the entire smart car control system with license plate recognition and path tracking begins to initialize and run, and the license plate recognition module 1, the path tracking module 2, the vehicle control module 3 and the related units inside each module, such as the image acquisition unit 4 and the image processing and recognition unit 5 of the license plate recognition module 1, are initialized accordingly; The camera in the image acquisition unit 4 of the license plate recognition module 1 is started and the initialization settings of the autofocus, aperture and exposure parameters are completed, preparing to capture the image in front of the vehicle at a frame rate of 30fps. The GPS receiver, IMU, wheel speed sensor 9 and vehicle-mounted camera 10 in the multi-sensor fusion positioning unit 6 of the path tracking module 2 also start to work normally and prepare to collect their respective related data.

[0027] In this embodiment, as shown in the attached Figure 1 As shown, the license plate recognition module 1 includes an image acquisition unit 4, which uses a high-resolution and high-frame-rate camera. The camera has the functions of automatic focus, automatic aperture adjustment and exposure, and the camera is installed in a position to ensure that the best viewing angle image of the license plate in front of the vehicle and the surrounding environment can be obtained; The camera captures images at a frame rate of 30fps and follows the following process: After the system is started, the camera is continuously turned on and captures an image sequence in front of the vehicle at a frame rate of 30fps. Each captured image data is transmitted to the license plate recognition module 1 through the vehicle network for subsequent processing.

[0028] Specifically, the image acquisition unit 4 of the license plate recognition module 1 performs image acquisition according to the set process. After the system is started, the camera with high resolution, high frame rate, automatic focus, automatic aperture and exposure adjustment functions is continuously turned on to acquire image sequences in front of the vehicle and the surrounding environment at a frame rate of 30fps. During the driving process of the vehicle, the camera continuously captures images containing license plates of other vehicles. The image data collected each time is transmitted in real time through the on-board network to the image processing and recognition unit 5 of the license plate recognition module 1 for subsequent processing.

[0029] In this embodiment, as shown in the attached Figure 1 , 4 As shown, the license plate recognition module 1 also includes an image processing and recognition unit 5, which is used to perform the following processing on the collected license plate image: S1. Image preprocessing: S1.1. Use the weighted average method to convert the collected color license plate image into a grayscale image. The specific formula is: G=0.299R+0.587G+0.114B Where G represents the grayscale value of the pixel obtained after calculation, and the value range is usually 0-255, 0 represents black, and 255 represents white; R represents the color value of the red channel of the pixel in the color image; G represents the color value of the green channel of the pixel in the color image; B represents the color value of the blue channel of the pixel in the color image; S2.2, use Gaussian filtering to filter and denoise the grayscale image, by performing weighted average operation on each pixel in the image and its neighboring pixels, with the weight determined by the Gaussian function, to remove Gaussian noise; The formula for the Gaussian function is: k e - where k Represents the coordinate offset relative to the current pixel on the image plane, such as, for the current pixel ( ), the coordinates of a point in the neighborhood are ( ); is the standard deviation of the Gaussian function, which determines the shape of the Gaussian function and the degree of filtering. The larger the value, the flatter the curve of the Gaussian function and the wider the filtering range, resulting in a more blurred image. The smaller the value, the smaller the filtering range, the better the image details are preserved, and the noise removal effect may be relatively weak; The specific steps of weighted average operation are: S2.21, determine the neighborhood range, select a square neighborhood of the image, such as 3×3, 5×5 or a larger area of ​​7×7 centered on the current pixel; S2.22. Calculate the weight of each pixel in the neighborhood. According to the Gaussian function, for each pixel in the neighborhood, the offset of its coordinates relative to the current pixel is substituted into the Gaussian function to calculate the weight value corresponding to the pixel. For example, for the current pixel point ( ) is the center of the 3×3 neighborhood, and the pixel coordinates in the neighborhood are , , , relative these coordinates to ( )’s offset is substituted into the Gaussian function to calculate their respective weights; S2.23, weighted average calculation, multiply the gray value of each pixel in the neighborhood by its corresponding weight, then add all the products, and finally divide by the sum of all weights. The result is the gray value of the current pixel after Gaussian filtering; For example, in a 3×3 neighborhood, the current pixel is ( ), the grayscale values ​​of its neighboring pixels are G-1, -1, G-1, 0, G-1, 1, and the corresponding weights are w-1, -1, w-1, 0, w-1, 1. Then after Gaussian filtering, the pixel ( ) gray value G filtered ( ) is calculated as follows: G filtered ( )= Median filtering is used to sort the pixel values ​​in the neighborhood of the pixel point, and the middle value is taken as the filtered value of the pixel point to remove salt and pepper noise. Specifically: S2.24, in the same manner as S2.21, select a square neighborhood of the image, such as 3×3, 5×5 centered on the current pixel; S2.25. Obtain and sort pixel values ​​in the neighborhood. For the current pixel, extract the grayscale values ​​of all pixels in its neighborhood to form a data sequence; For example, in a 3×3 neighborhood, the current pixel is ( ), collect all the grayscale values ​​of the 9 pixels in its neighborhood, g-1, -1, g-1,0, g-1,1, and then sort this data sequence from small to large or from large to small; S2.26. Take the middle value as the filtered value. After the sorting is completed, the middle value is determined according to whether the number of pixels in the neighborhood is an odd number or an even number. For example, if the neighborhood is an odd number of pixels such as 3×3 or 5×5, then the middle value is the value in the middle of the sorted data sequence. If the neighborhood is an even number of pixels such as 4×4, then the average of the two middle values ​​of the sorted data sequence is usually taken as the middle value, and this middle value is used as the gray value of the current pixel after median filtering. For example, for the current pixel ( ) is the 3×3 neighborhood centered on the g. The collected neighborhood pixel values ​​are sorted into g. sorted [1],g sorted [2]......,g sorted [9], then after median filtering, the pixel point ( ) gray value M filtered ( ) is: M filtered ( )=g sorted [5].

[0030] Specifically, after the image processing and recognition unit receives the collected color license plate image, it first performs image preprocessing and converts the color license plate image into a grayscale image using the weighted average method. The grayscale value of each pixel is calculated according to the formula Gray = 0.299 * R +0.587 * G + 0.114 * B. For example, for a certain pixel, its red channel value R is 100, the green channel value G is 150, and the blue channel value B is 80. By calculation, the grayscale value of the pixel is Gray = 0.299 * 100 + 0.587 * 150 + 0.114 * 80 = 125. Next, the grayscale image is filtered and denoised using a Gaussian filter. A 3×3 neighborhood centered on the current pixel is selected. For each pixel in the neighborhood, its weight is calculated according to the Gaussian function. For example, for a neighborhood pixel with a coordinate offset of (-1, -1), the weight value w-1, -1 is substituted into the Gaussian function. The weights of other neighborhood pixels are calculated in the same way. Then, the grayscale value of each pixel in the neighborhood is multiplied by its corresponding weight according to the weighted average calculation method. All products are then added and divided by the sum of all weights to obtain the grayscale value after Gaussian filtering. If salt and pepper noise is found in the image, a median filter is used for processing. Similarly, a 3×3 neighborhood is selected. The grayscale values ​​of the 9 pixels in the neighborhood are collected and sorted. The middle value after sorting, such as gsorted [5], is taken as the grayscale value of the pixel after median filtering.

[0031] S2, license plate positioning; S2.1 uses the Sobel operator to perform edge detection on the preprocessed image. The Sobel operator determines the edge points by calculating the grayscale change rate of the image in the horizontal and vertical directions; S2.2, performing morphological operations on the detected edge image, including corrosion and expansion, wherein the corrosion operation removes small noise and isolated points in the edge image, and the expansion operation connects adjacent edge segments to make the edge of the license plate area more complete and clear; S3.3. Based on the geometric features of the license plate, such as a rectangular shape and an aspect ratio between 3:1 and 5:1, search for possible license plate areas in the edge image after morphological processing, and filter out areas that meet the geometric features of the license plate as the initially located license plate area by traversing the connected areas in the edge image.

[0032] Specifically, the Sobel operator is used to perform edge detection on the preprocessed image, and the edge points are determined by calculating the grayscale change rate of the image in the horizontal and vertical directions. For example, for a row of pixels in the image, the grayscale difference of adjacent pixels is calculated from left to right. When the difference exceeds a certain threshold, it is determined to be an edge point; Morphological operations are performed on the detected edge image. First, an erosion operation is performed to remove small noise and isolated points in the edge image, and then an expansion operation is performed to connect adjacent edge segments to make the edge of the license plate area more complete and clear. According to the geometric features of the license plate, such as a rectangular shape and an aspect ratio between 3:1 and 5:1, possible license plate areas are searched in the edge image after morphological processing. By traversing the connected areas in the edge image, the areas that meet the geometric features of the license plate are screened out as the initially located license plate area.

[0033] S3, character segmentation and recognition; S3.1. For the located license plate area, its grayscale image is projected in the horizontal and vertical directions respectively. The horizontal projection can obtain the distribution of the characters in the vertical direction, and the vertical projection can obtain the distribution of the characters in the horizontal direction. The boundaries of the characters are determined according to the peaks and troughs of the projection curve. The area between two adjacent troughs is the area where a character is located, thereby segmenting the characters on the license plate; S3.2. Construct a character recognition model based on convolutional neural network. The model structure includes input layer, multiple convolutional layers, pooling layer, fully connected layer and output layer. Each segmented character image is normalized to make its size meet the input requirements of CNN model. The recognition accuracy is calculated by the formula Accuracy= Calculated; For example, the character images are uniformly adjusted to a size of 32×32 pixels, and the normalized character images are input into the CNN model for training and recognition. In the training stage, a large number of license plate character samples are used to train the model, including normal samples and samples with a certain degree of damage and deformation. The weight parameters of the model are continuously adjusted through the back propagation algorithm so that the model can accurately recognize different license plate characters. In the recognition stage, the character image to be recognized is input into the trained model, and the output layer of the model will give the recognition result of each character.

[0034] Specifically, for the located license plate area, its grayscale image is projected in the horizontal and vertical directions respectively. The horizontal projection can obtain the distribution of characters in the vertical direction, and the vertical projection can obtain the distribution of characters in the horizontal direction. The boundary of the character is determined according to the peak and trough positions of the projection curve. The area between two adjacent troughs is the area where a character is located, thereby segmenting the characters on the license plate; A character recognition model based on convolutional neural network is constructed. Each segmented character image is normalized and its size is uniformly adjusted to 32×32 pixels. Then the normalized character image is input into the CNN model for training and recognition. In the training stage, a large number of normal samples and license plate character samples with a certain degree of damage and deformation have been used to train the model. The weight parameters of the model are continuously adjusted through the back propagation algorithm so that the model can accurately recognize different license plate characters. In the recognition stage, the character image to be recognized is input into the trained model, and the output layer of the model will give the recognition result of each character, thereby completing the recognition of the license plate characters.

[0035] In this embodiment, as shown in the attached Figure 1 , 25. The path tracking module 2 includes a multi-sensor fusion positioning unit 6. The multi-sensor fusion positioning unit 6 integrates multiple sensors such as a global positioning system 7, an inertial measurement unit 8, a wheel speed sensor 9 and a vehicle-mounted camera 10, and realizes high-precision positioning of the vehicle in the following ways: S1, GPS data acquisition and processing, continuously receiving GPS signals from satellites through the vehicle-mounted GPS receiver to obtain the vehicle's latitude and longitude, altitude and timestamp information; Perform coordinate conversion on the received GPS data, converting it from the WGS84 coordinate system to a coordinate system suitable for local maps, such as the Beijing 50 coordinate system; Use differential GPS technology or correct GPS data to improve GPS positioning accuracy. The specific correction formula can be determined according to the correction method used; For example, if a simple linear correction is used, the corrected coordinates can be set to X' = X + ΔX, Y' = Y + ΔY, where X and Y are the original coordinates and ΔX and ΔY are the correction amounts.

[0036] Specifically, the vehicle-mounted GPS receiver continuously receives GPS signals from satellites to obtain the vehicle's latitude and longitude, altitude, and timestamp information. For example, the GPS signal received at a certain moment shows that the vehicle's latitude and longitude are 30° north latitude, 120° east longitude, and the altitude is 50m, and the timestamp is X:00:00 on X:X ... The received GPS data is converted from the WGS84 coordinate system to the Beijing 50 coordinate system suitable for local maps. Assuming a simple linear correction method, the original coordinates are X = 100, Y = 200, the correction amount ΔX = 5, ΔY = -3, then the corrected coordinates are X' = X + ΔX = 100 + 5 = 105, Y' = Y + ΔY = 200 - 3 = 197.

[0037] S2, IMU data acquisition and processing, the on-board IMU continuously collects the acceleration and angular velocity information of the vehicle, the acceleration includes linear acceleration and angular acceleration; The acceleration and angular velocity data collected by the IMU are fused using the quaternion method to accurately estimate the vehicle's attitude, such as the vehicle's roll angle, pitch angle, heading angle, etc. The quaternion value is continuously updated to reflect the vehicle's attitude changes in real time. The specific quaternion update formula can be determined according to the quaternion operation rules; For example, suppose the quaternion is q = [w, x, y, z], and the update formula at a certain time t is q (t) = q (t- 1) + Δq, where Δq is the quaternion increment calculated based on the acceleration and angular velocity data.

[0038] Specifically, the onboard IMU continuously collects the acceleration of the vehicle, including linear acceleration, angular acceleration and angular velocity information. For example, at a certain moment, the linear acceleration of the vehicle collected is 2 m / s², the angular acceleration is 0.5 rad / s², and the angular velocity is 1 rad / s; The acceleration and angular velocity data collected by the IMU are fused using the quaternion method to accurately estimate the vehicle's posture. Suppose the quaternion is q = [w, x, y, z], and the update formula at a certain time t is q (t) = q (t -1) + Δq, where Δq is the quaternion increment calculated based on the acceleration and angular velocity data. By continuously updating the quaternion value, the vehicle's posture changes, such as the vehicle's roll angle, pitch angle, and heading angle, are reflected in real time.

[0039] S3, wheel speed sensor data acquisition and processing, the wheel speed sensor is installed on the wheel of the vehicle, collects the wheel speed data in real time, and calculates the vehicle's driving speed, driving distance and other driving status information based on the wheel speed and the vehicle's tire size parameters. The specific calculation formula is: Speed ​​= wheel speed × tire circumference; The driving distance is calculated by integrating the wheel speed. If the wheel speed is assumed to be v (t), then the driving distance S = ∫v(t)dt).

[0040] Specifically, the wheel speed sensor is installed on the wheel of the vehicle to collect the wheel speed data in real time. Assuming that the wheel speed is 10 rpm at a certain moment and the circumference of the vehicle tire is known to be 2 m, the vehicle speed can be calculated as 10×2 = 20 m / s according to the formula speed = wheel speed × tire circumference. The travel distance is calculated by integrating the wheel speed. Assuming the wheel speed is v(t), the travel distance S = ∫v (t) dt. For example, if the change in wheel speed over a period of time is known, the distance traveled by the vehicle during this period of time can be obtained through integration.

[0041] S4. Visual sensor data acquisition and processing. The on-board camera collects road environment images around the vehicle at a frame rate of 30fps, pre-processes the collected road environment images, uses Huffman transform to detect lane lines and deep learning-based traffic sign recognition to identify lane lines and traffic signs on the road. The detected lane line information is used as a reference for vehicle positioning and path tracking.

[0042] Specifically, the on-board camera collects images of the road environment around the vehicle at a frame rate of 30fps, pre-processes the collected road environment images, and uses Hoffman transform to detect lane lines and deep learning-based traffic sign recognition to identify road features such as lane lines and traffic signs. For example, the position and direction of the lane line in front of the vehicle are detected through Hoffman transform, and the speed limit sign at the intersection ahead is identified as 50 km / h through deep learning traffic sign recognition. These road feature information is used as a reference for vehicle positioning and path tracking.

[0043] S5. Multi-sensor data fusion, constructing a Kalman filter, taking information from GPS, IMU, wheel speed sensor and visual sensor as input. The Kalman filter is divided into a prediction stage and an update stage. In the prediction stage, the vehicle's state at the current moment, such as position, speed and posture, is predicted based on the vehicle's dynamic model and the state estimate at the previous moment. In the update stage, the data actually collected by each sensor at the current moment is compared with the result of the prediction stage. The prediction result is corrected through Kalman gain calculation to obtain the optimal state estimate of the vehicle at the current moment, thereby achieving high-precision positioning. The specific calculation steps and formulas of the Kalman filter can be determined according to the standard Kalman filter algorithm and the specific situation of the vehicle; For example, in the prediction stage: atx k∣k−1 =Ax^ k−1∣k−1 +Bu k−1 P k∣k−1 =AP k−1∣k−1 A T +Q; Update phase: K k =P k∣k−1 H T (H Pk∣k−1 H T +R) −1 ;hatx k∣k =x^ k∣k−1 +K k (zk−Hx^ k∣k−1 );P k∣k =(1−K k H)P k∣k−1 Where x̂ represents the state estimate, P is the estimation error covariance, A and B are system matrices, H is the observation matrix, Q and R are the covariance matrices of process noise and observation noise respectively, and K k is the Kalman gain and zk is the observation value.

[0044] Specifically, a Kalman filter is constructed, and information from GPS, IMU, wheel speed sensor and visual sensor is used as input. In the prediction stage, the vehicle's state at the current moment, such as position, speed and attitude, is predicted based on the vehicle's dynamic model and the state estimate at the previous moment. For example, assuming that the vehicle's dynamic model is known, the state estimate of the vehicle at the previous moment is position (x1, y1), speed v1, attitude (roll angle α1, pitch angle β1, heading angle γ1), and the state estimate of the vehicle at the current moment is predicted through the calculation formula of the prediction stage; In the update phase, the data actually collected by each sensor at the current moment is compared with the results of the prediction phase. The prediction results are corrected through Kalman gain calculation to obtain the optimal state estimate of the vehicle at the current moment, thereby achieving high-precision positioning.

[0045] In this embodiment, as shown in the attached Figure 2 , 6 As shown, the path tracking module 2 also includes an intelligent path planning and tracking unit 11, and the intelligent path planning and tracking unit 11 implements the path planning and tracking functions in the following manner: S1. Route planning: Obtain real-time traffic information through the Internet of Vehicles, including road congestion, construction sections, and road speed limits. Combined with road features identified by visual sensors, such as lane lines and traffic signs, the system can fully understand the road environment around the vehicle. Using algorithm A for path planning, we first build a search space that includes the vehicle's current location, destination, and all possible road nodes. We set an evaluation function for each road node. The evaluation function consists of two parts: one is the actual path cost from the starting node to the node, such as the driving distance; the other is the estimated cost from the node to the destination, such as the estimated time to the destination based on real-time traffic information. Starting from the starting node, we search the nodes in ascending order according to the value of the evaluation function until we find the destination node, thereby planning an optimal path.

[0046] Specifically, the Internet of Vehicles can obtain real-time traffic information, including road congestion, construction sections, and road speed limits. For example, it can be obtained that a certain road ahead is congested due to construction, and the vehicle speed is expected to be slow, while another road has a speed limit of 60 km / h and moderate traffic volume. At the same time, combined with the road features recognized by the visual sensor, such as lane lines and traffic signs, the road environment around the vehicle can be fully understood; When using the A algorithm for path planning, we first build a search space that includes the vehicle's current location, destination, and all possible road nodes. We set an evaluation function for each road node. The evaluation function consists of two parts: one is the actual path cost from the starting node to the node, such as the driving distance; the other is the estimated cost from the node to the destination, such as the estimated time to the destination based on real-time traffic information. Starting from the starting node, we search for nodes in ascending order according to the value of the evaluation function until we find the destination node, thereby planning an optimal path. For example, after calculation and search, the optimal path is to first drive along the current lane for a distance, then turn left at an intersection to avoid construction congestion and finally reach the destination.

[0047] S2. Path tracking: construct a dynamic model of the vehicle, including the vehicle's mass, moment of inertia and tire parameters. The vehicle's position, speed and heading angle status information are used as input variables, and the vehicle's work, acceleration and deceleration control instructions are used as output variables. According to the vehicle's current state and the planned path, the MPC algorithm based on model predictive control is used to predict the vehicle's driving trajectory in the future, such as the next 5 seconds. The MPC algorithm sets an optimization objective function at each prediction moment, such as minimizing the deviation between the vehicle's actual driving trajectory and the planned path, while considering the vehicle's control constraints, such as the maximum steering angle, maximum acceleration and maximum deceleration. By solving the optimization objective function, the optimal control instructions at each prediction moment, namely steering, acceleration, deceleration and other instructions, are obtained. These instructions are sent to the vehicle control module to make the vehicle drive according to the planned path. At the same time, during the vehicle driving process, according to the real-time situation, such as the braking of the vehicle in front and the pedestrians crossing the road, the objective function and the predicted trajectory are continuously adjusted to ensure the safety of the vehicle. The specific calculation steps and formulas of the MPC algorithm can be determined according to the standard MPC algorithm and the specific situation of the vehicle; For example, let the vehicle's state vector be x, the control vector be u, the prediction domain be N, and the optimization objective function be J = ∑_{k = 1}^{N}(x_{k}^{} - x_{k})^{2} + ∑_{k = 1}^{N}(u_{k}^{} - u_{k})^{2}, where x_{k}^{} is the state vector under the planned path, x_{k} is the actual state vector, u_{k}^{} is the control vector under the planned path, and u_{k} is the actual control vector.

[0048] Specifically, a dynamic model of the vehicle is constructed, including the vehicle's mass, moment of inertia, and tire parameters. The position, speed and heading state information of the vehicle are taken as input variables, and the work, acceleration and deceleration control instructions of the vehicle are taken as output variables. According to the current state of the vehicle and the planned path, the MPC algorithm based on model predictive control is used to predict the vehicle's driving trajectory in the future, such as the next 5 seconds. For example, the state vector of the vehicle is x, the control vector is u, the prediction time domain is N, and the optimization objective function is J = ∑_{k = 1}^{N}(x_{k}^{} - x_{k})^{2} + ∑_{k = 1}^{N}(u_{k}^{} - u_{k})^{2}, where x_{k}^{} is the state vector under the planned path, x_{k} is the actual state vector, u_{k}^{} is the control vector under the planned path, and u_{k} is the actual control vector. By solving the optimization objective function, the optimal control instructions at each prediction moment, i.e., steering, acceleration, deceleration and other instructions, are obtained; These instructions are sent to the vehicle control module to make the vehicle drive according to the planned path. At the same time, during the driving process of the vehicle, the optimization objective function and predicted trajectory are continuously adjusted according to real-time conditions, such as the braking of the vehicle in front and pedestrians crossing the road, to ensure the safety of vehicle driving. For example, when the vehicle in front brakes suddenly, the optimization objective function in the MPC algorithm is adjusted in time, the weight of the distance between the vehicle and the vehicle in front is increased, the driving trajectory is re-predicted and new control instructions are sent to the vehicle control module to slow down the vehicle to avoid it.

[0049] In this embodiment, as shown in the attached Figure 3 , 7 As shown, the vehicle control module 3 includes an actuator control unit 12, and the actuator control unit 12 realizes precise control of the vehicle actuator in the following manner: S1, electric power steering or wire-controlled steering control. When the path tracking module issues a steering command, if an electric power steering system is used, the EPS electronic control unit ECU first calculates the required steering assist according to the command, and adjusts the output power of the motor through the PID control algorithm according to the vehicle's driving speed and steering angle parameters to achieve precise steering assist control; The basic formula of PID control algorithm is: u (t) = K_{p} e (t) + K_{i}\int_{0}^{t} e(s) ds + K_{d}\frac {de (t)}{dt}, where u (t) is the control output, K_{p}, K_{i}, K_{d} are the proportional, integral, and differential coefficients respectively, and e (t) is the error signal, that is, the difference between the target steering angle and the actual steering angle; If a steer-by-wire system is used, upon receiving the steering command from the path tracking module, the SBW ECU directly converts the command into an electronic signal and sends it to the steering actuator, such as the electric steering gear. Through precise signal conversion and control algorithms, it ensures that the steering actuator rotates at the required steering angle and speed, thus achieving full electronic control. The relationship between its control input and output can be expressed as: u=f control (e) Where u is the control input, e is the error, and f control is a control function, which is a PID control algorithm.

[0050] Specifically, when the path tracking module issues a steering command, if an electric power steering system is used, the EPS electronic control unit ECU first calculates the required steering assist according to the command. For example, based on the parameters of the vehicle's driving speed of 20 m / s and the steering angle of 10°, the output power of the motor is adjusted through the PID control algorithm to achieve precise steering assist control. Assume that the proportional coefficient K_{p} = 0.5, the integral coefficient K_{i} = 0.1, the differential coefficient K_{d} = 0.2 in the PID control algorithm, and the error signal e (t) (i.e., the difference between the target steering angle and the actual steering angle) is 0 (assuming that the actual steering angle is consistent with the target steering angle at the initial moment). According to the basic formula of the PID control algorithm u (t) = K_{p} e (t) + K_{i}\int_{0}^{t} e (s) ds + K_{d}\frac {de (t)}{dt}, the control output u (t) is calculated. The value of the steering wheel is adjusted to adjust the output power of the motor, making it easier for the driver to turn the steering wheel while ensuring that the vehicle turns according to the required steering angle; If a steer-by-wire system is used, upon receiving the steering command from the path tracking module, the SBW ECU directly converts the command into an electronic signal and sends it to the steering actuator (such as an electric steering gear). Through precise signal conversion and control algorithms, it ensures that the steering actuator rotates at the required steering angle and speed, achieving full electronic control.

[0051] S2. Electronically controlled braking and acceleration system control. When the vehicle needs to brake, the ECU of the electronically controlled braking system will calculate the required braking pressure according to the set braking strategy based on the path tracking module and the vehicle's driving conditions, such as the distance to the vehicle in front and the vehicle speed. By adjusting the solenoid valve actuator in the braking system, the application of braking pressure is accurately controlled to achieve smooth braking operation and avoid sudden braking that may cause discomfort to passengers or affect driving safety.

[0052] Specifically, when the vehicle needs to brake, the path tracking module and the vehicle's driving conditions are used to determine that, for example, if the distance to the vehicle ahead is 20 meters and the vehicle speed is 20 m / s, the ECU of the electronically controlled braking system will calculate the required braking pressure according to the set braking strategy. For example, based on factors such as the vehicle speed, the distance to the vehicle ahead, and the vehicle's braking performance, the ECU calculates that the braking pressure to be applied is 500 kPa. By adjusting the solenoid valve actuator in the braking system, the application of the braking pressure is precisely controlled to achieve a smooth braking operation, avoiding sudden braking that causes discomfort to passengers or affects driving safety. When the vehicle needs to accelerate, the ECU of the electronic throttle control system calculates the required throttle opening according to the acceleration command of the path tracking module and the current driving conditions of the vehicle (such as engine speed, vehicle speed, etc.). For example, based on the vehicle's current engine speed of 1500 rpm, vehicle speed of 20 m / s, and acceleration requirements, it is calculated that the throttle opening needs to be adjusted to 30%. By adjusting the electronic control mechanism of the accelerator pedal, the throttle opening is precisely controlled to achieve smooth acceleration operation, avoiding sudden acceleration that may cause discomfort to passengers or affect driving safety.

[0053] In this embodiment, as shown in the attached Figure 3 , 8 As shown, the vehicle control module 3 also includes a real-time state feedback and adjustment unit 13, which implements real-time feedback and adjustment of the state of the vehicle actuator in the following manner: S1, state sensor data acquisition, the steering angle sensor is installed on the steering wheel or steering mechanism of the vehicle, collects the steering wheel rotation angle data in real time, and transmits the collected steering angle data to the vehicle control module and the path tracking module through the vehicle network; The brake pressure sensor is installed in the brake system to collect the pressure change data of the brake system in real time, and transmit the collected pressure change data to the vehicle control module and the path tracking module through the vehicle network; The throttle opening sensor is installed near the accelerator pedal to collect throttle opening data in real time and transmit the collected throttle opening data to the vehicle control module and path tracking module through the vehicle network.

[0054] Specifically, the steering angle sensor is installed on the steering wheel or steering mechanism of the vehicle, collects the steering wheel rotation angle data in real time, and transmits the collected steering angle data to the vehicle control module and the path tracking module through the vehicle network. For example, during the vehicle steering process, the steering angle sensor collects the steering wheel rotation angle gradually changing from 0° to 10° in real time, and transmits these data in real time; The brake pressure sensor is installed in the brake system to collect the pressure change data of the brake system in real time, and transmit the collected pressure change data to the vehicle control module and the path tracking unit through the vehicle network. For example, when the vehicle is braking, the brake pressure sensor collects the pressure of the brake system gradually rising from 0 kPa to 500 kPa in real time, and transmits this data in real time; The throttle opening sensor is installed near the accelerator pedal to collect throttle opening data in real time, and transmit the collected throttle opening data to the vehicle control module and path tracking module through the vehicle network. For example, during vehicle acceleration, the throttle opening sensor collects the throttle opening gradually changing from 0% to 30% in real time, and transmits this data in real time.

[0055] S2, deviation analysis and adjustment, after the vehicle control module 3 and the path tracking module 2 receive the actual steering angle data transmitted by the steering angle sensor, they compare it with the steering angle required by the path tracking module 2 and calculate the steering deviation. If the steering deviation exceeds the set threshold, such as ±5, the vehicle control module 3 will recalculate the steering command through the PID control algorithm according to the deviation, adjust the steering assist size or directly adjust the action of the steering actuator; After receiving the pressure change data transmitted by the brake pressure sensor, it is compared with the ideal brake pressure calculated according to the braking strategy to calculate the brake deviation. If the brake deviation exceeds the set threshold, such as ±10%, the electronically controlled brake system will adjust the operation of the solenoid valve actuator according to the deviation to ensure that the brake pressure meets the requirements and achieve smooth braking; After receiving the actual throttle opening data transmitted by the throttle opening sensor, and the acceleration deviation calculated according to the acceleration command, if the acceleration deviation exceeds the set threshold, such as ±10%, the electronic throttle control system will adjust the operation of the electronic control mechanism of the accelerator pedal according to the deviation; The adjustment process can be expressed as: = - = + = + + in, is the error at the current moment, and are the actual output and expected output at the current moment, respectively. is the control input at the next moment, and They are proportional and integral gains respectively.

[0056] Specifically, after the vehicle control module 3 and the path tracking module 2 receive the actual steering angle data transmitted by the steering angle sensor, they compare it with the steering angle required by the path tracking module 2 and calculate the steering deviation. For example, if the steering angle required by the path tracking module 2 is 10° and the actual steering angle data is 9°, then the steering deviation is 10° - 9° = 1°. If the steering deviation exceeds the set threshold (such as ±5°), the vehicle control module will recalculate the steering command through the PID control algorithm according to the deviation, adjust the steering assist or directly adjust the action of the steering actuator; After receiving the pressure change data transmitted by the brake pressure sensor, it is compared with the ideal brake pressure calculated according to the braking strategy to calculate the brake deviation. For example, the ideal brake pressure calculated according to the braking strategy is 500 kPa, and the actual pressure change data shows that the current pressure is 480 kPa, then the brake deviation is (500 kPa - 480 kPa) / 500kPa × 100% = 4%. If the brake deviation exceeds the set threshold (such as ±10%), the electronically controlled brake system will adjust the operation of the solenoid valve actuator according to the deviation to ensure that the brake pressure meets the requirements and achieve smooth braking; After receiving the actual throttle opening data transmitted by the throttle opening sensor, and the acceleration deviation calculated based on the acceleration command, if the acceleration deviation exceeds the set threshold (such as ±10%), the electronic throttle control system will adjust the operation of the electronic control mechanism of the accelerator pedal according to the deviation to ensure that the throttle opening meets the requirements and achieve smooth acceleration.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. Intelligent vehicle control system with license plate recognition and path tracking, characterized by: It comprises a license plate recognition module (1), a path tracking module (2) and a vehicle control module (3); The license plate recognition module (1) is used to recognize the license plate information of the vehicle in front; The path tracking module (2) plans and tracks the vehicle's driving path based on the recognized license plate information or other navigation information; The vehicle control module (3) controls the steering, acceleration and braking of the vehicle according to the instructions of the path tracking module (2).

2. The intelligent automobile control system with license plate recognition and path tracking according to claim 1 is characterized in that: The license plate recognition module (1) comprises an image acquisition unit (4), wherein the image acquisition unit (4) adopts a camera with high resolution and high frame rate, the camera has the functions of automatic focusing, automatic aperture adjustment and exposure adjustment, and the camera is installed in a position to ensure that the best viewing angle image of the license plate in front of the vehicle and in the surrounding environment can be obtained; The camera captures images at a frame rate of 30fps and follows the following process: After the system is started, the camera is continuously turned on and captures an image sequence in front of the vehicle at a frame rate of 30fps. Each captured image data is transmitted to the license plate recognition module (1) through the vehicle network for subsequent processing.

3. The intelligent automobile control system with license plate recognition and path tracking according to claim 1 is characterized in that: The license plate recognition module (1) further comprises an image processing and recognition unit (5) for performing the following processing on the collected license plate image: S1, image preprocessing; S1.

1. Use the weighted average method to convert the collected color license plate image into a grayscale image. The specific formula is: G = 0.299R + 0.587G + 0.114B; Where G represents the grayscale value of the pixel obtained after calculation, and the value range is usually 0-255, 0 represents black, and 255 represents white; R represents the color value of the red channel of the pixel in the color image; G represents the color value of the green channel of the pixel in the color image; B represents the color value of the blue channel of the pixel in the color image; S2.2, use Gaussian filtering to filter and denoise the grayscale image, by performing weighted average operation on each pixel in the image and its neighboring pixels, with the weight determined by the Gaussian function, to remove Gaussian noise; The formula for the Gaussian function is: k yes - ; Among them, k Represents the coordinate offset relative to the current pixel on the image plane, such as, for the current pixel ( ), the coordinates of a point in the neighborhood are ( ), is the standard deviation of the Gaussian function; The specific steps of weighted average operation are: S2.21, determine the neighborhood range, select a square neighborhood of the image, such as 3×3, 5×5 or a larger area of ​​7×7 centered on the current pixel; S2.

22. Calculate the weight of each pixel in the neighborhood. According to the Gaussian function, for each pixel in the neighborhood, the offset of its coordinates relative to the current pixel is substituted into the Gaussian function to calculate the weight value corresponding to the pixel. For example, for the current pixel point ( ) is the center of the 3×3 neighborhood, and the pixel coordinates in the neighborhood are , , , relative these coordinates to ( ) is substituted into the Gaussian function to calculate the respective weights; S2.23, weighted average calculation, multiply the gray value of each pixel in the neighborhood by its corresponding weight, then add all the products, and finally divide by the sum of all weights. The result is the gray value of the current pixel after Gaussian filtering; For example, in a 3×3 neighborhood, the current pixel is ( ), the grayscale values ​​of its neighboring pixels are G-1, -1, G-1, 0, G-1, 1, and the corresponding weights are w-1, -1, w-1, 0, w-1, 1. Then after Gaussian filtering, the pixel ( ) gray value G filtered ( ) is calculated as follows: G filtered （ ）= ； Median filtering is used to sort the pixel values ​​in the neighborhood of the pixel point, and the middle value is taken as the filtered value of the pixel point to remove salt and pepper noise. Specifically: S2.24, in the same manner as S2.21, select a square neighborhood of the image, such as 3×3, 5×5 centered on the current pixel; S2.25, obtaining pixel values ​​in the neighborhood and sorting them, for the current pixel, extracting the grayscale values ​​of all pixels in its neighborhood to form a data sequence; For example, in a 3×3 neighborhood, the current pixel is ( ), collect all the grayscale values ​​of the 9 pixels in its neighborhood, g-1, -1, g-1,0, g-1,1, and then sort this data sequence from small to large or from large to small; S2.

26. Take the middle value as the filtered value. After the sorting is completed, the middle value is determined according to whether the number of pixels in the neighborhood is an odd number or an even number. For example, if the neighborhood is an odd number of pixels such as 3×3 or 5×5, then the middle value is the value in the middle of the sorted data sequence. If the neighborhood is an even number of pixels such as 4×4, then the average of the two middle values ​​of the sorted data sequence is usually taken as the middle value, and this middle value is used as the gray value of the current pixel after median filtering. For example, for the current pixel ( ) is the 3×3 neighborhood centered on the g. The collected neighborhood pixel values ​​are sorted into g. sorted [1],g sorted [2]......,g sorted [9], then after median filtering, the pixel point ( ) gray value M filtered ( ) is: M filtered ( )=g sorted [5]; S2, license plate positioning; S2.1 uses the Sobel operator to perform edge detection on the preprocessed image. The Sobel operator determines the edge points by calculating the grayscale change rate of the image in the horizontal and vertical directions; S2.2, performing morphological operations on the detected edge image, including corrosion and expansion, wherein the corrosion operation removes small noise and isolated points in the edge image, and the expansion operation connects adjacent edge segments to make the edge of the license plate area more complete and clear; S3.3, based on the geometric features of the license plate, such as a rectangular shape and an aspect ratio between 3:1 and 5:1, search for possible license plate regions in the edge image after morphological processing, and screen out regions that meet the geometric features of the license plate as the initially located license plate region by traversing the connected regions in the edge image; S3, character segmentation and recognition; S3.

1. For the located license plate area, its grayscale image is projected in the horizontal and vertical directions respectively. The horizontal projection can obtain the distribution of the characters in the vertical direction, and the vertical projection can obtain the distribution of the characters in the horizontal direction. The boundaries of the characters are determined according to the peaks and troughs of the projection curve. The area between two adjacent troughs is the area where a character is located, thereby segmenting the characters on the license plate; S3.

2. Construct a character recognition model based on convolutional neural network. The model structure includes input layer, multiple convolutional layers, pooling layer, fully connected layer and output layer. Each segmented character image is normalized to make its size meet the input requirements of CNN model. The recognition accuracy is calculated by the formula Accuracy= Calculated; For example, the character images are uniformly adjusted to a size of 32×32 pixels, and the normalized character images are input into the CNN model for training and recognition. In the training stage, a large number of license plate character samples are used to train the model, including normal samples and samples with a certain degree of damage and deformation. The weight parameters of the model are continuously adjusted through the back propagation algorithm so that the model can accurately recognize different license plate characters. In the recognition stage, the character image to be recognized is input into the trained model, and the output layer of the model will give the recognition result of each character.

4. The intelligent automobile control system with license plate recognition and path tracking according to claim 1 is characterized in that: The path tracking module (2) comprises a multi-sensor fusion positioning unit (6), wherein the multi-sensor fusion positioning unit (6) integrates a global positioning system (7), an inertial measurement unit (8), a wheel speed sensor (9) and a vehicle-mounted camera (10) and realizes high-precision positioning of the vehicle in the following manner: S1, GPS data acquisition and processing, the vehicle's latitude and longitude, altitude and timestamp information are obtained by continuously receiving GPS signals from satellites through the vehicle's onboard GPS receiver; Perform coordinate conversion on the received GPS data, converting it from the WGS84 coordinate system to a coordinate system suitable for local maps, such as the Beijing 50 coordinate system; Use differential GPS technology or correct GPS data to improve GPS positioning accuracy. The specific correction formula can be determined according to the correction method used; For example, if a simple linear correction is used, the corrected coordinates can be set to X' = X + ΔX, Y' = Y + ΔY, where X and Y are the original coordinates and ΔX and ΔY are the correction values; S2, IMU data acquisition and processing, the on-board IMU continuously collects the acceleration and angular velocity information of the vehicle, the acceleration includes linear acceleration and angular acceleration; The acceleration and angular velocity data collected by the IMU are fused using the quaternion method to accurately estimate the vehicle's attitude, such as the vehicle's roll angle, pitch angle, heading angle, etc. The quaternion value is continuously updated to reflect the vehicle's attitude changes in real time. The specific quaternion update formula can be determined according to the quaternion operation rules; For example, let the quaternion be q = [w, x, y, z], and the update formula at a certain time t is q (t) = q (t -1) + Δq, where Δq is the quaternion increment calculated based on the acceleration and angular velocity data; S3, wheel speed sensor (9) data acquisition and processing, the wheel speed sensor (9) is installed on the wheel of the vehicle, collects the wheel speed data in real time, and calculates the vehicle's driving speed, driving distance and other driving status information based on the wheel speed and the vehicle's tire size parameters. The specific calculation formula is: Speed ​​= wheel speed × tire circumference; The driving distance is calculated by integrating the wheel speed. If the wheel speed is v (t), the driving distance S = ∫v(t)dt); S4, visual sensor data acquisition and processing, the vehicle-mounted camera (10) collects road environment images around the vehicle at a frame rate of 30fps, pre-processes the collected road environment images, uses Huffman transform to detect lane lines and deep learning-based traffic sign recognition to identify lane lines and road features of traffic signs on the road, and uses the detected lane line information as a reference for vehicle positioning and path tracking; S5. Multi-sensor data fusion, constructing a Kalman filter, taking information from GPS, IMU, wheel speed sensor (9) and visual sensor as input. The Kalman filter is divided into a prediction stage and an update stage. In the prediction stage, the vehicle's state at the current moment, such as position, speed and posture, is predicted based on the vehicle's dynamic model and the state estimate at the previous moment. In the update stage, the data actually collected by each sensor at the current moment is compared with the result of the prediction stage. The prediction result is corrected through Kalman gain calculation to obtain the optimal state estimate of the vehicle at the current moment, thereby achieving high-precision positioning. The specific calculation steps and formulas of the Kalman filter can be determined according to the standard Kalman filter algorithm and the specific conditions of the vehicle. For example, in the prediction phase: hatx k∣k−1 =Ax^ k−1∣k−1 +Bu k−1 ; P k∣k−1 =AP k−1∣k−1 From T +Q; Update phase: K k =P k∣k−1 H T (H Pk∣k−1 H T +R) −1 ;hatx k∣k =x^ k∣k−1 +K k (zk−Hx^ k∣k−1 );P k∣k =(1−K k H)P k∣k−1 ; Where x̂ represents the state estimate, P is the estimation error covariance, A and B are system matrices, H is the observation matrix, Q and R are the covariance matrices of process noise and observation noise, respectively, and K k is the Kalman gain and zk is the observation value.

5. The intelligent automobile control system with license plate recognition and path tracking according to claim 1 is characterized in that: The path tracking module (2) further comprises an intelligent path planning and tracking unit (11), wherein the intelligent path planning and tracking unit (11) implements path planning and tracking functions in the following manner: S1. Route planning: Obtain real-time traffic information through the Internet of Vehicles, including road congestion, construction sections, and road speed limits. Combined with road features identified by visual sensors, such as lane lines and traffic signs, the system can fully understand the road environment around the vehicle. Using the A algorithm for path planning, first build a search space that includes the vehicle's current location, destination, and all possible road nodes, and set an evaluation function for each road node. The evaluation function consists of two parts: one is the actual path cost from the starting node to the node, such as the driving distance, and the other is the estimated cost from the node to the destination, such as the estimated time to the destination based on real-time traffic information. Starting from the starting node, search for nodes in ascending order according to the value of the evaluation function until the destination node is found, thereby planning an optimal path; S2. Path tracking: construct a dynamic model of the vehicle, including the vehicle's mass, moment of inertia and tire parameters. The vehicle's position, speed and heading angle status information are used as input variables, and the vehicle's work, acceleration and deceleration control instructions are used as output variables. According to the vehicle's current state and the planned path, the MPC algorithm based on model predictive control is used to predict the vehicle's driving trajectory in the future, such as the next 5 seconds. The MPC algorithm sets an optimization objective function at each prediction moment, such as minimizing the deviation between the vehicle's actual driving trajectory and the planned path, while considering the vehicle's control constraints, such as the maximum steering angle, maximum acceleration and maximum deceleration. By solving the optimization objective function, the optimal control instructions at each prediction moment, namely steering, acceleration, deceleration and other instructions, are obtained. These instructions are sent to the vehicle control module (3) to make the vehicle drive according to the planned path. At the same time, during the driving process of the vehicle, according to the real-time situation, such as the braking of the vehicle in front and the pedestrians crossing the road, the optimization objective function and the predicted trajectory are continuously adjusted to ensure the driving safety of the vehicle. The specific calculation steps and formulas of the MPC algorithm can be determined according to the standard MPC algorithm and the specific situation of the vehicle; For example, let the vehicle's state vector be x, the control vector be u, the prediction domain be N, and the optimization objective function be J = ∑_{k = 1}^{N}(x_{k}^{} - x_{k})^{2} + ∑_{k = 1}^{N}(u_{k}^{} - u_{k})^{2}, where x_{k}^{} is the state vector under the planned path, x_{k} is the actual state vector, u_{k}^{} is the control vector under the planned path, and u_{k} is the actual control vector.

6. The intelligent automobile control system with license plate recognition and path tracking according to claim 1 is characterized in that: The vehicle control module (3) comprises an actuator control unit (12), wherein the actuator control unit (12) realizes precise control of the vehicle actuator in the following manner: S1, electric power steering or wire-controlled steering control. When the path tracking module (2) issues a steering command, if an electric power steering system is used, the EPS electronic control unit ECU first calculates the required steering assist according to the command, and adjusts the output power of the motor through the PID control algorithm according to the vehicle's speed and steering angle parameters to achieve precise steering assist control; The basic formula of PID control algorithm is: u (t) = K_{p} e (t) + K_{i}\int_{0}^{t} e (s) ds+ K_{d}\frac {de (t)}{dt}, where u (t) is the control output, K_{p}, K_{i}, K_{d} are the proportional, integral, and differential coefficients, respectively, and e (t) is the error signal, that is, the difference between the target steering angle and the actual steering angle; If a steer-by-wire system is used, upon receiving the steering command from the path tracking module (2), the SBW ECU directly converts the command into an electronic signal and sends it to the steering actuator, such as the electric steering gear. Through precise signal conversion and control algorithms, the steering actuator is ensured to rotate at the required steering angle and speed, thus achieving full electronic control. The relationship between its control input and output can be expressed as: u=f control (e); Where u is the control input, e is the error, and f control is a control function, wherein the control function is a PID control algorithm; S2, electronically controlled braking and acceleration system control. When the vehicle needs to brake, based on the path tracking module (2) and the vehicle's driving conditions, such as the distance to the vehicle ahead and the vehicle speed, the ECU of the electronically controlled braking system will calculate the required braking pressure according to the set braking strategy, and accurately control the application of braking pressure by adjusting the solenoid valve actuator in the braking system to achieve smooth braking operation and avoid sudden braking that may cause discomfort to passengers or affect driving safety.

7. The intelligent automobile control system with license plate recognition and path tracking according to claim 1 is characterized in that: The vehicle control module (3) further comprises a real-time state feedback and adjustment unit (13), wherein the real-time state feedback and adjustment unit (13) implements real-time feedback and adjustment of the state of the vehicle actuator in the following manner: S1, state sensor data acquisition, the steering angle sensor is installed on the steering wheel or steering mechanism of the vehicle, collects the steering wheel rotation angle data in real time, and transmits the collected steering angle data to the vehicle control module (3) and the path tracking module (2) through the vehicle network; The brake pressure sensor is installed in the brake system to collect pressure change data of the brake system in real time, and transmits the collected pressure change data to the vehicle control module (3) and the path tracking module (2) through the vehicle network; The throttle opening sensor is installed near the accelerator pedal to collect throttle opening data in real time, and transmits the collected throttle opening data to the vehicle control module (3) and the path tracking module (2) through the vehicle network; S2, deviation analysis and adjustment. After the vehicle control module (3) and the path tracking module (2) receive the actual steering angle data transmitted by the steering angle sensor, they compare it with the steering angle required by the path tracking module (2) and calculate the steering deviation. If the steering deviation exceeds a set threshold, such as ±5, the vehicle control module (3) will recalculate the steering command through the PID control algorithm according to the deviation, adjust the steering assist or directly adjust the action of the steering actuator; After receiving the pressure change data transmitted by the brake pressure sensor, it is compared with the ideal brake pressure calculated according to the braking strategy to calculate the brake deviation. If the brake deviation exceeds the set threshold, such as ±10%, the electronically controlled brake system will adjust the operation of the solenoid valve actuator according to the deviation to ensure that the brake pressure meets the requirements and achieve smooth braking; After receiving the actual throttle opening data transmitted by the throttle opening sensor, and the acceleration deviation calculated according to the acceleration command, if the acceleration deviation exceeds the set threshold, such as ±10%, the electronic throttle control system will adjust the operation of the electronic control mechanism of the accelerator pedal according to the deviation; The adjustment process can be expressed as: = - ; = + = + + ； in, is the error at the current moment, and are the actual output and expected output at the current moment, respectively. is the control input at the next moment, and They are proportional and integral gains respectively.

Images