Intelligent vehicle side parking method based on multi-vision sensor information fusion

Abstract

The invention discloses an intelligent vehicle side parking method based on multi-vision sensor information fusion. The method comprises the following steps: constructing an intelligent vehicle side parking system; original parking space images of the surrounding environment of the intelligent vehicle are collected; obtaining a parking space line position in the preprocessed parking space image; obtaining position coordinates of the obstacle; calculating parking space parameters; obtaining a path point sequence; performing motion control and wheel driving on the intelligent vehicle; the point-arriving wheel hitting operation is carried out; and controlling the intelligent vehicle to accurately enter the garage. Clear peripheral view can be provided for the intelligent vehicle, so that the driving track of the intelligent vehicle is more accurate, and the side parking precision of the intelligent vehicle is improved; a track prediction and parking space boundary detection technology is improved, a deep learning detection algorithm is used as a basis, and an adaptive feature fusion module is designed to carry out real-time detection of parking space lines and obstacles. The method is a novel visual architecture which gives consideration to real-time performance, robustness and scene generalization ability, has a self-adaptive detection function, improves the target detection precision, and improves the side parking precision of the intelligent vehicle.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle parking technology, and specifically relates to an intelligent vehicle side parking method that integrates information from multiple visual sensors. Background Technology

[0002] To improve intelligent vehicles' environmental perception and real-time monitoring of parking spaces, on the one hand, visual sensors should have high resolution, wide dynamic range, and strong low-light performance to ensure stable image information is obtained in both bright daylight and low-light nighttime conditions. On the other hand, the acquired image information should be identified and extracted using deep learning detection algorithms, enabling intelligent vehicles to quickly identify parking spaces and monitor dynamic changes around the vehicle in real time to ensure a safe and smooth parking process.

[0003] Currently, side-parking systems in the field of autonomous driving generally adopt solutions based on geometric models and optimization control. Mainstream technologies construct an initial reference path through Ritz-Shep curves or their simplified geometric models, realize "outward swing-in" type trajectory planning by combining arcs and straight lines, and verify feasibility based on the vehicle's minimum turning radius and parking space boundary constraints, but the trajectory tracking control accuracy needs to be improved.

[0004] Although some studies in recent years have introduced algorithms such as reinforcement learning to handle unstructured scenarios, mass-produced systems are still severely limited by the inherent defects of geometric models: First, they have poor adaptability to non-standard parking spaces (such as angled parking spaces and obstacles), and require the pre-setting of idealized boundary conditions; Second, the computational load of optimization control is high, and it is difficult to meet the real-time requirements on low-cost embedded platforms; Third, the generalization ability of learning algorithms is insufficient, and the collision risk in narrow scenarios still relies on conservative rule bases for avoidance. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a method for intelligent vehicles to enter parking spaces from the side using multi-visual sensor information fusion.

[0006] To achieve the above objectives, the intelligent vehicle side-entry parking method based on multi-visual sensor information fusion provided by the present invention includes the following steps performed in sequence:

[0007] 1) Construct an intelligent vehicle side-entry system; the intelligent vehicle side-entry system is installed on the intelligent vehicle and includes an image processing module, a parking space detection module, an obstacle detection module, an information processor, a motion execution module, and a path planning module; the image processing module is electrically connected to the parking space detection module and the obstacle detection module respectively; the information processor is data connected to the parking space detection module and the obstacle detection module respectively, and is also electrically connected to the motion execution module, the path planning module, and the on-board inertial measurement unit and encoder of the intelligent vehicle;

[0008] 2) The vision sensor in the image processing module is calibrated and distortion corrected. Then, the vision sensor is used to collect the original parking space images of the surrounding environment of the intelligent vehicle in real time and transmit them to the parking space detection module and the obstacle detection module respectively.

[0009] 3) The original parking space image is processed by the parking space detection module to obtain a preprocessed parking space image. Then, the parking line is detected on the preprocessed parking space image to obtain the position of the parking line in the preprocessed parking space image and transmit it to the information processor.

[0010] 4) The obstacle detection module detects obstacles in the original parking space image, obtains the obstacle position coordinates, and transmits them to the information processor;

[0011] 5) The information processor processes the parking line positions obtained in step 3) and the obstacle position coordinates obtained in step 4) and calculates the parking parameters.

[0012] 6) The path planning module performs path planning based on the above parking space parameters, obtains the path point sequence P, and transmits it to the motion execution module via the information processor;

[0013] 7) The motion execution module performs motion control and wheel drive on the intelligent vehicle based on the above path point sequence P;

[0014] 8) The information processor calculates the timing for turning the wheel and issues a command. The motion execution module controls the intelligent vehicle to perform the turning operation at the designated point according to the command.

[0015] 9) The motion execution module controls the intelligent vehicle to accurately enter the parking space according to the planned path and the above-mentioned wheel turning operation, thereby completing automatic parking.

[0016] In step 1), the image processing module includes a sensor calibration module, an image distortion correction module, and a multi-vision sensor image acquisition module connected in sequence; the multi-vision sensor image acquisition module includes vision sensors respectively installed in the upper part of the windshield of the smart car, one above the front license plate, one to two on the left and right sides, one in the upper part of the rear windshield, and one above the rear license plate.

[0017] The parking space detection module includes a multi-path image preprocessing module, a backbone network feature extraction module, an adaptive feature fusion module, and a parking line regression network module connected in sequence. It is pre-built by performing multiple rounds of iterative training using a publicly available parking line dataset.

[0018] The obstacle detection module consists of a backbone feature extraction module, a neck multi-scale feature fusion module, an obstacle detection head, and a post-processing module, which are connected in sequence. It is pre-built by performing multiple rounds of iterative training using a publicly available obstacle dataset.

[0019] Information processors are used to process images and data and generate control commands;

[0020] The motion execution module includes a motion controller and a wheel drive unit connected in sequence, and the wheel drive unit is mechanically connected to the wheels of the intelligent vehicle.

[0021] The path planning module consists of the A* algorithm module and the path planning dynamic adjustment module, which are connected in sequence.

[0022] In step 2), the method of calibrating and correcting the distortion of the visual sensor in the image processing module, and then using the visual sensor to acquire raw parking space images of the surrounding environment of the intelligent vehicle in real time and transmitting them to the parking space detection module and the obstacle detection module respectively is as follows:

[0023] 2.1) Multiple vision sensors in the image acquisition module acquire images of the calibration board. Then, the sensor calibration module uses Zhang Zhengyou's calibration method to calibrate the images of the calibration board and calculates the intrinsic and extrinsic parameter matrix and distortion coefficient of the vision sensor.

[0024] 2.2) The image distortion correction module uses the aforementioned intrinsic parameter matrix and distortion coefficients to correct the distortion of the visual sensor;

[0025] 2.3) During the parallel parking process of the intelligent vehicle, the front, rear, left, and right multi-channel visual sensors in the image acquisition module, after distortion correction, acquire original parking space images of the surrounding environment of the intelligent vehicle in real time. raw ∈R 3×H×W The data is then transmitted to the parking space detection module and the obstacle detection module, respectively; where H and W represent the height and width of the original parking space image, respectively.

[0026] In step 3), the method of using the parking space detection module to perform inverse perspective processing on the original parking space image to obtain a preprocessed parking space image, and then performing parking line detection on the preprocessed parking space image to obtain the position of the parking line in the preprocessed parking space image and transmitting it to the information processor is as follows:

[0027] 3.1) The multi-channel image preprocessing module in the parking space detection module processes the original parking space image p based on the extrinsic parameter matrix of the visual sensor obtained in step 2-1). raw ∈R 3×H×W Preprocessed parking space image p is obtained by performing inverse perspective processing. rgb ∈R 3 ×H×W ;

[0028] 3.2) The backbone network feature extraction module processes the preprocessed parking space image p. rgb ∈R 3×H×W Perform feature extraction to obtain feature maps x∈R C×H×W ;

[0029] 3.3) The adaptive feature fusion module processes the above feature map x∈R C×H×W Perform feature enhancement to obtain the enhanced feature map.

[0030] The adaptive feature fusion module includes a local attention branch, a global attention branch, a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function; wherein the local attention branch and the global attention branch are set in parallel, followed by a cascaded 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function;

[0031] The local attention branch includes parallel 3×3 and 5×5 convolutional layers, followed by a batch normalization layer and a ReLU activation function.

[0032] The global attention branch includes cascaded deep convolutional layers, group normalization layers, multi-head self-attention mechanism layers, group normalization layers, and multilayer perceptrons.

[0033] 3.4) The enhanced feature map x is processed by the parking line regression network module. F ∈R C×H×W Processing is performed to obtain a preprocessed parking space image p rgb ∈R 3×H×W The coordinate vector (x,y,s,β) of each parking space line marker is given, where (x,y) represents the position of the parking space line; s is a binary value representing the shape of the parking space line pattern, where 0 indicates that the parking space line pattern is "T" shaped and 1 indicates that the parking space line pattern is "L" shaped; and β represents the azimuth coordinate of the parking space line pattern in the polar coordinate system.

[0034] In step 4), the method for the obstacle detection module to detect obstacles in the original parking space image, obtain the obstacle position coordinates, and transmit them to the information processor is as follows:

[0035] 4.1) The backbone feature extraction module in the obstacle detection module extracts the original parking space image p raw ∈R 3×H×W Feature extraction is performed to obtain three different output tensors β1, β2, and β3.

[0036] 4.2) The neck multi-scale feature fusion module performs multi-scale feature extraction on the above output tensors β1, β2, and β3 to obtain three enhanced feature maps β. 1f ,β 2f ,β 3f ;

[0037] 4.3) The enhanced feature map β above is processed by the obstacle detection head. 1f ,β 2f ,β 3fProcess the data to obtain candidate bounding boxes for all obstacles;

[0038] 4.4) The post-processing module filters the above candidate bounding boxes using confidence filtering and non-maximum suppression methods to obtain the original parking space image p. raw ∈R 3×H×W The coordinate vector (x1, y1, x2, y2, class_id, confidence) of each obstacle detected is given, where (x1, y1) and (x2, y2) represent the coordinates of the top left and bottom right corners of the candidate bounding box of the obstacle, respectively, class_id represents the category of the obstacle, and confidence represents the confidence of the category.

[0039] In step 5), the method by which the information processor processes the parking space line positions obtained in step 3) and the obstacle position coordinates obtained in step 4) and calculates parking space parameters is as follows:

[0040] 5.1) Taking the center of the intelligent vehicle as the origin, the direction of the intelligent vehicle's movement as the X-axis, and the Y-axis perpendicular to the X-axis and pointing to the left side of the vehicle body as the Y-axis, a transformation model from the image coordinate system to the intelligent vehicle coordinate system is established. Based on the coordinates of two vertices on two L-shaped parking lines or the coordinates of two intersection points on two T-shaped parking lines obtained in step 3), p1(u1, v1) and p2(u2, v2), the coordinates are first distorted and normalized using the intrinsic parameter matrix and distortion coefficients of the visual sensor obtained in step 2), resulting in the normalized coordinates of these two vertices or intersection points in the image coordinate system:

[0041]

[0042] Using the coordinates (X1, Y1) and (X2, Y2) of the two vertices or intersection points in the intelligent vehicle coordinate system, the slope k of the parking line can be calculated:

[0043]

[0044] Wherein, (X, Y) is obtained from the image coordinate system to the intelligent vehicle coordinate system through coordinate transformation;

[0045] 5.2) Based on the principle of multi-vision sensor stereo vision, and combined with the intrinsic and extrinsic parameters of the vision sensors calibrated in step 2), the baseline distance of the vision sensor placed on one side of the vehicle body is determined to be B, and the focal length is f. If one vision sensor is placed on each of the left and right sides of the vehicle body, it is called using a panoramic binocular vision sensor. If two vision sensors are placed on one side of the vehicle body, it is called using two vision sensors that look forward or backward. Let the horizontal pixel coordinates of the same feature point in the two frames of images acquired by the two vision sensors be u and u, respectively. l and u rThen disparity disp = u l -u r Then the distance d between the parking space lines is:

[0046]

[0047] 5.3) In the intelligent vehicle coordinate system, the direction vector of the parking space line is (ΔX, ΔY), where ΔX represents the increment of the parking space line along the X-axis and ΔY represents the increment along the Y-axis. The angle between the direction vector of the parking space line and the X-axis in the intelligent vehicle coordinate system is taken as the intelligent vehicle angle θ, and the formula is:

[0048]

[0049] 5.4) Based on the obstacle location information output by the obstacle detection module, establish a location distribution map of obstacles in the intelligent vehicle coordinate system;

[0050] 5.5) The above parking space line slope k, parking space line distance d, intelligent vehicle angle θ, and obstacle location distribution map are used as parking space parameters and transmitted to the path planning module via the information processor.

[0051] In step 6), the method by which the path planning module performs path planning based on the above parking space parameters, obtains the path point sequence P, and transmits it to the motion execution module via the information processor is as follows:

[0052] 6.1) The A* algorithm module generates an initial parking path based on the parking line slope k, parking line distance d, intelligent vehicle angle θ, and obstacle location distribution map obtained in step 5).

[0053] 6.2) Based on the original parking space images acquired in real time by the vision sensor, the path planning dynamic adjustment module updates the obstacle positions and parking line status, and optimizes the initial parking path in real time, generating an optimized path point sequence P = {p1, p2, ..., p...} n The information is then transmitted to the motion execution module via the information processor.

[0054] In step 7), the method by which the motion execution module performs motion control and wheel drive on the intelligent vehicle based on the above path point sequence P is as follows:

[0055] 7.1) The motion controller reads the above path point sequence P and the acceleration a and angular velocity ω of the intelligent vehicle measured by the on-board inertial measurement unit on the intelligent vehicle;

[0056] 7.2) The displacement sequence s is obtained by integrating the acceleration a twice, and the voltage value or pulse number returned by the encoder on the intelligent vehicle is used for auxiliary calibration to obtain the accurate displacement sequence s. cal ;

[0057] 7.3) Set a fixed-length time window for the accurate displacement sequence s cal Perform moving average calculation to obtain the smoothed displacement sequence s smooth ;

[0058] 7.4) Smooth the displacement sequence s smooth The average speed v of the intelligent vehicle is obtained by dividing the difference between adjacent smoothed displacement values ​​by the time interval Δt.

[0059]

[0060] Among them, S smooth，i Let s be the smoothed displacement value at time i. smooth，i+1 This represents the smoothed displacement value at time i+1;

[0061] 7.5) Based on the average speed v, path point sequence P, and obstacle information detected in real time by the vision sensor of the intelligent vehicle, the steering angle control quantity, vehicle speed adjustment command, and gear shift signal are generated as control commands and transmitted to the wheel drive device.

[0062] 7.6) Upon receiving the above control commands, the wheel drive unit drives the intelligent vehicle to travel along the planned path and avoids dynamic obstacles in real time.

[0063] In step 8), the information processor calculates the timing for turning the wheel and issues a command. The motion execution module controls the intelligent vehicle to perform the turning operation at the designated point according to the command.

[0064] 8.1) The information processor combines IMU or encoder data with data detected by the vision sensor to calculate the intelligent vehicle's own position and the timing for turning the wheel when it reaches the designated point, and then issues the command.

[0065] 8.2) After receiving the instruction, the motion controller controls the direction and angle of the intelligent vehicle's steering wheel based on the parking space line distance d and the intelligent vehicle angle θ obtained in step 6).

[0066] 8.3) When an obstacle is detected, the motion controller controls the intelligent vehicle to automatically fine-tune the wheel angle to avoid the obstacle;

[0067] 8.4) The wheel drive unit drives the wheels to rotate according to the instructions of the information processor, so as to achieve the turning of the wheel at the designated time.

[0068] In step 9), the motion execution module controls the intelligent vehicle to accurately enter the parking space according to the planned path and the above-mentioned wheel-turning operation, thereby completing the automatic parking method:

[0069] The motion controller adjusts the wheel angles based on the slope k and distance d of the parking space lines. When the intelligent vehicle reaches the target parking space, the motion controller issues a parking command, and the wheel drive unit performs braking, thereby completing automatic parking.

[0070] The intelligent vehicle side-entry parking method based on multi-visual sensor information fusion provided by this invention has the following beneficial effects:

[0071] This invention provides intelligent vehicles with a clear four-way field of vision, making their driving trajectory more accurate and improving the accuracy of side-parking maneuvers. It improves trajectory prediction and parking space boundary detection technologies by designing an adaptive feature fusion module based on deep learning detection algorithms for real-time detection of parking lines and obstacles. This invention features a novel visual architecture that balances real-time performance, robustness, and scene generalization capabilities. Building upon existing deep learning detection algorithms, it possesses adaptive detection capabilities, improving target detection accuracy and enhancing the accuracy of side-parking maneuvers for intelligent vehicles.

[0072] This system utilizes a deep learning algorithm that incorporates a lightweight attention mechanism for deep network training. Before formal application, the algorithm undergoes multiple rounds of iterative training to comprehensively evaluate its feasibility and effectiveness. This significantly improves the model's performance in object detection tasks, enabling accurate identification of various targets while optimizing detection speed and resource consumption. It also demonstrates excellent object detection performance under conditions such as low light, rain, and partial occlusion. This novel visual algorithm allows intelligent vehicles to identify obstacles or other unexpected situations during side-parking maneuvers, dynamically adjusting their parking strategy in conjunction with motion control. Deep learning-based object detection algorithms improve the accuracy of image information and enhance the performance of the autonomous side-parking system for intelligent vehicles. Attached Figure Description

[0073] Figure 1 This is a structural diagram of the intelligent vehicle side-entry system provided by the present invention.

[0074] Figure 2 This is a flowchart illustrating the visual analysis processing of the image processing module and the parking space detection module in this invention.

[0075] Figure 3 This is a structural diagram of the AFFM module in the parking space detection module of the present invention.

[0076] Figure 4 (a) and (b) are flowcharts of the local attention branch and global attention branch processing procedures in the AFFM module of this invention, respectively.

[0077] Figure 5 Preprocessing of parking space images before visual analysis.

[0078] Figure 6 The parking space image was processed for visual analysis.

[0079] Figure 7 This is a flowchart of the obstacle detection module in this invention.

[0080] Figure 8 This is the original parking space image used in this invention.

[0081] Figure 9 This is an image of a parking space after obstacles have been marked in this invention.

[0082] Figure 10 This is a flowchart of the adaptive channel attention module processing in this invention.

[0083] Figure 11 The flowchart of the intelligent vehicle side-entry method provided by the present invention. Detailed Implementation

[0084] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0085] like Figure 11 As shown, the intelligent vehicle side-entry parking method based on multi-vision sensor information fusion provided by the present invention includes the following steps performed in sequence:

[0086] 1) Constructing such Figure 1 The intelligent vehicle side-entry system shown is installed on the intelligent vehicle and includes an image processing module 1, a parking space detection module 2, an obstacle detection module 3, an information processor 4, a motion execution module 5, and a path planning module 6. The image processing module 1 is electrically connected to the parking space detection module 3 and the obstacle detection module 3, respectively. The information processor 4 is data connected to the parking space detection module 2 and the obstacle detection module 3, and is also electrically connected to the motion execution module 5, the path planning module 6, and the onboard inertial measurement unit (IMU) and encoder on the intelligent vehicle.

[0087] The image processing module 1 includes a sensor calibration module 1-1, an image distortion correction module 1-2, and a multi-vision sensor image acquisition module 1-3 connected in sequence; the multi-vision sensor image acquisition module 1-3 includes vision sensors respectively installed in the upper part of the front windshield of the intelligent vehicle, one above the front license plate, one to two on the left and right sides, one in the upper part of the rear windshield, and one above the rear license plate.

[0088] The parking space detection module 2 comprises a multi-path image preprocessing module 2-1, a backbone network feature extraction module 2-2, an adaptive feature fusion module 2-3, and a parking line regression network module 2-4, connected in sequence. It adds an adaptive feature fusion module 2-3 between the backbone network feature extraction module 2-2 and the parking line regression network module 2-4 of the original DMPR-PS model, and is pre-built using a publicly available parking line dataset through multiple rounds of iterative training. The adaptive feature fusion module 2-3 improves parking space detection performance under conditions such as low light, rain, and partial occlusion. This module focuses on the synergistic effect of global and local attention, adaptively aggregating global and local features, enabling the model to focus on key areas of the target (such as edges and textures) and weaken background interference areas, thereby improving detection performance.

[0089] The obstacle detection module 3 includes a backbone feature extraction module 3-1, a neck multi-scale feature fusion module 3-2, an obstacle detection head 3-3, and a post-processing module 3-4 connected in sequence. It is constructed in advance by performing multiple rounds of iterative training using a publicly available obstacle dataset. The obstacle dataset typically includes objects such as people, spherical roadblocks, warning posts, bicycles, stop signs, and cars.

[0090] Information processor 4 is used to process images and data and generate control commands;

[0091] The motion execution module 5 includes a motion controller 5-1 and a wheel drive device 5-2 connected in sequence, and the wheel drive device 5-2 is mechanically connected to the wheels of the intelligent vehicle.

[0092] The path planning module 6 includes the A* algorithm module 6-1 and the path planning dynamic adjustment module 6-2, which are connected in sequence.

[0093] 2) such as Figure 2 As shown, the visual sensor in the image processing module 1 is calibrated and distortion corrected. Then, the visual sensor is used to collect the original parking space images of the surrounding environment of the intelligent vehicle in real time and transmit them to the parking space detection module 2 and the obstacle detection module 3 respectively.

[0094] 2.1) Multiple vision sensors in the image acquisition module 1-3 acquire images of the calibration board, and then the sensor calibration module 1-1 calibrates the above calibration board images using the Zhang Zhengyou calibration method to calculate the intrinsic and extrinsic parameter matrix and distortion coefficient of the vision sensor.

[0095] 2.2) The image distortion correction module 1-2 uses the above-mentioned intrinsic parameter matrix and distortion coefficient to perform distortion correction on the vision sensor; this makes the parking space image collected by the vision sensor closer to the real situation, effectively reduces the impact of distortion on parking space detection, and facilitates accurate detection of parking space boundaries and intelligent vehicle position.

[0096] 2.3) During the parallel parking process of the intelligent vehicle, the distortion-corrected front, rear, left, and right multi-channel visual sensors in image acquisition modules 1-3 are used to acquire real-time original parking space images of the surrounding environment of the intelligent vehicle. raw ∈R 3×H×W The data is then transmitted to the parking space detection module 2 and the obstacle detection module 3, respectively; where H and W represent the height and width of the original parking space image, respectively.

[0097] 3) The original parking space image is processed by the parking space detection module 2 to obtain a preprocessed parking space image. Then, the parking line is detected on the preprocessed parking space image to obtain the position of the parking line in the preprocessed parking space image and transmit it to the information processor 4.

[0098] 3.1) The multi-channel image preprocessing module 2-1 in the parking space detection module 2 processes the original parking space image p based on the extrinsic parameter matrix of the visual sensor obtained in step 2-1). raw ∈R 3×H×W To obtain such by performing reverse perspective processing Figure 5 The preprocessed parking space image p shown rgb ∈R 3×H×W ;

[0099] 3.2) The backbone network feature extraction module 2-2 processes the above preprocessed parking space image p rgb ∈R 3×H×W Perform feature extraction to obtain feature maps x∈R C×H×W ;

[0100] 3.3) The adaptive feature fusion module 2-3 processes the above feature map x∈R C×H×W Perform feature enhancement to obtain the enhanced feature map x. F ∈R C×H×W ;

[0101] like Figure 3 As shown, the Adaptive Feature Fusion Module (AFFM) 2-2 includes a local attention branch, a global attention branch, a 1×1 convolutional layer (Conv1×1), a batch normalization layer (BN), and a ReLU activation function; wherein the local attention branch and the global attention branch are set in parallel, followed by a cascaded 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function;

[0102] First, the feature map x∈R output by the backbone network feature extraction module 2-2 is processed by the local attention branch and the global attention branch, respectively. C×H×W Local attention φ (generated by convolution) and global attention σ (generated by self-attention) are calculated to obtain the local attention map x.L ∈R C×H×W and global attention graph x G ∈R C×H×W Then, the local attention map x L ∈R C ×H×W and global attention graph x G ∈R C×H×W The total feature map x is obtained by concatenating the features along the dimensional lines. S ∈R 2C×H×W Then, the enhanced feature map x is generated after processing through a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function. F ∈R C×H×W This can be expressed as a formula:

[0103]

[0104] x G =σ(x)

[0105] x S =Concat([x L ,x G ])

[0106] x F =ReLU(BN(Conv1×1(x S )))

[0107] Where Concat represents the concatenation operation, Conv1×1 represents a convolutional layer with a kernel size of 1, a stride of 1, and padding of 0, BN represents a batch normalization layer, and ReLU represents the ReLU activation function;

[0108] like Figure 4 As shown in (a), the local attention branch includes parallel 3×3 and 5×5 convolutional layers, followed by batch normalization layers and a ReLU activation function. It first uses 3×3 and 5×5 convolutional layers respectively for feature extraction to increase the receptive field. Then, the extracted feature maps are concatenated and processed by batch normalization layers and a ReLU activation function to obtain the final local attention map, expressed by the formula:

[0109]

[0110] Wherein, Conv3×3 represents a convolutional layer with a kernel size of 3, a stride of 1, padding of 1, and a group size of input channel number / 2, and Conv5×5 represents a convolutional layer with a kernel size of 5, a stride of 1, padding of 2, and a group size of input channel number / 2;

[0111] like Figure 4As shown in (b), the global attention branch includes cascaded deep convolutional layers, group normalization layers, multi-head self-attention mechanism layers, group normalization layers, and multilayer perceptrons; it utilizes the deep (Dw) convolutional layers as input feature maps x∈R. C×H×W The location embedding is processed through a group normalization (GN) layer, then fed into a multi-head self-attention (MHSA) layer to extract global information, followed by another group normalization layer, and finally a multilayer perceptron (MLP) to generate a global attention map, expressed by the formula:

[0112]

[0113] in, To represent element-wise addition, in order to reduce the amount of computation, the multilayer perceptron is constructed using two 1×1 convolutional layers and the ReLU activation function. After the first convolutional layer, the number of channels is changed to r (usually 4) times the original number, and then the second convolutional layer is used to restore the number of channels.

[0114] 3.4) The enhanced feature map x is processed by the parking line regression network module 2-4. F ∈R C×H×W Processing is performed to obtain a preprocessed parking space image p rgb ∈R 3×H×W The coordinate vector (x, y, s, β) of each parking space line marker is given, where (x, y) represents the position of the parking space line; s is a binary value representing the shape of the parking space line pattern, where 0 indicates a "T" shape and 1 indicates an "L" shape; and β represents the azimuth coordinate of the parking space line pattern in polar coordinates. Based on this information, the preprocessed parking space image p can be processed. rgb ∈R 3×H×W Fit the parking space lines in the image and mark them, such as... Figure 6 As shown.

[0115] Comparative observation Figure 5 and Figure 6 After visual analysis, the pre-processed parking space image p can be automatically labeled. rgb ∈R 3 ×H×W The parking space lines are identified to determine their location, shape, and size, which facilitates subsequent intelligent vehicle motion control.

[0116] 4) The obstacle detection module 3 detects obstacles in the original parking space image, obtains the obstacle position coordinates, and transmits them to the information processor 4;

[0117] The obstacle detection module 3 employs an improved YOLOX (You Only Look Once X) model. While the original YOLOX model uses the Darknet53 network for image feature extraction, this invention replaces the Darknet53 network with the lightweight MobileNetV3 network to address real-time performance concerns. To enhance obstacle detection performance without sacrificing real-time accuracy, a lightweight Adaptive Channel Attention (ACA) module is designed to replace the SE attention module in MobileNetV3. This ACA module decomposes channel attention into two one-dimensional feature codes, aggregating features from both spatial directions before fusing them. This encodes positional information into the channel attention, enhancing the target representation and improving detection performance.

[0118] like Figure 7 As shown, the method is as follows:

[0119] 4.1) The backbone feature extraction module 3-1 in the obstacle detection module 3 performs the following steps: Figure 8 The original parking space image p shown raw ∈R 3×H×W Feature extraction is performed to obtain three different output tensors β1, β2, and β3.

[0120] 4.2) The neck multi-scale feature fusion module 3-2 performs multi-scale feature extraction on the above output tensors β1, β2, and β3 to obtain three enhanced feature maps β. 1f ,β 2f ,β 3f ;

[0121] 4.3) The enhanced feature map β above is processed by obstacle detection head 3-3. 1f ,β 2f ,β 3f Process the data to obtain candidate bounding boxes for all obstacles;

[0122] 4.4) The post-processing module 3-4 filters the above candidate bounding boxes using confidence filtering and non-maximum suppression methods to obtain the original parking space image p. raw ∈R 3×H×W The coordinate vector (x1, y1, x2, y2, class_id, confidence) of each detected obstacle is generated, where (x1, y1) and (x2, y2) represent the top-left and bottom-right coordinates of the candidate bounding box of the obstacle, respectively; class_id represents the category of the obstacle; and confidence represents the confidence level of the category. Based on this information, the original parking space image p can be analyzed. raw∈R 3×H×W Mark the obstacles in the map, such as Figure 9 As shown.

[0123] like Figure 10 As shown, for the input feature map x s Each channel in the adaptive channel attention module is encoded using spatial range pooling kernels in both horizontal and vertical directions. The horizontal direction uses a (H,1) kernel along the coordinate axis, and the vertical direction uses a (1,W) kernel along the coordinate axis. Therefore, the encoded tensor output by the c-th channel at height h is represented as:

[0124]

[0125] Similarly, the encoded tensor output by the c-th channel at width w is represented as:

[0126]

[0127] Subsequently, the two encoded tensors are processed by group normalization and depthwise convolution, and then matrix multiplication is performed to achieve information fusion in two spatial directions. The mathematical expression is as follows:

[0128] z h =DwConv(GN 1 (z h ))

[0129] z w =DwConv(GN 1 (z w ))

[0130] z f =MatMul(z h ,z w )

[0131] Where MatMul(·) denotes matrix multiplication, GN 1 (·) indicates grouping normalization for groups of 1;

[0132] Finally, after processing through another group normalization layer, ReLU activation function, and deep convolutional layer, the channel attention map z is obtained. att ∈R C×1×1 The formula is as follows:

[0133]

[0134] in, This represents the ReLU activation function.

[0135] 5) The information processor 4 processes the parking line position obtained in step 3) and the obstacle position coordinates obtained in step 4) and calculates the parking parameters.

[0136] 5.1) Taking the center of the intelligent vehicle as the origin, the direction of the intelligent vehicle's movement as the X-axis, and the Y-axis perpendicular to the X-axis and pointing to the left side of the vehicle body as the Y-axis, a transformation model from the image coordinate system to the intelligent vehicle coordinate system is established. Based on the coordinates of two vertices on two L-shaped parking lines or the coordinates of two intersection points on two T-shaped parking lines obtained in step 3), p1(u1, v1) and p2(u2, v2), the coordinates are first distorted and normalized using the intrinsic parameter matrix and distortion coefficients of the visual sensor obtained in step 2), resulting in the normalized coordinates of these two vertices or intersection points in the image coordinate system:

[0137]

[0138] Using the coordinates (X1, Y1) and (X2, Y2) of the two vertices or intersection points in the intelligent vehicle coordinate system, the slope k of the parking line can be calculated:

[0139]

[0140] Among them, (X,Y) is obtained from the image coordinate system to the intelligent vehicle coordinate system through coordinate transformation;

[0141] 5.2) Based on the principle of multi-vision sensor stereo vision, and combined with the intrinsic and extrinsic parameters of the vision sensors calibrated in step 2), the baseline distance of the vision sensor placed on one side of the vehicle body is determined to be B, and the focal length is f. If one vision sensor is placed on each of the left and right sides of the vehicle body, it is called using a panoramic binocular vision sensor. If two vision sensors are placed on one side of the vehicle body, it is called using two vision sensors that look forward or backward. Let the horizontal pixel coordinates of the same feature point in the two frames of images acquired by the two vision sensors be u and u, respectively. l and u r Then disparity disp = u l -u r Then the distance d between the parking space lines is:

[0142]

[0143] 5.3) In the intelligent vehicle coordinate system, the direction vector of the parking space line is (ΔX, ΔY), where ΔX represents the increment of the parking space line along the X-axis and ΔY represents the increment along the Y-axis. The angle between the direction vector of the parking space line and the X-axis in the intelligent vehicle coordinate system is taken as the intelligent vehicle angle θ, and the formula is:

[0144]

[0145] By fusing the geometric feature information (such as the perimeter, area, and rectangularity of parking lines) in the images of the intelligent vehicle's surrounding environment acquired by the visual sensors with the measurement data from the onboard inertial measurement unit (IMU) of the intelligent vehicle using Kalman filtering, the deflection relationship of the parking lines relative to the wheels can be reflected more accurately, providing key geometric constraints for path planning.

[0146] 5.4) Based on the obstacle location information output by obstacle detection module 3, establish a location distribution map of obstacles in the intelligent vehicle coordinate system;

[0147] 5.5) The above parking space line slope k, parking space line distance d, intelligent vehicle angle θ, and obstacle location distribution map are used as parking space parameters and transmitted to the path planning module 6 via the information processor 4.

[0148] 6) The path planning module 6 performs path planning based on the above parking space parameters, obtains the path point sequence P, and transmits it to the motion execution module 5 via the information processor 4;

[0149] 6.1) The A* algorithm module 6-1 generates an initial parking path based on the parking line slope k, parking line distance d, intelligent vehicle angle θ, and obstacle location distribution map obtained in step 5).

[0150] 6.2) Based on the original parking space images acquired in real time by the vision sensor, the path planning dynamic adjustment module 6-2 updates the obstacle positions and parking line status, and optimizes the initial parking path in real time, generating an optimized path point sequence P = {p1, p2, ..., p...} n The information is then transmitted to the motion execution module 5 via the information processor 4.

[0151] 7) Motion execution module 5 performs motion control and wheel drive on the intelligent vehicle based on the above path point sequence P;

[0152] 7.1) The motion controller 5-1 reads the above path point sequence P and the acceleration a and angular velocity ω of the intelligent vehicle measured by the on-board inertial measurement unit on the intelligent vehicle;

[0153] 7.2) The displacement sequence s is obtained by integrating the acceleration a twice, and the voltage value or pulse number returned by the encoder on the intelligent vehicle is used for auxiliary calibration to obtain the accurate displacement sequence s. cal ;

[0154] 7.3) Set a fixed-length time window for the accurate displacement sequence s cal Perform moving average calculation to obtain the smoothed displacement sequence s smooth ;

[0155] 7.4) Smooth the displacement sequence s smooth The average speed v of the intelligent vehicle is obtained by dividing the difference between adjacent smoothed displacement values ​​by the time interval Δt.

[0156]

[0157] Among them, S smooth,i Let S be the smooth displacement value at time i. smooth,i+1 This represents the smoothed displacement value at time i+1;

[0158] 7.5) Based on the average speed v, path point sequence P, and obstacle information detected in real time by the vision sensor of the intelligent vehicle, the steering angle control quantity, vehicle speed adjustment command, and gear shift signal are generated as control commands and transmitted to the wheel drive device 5-2.

[0159] 7.6) The wheel drive unit 5-2 receives the above control command, drives the intelligent vehicle to travel along the planned path, and avoids dynamic obstacles in real time.

[0160] 8) The information processor 4 calculates the timing of the wheel turn and issues a command. The motion execution module 5 controls the intelligent vehicle to perform the wheel turn operation at the designated point according to the command.

[0161] 8.1) The information processor 4 combines IMU or encoder data with data detected by the vision sensor to calculate the intelligent vehicle's own position and the timing for turning the wheel when it reaches the designated point, and then issues a command.

[0162] 8.2) After receiving the instruction, the motion controller 5-1 controls the direction and angle of the intelligent vehicle's steering wheel according to the parking space line distance d and the intelligent vehicle angle θ obtained in step 6);

[0163] 8.3) When an obstacle is detected, the motion controller 5-1 controls the intelligent vehicle to automatically fine-tune the wheel angle to avoid the obstacle;

[0164] 8.4) The wheel drive device 5-2 drives the wheels to rotate according to the instructions of the information processor 4, so as to achieve the turning of the wheel at the designated time.

[0165] 9) The motion execution module 5 controls the intelligent vehicle to accurately enter the parking space according to the planned path and the above-mentioned wheel operation, thereby completing the automatic parking;

[0166] The motion controller 5-1 adjusts the wheel angle according to the slope k and distance d of the parking space line to ensure that the wheel turning action is completed in complex layouts. When the intelligent vehicle reaches the target parking space, the motion controller 5-1 issues a parking command and the wheel drive device 5-2 performs braking, thereby completing automatic parking.

Claims

1. A multi-vision sensor information fusion intelligent vehicle side entry warehouse method, characterized in that: The multi-vision sensor information fusion intelligent vehicle side parking method comprises the following steps in sequence: 1) Construct an intelligent vehicle side parking system; the intelligent vehicle side parking system is arranged on an intelligent vehicle and comprises an image processing module (1), a parking space detection module (2), an obstacle detection module (3), an information processor (4), a motion execution module (5) and a path planning module (6); the image processing module (1) is electrically connected with the parking space detection module (3) and the obstacle detection module (3) respectively; the information processor (4) is data-connected with the parking space detection module (2) and the obstacle detection module (3) respectively, and is electrically connected with the motion execution module (5), the path planning module (6) and an on-board inertial measurement unit and an encoder on the intelligent vehicle; 2) Calibrate and correct distortion of the vision sensor in the image processing module (1), then use the vision sensor to collect original parking space images of the environment around the intelligent vehicle in real time and transmit the original parking space images to the parking space detection module (2) and the obstacle detection module (3) respectively; 3) Use the parking space detection module (2) to perform inverse perspective processing on the original parking space images to obtain preprocessed parking space images, then perform parking space line detection on the preprocessed parking space images to obtain parking space line positions in the preprocessed parking space images and transmit the parking space line positions to the information processor (4); 4) Use the obstacle detection module (3) to detect obstacles in the original parking space images to obtain obstacle position coordinates and transmit the obstacle position coordinates to the information processor (4); 5) Use the information processor (4) to perform information processing and parking space parameter calculation on the parking space line positions obtained in step 3) and the obstacle position coordinates obtained in step 4); 6) Use the path planning module (6) to perform path planning according to the parking space parameters to obtain a path point sequence P and transmit the path point sequence P to the motion execution module (5) through the information processor (4); 7) Use the motion execution module (5) to perform motion control and wheel driving on the intelligent vehicle based on the path point sequence P; 8) Use the information processor (4) to perform round calculation and issue an instruction, and use the motion execution module (5) to control the intelligent vehicle to perform a point round operation according to the instruction; 9) Use the motion execution module (5) to control the intelligent vehicle to accurately park into a parking space according to the planned path and the round operation, thereby completing automatic parking. 2.The intelligent vehicle lateral warehouse entry method based on multi-vision sensor information fusion according to claim 1, characterized in that: In step 1), the image processing module (1) comprises a sensor calibration module (1-1), an image distortion correction module (1-2) and a multi-vision sensor image acquisition module (1-3) connected in sequence; the multi-vision sensor image acquisition module (1-3) comprises vision sensors respectively installed on the upper middle part of a front windshield, the upper side of a front license plate, the left and right sides of the intelligent vehicle, the upper middle part of a rear windshield and the upper side of a rear license plate; The parking space detection module (2) comprises a multi-path image preprocessing module (2-1), a backbone network feature extraction module (2-2), an adaptive feature fusion module (2-3) and a parking space line regression network module (2-4) connected in sequence, and is constructed by using a public parking space line data set for multiple rounds of iterative training in advance; The obstacle detection module (3) comprises a backbone feature extraction module (3-1), a neck multi-scale feature fusion module (3-2), an obstacle detection head (3-3) and a post-processing module (3-4) connected in sequence, and is constructed by using the disclosed obstacle data set for multiple rounds of iterative training in advance; The information processor (4) is used for processing images and data to generate control instructions; The motion execution module (5) comprises a motion controller (5-1) and a wheel driving device (5-2) connected in sequence, and the wheel driving device (5-2) is mechanically connected with the wheels of the intelligent vehicle; The path planning module (6) comprises an A* algorithm module (6-1) and a path planning dynamic adjustment module (6-2) connected in sequence. 3.The intelligent vehicle lateral parking method based on multi-vision sensor information fusion according to claim 2, characterized in that: In step 2), the method for calibrating and correcting distortion of the visual sensor in the image processing module (1), then using the visual sensor to collect the original parking space image of the surrounding environment of the intelligent vehicle in real time and transmitting it to the parking space detection module (2) and the obstacle detection module (3) is: 2.1) using the multiple visual sensors in the image acquisition module (1-3) to collect the calibration board image, then using the Zhang Zhengyou calibration method in the sensor calibration module (1-1) to calibrate the above calibration board image, and calculating the internal and external parameter matrices and distortion coefficients of the visual sensor; 2.2) using the internal parameter matrix and distortion coefficients to correct the distortion of the visual sensor by the image distortion correction module (1-2); 2.3) In the process of lateral parking of the intelligent vehicle, the original parking space image p of the surrounding environment of the intelligent vehicle is collected in real time by using the front, rear, left and right multi-channel visual sensors after distortion correction in the image acquisition module (1-3) raw ∈R 3×H×W and respectively transmitted to the parking space detection module (2) and the obstacle detection module (3); wherein H and W represent the height and width of the original parking space image respectively.

4. The intelligent vehicle lateral parking method of multi-vision sensor information fusion according to claim 2, characterized in that: In step 3), the method for using the parking space detection module (2) to perform inverse perspective processing on the above original parking space image to obtain a preprocessed parking space image, then performing parking space line detection on the preprocessed parking space image to obtain the parking space line position in the preprocessed parking space image and transmitting it to the information processor (4) is: 3.1) Obtain the extrinsic matrix of the vision sensor from the raw parking space image p raw ∈R 3×H×W by the multi-path image preprocessing module (2-1) in the parking space detection module (2) according to step 2-1) rgb ∈R 3×H×W ; 3.2) Feature extraction by backbone network feature extraction module (2-2) on the above pre-processed parking space image p rgb ∈R 3×H×W Feature extraction is performed to obtain feature map x∈R C×H×W ; 3.3) Feature enhancement by the adaptive feature fusion module (2-3) on the above feature map x e R C×H×W to obtain the enhanced feature map x F e R C×H×W ; The adaptive feature fusion module (2-2) comprises a local attention branch, a global attention branch, a 1×1 convolution layer, a batch normalization layer and a ReLU activation function; The local attention branch and the global attention branch are arranged in parallel, followed by a cascaded 1×1 convolution layer, a batch normalization layer and a ReLU activation function; The local attention branch comprises a 3×3 convolution layer and a 5×5 convolution layer arranged in parallel, followed by a batch normalization layer and a ReLU activation function; The global attention branch comprises a deep convolution layer, a group normalization layer, a multi-head self-attention mechanism layer, a group normalization layer and a multi-layer perception; 3.4) The enhanced feature map x F ∈R C×H×W is processed by the parking space line regression network module (2-4) to obtain the pre-processed parking space image p rgb ∈R 3×H×W The coordinate vector (x, y, s, β) of each parking space line marker point is obtained, where (x, y) represents the position of the parking space line; s is a binary value representing the shape of the parking space line pattern, where 0 represents that the shape of the parking space line pattern is "T", and 1 represents that the shape of the parking space line pattern is "L"; and β represents the azimuth coordinate of the parking space line pattern in the polar coordinate system.

5. The intelligent vehicle lateral in-lot method of multi-vision sensor information fusion according to claim 2, characterized in that: In step 4), the method for detecting the obstacle in the original parking space image by the obstacle detection module (3) to obtain the obstacle position coordinates and transmitting them to the information processor (4) is: 4.1) performing feature extraction by the backbone feature extraction module (3-1) in the obstacle detection module (3) on the original parking space image p raw ∈R 3×H×W to obtain three different output tensors β1, β2, β3; 4.2) Multi-scale feature extraction is performed on the above output tensors β1, β2, β3 by the neck multi-scale feature fusion module (3-2) to obtain three enhanced feature maps β 1f , β 2f , β 3f ; 4.3) The enhanced feature map β 1f , β 2f , β 3f is processed by the obstacle detection head (3-3) to obtain the candidate bounding boxes of all obstacles. 4.4) Filtering the above candidate bounding boxes by the post-processing module (3-4) through confidence filtering and non-maximum suppression method to obtain the original parking space image p raw ∈R 3×H×W The coordinate vector (x1, y1, x2, y2, class_id, confidence) of each obstacle detected in the image, wherein (x1, y1) and (x2, y2) represent the upper left corner coordinates and the lower right corner coordinates of the candidate bounding box of the obstacle respectively, class_id represents the category of the obstacle, and confidence represents the confidence of the category. 6.The intelligent vehicle lateral parking method based on multi-vision sensor information fusion according to claim 2, characterized in that: In step 5), the method for the information processor (4) to process the parking space line position obtained in step 3) and the obstacle position coordinates obtained in step 4) and calculate the parking space parameters is: 5.1) Take the center of the smart car body as the origin, the forward direction of the smart car as the X-axis, and the Y-axis as the vertical direction of the X-axis and pointing to the left side of the car body, to establish a conversion model of the image coordinate system to the smart car coordinate system. According to the coordinates of the two vertices on the two L-shaped parking space lines or the coordinates of the two intersection points on the two T-shaped parking space lines p1(u1, v1) and p2(u2, v2) obtained in step 3), first, the internal parameter matrix and distortion coefficient of the visual sensor obtained in step 2) are used to perform de-distortion and normalization on the above-mentioned coordinates, to obtain the normalized coordinates of the two vertices or intersection points in the image coordinate system: Through the coordinates (X1, Y1) and (X2, Y2) of the two vertices or intersection points in the smart car coordinate system, the slope k of the parking space line is calculated: Where (X, Y) is obtained by coordinate conversion from the image coordinate system to the smart car coordinate system; 5.2) based on the principle of multi-vision sensor stereo vision, combined with the step 2) calibrated vision sensor internal and external parameters, determine the baseline distance of the vision sensor placed on one side of the vehicle body is B, the focal length is f, when the left and right sides of the vehicle body are placed with one vision sensor, it is called panoramic binocular vision sensor, when two vision sensors are placed on one side of the vehicle body, it is called two forward-looking or backward-looking vision sensors; let the horizontal pixel coordinates of the same feature point in the two frames of images collected by the two vision sensors be u l and u r , then the parallax disp = u l -u r , then the parking line distance d is: 5.3) In the smart car coordinate system, the direction vector of the parking space line is (ΔX, ΔY), ΔX represents the increment of the parking space line along the X-axis direction, and ΔY represents the increment along the Y-axis direction of the parking space line. Then, the angle between the direction vector of the parking space line and the X-axis in the smart car coordinate system is taken as the smart car angle θ, and the formula is: 5.4) Combine the obstacle position information output by the obstacle detection module (3) to establish the position distribution diagram of the obstacle in the smart car coordinate system; 5.5) The above-mentioned parking space line slope k, parking space line distance d, smart car angle θ, and obstacle position distribution diagram are taken as parking space parameters and transmitted to the path planning module (6) through the information processor (4).

7. The intelligent vehicle lateral in-lot method of multi-vision sensor information fusion according to claim 2, characterized in that: In step 6), the path planning module (6) performs path planning according to the above-mentioned parking space parameters to obtain a path point sequence P and transmits it to the motion execution module (5) through the information processor (4). The method is: 6.1) The A* algorithm module (6-1) generates an initial parking path based on the parking space line slope k, parking space line distance d, smart car angle θ, and obstacle position distribution diagram obtained in step 5); 6.2) According to the real-time acquisition of the original parking space image by the visual sensor, the path planning dynamic adjustment module (6-2) updates the obstacle position and the parking line state, and optimizes the initial parking path in real time to generate the optimized path point sequence P={p1,p2,...,p n} and is transmitted to the motion execution module (5) through the information processor (4). 8.The intelligent vehicle lateral parking method based on multi-vision sensor information fusion according to claim 2, characterized in that: In step 7), the motion execution module (5) performs motion control and wheel driving of the smart car based on the above-mentioned path point sequence P. The method is: 7.1) The motion controller (5-1) reads the above-mentioned path point sequence P and the acceleration a and angular velocity ω of the smart car measured by the on-board inertial measurement unit of the smart car; 7.2) Get displacement sequence s by twice integration of acceleration a, and use the voltage value or pulse number returned by the encoder on the smart car to assist calibration, get accurate displacement sequence s cal ; 7.3) Set a fixed length time window, align the accurate displacement sequence s cal Perform a sliding average calculation to obtain a smoothed displacement sequence s smooth ; 7.4) The smooth displacement sequence s smooth The average speed v of the smart car is obtained by dividing the difference between the adjacent smooth displacement values by the time interval At. wherein S smooth，i is the smoothed displacement value at time i, S smooth，i+1 is the smoothed displacement value at time i+1. 7.5) According to the average speed v of the smart car, the path point sequence P, and the real-time obstacle information detected by the visual sensor, a steering angle control quantity, a vehicle speed adjustment instruction, and a gear shifting signal are generated as control instructions and transmitted to the wheel driving device (5-2); 7.6) The wheel driving device (5-2) receives the above-mentioned control instructions, drives the smart car to travel along the planned path, and avoids dynamic obstacles in real time. 9.The intelligent vehicle lateral parking method of multi-vision sensor information fusion according to claim 2, characterized in that: In step 8), the information processor (4) performs round calculation and issues instructions, and the motion execution module (5) controls the smart car to perform point round operation according to the instructions. The method is: 8.1) The information processor (4) combines the IMU or encoder data and the data detected by the visual sensor to calculate the position of the smart car and the timing of the point round and issue instructions; 8.2) After receiving the instruction, the motion controller (5-1) controls the direction and angle of the smart car's wheel according to the distance d and the angle θ of the smart car obtained in step 6); 8.3) When detecting obstacles, the motion controller (5-1) controls the smart car to automatically fine-tune the wheel angle to avoid obstacles; 8.4) The wheel driving device (5-2) drives the wheel to rotate according to the instruction of the information processor (4), realizing point wheeling.

10. The intelligent vehicle lateral in garage method of multi-vision sensor information fusion according to claim 2, characterized in that: In step 9), the motion execution module (5) controls the smart car to accurately enter the garage according to the planned path and the above wheeling operation, thereby completing the automatic parking method: The motion controller (5-1) adjusts the wheel angle according to the slope k and the distance d of the parking line; When the smart car reaches the target parking position, the motion controller (5-1) issues a stop instruction, and the wheel driving device (5-2) executes the brake, thereby completing the automatic parking.

Images