Steering assistance apparatus and method

Abstract

The invention belongs to the field of vehicle safety, and relates to a steering auxiliary device and method, which can accurately obtain a dangerous area of vehicle operation, analyze the condition of the dangerous area and provide early warning assistance for a driver. According to the technical scheme, the system comprises a fisheye camera, a first monocular camera, a second monocular camera, a microprocessor and a prompter. The fisheye camera is arranged at the uppermost end of the front end of the vehicle, and the view field of the fisheye camera comprises the vehicle and areas right ahead of dangerous areas on the two sides of the vehicle. The first monocular camera is arranged at the rear end of one side of the vehicle, the view field of the first monocular camera comprises an area from the upper edge of the vehicle to the rear edge of the last axle tire of the vehicle, and when the vehicle is at the maximum steering angle, the view field of the first monocular camera further comprises the edge of the side, away from the vehicle, of the side danger area. And the second monocular camera and the first monocular camera are symmetrically arranged.

Description

Technical Field

[0001] This invention belongs to the field of vehicle safety and relates to a steering assist device and method. Background Technology

[0002] When vehicles turn, especially large trucks and buses, due to their large size and length, there are blind spots. This can easily cause dangerous objects in the blind spots to be run over by the wheels. Dangerous objects include pedestrians, vehicles, and other obstacles.

[0003] Current technologies include installing multiple cameras on the upper part of the vehicle to create a full-view image of the vehicle through video stitching and displaying it in the driver's cabin. Alternatively, radar or other sensors can be used to determine the presence of hazards in blind spots, providing some assistance to the driver in operating the vehicle.

[0004] However, in most cases, the space for maneuver when turning is limited, especially for large trucks and buses, which often need to complete turning operations within a small space. Currently, the distortion in images captured by cameras becomes more pronounced closer to the vehicle, making it impossible for drivers to effectively judge the precise distance between hazard and the vehicle using a full-view image from inside the cab, nor can they accurately determine whether a hazard is on the vehicle's path. In other words, current camera images cannot effectively provide drivers with accurate and effective hazard assessment and warnings when turning. Radar, on the other hand, often suffers from inaccurate detection and wide detection angles, causing feedback information to interfere with the driver's judgment. Summary of the Invention

[0005] To overcome the shortcomings of the aforementioned related technologies, this invention proposes a steering assist device that can accurately acquire the dangerous zone of the vehicle's operation and analyze the situation of the dangerous zone to provide early warning assistance to the driver.

[0006] To achieve the above-mentioned technical objectives, the present invention provides a steering assist device. The steering assist device is installed on the vehicle, and multiple wheels of the vehicle are connected end-to-end to form a square area. When the vehicle is running, the area of ​​the square area covering the ground is a crushing zone, and the area of ​​the square area covering the ground from the starting point to when the vehicle is about to travel a first distance is a danger zone.

[0007] When the vehicle turns, the first set of wheels in the direction of vehicle travels to the ground covered by a first distance is the positive danger zone, and the other danger zones besides the positive danger zone are the side danger zones.

[0008] The steering assist device includes: a fisheye camera, a first monocular camera, a second monocular camera, a microprocessor, and a prompter. The fisheye camera is positioned at the top front of the vehicle, and its field of view includes the vehicle and the area directly in front of the side hazard zones. The first monocular camera is positioned at the rear end of one side of the vehicle, and its field of view includes the area from the upper edge of the vehicle to the rear edge of the last axle tire. At the vehicle's maximum steering angle, the first monocular camera's field of view also includes the edge of the side hazard zone furthest from the vehicle. The second monocular camera is positioned at the rear end of the other side of the vehicle, and its field of view includes the area from the upper edge of the vehicle to the rear edge of the last axle tire. At the vehicle's maximum steering angle, the second monocular camera's field of view also includes the edge of the side hazard zone furthest from the vehicle.

[0009] The microprocessor is electrically connected to the fisheye camera, the first monocular camera, and the second monocular camera. The indicator is electrically connected to the microprocessor and is fixed to the driver's cab of the vehicle. The indicator is configured to receive signals from the microprocessor and alert the driver of the risk of a run-over accident when the vehicle is turning.

[0010] Preferably, the steering assist device further includes a steering wheel angle sensor, which is fixed below the steering wheel of the vehicle and configured to detect the rotation angle of the steering wheel. The steering wheel angle sensor is electrically connected to the microprocessor.

[0011] On the other hand, the present invention also provides a steering assist method applicable to the above-mentioned steering assist device.

[0012] When the vehicle turns, the turning radius R1 of the first wheel on the inner side of the vehicle is the ratio of the distance L1 between the first wheel and the turning center of the wheel in the length direction of the vehicle to sinα.

[0013] The turning radius R2 of the last wheel inside the vehicle is the ratio of the distance L2 between the last wheel and the turning center of the wheel in the length direction of the vehicle to sinα.

[0014] The side danger zone inside the vehicle has a fan-shaped structure, and the width H of the side danger zone inside the vehicle is the difference between the turning radius R1 of the first wheel and the turning radius R2 of the last wheel.

[0015] The edge of the side hazard zone on the outer side of the vehicle includes: a surface region formed by an arc passing through the last wheel and a line segment connecting several tires on the other side of the vehicle, with the steering center as the center point.

[0016] Wherein, the steering angle α is the angle at which the vehicle turns.

[0017] The steering assistance method includes: the fisheye camera acquiring images of the vehicle and the area directly in front of the hazard zones on both sides when the vehicle is at its maximum steering angle; the first monocular camera acquiring images of the lateral hazard zone on the inside of the vehicle when the vehicle is at its maximum steering angle; and the second monocular camera acquiring images of the lateral hazard zone on the outside of the vehicle when the vehicle is at its maximum steering angle.

[0018] The system acquires the vehicle's steering angle α when turning, calculates the range of the inner lateral hazard zone of the vehicle at the current steering angle α, and also calculates the range of the outer lateral hazard zone of the vehicle at the current steering angle α. It analyzes the image of the area directly in front of the vehicle captured by the fisheye camera to determine if any hazards exist. It analyzes the calculated image of the inner lateral hazard zone of the vehicle within the area directly in front of the two lateral hazard zones captured by the fisheye camera to determine if any hazards exist. It analyzes the image of the calculated inner lateral hazard zone of the vehicle captured by the first monocular camera to determine if any hazards exist. It analyzes the image of the calculated inner lateral hazard zone of the vehicle captured by the second monocular camera to determine if any hazards exist. If any hazards are found, a warning is issued to the driver of the vehicle.

[0019] Preferably, before analyzing the image in the area directly in front of the vehicle captured by the fisheye camera, the steering assist method further includes: performing orthogonal processing on the image captured by the fisheye camera, the image captured by the first monocular camera, and the image captured by the second monocular camera.

[0020] Before analyzing the image of the area directly in front of the vehicle captured by the fisheye camera, the steering assist method further includes:

[0021] The images captured by the fisheye camera, the first monocular camera, and the second monocular camera are subjected to orthogonal processing.

[0022] The method for orthogonalizing the images captured by the fisheye camera includes:

[0023] The world coordinate system coordinates are converted to the camera coordinate system coordinates through a rigid body transformation, wherein the rigid body transformation satisfies:

[0024]

[0025] in, Using the world coordinate system; For camera coordinate system coordinates, For rotation matrix, It is a translation vector;

[0026] The camera coordinate system coordinates are converted to image physical coordinates, and then to pixel coordinates. The conversion process is achieved using the following formula:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] in, This represents the distance from a pixel to the image center in the absence of distortion. For the corresponding angle, Let x and y be the components of the pixel length in the x and y directions of the photosensitive plane, respectively. cx and cy are the coordinates of the center of the photosensitive plane in the pixel coordinate system, and (u, v) are the pixel coordinates.

[0033] Preferably, the calibration process of the fisheye camera module includes:

[0034] Images of the calibration board under different postures are acquired, the images are binarized, and the pixel coordinates of the checkerboard corners are obtained through a corner detection algorithm.

[0035] Set the top left corner of the calibration plate as the origin of its own world coordinate system, and determine the world coordinates of the other corner points;

[0036] Based on the pixel coordinates and world coordinates of the corner points, the intrinsic parameter matrix and distortion parameters of the fisheye camera are obtained. The intrinsic parameter matrix includes the principal point coordinates and focal length. The corner detection algorithm is selected from the contour curve detection algorithm, the binary image detection algorithm, or the grayscale image detection algorithm.

[0037] Preferably, the calibration and ranging process of the first monocular camera and the second monocular camera includes:

[0038] The Zhang calibration method is used to capture multiple images by changing the chessboard grid posture to obtain the intrinsic and extrinsic parameters of the binocular camera. The intrinsic parameters include focal length and principal point coordinates, and the extrinsic parameters include rotation matrix and translation vector.

[0039] The left and right views are stereoscopically corrected using the aforementioned intrinsic and extrinsic parameters so that the image matching points are located on the same epipolar line;

[0040] The corrected image pairs are processed using a stereo matching algorithm to calculate the disparity value of each pixel. The stereo matching algorithm is selected from local stereo matching algorithm, global stereo matching algorithm, semi-global stereo matching algorithm or deep learning-based stereo matching algorithm.

[0041] Based on the principle of triangulation, using the baseline length B, focal length f, and disparity value d, the formula is used... Calculate the spatial distance Z of the target point.

[0042] Preferably, the method for analyzing the images from the fisheye camera, the first monocular camera, and the second monocular camera includes:

[0043] Feature extraction is performed on the input image, and the features include histogram of oriented gradients, edge features, diagonal features, or deep features extracted based on convolutional neural networks;

[0044] Candidate boxes are generated based on the extracted features, and the targets within the candidate boxes are classified by a pre-trained classifier, which is selected from random forest, decision tree or deep learning-based classification model.

[0045] Non-maximum suppression technique is used to filter candidate boxes and determine the optimal position and category of the target.

[0046] Preferably, the deep learning process employs a single-stage detection algorithm or a two-stage detection algorithm;

[0047] The single-stage detection algorithm uses a convolutional network to perform feature extraction, classification, and localization, achieving end-to-end detection.

[0048] The two-stage detection algorithm first obtains candidate regions in the first stage, and then classifies and locates the candidate regions in the second stage.

[0049] Preferably, the single-stage detection algorithm employs Mosaic data augmentation during training, stitching together multiple images and inputting them into the neural network for learning;

[0050] The data augmentation also includes adding noise, rotation, scaling, or cropping.

[0051] The beneficial effects of this invention are as follows:

[0052] This invention employs a fisheye camera, a first monocular camera, and a second monocular camera, along with their installation positions, to encompass the dangerous areas when a vehicle is turning within the camera's field of view, providing a foundation for effective warnings when the vehicle is turning.

[0053] This invention uses a steering wheel angle sensor to obtain the vehicle's steering angle and calculate the range of the danger zone based on the vehicle's steering angle. It can provide accurate warnings and avoid issuing warnings to the driver when the vehicle is not in the danger zone, thus preventing interference with the driver's normal driving. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a structural diagram of the present invention;

[0056] Figure 2 This is a flowchart illustrating the operation process of the present invention;

[0057] Figure 3 This is a diagram illustrating the operation of the vehicle at different steering angles according to the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0060] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0061] like Figure 1 As shown, some embodiments of the present invention provide a steering assist device that can accurately acquire the dangerous zone where the vehicle is operating and analyze the situation in the dangerous zone to provide warning assistance to the driver.

[0062] To achieve the above-mentioned technical objectives, the present invention provides a steering assist device. The steering assist device is installed on the vehicle, and multiple wheels of the vehicle are connected end-to-end to form a square area. When the vehicle is running, the area of ​​the square area covering the ground is a crushing zone, and the area of ​​the square area covering the ground from the starting point to when the vehicle is about to travel a first distance is a danger zone.

[0063] When the vehicle turns, the first set of wheels in the direction of vehicle travels to the ground covered by a first distance is the positive danger zone, and the other danger zones besides the positive danger zone are the side danger zones.

[0064] The steering assist device includes: a fisheye camera 1, a first monocular camera 2, a second monocular camera 3, a microprocessor, and a prompter. The fisheye camera 1 is positioned at the top front of the vehicle, and its field of view includes the vehicle and the area directly in front of the side hazard zones. The first monocular camera 2 is positioned at the rear end of one side of the vehicle, and its field of view includes the area from the upper edge of the vehicle to the rear edge of the last axle tire. At the vehicle's maximum steering angle, the field of view of the first monocular camera 2 also includes the edge of the side hazard zone furthest from the vehicle. The second monocular camera 3 is positioned at the rear end of the other side of the vehicle, and its field of view includes the area from the upper edge of the vehicle to the rear edge of the last axle tire. At the vehicle's maximum steering angle, the field of view of the second monocular camera 3 also includes the edge of the side hazard zone furthest from the vehicle.

[0065] The microprocessor is electrically connected to the fisheye camera 1, the first monocular camera 2, and the second monocular camera 3. The indicator is electrically connected to the microprocessor and is fixed to the driver's cab of the vehicle. The indicator is configured to receive signals from the microprocessor and alert the driver of the risk of a run-over accident when the vehicle is turning.

[0066] In some examples, a fisheye camera 1 is fixed at the center of the top of the hood of a large truck. The fisheye camera 1 has a field of view of 180° and can capture images within a range of n meters to m meters in front of the hood, where n can be 0.5 and m can be 3, 4, or 5. A monocular camera is installed on each side of the truck bed, with each monocular camera positioned at a corresponding corner at the rear of the bed. The pitch angle of the monocular camera is such that its upper boundary can capture the upper edge of the vehicle, and its lower boundary can reach the rear edge of the last axle tire. This allows for a comprehensive view of the front side of the vehicle and also enables the assessment of whether the truck bed will collide with surrounding obstacles.

[0067] When a vehicle turns, especially a large vehicle, the turning radius of the rear wheels is smaller than that of the front wheels. This means that when the vehicle turns, the travel paths of the first and last axles on the inner side of the vehicle form a fan-shaped loop. The larger the turning angle, the wider this fan-shaped loop becomes. To ensure that the monocular camera can consistently capture the danger zones on both sides of the vehicle during turns, the monocular camera's field of view on the ground extends outwards from the side of the vehicle, and this outward distance is greater than or equal to the maximum width of the fan-shaped loop.

[0068] The maximum width of the fan ring is related to the vehicle's maximum turning radius and turning center.

[0069] This application can acquire images of the vehicle's danger zone. Through image analysis, it can be determined whether the vehicle will run over dangerous obstacles when turning. Dangerous obstacles may include pedestrians, curbs, walls, utility poles, etc.

[0070] Understandably, in practice, the wheels are often inside the carriage, so the danger zone can be set according to the last edge of the carriage.

[0071] In some embodiments, the steering assist device further includes a steering wheel angle sensor fixed below the steering wheel of the vehicle, the steering wheel angle sensor being configured to detect the rotation angle of the steering wheel, and the steering wheel angle sensor being electrically connected to the microprocessor.

[0072] In some examples, the extent of the danger zone changes as the vehicle turns. In order for the steering assist device described in this application to provide accurate warnings, the extent of the danger zone is calculated based on the vehicle's turning direction, and potential dangerous obstacles within the danger zone are identified.

[0073] In addition, the wheel steering angle can be obtained by installing an angle sensor on the steering wheel kingpin or a displacement sensor on the power steering cylinder. No specific limitation is made here regarding the acquisition of the wheel steering angle.

[0074] On the other hand, such as Figure 2 and Figure 3 As shown, the present invention also provides a steering assist method applicable to the above-mentioned steering assist device.

[0075] When the vehicle turns, the turning radius R1 of the first wheel on the inner side of the vehicle is the ratio of the distance L1 between the first wheel and the turning center of the wheel in the length direction of the vehicle to sinα.

[0076] The turning radius R2 of the last wheel inside the vehicle is the ratio of the distance L2 between the last wheel and the turning center of the wheel in the length direction of the vehicle to sinα.

[0077] The side danger zone inside the vehicle has a fan-shaped structure, and the width H of the side danger zone inside the vehicle is the difference between the turning radius R1 of the first wheel and the turning radius R2 of the last wheel.

[0078] The edge of the side hazard zone on the outer side of the vehicle includes: a surface region formed by an arc passing through the last wheel and a line segment connecting several tires on the other side of the vehicle, with the steering center as the center point.

[0079] Wherein, the steering angle α is the angle at which the vehicle turns.

[0080] The steering assistance method includes:

[0081] S1. The fisheye camera captures images of the vehicle and the area directly in front of the danger zones on both sides when the vehicle is at its maximum steering angle.

[0082] S2. The first monocular camera captures images of the side danger zone inside the vehicle when the vehicle is at its maximum steering angle.

[0083] S3. The second monocular camera captures images of the side danger zone on the outside of the vehicle when the vehicle is at its maximum steering angle.

[0084] S4. Obtain the steering angle α when the vehicle is turning, calculate the range of the side danger zone inside the vehicle under the current steering angle α, and calculate the range of the side danger zone outside the vehicle under the current steering angle α.

[0085] S5. Analyze the images of the area directly in front of the vehicle captured by the fisheye camera to determine whether there are any dangerous objects.

[0086] S6. Analyze and calculate the image of the side danger zone inside the vehicle in the area directly in front of the two side danger zones captured by the fisheye camera to determine whether there are any dangerous objects.

[0087] S7. Analyze the image of the side danger zone inside the vehicle calculated from the image captured by the first monocular camera to determine whether there are any dangerous objects.

[0088] S8. Analyze the image of the side danger zone inside the vehicle calculated from the image captured by the second monocular camera to determine whether there are any dangerous objects.

[0089] S9. If a hazardous material is present, issue a warning to the driver of the vehicle.

[0090] In some embodiments, before analyzing the image in the area directly in front of the vehicle captured by the fisheye camera, the steering assist method further includes: performing orthogonal processing on the image captured by the fisheye camera, the image captured by the first monocular camera, and the image captured by the second monocular camera.

[0091] Before analyzing the image in the area directly in front of the vehicle captured by the fisheye camera, the steering assist method further includes: performing orthogonal processing on the image captured by the fisheye camera, the image captured by the first monocular camera, and the image captured by the second monocular camera.

[0092] Before analyzing the image of the area directly in front of the vehicle captured by the fisheye camera, the steering assist method further includes:

[0093] The images captured by the fisheye camera, the first monocular camera, and the second monocular camera are subjected to orthogonal processing.

[0094] The method for orthogonalizing the images captured by the fisheye camera includes:

[0095] The world coordinate system coordinates are converted to the camera coordinate system coordinates through a rigid body transformation, wherein the rigid body transformation satisfies:

[0096]

[0097] in, Using the world coordinate system; For camera coordinate system coordinates, For rotation matrix, It is a translation vector;

[0098] The camera coordinate system coordinates are converted to image physical coordinates, and then to pixel coordinates. The conversion process is achieved using the following formula:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, This represents the distance from a pixel to the image center in the absence of distortion. For the corresponding angle, Let x and y be the components of the pixel length in the x and y directions of the photosensitive plane, respectively. cx and cy are the coordinates of the center of the photosensitive plane in the pixel coordinate system, and (u, v) are the pixel coordinates.

[0105] The calibration process for the fisheye camera module includes:

[0106] Images of the calibration board under different postures are acquired, the images are binarized, and the pixel coordinates of the checkerboard corners are obtained through a corner detection algorithm.

[0107] Set the top left corner of the calibration plate as the origin of its own world coordinate system, and determine the world coordinates of the other corner points;

[0108] Based on the pixel coordinates and world coordinates of the corner points, the intrinsic parameter matrix and distortion parameters of the fisheye camera are obtained. The intrinsic parameter matrix includes the principal point coordinates and focal length. The corner detection algorithm is selected from the contour curve detection algorithm, the binary image detection algorithm, or the grayscale image detection algorithm.

[0109] The calibration and ranging process of the first and second monocular cameras includes:

[0110] S71. Using Zhang's calibration method, multiple images are captured by changing the chessboard posture to obtain the intrinsic and extrinsic parameters of the binocular camera. The intrinsic parameters include focal length and principal point coordinates, and the extrinsic parameters include rotation matrix and translation vector.

[0111] S72. Use the internal and external parameters to perform stereoscopic correction on the left and right views so that the image matching points are located on the same polar line;

[0112] S73. The corrected image pair is processed using a stereo matching algorithm to calculate the disparity value of each pixel. The stereo matching algorithm is selected from local stereo matching algorithm, global stereo matching algorithm, semi-global stereo matching algorithm or stereo matching algorithm based on deep learning.

[0113] S74. Based on the principle of triangulation, using the baseline length B, focal length f, and disparity value d, through the formula... Calculate the spatial distance Z of the target point.

[0114] The method for analyzing the images from the fisheye camera, the first monocular camera, and the second monocular camera includes:

[0115] Feature extraction is performed on the input image, and the features include histogram of oriented gradients, edge features, diagonal features, or deep features extracted based on convolutional neural networks;

[0116] Candidate boxes are generated based on the extracted features, and the targets within the candidate boxes are classified by a pre-trained classifier, which is selected from random forest, decision tree or deep learning-based classification model.

[0117] Non-maximum suppression technique is used to filter candidate boxes and determine the optimal position and category of the target.

[0118] The deep learning process employs a single-stage detection algorithm or a two-stage detection algorithm.

[0119] The single-stage detection algorithm uses a convolutional network to perform feature extraction, classification, and localization, achieving end-to-end detection.

[0120] The two-stage detection algorithm first obtains candidate regions in the first stage, and then classifies and locates the candidate regions in the second stage.

[0121] The single-stage detection algorithm employs Mosaic data augmentation during training, stitching together multiple images and inputting them into the neural network for learning.

[0122] The data augmentation also includes adding noise, rotation, scaling, or cropping.

[0123] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0124] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A steering assist device, installed on the vehicle, wherein multiple wheels of the vehicle are sequentially connected end-to-end to form a square area; when the vehicle is running, the area of ​​the square area covering the ground is a crush zone, and the area of ​​the square area covering the ground from the starting point to when the vehicle is about to travel a first distance is a danger zone. When the vehicle turns, the first set of wheels in the direction of vehicle travels to the ground covered by a first distance is the positive danger zone, and the other danger zones besides the positive danger zone are the side danger zones. Its features are, The steering assist device includes: A fisheye camera is installed at the top front of the vehicle, and the field of view of the fisheye camera includes the vehicle and the area directly in front of the danger zones on both sides of the vehicle. The first monocular camera is located at the rear end of one side of the vehicle. The field of view of the first monocular camera includes the area from the upper edge of the vehicle to the rear edge of the last axle tire of the vehicle. When the vehicle is at its maximum steering angle, the field of view of the first monocular camera also includes the edge of the side hazard zone away from the vehicle. The second monocular camera is located at the rear end of the other side of the vehicle. The field of view of the second monocular camera includes the area from the upper edge of the vehicle to the rear edge of the last axle tire of the vehicle. When the vehicle is at its maximum steering angle, the field of view of the second monocular camera also includes the edge of the side hazard zone away from the vehicle. A microprocessor, which is electrically connected to the fisheye camera, the first monocular camera, and the second monocular camera; A warning device, electrically connected to the microprocessor, is fixed to the driver's cab of the vehicle and configured to receive signals from the microprocessor and warn the driver of the risk of a run-over accident when the vehicle is turning.

2. The steering assist device according to claim 1, characterized in that, The steering assist device also includes a steering wheel angle sensor, which is fixed below the steering wheel of the vehicle and configured to detect the rotation angle of the steering wheel. The steering wheel angle sensor is electrically connected to the microprocessor.

3. A steering assist method, applicable to the steering assist device described in claim 1 or 2 above. When the vehicle turns The turning radius R1 of the first wheel on the inner side of the vehicle is the ratio of the distance L1 between the first wheel and the turning center of the wheel in the length direction of the vehicle to sinα. The turning radius R2 of the last wheel inside the vehicle is the ratio of the distance L2 between the last wheel and the turning center of the wheel in the length direction of the vehicle to sinα. The side danger zone inside the vehicle has a fan-shaped structure, and the width H of the side danger zone inside the vehicle is the difference between the turning radius R1 of the first wheel and the turning radius R2 of the last wheel. The edge of the side hazard zone on the outer side of the vehicle includes: The center point is the steering center, and the area formed by the arc of the last wheel and the line segments that are connected sequentially by several tires on the other side of the vehicle. Wherein, the steering angle α when the vehicle turns; The steering assistance method is characterized by comprising: The fisheye camera captures images of the vehicle and the area directly in front of the danger zones on both sides when the vehicle is at its maximum steering angle. The first monocular camera captures images of the side danger zone inside the vehicle when the vehicle is at its maximum steering angle; The second monocular camera captures images of the side hazard zone on the outside of the vehicle at the vehicle's maximum steering angle; Obtain the steering angle α when the vehicle is turning, calculate the range of the side danger zone inside the vehicle under the current steering angle α, and calculate the range of the side danger zone outside the vehicle under the current steering angle α. Analyze the images captured by the fisheye camera in the area directly in front of the vehicle to determine if there are any hazards. The image of the side hazard zone inside the vehicle, which is located in front of the two hazard zones captured by the fisheye camera, is analyzed and calculated to determine whether there are any hazards. Analyze the images of the side danger zone inside the vehicle calculated from the images captured by the first monocular camera to determine whether there are any dangerous objects; Analyze the images of the side danger zone inside the vehicle calculated from the images captured by the second monocular camera to determine whether there are any dangerous objects; If a hazard is present, a warning is issued to the driver of the vehicle.

4. The steering assistance method according to claim 3, characterized in that, Before analyzing the image of the area directly in front of the vehicle captured by the fisheye camera, the steering assist method further includes: The images captured by the fisheye camera, the first monocular camera, and the second monocular camera are subjected to orthogonal processing. The method for orthogonalizing the images captured by the fisheye camera includes: The world coordinate system coordinates are converted to the camera coordinate system coordinates through a rigid body transformation, wherein the rigid body transformation satisfies: in, Using the world coordinate system; For camera coordinate system coordinates, For rotation matrix, It is a translation vector; The camera coordinate system coordinates are converted to image physical coordinates, and then to pixel coordinates. The conversion process is achieved using the following formula: in, This represents the distance from a pixel to the image center in the absence of distortion. For the corresponding angle, Let x and y be the components of the pixel length in the x and y directions of the photosensitive plane, respectively. cx and cy are the coordinates of the center of the photosensitive plane in the pixel coordinate system, and (u, v) are the pixel coordinates.

5. The steering assistance method according to claim 3, characterized in that, The calibration process for the fisheye camera module includes: Images of the calibration board under different postures are acquired, the images are binarized, and the pixel coordinates of the checkerboard corners are obtained through a corner detection algorithm. Set the top left corner of the calibration plate as the origin of its own world coordinate system, and determine the world coordinates of the other corner points; Based on the pixel coordinates and world coordinates of the corner points, the intrinsic parameter matrix and distortion parameters of the fisheye camera are obtained. The intrinsic parameter matrix includes the principal point coordinates and focal length. The corner detection algorithm is selected from the contour curve detection algorithm, the binary image detection algorithm, or the grayscale image detection algorithm.

6. The steering assistance method according to claim 3, characterized in that, The calibration and ranging process of the first and second monocular cameras includes: The Zhang calibration method is used to capture multiple images by changing the chessboard grid posture to obtain the intrinsic and extrinsic parameters of the binocular camera. The intrinsic parameters include focal length and principal point coordinates, and the extrinsic parameters include rotation matrix and translation vector. The left and right views are stereoscopically corrected using the aforementioned intrinsic and extrinsic parameters so that the image matching points are located on the same epipolar line; The corrected image pairs are processed using a stereo matching algorithm to calculate the disparity value of each pixel. The stereo matching algorithm is selected from local stereo matching algorithm, global stereo matching algorithm, semi-global stereo matching algorithm or deep learning-based stereo matching algorithm. Based on the principles of triangulation, using the baseline length B, focal length f, and disparity value d, the formula is used... Calculate the spatial distance Z of the target point.

7. The steering assistance method according to claim 3, characterized in that, The method for analyzing the images from the fisheye camera, the first monocular camera, and the second monocular camera includes: Feature extraction is performed on the input image, and the features include histogram of oriented gradients, edge features, diagonal features, or deep features extracted based on convolutional neural networks; Candidate boxes are generated based on the extracted features, and the targets within the candidate boxes are classified by a pre-trained classifier, which is selected from random forest, decision tree or deep learning-based classification model. Non-maximum suppression technique is used to filter candidate boxes and determine the optimal position and category of the target.

8. The steering assistance method according to claim 3, characterized in that, The deep learning process employs a single-stage detection algorithm or a two-stage detection algorithm. The single-stage detection algorithm uses a convolutional network to perform feature extraction, classification, and localization, achieving end-to-end detection. The two-stage detection algorithm first obtains candidate regions in the first stage, and then classifies and locates the candidate regions in the second stage.

9. The steering assistance method according to claim 3, characterized in that, The single-stage detection algorithm employs Mosaic data augmentation during training, stitching together multiple images and inputting them into the neural network for learning. The data augmentation also includes adding noise, rotation, scaling, or cropping.

Images