Panoramic surround view assisted driving method and system based on machine vision

Abstract

The application provides a machine vision-based panoramic surround-view auxiliary driving method and system, which comprises the following steps: acquiring a plurality of first images around a special vehicle by using a plurality of fisheye cameras; acquiring a distortion model of the fisheye camera; correcting the first images by using a polynomial model correction method through the distortion model to obtain second images; performing pitch transformation on the second images by using a direct linear transformation algorithm to obtain third images; eliminating splicing gaps of all the third images by using a weighted average algorithm to obtain a panoramic surround-view image; performing target detection on the panoramic surround-view image based on a multi-task perception algorithm of YOLOv5s, wherein the target comprises drivable area segmentation results and lane line segmentation results; and displaying the panoramic surround-view image and the target in the panoramic surround-view image. The application can completely display a virtual vehicle, a vehicle bottom and a surrounding road surface fusion image, intelligently recognize road scene information, eliminate a visual field blind area and enhance the visual field of a special vehicle driver.

Description

Technical Field

[0001] This invention relates to the field of driver assistance technology, and more specifically to a panoramic surround-view driver assistance method and system based on machine vision. Background Technology

[0002] With the development of internet technology, research on intelligent driver assistance systems (ADAS) has made significant progress. Vision-based environmental perception technology has been applied to automotive products, and its high cost-effectiveness has further promoted the development of ADAS. Distance illusion, width illusion, and curve illusion, among other human sensory errors, are significant factors leading to traffic hazards. To overcome these problems, various augmented reality technologies are rapidly developing and beginning to be applied in automotive products. Augmented reality technology integrates virtual vehicle models with the real road environment, helping drivers more accurately understand the vehicle's position relative to the road, reducing driving difficulty, and improving driving safety. This is currently a hot research topic in in-vehicle intelligent driver assistance systems. Currently, Mobileye in Israel is a relatively mature company in this field; Jaguar Land Rover in the UK has proposed some newer concepts based on existing driver assistance systems and augmented reality to further improve driving convenience. In China, driver assistance systems have also developed rapidly in recent years, with Minieye being a prominent example after a period of development.

[0003] However, current research on vehicle vision systems mainly focuses on passenger cars and commercial vehicles, with a lack of research on driver assistance systems for other vehicle types. In particular, for super-heavy off-road vehicles, due to the limitations of overall vehicle function and structure, their external dimensions are often too large, with the overall width, overall length, front overhang length, and rear overhang length all exceeding the limits. They generally suffer from poor driving visibility, requiring higher operating skills from drivers, and resulting in greater driving difficulty and operational intensity. Specifically, this manifests in several ways: When special vehicles climb steep slopes, the driver has difficulty observing the road ahead due to the upturned front of the vehicle. They must rely on landmarks on either side of the road or on-the-ground observation by others to assist them. This poses a significant risk when traversing unfamiliar or winding sections of road. When special vehicles pass through narrow passages, their large size means the passage may only be wide enough for the vehicle to pass through. Because the driver cannot accurately observe the position of the wheels relative to the road, existing vehicles typically rely on other personnel on the ground to observe and direct the driver. In this situation, vehicle efficiency is extremely low, requiring a high level of coordination between the directing personnel and the driver. When special vehicles cross ditches, the distance between the cab and the wheels makes it easy for the driver to misjudge the position of each wheel, compromising safe driving and potentially causing damage to the vehicle.

[0004] Because special-purpose vehicles differ significantly from other vehicles, research findings are difficult to apply directly. Therefore, it is essential to explore new application models and methods tailored to the specific characteristics of special-purpose vehicles. This requires developing core technologies such as multi-view image fusion and road scene perception to enhance the research and development capabilities of augmented reality-based assisted driving technologies for special-purpose vehicles. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a panoramic surround-view assisted driving method and system based on machine vision, applicable to special vehicles. This machine vision-based in-vehicle environmental perception application technology employs image processing and pattern recognition technologies, combined with augmented reality methods, to fully display fused images, intelligently identify road scene information, eliminate blind spots, enhance the driver's field of vision for special vehicles, and better suit the driving environment of special vehicles.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] Machine vision-based panoramic surround view assisted driving methods include:

[0008] Multiple fisheye cameras were used to acquire multiple first images of the area surrounding the special vehicle;

[0009] Obtain the distortion model of the fisheye camera;

[0010] The first image is corrected using a polynomial model correction method through a distortion model to obtain the second image;

[0011] The second image is transformed into a pitch diagram using the direct linear transformation algorithm to obtain the third image.

[0012] A weighted average algorithm is used to eliminate stitching gaps in all third images to obtain a panoramic view image.

[0013] The multi-task perception algorithm based on YOLOv5s is used to perform target detection on the panoramic surround view image, and obtain the drivable area segmentation result and lane line segmentation result in the panoramic surround view image.

[0014] Display the panoramic view image and the targets within it.

[0015] Furthermore, the number of fisheye cameras is greater than or equal to 5, and multiple fisheye cameras are installed around the special vehicle and at the bottom of the special vehicle.

[0016] Furthermore, the specific method for obtaining the distortion model of the fisheye camera is as follows:

[0017] The internal parameters and distortion coefficients of the fisheye camera were solved using Zhang Zhengyou's planar calibration method. The mapping relationship between the world coordinate system in three-dimensional space and the image coordinate system on the two-dimensional plane was established. Radial distortion model and tangential distortion model were established for the fisheye camera respectively. The distortion model was obtained by combining the radial distortion model and the tangential distortion model.

[0018] Furthermore, the specific method for correcting the first image using a polynomial model correction method through a distortion model to obtain the second image is as follows:

[0019] A distortion conversion model is established between the image points of the ideal linear model imaging and the distorted image points of the nonlinear imaging model imaging. The distortion conversion model is used to solve the mapping relationship between the positions of each pixel point on the first image when it is imaged under the linear model, so as to restore the distorted pixels to their normal positions and obtain the second image.

[0020] Furthermore, the specific method for performing a pitch transformation on the second image using the direct linear transformation algorithm to obtain the third image is as follows:

[0021] Solving the homography transformation matrix establishes a linear relationship between the world coordinate system and the image coordinate system. This is used to transform the second image between the perspective coordinate system and the image plane coordinate system, converting the second image from a shooting angle with a certain angle to the ground to a top-down angle, thus obtaining the third image.

[0022] Furthermore, the specific method for eliminating stitching gaps in all third images using a weighted average algorithm to obtain the panoramic view image is as follows:

[0023] By adding weights to the average value method, the pixel gray values ​​of feature points in the overlapping area of ​​two adjacent third images after registration are weighted and averaged according to certain weights to obtain the solution. The solution is used as the new gray value of the feature points after fusion to eliminate the stitching gap between the two adjacent third images. This process is repeated until all stitching gaps between two adjacent third images are eliminated, resulting in a panoramic view image.

[0024] Furthermore, during the target detection process of the panoramic surround view image using YOLOv5s' multi-task perception algorithm, an image detection decoder is established. The image detection decoder includes 4 residual blocks, 3 upsampling modules, and 1 spatial pyramid pooling module. The drivable area segmentation result is obtained through the 4 residual blocks and 3 upsampling modules. In the 1 spatial pyramid pooling module, multiple small-sized convolutional kernels are applied to replace a single large-sized convolutional kernel to obtain the lane line segmentation result.

[0025] A machine vision-based panoramic surround-view assisted driving system includes:

[0026] The imaging module is used to acquire multiple first images of the area around the special vehicle using multiple fisheye cameras;

[0027] The calibration module is used to obtain the distortion model of the fisheye camera;

[0028] The correction module is used to correct the first image using a polynomial model correction method through a distortion model to obtain the second image;

[0029] The transformation module is used to perform elevation transformation on the second image using a direct linear transformation algorithm to obtain the third image.

[0030] The image fusion module is used to eliminate stitching gaps in all third images using a weighted average algorithm to obtain a panoramic view image.

[0031] The target detection module is used to perform target detection on the panoramic surround view image based on the YOLOv5s multi-task perception algorithm, and to obtain the drivable area segmentation result and lane line segmentation result in the panoramic surround view image.

[0032] The display module is used to display the panoramic view image and the targets within the panoramic view image.

[0033] An electronic device includes a plurality of fisheye cameras, a memory, a processor, and a computer program stored in the memory and executable on the processor. The plurality of fisheye cameras are used to acquire multiple first images of the surroundings of a special vehicle. When the processor executes the program, the aforementioned machine vision-based panoramic surround-view assisted driving method is described.

[0034] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described machine vision-based panoramic surround-view assisted driving method.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] (1) This invention proposes a panoramic surround-view assisted driving method based on machine vision, which is designed specifically for special vehicles. The software uses image processing and pattern recognition technology, combined with augmented reality methods, to fully display a fused image of the virtual vehicle, the underside of the vehicle, and the surrounding road surface, intelligently identify road scene information, eliminate blind spots, and enhance the field of vision of special vehicle drivers.

[0037] (2) The core functions of this invention include environmental perception technologies such as multi-view image fusion, chassis perspective, and road scene perception. By fusing images captured by multiple cameras, a panoramic image is generated, allowing the driver to observe the vehicle and its surrounding environment from all angles. To enable the driver to clearly understand the situation below the vehicle and provide a safer driving experience, this invention employs a chassis perspective function, which can display a fused image of the virtual vehicle and its undercarriage. To help the driver better understand and judge road conditions and reduce the risk of accidents, the software is designed to intelligently identify road scene information, such as lane lines and traffic signs.

[0038] (3) The panoramic surround-view assisted driving method of the present invention uses image processing technology and YOLOv5s-based image detection technology to design software, thereby enhancing the field of vision and providing safe driving assistance for special vehicle drivers. This method can provide a reliable and efficient solution to help special vehicle drivers better understand the situation around the vehicle, enhance the driver's control ability, reduce the risk of accidents, and provide a good driving experience. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and are intended to explain the invention, but do not constitute an undue limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart of the panoramic surround-view assisted driving method based on machine vision according to the present invention;

[0041] Figure 2 This is a first schematic diagram of the panoramic surround-view assisted driving system based on machine vision according to the present invention;

[0042] Figure 3 This is a second schematic diagram of the panoramic surround-view assisted driving system based on machine vision according to the present invention;

[0043] Figure 4 This is the initial interface of the panoramic surround-view assisted driving system based on machine vision in a specific embodiment of the present invention;

[0044] Figure 5(a) is a schematic diagram of a panoramic image with chassis perspective in a specific embodiment of the present invention;

[0045] Figure 5(b) is a schematic diagram of the front view in a specific embodiment of the present invention;

[0046] Figure 5(c) is a schematic diagram of the rear view in a specific embodiment of the present invention;

[0047] Figure 5(d) is a schematic diagram of the left-hand image in a specific embodiment of the present invention;

[0048] Figure 5(e) is a schematic diagram of the right-hand image in a specific embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of target detection in a specific embodiment of the present invention;

[0050] Figure 7 This is a software shutdown interface for a specific embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0052] Example 1

[0053] Example 1 provides a machine vision-based panoramic surround-view assisted driving method, such as... Figure 1 As shown, it includes:

[0054] Step S1: Use a fisheye camera to acquire multiple first images of the area surrounding the special vehicle;

[0055] Step S2: Obtain the distortion model of the fisheye camera;

[0056] Step S3: The first image is corrected using a polynomial model correction method through a distortion model to obtain the second image;

[0057] Step S4: Use the direct linear transformation algorithm to perform an elevation transformation on the second image to obtain the third image;

[0058] Step S5: Use a weighted average algorithm to eliminate stitching gaps in all third images to obtain a panoramic view image;

[0059] Step S6: Perform target detection on the panoramic surround view image based on the YOLOv5s multi-task perception algorithm to obtain the drivable area segmentation result and lane line segmentation result in the panoramic surround view image;

[0060] Step S7: Display the panoramic view image and the target in the panoramic view image.

[0061] In this embodiment, by understanding the current research status of intelligent assisted driving systems at home and abroad, a panoramic surround-view assisted driving method based on machine vision is proposed, specifically designed for special vehicles. This software employs image processing and pattern recognition technologies, combined with augmented reality methods, to fully display a fused image of the virtual vehicle, its undercarriage, and the surrounding road surface. It intelligently identifies road scene information, eliminates blind spots, and enhances the field of vision for special vehicle drivers.

[0062] In this embodiment, core functions include environmental perception technologies such as multi-view image fusion, chassis perspective, and road scene perception. By fusing images captured by multiple cameras, a panoramic image is generated, allowing the driver to observe the vehicle and its surrounding environment from all angles. To provide the driver with a clear understanding of the situation below the vehicle and a safer driving experience, this design employs a chassis perspective function, which can display a fused image of the virtual vehicle and its undercarriage. To help the driver better understand and judge road conditions and reduce the risk of accidents, the software is designed to intelligently identify road scene information, such as lane lines and traffic signs.

[0063] In this embodiment, the panoramic surround-view assisted driving method utilizes image processing technology and YOLOv5s-based image detection technology to design software that enhances the driver's field of vision and provides safer driving assistance for special vehicle drivers. This method offers a reliable and efficient solution to help special vehicle drivers better understand their surroundings, enhance their control capabilities, reduce the risk of accidents, and provide a superior driving experience.

[0064] In this embodiment, the number of fisheye cameras is greater than or equal to 5, and multiple fisheye cameras are installed on the front, rear, left, right sides and the bottom of the special vehicle.

[0065] Images captured using fisheye cameras exhibit significant distortion, primarily categorized into two types: radial distortion and tangential distortion. Radial distortion is mainly caused by the shape of the fisheye lens altering its radial index, causing light rays to deviate from their ideal linear path, resulting in noticeable radial distortion at the image edges. Tangential distortion is primarily caused by defects in the lens's manufacturing and assembly processes, creating an angle between the lens plane and the image plane.

[0066] In this embodiment, a point in three-dimensional space is projected onto a two-dimensional image using a fisheye camera. There is often a mapping relationship between the spatial coordinates of this point and the coordinates of the imaging plane. Fisheye camera calibration technology utilizes modeling to obtain information about points or lines from the image captured by the fisheye camera, and then uses calibration algorithms to solve for the mapping relationship and related parameters between the spatial coordinates of the point and the imaging plane coordinates. The relevant parameters of a fisheye camera typically refer to intrinsic parameters, extrinsic parameters, and distortion coefficients. Extrinsic parameters reveal how the point is rotated and translated from world coordinates to the fisheye camera coordinates. Intrinsic parameters show how the point is imaged through the fisheye camera lens within the fisheye camera coordinate system, ultimately being converted into pixels. The distortion coefficient reflects the offset of the imaged point. Using fisheye camera calibration technology, a mapping relationship between the world coordinate system in three-dimensional space and the image coordinate system on the two-dimensional plane can be established.

[0067] Zhang Zhengyou's planar calibration method is more flexible than the two-step calibration method mentioned above. This method only requires taking pictures of a calibration board with a fisheye camera to solve for the internal parameters and distortion coefficients of the fisheye camera. The flexibility of this method compared to traditional calibration methods lies in the fact that it does not require fixing the fisheye camera. The fisheye camera can move or rotate freely when taking pictures, and accurate calibration results can still be obtained even without knowing the movement information or rotation angle. Zhang Zhengyou's planar calibration method points out that the calibration process requires at least three images of the calibration board taken from different angles to obtain calibration results. In actual calibration, it is often necessary to simultaneously calibrate a dozen or more images of the calibration board taken from different angles, thus obtaining more accurate and reliable calibration results.

[0068] In this embodiment, the image distortion only produces a certain deformation, making it no longer similar to the original scene, and does not change the image clarity. This deformation is a regular deformation that does not affect the acquisition of scene information from the image. After obtaining the intrinsic parameter matrix and distortion coefficients through the fisheye camera calibration, the fisheye image can be corrected using the distortion correction algorithm to finally obtain a distortion-free image. This allows the fisheye image to provide greater visual information while maintaining normal visual habits.

[0069] The basic idea of ​​the polynomial model correction method is to establish a distortion transformation model between the image points of the ideal linear model imaging and the image points of the distorted image imaging in the nonlinear imaging model, so as to solve the mapping relationship of the position of each pixel point on the distorted image when it is imaged under the linear model, thereby restoring the distorted pixel points to the normal position and achieving the effect of eliminating distortion.

[0070] In this embodiment, the specific method for obtaining the distortion model of the fisheye camera is as follows:

[0071] The internal parameters and distortion coefficients of the fisheye camera were solved using Zhang Zhengyou's planar calibration method. The mapping relationship between the world coordinate system in three-dimensional space and the image coordinate system on the two-dimensional plane was established. Radial distortion model and tangential distortion model were established for the fisheye camera respectively. The distortion model was obtained by combining the radial distortion model and the tangential distortion model.

[0072] The specific method for correcting the first image using a polynomial model to obtain the second image is as follows:

[0073] A distortion conversion model is established between the image points of the ideal linear model imaging and the distorted image points of the nonlinear imaging model imaging. The distortion conversion model is used to solve the mapping relationship between the positions of each pixel point on the first image when it is imaged under the linear model, so as to restore the distorted pixels to their normal positions and obtain the second image.

[0074] In this embodiment, after distortion correction and appropriate interpolation correction are performed on the images captured by the fisheye camera, more than five non-perpendicular view images of the vehicle body at a certain angle to the ground are obtained. If these images are directly stitched together, the resulting image is a distorted image that does not conform to normal human vision. Presenting such an image to the driver would not clearly and intuitively show the normal scene around the car, and could even lead to other traffic accidents due to the lack of accurate road information. Therefore, to effectively improve the image stitching effect, after obtaining the non-perpendicular view images at a certain angle to the ground, the images need to be converted into a top-down view, i.e., a top-down projection transformation is performed. The image after the top-down projection transformation is a view of the car and the surrounding road surface from above, which conforms to normal visual habits and is more easily accepted by the driver.

[0075] The specific method for performing elevation transformation on the second image using the direct linear transformation algorithm to obtain the third and fourth images is as follows:

[0076] Solving the homography transformation matrix establishes a linear relationship between the world coordinate system and the image coordinate system. This is used to transform the second image between the perspective coordinate system and the image plane coordinate system, converting the second image from a shooting angle with a certain angle to the ground to a top-down angle, thus obtaining the third image.

[0077] Its basic formula is shown in equation (1):

[0078]

[0079] Where (X, Y, Z) is the position of the image in the perspective coordinate system before the direct linear transformation, (u, v) are the coordinates on the image plane, representing the position of the image after the direct linear transformation, and l1, l2, l3, l4, l5, l6, l7, l8, l9, l 10 l 11 These are image transformation parameters. By solving the 11 unknown parameters in equation (1), the linear relationship between coordinate systems can be obtained. Typically, assuming the fisheye camera captures a horizontal ground, then Z = 0 in the (X, Y, Z) coordinate system, and equation (3.6) is simplified to equation (2):

[0080]

[0081] Setting Z=0 simplifies equation (2) to only have 8 unknown parameters: l1, l2, l4, l5, l6, l8, l9, l 10To solve for these 8 unknown parameters, four points need to be arbitrarily selected on the perspective image. The coordinates of these four points in the image coordinate system are obtained by using the corner detection method. Using the coordinates of these four corresponding points, eight equations can be constructed to solve for the eight unknown parameters, thereby solving for the homography transformation matrix H. The transformation from the perspective image to the top view image can be expressed as equation (3):

[0082] I 俯视图 =HI 透视图 (3)

[0083] In this embodiment, during the practical application of the in-vehicle panoramic surround view system, fisheye cameras are installed in five directions on the car. The images captured by fisheye cameras at different locations exhibit brightness differences due to factors such as lighting angle, resulting in uneven brightness in the stitched image. Furthermore, capturing moving images can cause distortions such as ghosting in the stitched image. To address these issues, image fusion technology is needed to effectively improve the quality of the stitched image, providing the driver with a better panoramic image in the in-vehicle panoramic surround view system. After initial image registration and stitching, two images will have a noticeable seam in the stitching area. If the brightness on both sides of the seam is different, the stitched image will have uneven brightness, affecting the visual effect. Therefore, image fusion technology is needed to ensure a natural transition in the overlapping area, balanced overall brightness, and no obvious stitching marks in the stitched image. Since the in-vehicle panoramic surround view system needs to provide the driver with high-quality stitched images, this embodiment chooses a pixel-level image fusion algorithm—the weighted average method—which, although slower in computation, offers reliable fusion results and is easy to implement.

[0084] The specific method for eliminating stitching gaps in all third images using a weighted average algorithm to obtain a panoramic view image is as follows:

[0085] By adding weights to the average value method, the pixel gray values ​​of feature points in the overlapping area of ​​two adjacent third images after registration are weighted and averaged according to certain weights to obtain the solution. The solution is used as the new gray value of the feature points after fusion to eliminate the stitching gap between the two adjacent third images. This process is repeated until all stitching gaps between two adjacent third images are eliminated, resulting in a panoramic view image.

[0086] In this embodiment, the YOLOv5s multi-task perception algorithm's network structure consists of an encoder and a decoder. The encoder comprises a backbone network and a feature fusion network, while the decoder consists of a drivable region decoder, a lane line decoder, and a vehicle detection decoder. The encoder extracts image features required for different tasks and aggregates these features, enabling the model to learn multi-scale feature information. The decoder is designed with corresponding semantic segmentation or object detection structures according to the characteristics of different tasks, allowing the network to detect three types of objects simultaneously.

[0087] In this embodiment, to avoid data transmission being affected by network signal latency, the road scene perception algorithm is generally deployed offline in an embedded device on the vehicle platform. Therefore, the algorithm is required to maintain detection accuracy while possessing fast inference speed under relatively weak hardware computing power to meet the real-time requirements of the autonomous driving system. This embodiment improves the C3 module and standard 3×3 convolutional layer of the YOLOv5s backbone network structure, making the model more lightweight to meet the real-time requirements of multi-task models. The YOLOv5s backbone network uses residual units as the basic unit of the C3 module. First, 1×1 convolutions and 3×3 convolutions are combined. Then, the 1×1 convolution reduces the number of channels to half of the input feature map, and the 3×3 convolution restores the number of channels. Then, skip connections are used to superimpose channel-dimensional features with the input features to obtain the output of the residual unit. Skip connections directly pass the input features to the output, so that the gradient of backpropagation can be effectively passed to the preceding network, thereby effectively training the parameters of the shallow network layers. Therefore, the residual unit ensures feature extraction capability while reducing the number of parameters.

[0088] To meet the real-time and accuracy requirements of multi-task environment awareness models on mobile hardware such as edge devices, this embodiment introduces an RVB structure to replace the residual units of the C3 module, resulting in the RepC3 module. Compared to the original C3 module structure, the RepC3 module adds a gradient backpropagation branch during the training phase; during the model inference phase, the three-branch convolutional layer is converted into a single-branch structure, which better improves the memory utilization of the device, thereby increasing the inference speed of the model.

[0089] YOLOv5s's Neck structure constructs a Feature Pyramid Network (FPN) and a Feature Aggregation Network (PAN) to fuse and redistribute extracted multi-scale features, improving the reliability of the network's detection results for targets at different scales. The PAN structure uses Concat to concatenate feature maps of different scales along the channel dimension, increasing the number of channels in the output feature map. Furthermore, the relative displacement between objects in a road scene can alter the pose of targets, hindering the detection of small targets. To improve the inference efficiency of multi-task network algorithms on in-vehicle computing platforms and enhance their ability to focus on important small-scale targets, this embodiment combines RepC3 and a positional attention mechanism module to construct a more effective feature fusion structure.

[0090] First, the C3 module is replaced with RepC3, and the 3×3 convolution of the PAN structure is adjusted to an RVB module. Next, Add is used instead of Concat without changing the number of channels in the feature map. However, while Add reduces the number of parameters, it also reduces model accuracy. To eliminate this problem, this embodiment adjusts the medium-scale fusion structure of the feature fusion network. To enhance the model's ability to detect small targets, this embodiment introduces an attention mechanism into the feature fusion network. The attention mechanism allows the model to focus more on features that are more important for the prediction result. However, in road scene environment perception models, the attention module reduces the model's real-time performance, and small targets that are relatively far from the driver are more prone to missed or false detections in road scenes. Based on these influencing factors, this embodiment inserts an attention module into the small-scale target detection branch of the feature fusion network structure. To address the problem of vehicle occlusion in road scenes causing a loss of model detection accuracy, this embodiment uses a positional attention mechanism module that combines positional and channel information. This attention mechanism assigns high weights to occluded small targets that are more difficult to detect to increase attention, and assigns low weights to road background objects to suppress them, thereby improving the detection accuracy of vehicles in road scenes.

[0091] In this embodiment, during the target detection process of the panoramic surround view image using the YOLOv5s multi-task perception algorithm, a drivable area decoder and a lane line decoder are established. The drivable area decoder includes 4 residual blocks and 3 upsampling modules, and the lane line decoder includes 1 spatial pyramid pooling module. The drivable area segmentation result is obtained through the 4 residual blocks and 3 upsampling modules. In the 1 spatial pyramid pooling module, multiple small-sized convolutional kernels are applied to replace a single large-sized convolutional kernel to obtain the lane line segmentation result.

[0092] To accomplish lane detection and drivable area segmentation, a semantic segmentation decoder needs to be added to the YOLOv5s object detection decoder. Since the drivable area constitutes a large proportion of pixels in road images, and there is no imbalance between positive and negative samples during training, this embodiment designs a decoder with fewer network layers to classify each pixel in the drivable area to improve its real-time performance. The drivable area decoder mainly consists of four RVBs and three upsampling modules. Assuming the input image size is (w, h, 3), the decoder first receives the shallow features extracted by the backbone network, with dimensions (w / 8, h / 8, 128); then, it applies RVB for dimensionality reduction, adjusting the number of channels to 64; subsequently, it uses a structure combining RVB and bilinear interpolation to sequentially upsample the feature map to (w / 4, h / 4, 32), (w / 2, h / 2, 16), and (w, h, 2), obtaining the drivable area segmentation result.

[0093] Because lane lines are long and thin, and their texture differs from the elongated markings on the road surface, they occupy a much smaller area than the drivable region in a road scene. Therefore, an SPPF module is introduced to enhance the feature extraction capability of the drivable region segmentation decoder. Multiple small-sized convolutional kernels replace the single large-sized convolutional kernel in the SPP module, thus fusing receptive fields of different scales and improving processing speed. Compared to the drivable region decoder structure, the only difference is the addition of an SPPF module after adjusting the channel dimensions; the rest of the structure is basically the same.

[0094] Since the model in this embodiment needs to simultaneously perform traffic vehicle detection, lane line detection, and drivable area segmentation, a weighted loss function corresponding to each of the three tasks is used to train the model, as shown in Equation (4).

[0095] L all =λ1L det +λ2L daseg +λ3L llseg (4)

[0096] Where λ1, λ2, and λ3 are the single-task loss weight values; L det L is the loss function for the target detection branch; daseg L is the loss function for segmenting the driving region; llseg L is the lane line segmentation loss function; all This is the overall network structure loss function.

[0097] The object detection branch loss function consists of three parts: classification loss, confidence loss, and bounding box regression loss.

[0098] L det =α1L class +α2Lconf +α3L EIoU (5)

[0099] Where α1, α2, and α3 are the weight values ​​of each loss function term; L class and L conf For classification loss and confidence loss, a binary cross-entropy loss function is used; the target box regression loss function L... EIOU Using EIOU Loss, constraints are applied to the overlap area, center distance, width, and height differences between the predicted and ground truth bounding boxes.

[0100] The lane detection branch loss function consists of cross-entropy loss and Dice loss. Cross-entropy is obtained by comparing each pixel individually. Since the number of lane line pixels is much smaller than the number of background pixels, using only the cross-entropy loss function would cause the loss function to tend towards the background pixels. Therefore, this embodiment uses the Dice loss function to suppress the sample class imbalance problem. Its formula is as follows:

[0101] L llseg =L CE +γ1L dice (6)

[0102]

[0103] Where γ1 is the weight value of the Dice loss function; p i j denoted as the number of pixels whose true label is category i and predicted to be category j; k is the maximum number of semantic segmentation category labels, k+1 is the total number of categories, and k=1 in this experiment.

[0104] Because the number of target pixels is large and the detection difficulty is relatively low, only cross-entropy is used as the loss function for drivable area detection:

[0105] L daseg =L CE (8)

[0106] Example 2

[0107] Example 2 provides a machine vision-based panoramic surround view assisted driving system, such as... Figure 2 and Figure 3 As shown, it includes:

[0108] The imaging module is used to acquire multiple first images of the area around the special vehicle using a fisheye camera;

[0109] The calibration module is used to obtain the distortion model of the fisheye camera;

[0110] The correction module is used to correct the first image using a polynomial model correction method through a distortion model to obtain the second image;

[0111] The transformation module is used to perform elevation transformation on the second image using a direct linear transformation algorithm to obtain the third image.

[0112] The image fusion module is used to eliminate stitching gaps in all third images using a weighted average algorithm to obtain a panoramic view image.

[0113] The target detection module is used to perform target detection on the panoramic surround view image based on the YOLOv5s multi-task perception algorithm, and to obtain the drivable area segmentation result and lane line segmentation result in the panoramic surround view image.

[0114] The display module is used to display the panoramic view image and the targets within the panoramic view image.

[0115] In one specific embodiment of this invention, the machine vision-based panoramic surround-view assisted driving system aims to eliminate as many blind spots as possible for the driver, thereby reducing the occurrence of traffic accidents to some extent. The system acquires images from the front, rear, left, and right sides of the vehicle by installing fisheye cameras in four directions. After a series of image processing steps, the four images are stitched together into a panoramic image and displayed on the vehicle's central control screen.

[0116] like Figure 4 The image shown is the initial interface of the panoramic surround-view assisted driving system based on machine vision in a specific embodiment of the present invention. After the system is installed and loaded, select the entry mode on the homepage to use it.

[0117] like Figures 5(a)-5(e) As shown, when the system starts to be used, different detection categories can be selected, and cameras in different directions will be called to collect images. The system will perform target detection on the collected images in real time and display the detection results in the result box below. If vehicles and people are detected, the results will also be displayed in the bar chart on the right.

[0118] like Figure 6 The diagram shown is a target detection schematic in a specific embodiment of the present invention, demonstrating that the system is capable of target detection.

[0119] like Figure 7 As shown, this is the software closing interface of a specific embodiment of the present invention. When the system is finished, the user can click the "×" button in the upper right corner to select to exit the software or cancel the exit.

[0120] Example 3

[0121] Example 3 provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described machine vision-based panoramic surround-view assisted driving method.

[0122] Example 4

[0123] Example 4 provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the above-described machine vision-based panoramic surround-view assisted driving method.

[0124] The memory in this embodiment of the invention is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0125] The machine vision-based panoramic surround-view assisted driving method disclosed in this invention can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the machine vision-based panoramic surround-view assisted driving method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The aforementioned processor can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in this invention's embodiments. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention's embodiments can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can reside in a storage medium, specifically a memory. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the machine vision-based panoramic surround-view assisted driving method provided in this invention's embodiments.

[0126] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0127] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0128] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A panoramic surround-view assisted driving method based on machine vision, characterized in that, include: Multiple fisheye cameras were used to acquire multiple first images of the area surrounding the special vehicle; Obtain the distortion model of the fisheye camera; The first image is corrected using a polynomial model correction method through a distortion model to obtain the second image; The second image is transformed into a pitch diagram using the direct linear transformation algorithm to obtain the third image. A weighted average algorithm is used to eliminate stitching gaps in all third images to obtain a panoramic view image. The multi-task perception algorithm based on YOLOv5s is used to perform target detection on the panoramic surround view image, and obtain the drivable area segmentation result and lane line segmentation result in the panoramic surround view image. Display the panoramic view image and the targets within it; The network structure of the YOLOv5s-based multi-task perception algorithm consists of an encoder and a decoder. The encoder is composed of a backbone network and a feature fusion network, while the decoder consists of a drivable area decoder, a lane line decoder, and a vehicle detection decoder. The encoder is used to extract image features required for different tasks and aggregate the features to learn multi-scale feature information. The decoder is designed with corresponding semantic segmentation structures or object detection structures according to the characteristics of different tasks to simultaneously detect three objects: drivable areas, lane lines, and vehicles. In the network structure of the YOLOv5s-based multi-task perception algorithm, an RVB module is introduced to replace the residual units of the C3 module of the YOLOv5s backbone network, resulting in the RepC3 module. During the training phase, the RepC3 module adds a gradient backpropagation branch, and during the model inference phase, it converts the 3-branch convolutional layer into a single-branch structure. The C3 module of the YOLOv5s feature fusion network is replaced with RepC3, and the 3×3 convolution of the feature fusion network is adjusted to an RVB module; Add is used instead of Concat without changing the number of channels in the feature map; a position attention mechanism module that combines position information and channel information is inserted into the small-scale target detection branch of the feature fusion network. The drivable region decoder includes four residual blocks and three upsampling modules, which are used to obtain the drivable region segmentation result. The lane line decoder includes one spatial pyramid pooling module, in which multiple small-sized convolutional kernels are applied to replace a single large-sized convolutional kernel to obtain the lane line segmentation result.

2. The panoramic surround-view assisted driving method based on machine vision according to claim 1, characterized in that: The number of fisheye cameras is greater than or equal to 5, and multiple fisheye cameras are installed around the special vehicle and at the bottom of the special vehicle.

3. The panoramic surround-view assisted driving method based on machine vision according to claim 1, characterized in that, The specific method for obtaining the distortion model of a fisheye camera is as follows: The internal parameters and distortion coefficients of the fisheye camera were solved using Zhang Zhengyou's planar calibration method. The mapping relationship between the world coordinate system in three-dimensional space and the image coordinate system on the two-dimensional plane was established. Radial distortion model and tangential distortion model were established for the fisheye camera respectively. The distortion model was obtained by combining the radial distortion model and the tangential distortion model.

4. The panoramic surround-view assisted driving method based on machine vision according to claim 1, characterized in that, The specific method for correcting the first image using a polynomial model to obtain the second image is as follows: A distortion conversion model is established between the image points of the ideal linear model imaging and the distorted image points of the nonlinear imaging model imaging. The distortion conversion model is used to solve the mapping relationship between the positions of each pixel point on the first image when it is imaged under the linear model, so as to restore the distorted pixels to their normal positions and obtain the second image.

5. The panoramic surround-view assisted driving method based on machine vision according to claim 1, characterized in that, The specific method for performing elevation transformation on the second image using the direct linear transformation algorithm to obtain the third image is as follows: Solving the homography transformation matrix establishes a linear relationship between the world coordinate system and the image coordinate system. This is used to transform the second image between the perspective coordinate system and the image plane coordinate system, converting the second image from a shooting angle with a certain angle to the ground to a top-down angle, thus obtaining the third image.

6. The panoramic surround-view assisted driving method based on machine vision according to claim 1, characterized in that, The specific method for eliminating stitching gaps in all third images using a weighted average algorithm to obtain a panoramic view image is as follows: By adding weights to the average value method, the pixel gray values ​​of feature points in the overlapping area of ​​two adjacent third images after registration are weighted and averaged according to certain weights to obtain the solution. The solution is used as the new gray value of the feature points after fusion to eliminate the stitching gap between the two adjacent third images. This process is repeated until all stitching gaps between two adjacent third images are eliminated, resulting in a panoramic view image.

7. A panoramic surround-view assisted driving system based on machine vision using the method described in any one of claims 1 to 6, characterized in that, include: The imaging module is used to acquire multiple first images of the area around the special vehicle using multiple fisheye cameras; The calibration module is used to obtain the distortion model of the fisheye camera; The correction module is used to correct the first image using a polynomial model correction method through a distortion model to obtain the second image; The transformation module is used to perform elevation transformation on the second image using a direct linear transformation algorithm to obtain the third image. The image fusion module is used to eliminate stitching gaps in all third images using a weighted average algorithm to obtain a panoramic view image. The target detection module is used to perform target detection on the panoramic surround view image based on the YOLOv5s multi-task perception algorithm, and to obtain the drivable area segmentation result and lane line segmentation result in the panoramic surround view image. The display module is used to display the panoramic view image and the targets within the panoramic view image.

8. An electronic device, characterized in that, The system includes multiple fisheye cameras, a memory, a processor, and a computer program stored in the memory and executable on the processor. The multiple fisheye cameras are used to acquire multiple first images of the area surrounding the special vehicle. When the processor executes the program, it implements the machine vision-based panoramic surround-view assisted driving method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the machine vision-based panoramic surround-view assisted driving method as described in any one of claims 1 to 6.

Images