Panoramic look-around auxiliary driving method and system based on machine vision

Abstract

The invention provides a machine vision-based panoramic look-around auxiliary driving method and system, and the method comprises the steps: employing a plurality of fisheye cameras to obtain a plurality of first images around a special vehicle, obtaining a distortion model of the fisheye cameras, employing a polynomial model correction method to correct the first images through the distortion model to obtain second images, and obtaining the second images; a direct linear transformation algorithm is used to transform the pitch map of the second image to obtain third images, a weighted average algorithm is used to eliminate splicing gaps of all the third images to obtain a panoramic look-around image, a YOLOv5s-based multi-task perception algorithm is used to detect the target of the panoramic look-around image, the target comprises a drivable area segmentation result and a lane line segmentation result, and the target comprises a driving area segmentation result and a lane line segmentation result. And displaying the panoramic look-around image and the target in the panoramic look-around image. The method can completely display the fusion image of the virtual vehicle, the vehicle bottom and the surrounding road surface, intelligently recognizes the road scene information, eliminates the view blind area, and enhances the view of the driver of the special vehicle.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a panoramic surround view assisted driving method and system based on machine vision. Background Art

[0002] With the development of Internet technology, the research on intelligent assisted driving systems has made great progress. Vision-based environmental perception technology has been applied to automotive products, and its cost-effectiveness has further promoted the development of intelligent assisted driving systems. Human sensory errors such as distance illusion, width illusion, and curve illusion are important factors leading to traffic hazards. In order to overcome these human sensory problems, various augmented reality technologies are developing rapidly and beginning to be promoted and applied in automotive products. The application of augmented reality technology can integrate virtual vehicle models with real road environments, help drivers more accurately understand the positional relationship between vehicles and road environments, reduce the difficulty of driving operations, and improve driving safety. It is currently a hot topic in the research of on-board intelligent assisted driving systems. At present, the more mature one abroad is Israel's Mobileye; Jaguar Land Rover in the UK has proposed some newer concepts based on the existing driving assistance system and combined with augmented reality to further improve the convenience of driving. In recent years, the domestic assisted driving system has also developed rapidly. After a period of precipitation, Minieye is more prominent.

[0003] However, the current research on vehicle vision system technology is mainly aimed at passenger cars and commercial vehicles, and there is a lack of research on auxiliary driving systems for other types of vehicles. Especially for super-heavy off-road vehicles, due to the limitations of vehicle functions and structures, their dimensions are often large, and the width, length, front overhang and rear overhang of the vehicle are all over the limit. There is a general disadvantage of poor driving vision, high requirements on the operating skills of the driver, and high driving difficulty and operation intensity. Specifically, when special vehicles climb steep slopes, due to the upturned front of the vehicle, it is difficult for the driver to observe the road conditions in front of the vehicle. The driver can only rely on reference objects on both sides of the road or other personnel observing on the ground to help him pass. When passing through unfamiliar sections or curved sections, there is a great risk; when special vehicles pass through narrow passages, due to the wide body of the vehicle, the passage may only be able to meet the requirements of the vehicle just passing through. Since the driver cannot accurately observe the position between the wheels and the road, existing vehicles are generally directed by other personnel to drive the vehicle through after observation on the ground. Under this condition, the vehicle passage efficiency is extremely low, and high coordination and cooperation between the commander and the driver are required; when special vehicles cross a ditch, since the cab is far away from the wheels, the driver's judgment on the position of the wheels of each bridge is prone to deviation, which is not conducive to safe driving and may even cause damage to the vehicle in serious cases.

[0004] Since special vehicles are very different from other vehicles, it is difficult to directly apply relevant research results. Therefore, it is necessary to explore new application modes and methods based on the characteristics of special vehicles. By developing relevant core technologies such as multi-view image fusion and road scene perception, we will develop the research and development capabilities of special vehicle assisted driving technology based on augmented reality. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a panoramic surround view assisted driving method and system based on machine vision, which is suitable for special vehicles. The vehicle-mounted environment perception application technology based on machine vision adopts image processing and pattern recognition technology, combined with reality enhancement method, to fully display the fused image, intelligently identify road scene information, eliminate blind spots, enhance the field of vision of special vehicle drivers, and better adapt to the driving environment of special vehicles.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The panoramic surround-view assisted driving method based on machine vision includes:

[0008] Using multiple fisheye cameras to obtain multiple first images around the special vehicle;

[0009] Get the distortion model of the fisheye camera;

[0010] The first image is corrected by using a polynomial model correction method through a distortion model to obtain a second image;

[0011] Performing a pitch transformation on the second image using a direct linear transformation algorithm to obtain a third image;

[0012] A weighted average algorithm is used to eliminate the stitching gaps of all third images to obtain a panoramic image;

[0013] Perform target detection on the panoramic surround image based on the YOLOv5s multi-task perception algorithm, and obtain the drivable area segmentation results and lane line segmentation results in the panoramic surround image;

[0014] The panoramic view image and the targets in the panoramic view image are displayed.

[0015] Furthermore, the number of the fisheye cameras is greater than or equal to 5, and the multiple fisheye cameras are installed around the special vehicle and on the bottom of the special vehicle.

[0016] Furthermore, the specific method for obtaining the distortion model of the fisheye camera is:

[0017] The internal parameters and distortion coefficients of the fisheye camera are solved through Zhang Zhengyou's plane calibration method, and the mapping relationship between the world coordinate system in the three-dimensional space and the image coordinate system on the two-dimensional plane is established. The radial distortion model and the tangential distortion model are established for the fisheye camera respectively, and the distortion model is obtained by combining the radial distortion model and the tangential distortion model.

[0018] Furthermore, the polynomial model correction method is used to correct the first image through the distortion model, and the specific method of obtaining the second image is:

[0019] A distortion transformation model is established between the image points imaged by the ideal linear model and the image points of the distorted image imaged by the nonlinear imaging model. The distortion transformation model is used to solve the mapping relationship between the positions of each pixel point on the first image and the positions where they should be when imaged under the linear model, so as to restore the pixel points at the distorted position to the normal position and obtain the second image.

[0020] Furthermore, the specific method of using the direct linear transformation algorithm to perform pitch transformation on the second image to obtain the third image is:

[0021] Solve the homography transformation matrix to establish a linear relationship between the world coordinate system and the image coordinate system to transform the second image between the perspective coordinate system and the image plane coordinate system, and transform the second image from a shooting angle with a certain angle to the ground to a bird's-eye view to obtain the third image.

[0022] Furthermore, the specific method of using a weighted average algorithm to eliminate the stitching gaps of all third images to obtain a panoramic surround image is as follows:

[0023] Weights are added on the basis of the average value method, and the pixel grayscale values ​​of the feature points in the overlapping area of ​​the two adjacent third images after registration are weighted averaged according to certain weights to obtain the solution result, and the solution result is used as the new grayscale value of the fused feature point to eliminate the stitching gaps of the two adjacent third images, until all the stitching gaps of the two adjacent third images are eliminated to obtain a panoramic surround image.

[0024] Furthermore, in the process of performing target detection on panoramic surround images through the multi-task perception algorithm of YOLOv5s, an image detection decoder is established. The image detection decoder includes 4 residual blocks, 3 upsampling modules and 1 spatial pyramid pooling module. The drivable area segmentation result is obtained through 4 residual blocks and 3 upsampling modules. In 1 spatial pyramid pooling module, multiple small-size convolution kernels are used to replace a single large-size convolution kernel to obtain the lane line segmentation result.

[0025] A panoramic surround-view assisted driving system based on machine vision, comprising:

[0026] A shooting module, used for acquiring a plurality of first images around the special vehicle using a plurality of fisheye cameras;

[0027] Calibration module, used to obtain the distortion model of the fisheye camera;

[0028] A correction module, used to correct the first image through a distortion model using a polynomial model correction method to obtain a second image;

[0029] A transformation module, used for performing a pitch image transformation on the second image using a direct linear transformation algorithm to obtain a third image;

[0030] An image fusion module is used to eliminate the stitching gaps of all third images using a weighted average algorithm to obtain a panoramic surround image;

[0031] The target detection module is used to detect targets in the panoramic surround image based on the YOLOv5s multi-task perception algorithm, and obtain the segmentation results of the drivable area and lane lines in the panoramic surround image;

[0032] The display module is used to display the panoramic surround image and the targets in the panoramic surround image.

[0033] An electronic device includes multiple fisheye cameras, a memory, a processor, and a computer program stored in the memory and executable on the processor. The multiple fisheye cameras are used to obtain multiple first images around a special vehicle. When the processor executes the program, the above-mentioned panoramic surround view assisted driving method based on machine vision is implemented.

[0034] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned machine vision-based panoramic surround view assisted driving method.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] (1) The present invention proposes a panoramic surround-view assisted driving method based on machine vision, which is specially designed for special vehicles. The software uses image processing and pattern recognition technology, combined with reality enhancement methods, to fully display the fused image of the virtual vehicle, the bottom of the vehicle and the surrounding road surface, intelligently identify road scene information, eliminate blind spots in the field of vision, and enhance the field of vision of special vehicle drivers.

[0037] (2) The core functions of the present invention include environmental perception technologies such as multi-view image fusion, chassis perspective, and road scene perception. By fusing the images captured by multiple cameras to generate a panoramic image, the driver can observe the vehicle and the surrounding environment in all directions. In order to allow the driver to clearly understand the situation under the vehicle and provide a safer driving experience, the present invention uses a chassis perspective function to display a fused image of the virtual vehicle and the bottom of the vehicle. In order to help the driver better understand and judge the road conditions and reduce the risk of accidents, the design software can intelligently identify road scene information, such as lane lines, traffic signs, etc.

[0038] (3) The panoramic surround view assisted driving method of the present invention realizes the field of vision enhancement and safe assisted driving of special vehicle drivers through image processing technology and YOLOv5s-based image detection technology design software. The method can provide a reliable and efficient solution to help special vehicle drivers better understand the situation around the vehicle, enhance the driver's control ability, reduce the risk of accidents, and provide a good driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 This is a flow chart of a panoramic surround-view assisted driving method based on machine vision of the present invention;

[0041] Figure 2 is a first schematic diagram of a panoramic surround view assisted driving system based on machine vision of the present invention;

[0042] Figure 3 is a second schematic diagram of a panoramic surround view assisted driving system based on machine vision of the present invention;

[0043] Figure 4 It is the initial interface of the panoramic surround view assisted driving system based on machine vision in the specific implementation mode of the present invention;

[0044] FIG5( a ) is a schematic diagram of a perspective panoramic image with a chassis in a specific embodiment of the present invention;

[0045] FIG5( b ) is a schematic diagram of a front image in a specific embodiment of the present invention;

[0046] FIG5( c ) is a schematic diagram of a rear image in a specific embodiment of the present invention;

[0047] FIG5( d ) is a schematic diagram of an image on the left side in a specific embodiment of the present invention;

[0048] FIG5( e ) is a schematic diagram of an image on the right side in a specific embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of target detection in a specific embodiment of the present invention;

[0050] Figure 7 This is the software closing interface of the specific implementation mode of the present invention. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0052] Example 1

[0053] Embodiment 1 provides a panoramic surround-view assisted driving method based on machine vision, such as Figure 1 As shown, including:

[0054] Step S1: using a fisheye camera to obtain a plurality of first images around the special vehicle;

[0055] Step S2: Obtaining a distortion model of the fisheye camera;

[0056] Step S3: using a polynomial model correction method to correct the first image through a distortion model to obtain a second image;

[0057] Step S4: using a direct linear transformation algorithm to perform a pitch transformation on the second image to obtain a third image;

[0058] Step S5: using a weighted average algorithm to eliminate the stitching gaps of all third images to obtain a panoramic image;

[0059] Step S6: Detect targets on the panoramic surround image based on the multi-task perception algorithm of YOLOv5s, and obtain segmentation results of the drivable area and lane lines in the panoramic surround image;

[0060] Step S7: displaying the panoramic surround image and the target in the panoramic surround image.

[0061] In this embodiment, by understanding the current status of research on intelligent assisted driving systems at home and abroad, a panoramic surround assisted driving method based on machine vision is proposed, which is specially designed for special vehicles. The software uses image processing and pattern recognition technology, combined with reality enhancement methods, to fully display the virtual vehicle, the bottom of the vehicle and the surrounding road surface fusion image, intelligently identify road scene information, eliminate blind spots, and enhance the vision of special vehicle drivers.

[0062] In this embodiment, the core functions include environmental perception technologies such as multi-view image fusion, chassis perspective, and road scene perception. By fusing the images captured by multiple cameras, a panoramic image is generated, allowing the driver to observe the vehicle and the surrounding environment in all directions. In order to allow the driver to clearly understand the situation under the vehicle and provide a safer driving experience, this design uses a chassis perspective function to display a fused image of the virtual vehicle and the bottom of the vehicle. In order to help the driver better understand and judge the road conditions and reduce the risk of accidents, the design software can intelligently identify road scene information, such as lane lines, traffic signs, etc.

[0063] In this embodiment, the panoramic surround view assisted driving method realizes the field of vision enhancement and safe assisted driving of special vehicle drivers through image processing technology and YOLOv5s-based image detection technology design software. This method can provide a reliable and efficient solution to help special vehicle drivers better understand the situation around the vehicle, enhance the driver's control ability, reduce the risk of accidents, and provide a good driving experience.

[0064] In this embodiment, the number of fisheye cameras is greater than or equal to 5, and the multiple fisheye cameras are installed on the front, back, left, right and bottom of the special vehicle.

[0065] Images captured by fisheye cameras will have serious distortions, which are mainly divided into two types: radial distortion and tangential distortion. The main reason for radial distortion is that the shape of the fisheye lens changes the radial rate, causing the light to deviate from the ideal linear path, resulting in obvious radial distortion at the edge of the image. Tangential distortion is mainly caused by some defects in the production and assembly process of the lens, which causes a certain angle between the lens plane and the image plane.

[0066] In this embodiment, an object point in three-dimensional space is projected onto a two-dimensional plane image through a fisheye camera, and there is often a mapping relationship between the spatial coordinates of the object point and the coordinates of the imaging plane. The calibration technology of the fisheye camera is to use the idea of ​​modeling to obtain some point or line information from the image taken by the fisheye camera, and use the calibration algorithm to solve the mapping relationship and related parameters between the spatial coordinates of the object point and the coordinates of the imaging plane. The relevant parameters of the fisheye camera usually refer to internal parameters, external parameters and distortion coefficients. Through the external parameters, it can be known how the object point is rotated and translated from the world coordinates to the fisheye camera coordinates. Through the internal parameters, it can be known how the object point is imaged through the fisheye camera lens on the basis of the fisheye camera coordinate system, and finally electronically converted into pixel points. The distortion coefficient reflects the offset of the imaging point. The calibration technology of the fisheye camera can establish a mapping relationship between the world coordinate system in three-dimensional space and the image coordinate system on the two-dimensional plane.

[0067] Zhang Zhengyou's plane calibration method is more flexible than the above two-step calibration method. This method only needs to shoot a calibration plate with a fisheye camera to solve the internal parameters and distortion coefficients of the fisheye camera. This method is more flexible than the traditional calibration method in that it does not require the fisheye camera to be fixed. The fisheye camera can move or rotate freely when shooting images, and accurate calibration results can be obtained without knowing the movement information or rotation angle. Zhang Zhengyou's plane calibration method points out that the calibration process requires at least 3 calibration plate image information taken at different angles to obtain the calibration result. In the actual calibration process, more than a dozen or more calibration plate images at different angles are usually calibrated at the same time, so that the calibration results are more accurate and reliable.

[0068] In this embodiment, the distortion of the image only produces a certain deformation, making it no longer similar to the original scene, and does not change the clarity of the image. This deformation is a regular deformation that does not affect the acquisition of scene information from the image. After the intrinsic parameter matrix and distortion coefficient are obtained through the fisheye camera calibration, the fisheye image can be corrected using the distortion correction algorithm to finally obtain a distortion-free image, so that the fisheye image can provide a larger viewing angle information while maintaining normal visual habits.

[0069] The basic idea of ​​the polynomial model correction method is to establish a distortion transformation model between the image points of the ideal linear model and the image points of the distorted image imaged by the nonlinear imaging model, so as to solve the mapping relationship between the positions of each pixel point on the distorted image and the original position when imaging under the linear model, thereby restoring the pixel points at the distorted position to the normal position to achieve the effect of eliminating distortion.

[0070] In this embodiment, the specific method for obtaining the distortion model of the fisheye camera is:

[0071] The internal parameters and distortion coefficients of the fisheye camera are solved through Zhang Zhengyou's plane calibration method, and the mapping relationship between the world coordinate system in the three-dimensional space and the image coordinate system on the two-dimensional plane is established. The radial distortion model and the tangential distortion model are established for the fisheye camera respectively, and the distortion model is obtained by combining the radial distortion model and the tangential distortion model.

[0072] The polynomial model correction method is used to correct the first image through the distortion model, and the specific method of obtaining the second image is as follows:

[0073] A distortion transformation model is established between the image points imaged by the ideal linear model and the image points of the distorted image imaged by the nonlinear imaging model. The distortion transformation model is used to solve the mapping relationship between the positions of each pixel point on the first image and the positions where they should be when imaged under the linear model, so as to restore the pixel points at the distorted position to the normal position and obtain the second image.

[0074] In this embodiment, after the image collected by the fisheye camera is corrected for distortion and appropriate interpolation correction, more than 5 non-vertical perspective images with a certain angle to the ground are obtained around the vehicle body. If such images are directly spliced, the actual effect picture obtained is a deformed image that does not conform to the normal visual effect of the human eye. If such an image is presented to the driver, it cannot clearly and intuitively show the normal scene around the car. Instead, it will cause other traffic accidents because of the inability to obtain accurate road information. Therefore, in order to effectively improve the effect of image splicing, after obtaining a non-vertical perspective image with a certain angle to the ground, it is necessary to convert the image into a top view, that is, to perform a top-down projection transformation. The image effect after the top-down projection transformation is a top-down view of the car and the surrounding road surface from above the car, which conforms to normal visual habits and makes it easier for the driver to accept.

[0075] The specific method of using the direct linear transformation algorithm to transform the second image into a pitch image and obtain the third and fourth images is as follows:

[0076] Solve the homography transformation matrix to establish a linear relationship between the world coordinate system and the image coordinate system to transform the second image between the perspective coordinate system and the image plane coordinate system, and transform the second image from a shooting angle with a certain angle to the ground to a bird's-eye view to obtain the third image.

[0077] Its basic formula is shown in formula (1):

[0078]

[0079] Among them, (X, Y, Z) is the position of the image in the perspective coordinate system before the direct linear transformation, (u, v) is the coordinate on the image plane, indicating the position of the image after the direct linear transformation, l1, l2, l3, l4, l5, l6, l7, l8, l9, l 10 , l 11 is the image transformation parameter. By solving the 11 unknown parameters in equation (1), we can get the linear relationship between the coordinate systems. Usually, assuming that the fisheye camera is shooting a horizontal ground, it is assumed that Z=0 in the (X, Y, Z) coordinate system, and equation (3.6) is simplified to equation (2):

[0080]

[0081] After Z=0, there are only 8 unknown parameters left in the simplified formula (2): l1, l2, l4, l5, l6, l8, l9, l 10, to solve these 8 unknown parameters, we need to select 4 points at random on the perspective image, and obtain the corresponding coordinates of these 4 points in the image coordinate system by the corner point detection method. Using the coordinates of these 4 corresponding points, we can construct 8 equations to solve the 8 unknown parameters, and thus solve the homography transformation matrix H. The transformation from perspective image to top view image can be expressed as formula (3):

[0082] I 俯视图 =HI 透视图 (3)

[0083] In this embodiment, in the actual application of the vehicle-mounted panoramic view system, fisheye cameras are installed in 5 directions of the car. The images collected by the fisheye cameras at different positions have certain differences in brightness due to factors such as the illumination angle, which makes the spliced ​​image appear different in brightness: and when collecting moving images, it will cause the spliced ​​image to appear ghosting and other distortion phenomena. In order to solve such problems, it is necessary to use image fusion technology to effectively improve the quality of the spliced ​​image, and provide the driver with a better panoramic image in the vehicle-mounted panoramic view system. The two images that are initially spliced ​​together after image registration will have an obvious splicing gap in the splicing area, and if the brightness on both sides of the splicing gap is different, the spliced ​​image will be uneven in brightness and dark, affecting the visual effect. At this time, it is necessary to use image fusion technology to make the spliced ​​image naturally transition in the overlapping area, the overall brightness is balanced, and no obvious splicing marks can be seen. Since the vehicle-mounted panoramic view system needs to provide the driver with high-quality spliced ​​images, this embodiment chooses to use a pixel-level image fusion algorithm-weighted average method, which has a slow calculation process but a reliable fusion effect and is easy to implement.

[0084] The specific method of using the weighted average algorithm to eliminate the stitching gaps of all third images and obtain the panoramic surround image is as follows:

[0085] Weights are added on the basis of the average value method, and the pixel grayscale values ​​of the feature points in the overlapping area of ​​the two adjacent third images after registration are weighted averaged according to certain weights to obtain the solution result, and the solution result is used as the new grayscale value of the fused feature point to eliminate the stitching gaps of the two adjacent third images, until all the stitching gaps of the two adjacent third images are eliminated to obtain a panoramic surround image.

[0086] In this embodiment, the network structure of the multi-task perception algorithm of YOLOv5s consists of an encoder and a decoder, wherein the encoder consists of a backbone network and a feature fusion network, and the decoder consists of a drivable area decoder, a lane line decoder, and a vehicle detection decoder. The encoder is used to extract image features required for different tasks and aggregate the features so that the model can learn multi-scale feature information. The decoder designs corresponding semantic segmentation structures or target detection structures according to the characteristics of different tasks so that the network can detect three objects at the same time.

[0087] In this embodiment, in order to avoid the influence of network signal delay on data transmission, the road scene perception algorithm is generally deployed offline in the embedded device of the vehicle platform. Therefore, the algorithm is required to maintain detection accuracy under the support of hardware computing power with weak performance, and also have a faster reasoning speed to meet the real-time requirements of the autonomous driving system. In this embodiment, the C3 module and the standard 3×3 convolution layer of the YOLOv5s backbone network structure are improved to make the model more lightweight to meet the real-time requirements of the multi-task model. The backbone network of YOLOv5s uses the residual unit as the basic unit of the C3 module. First, 1×1 convolution and 3×3 convolution are combined, and then 1×1 convolution reduces the number of channels to half of the input feature map, and 3×3 convolution is used to restore the number of channels, and then the jump connection is used to perform channel dimension feature superposition with the input feature to obtain the residual unit output. The jump connection directly transfers the input feature to the output, so that the gradient of the back propagation can be effectively transferred to the front network, thereby effectively training the shallow network layer parameters. Therefore, the residual unit guarantees the feature extraction capability while reducing the number of parameters.

[0088] In order to meet the real-time and accuracy requirements of the multi-task environmental perception model on mobile hardware such as edge devices, this embodiment introduces the RVB structure to replace the residual unit of the C3 module to obtain the RepC3 module. Compared with the original C3 module structure, in the training phase, the RepC3 module adds a gradient return branch; in the model inference phase, the 3-branch convolution layer is converted to a single-branch structure to better improve the memory utilization of the device, thereby improving the inference speed of the model.

[0089] The Neck structure of YOLOv5s constructs a feature pyramid network (FPN) and a feature aggregation network (PAN) to fuse and redistribute the extracted multi-scale features, thereby improving the reliability of the network's detection results for targets of different scales. The PAN structure uses Concat to splice feature maps of different scales in the channel dimension, resulting in an increase in the number of channels in the output feature map. In addition, due to the relative displacement between objects in the road scene, the posture of the target will change, which is not conducive to the detection of small targets. In order to improve the reasoning efficiency of the multi-task network algorithm on the vehicle computing platform and the ability to focus on important small-scale targets, this embodiment combines RepC3 and the position attention mechanism module to construct a more effective feature fusion structure.

[0090] First, the C3 module is replaced by RepC3, and the 3×3 convolution of the PAN structure is adjusted to the RVB module. Then, Add is used to replace Concat without changing the number of channels of the feature map. However, while reducing the number of parameters, the Add feature fusion will reduce the model accuracy. In order to eliminate the above problems, this embodiment adjusts the medium-scale fusion structure of the feature fusion network. In order to enhance the model's detection ability for small targets, this embodiment introduces an attention mechanism in the feature fusion network. The attention mechanism enables the model to pay more attention to features that are more important for the prediction results. However, in the road scene environment perception model, the attention module will reduce the real-time performance of the model. At the same time, the small target vehicles that are easily missed and misdetected in the road scene are mostly small target vehicles far away from the driver. Based on the above influencing factors, this embodiment inserts an attention module into the small-scale target detection branch of the feature fusion network structure. In order to solve the problem that vehicles are prone to occlusion in road scenes, resulting in the loss of model detection accuracy, this embodiment uses a position attention mechanism module that combines position information and channel information. The attention mechanism assigns high weights to small occluded targets that are difficult to detect to increase attention, and assigns low weights to road background objects to suppress them, so as to improve the detection accuracy of vehicles in road scenes.

[0091] In this embodiment, during the process of performing target detection on the panoramic surround image through the multi-task perception algorithm of YOLOv5s, a drivable area decoder and a lane line decoder are established. The drivable area decoder includes 4 residual blocks and 3 upsampling modules, and the lane line decoder includes 1 spatial pyramid pooling module. The drivable area segmentation result is obtained through 4 residual blocks and 3 upsampling modules. In the 1 spatial pyramid pooling module, multiple small-size convolution kernels are applied to replace a single large-size convolution kernel to obtain the lane line segmentation result.

[0092] In order to complete the lane detection and drivable area segmentation tasks, it is necessary to add a corresponding semantic segmentation task decoder based on the YOLOv5s target detection decoder. Since the drivable area accounts for a large proportion of the number of pixels in the road image, there will be no imbalance between positive and negative samples during the training process. Therefore, in order to improve its real-time performance, this embodiment designs a decoder with fewer network layers to classify each pixel in the drivable area. The drivable area decoder is mainly composed of 4 RVBs and 3 upsampling modules. Assuming that the size of the input image is (w, h, 3), the decoder first receives the shallow features extracted by the backbone network, whose size is (w / 8, h / 8, 128); then RVB is applied for dimensionality reduction, and the number of channels is adjusted to 64; then the structure combined with RVB and bilinear interpolation is applied to upsample the feature map to (w / 4, h / 4, 32), (w / 2, h / 2, 16), (w, h, 2) in turn, and the drivable area segmentation result is obtained.

[0093] Since lane lines are long and thin and their texture is different from the long strip signs on the road surface, they account for a much smaller proportion of the road scene than the drivable area. Therefore, the SPPF module is introduced on the basis of the drivable area segmentation decoder to enhance its feature extraction capability, and multiple small-sized convolution kernels are used to replace the single large-sized convolution kernel in the SPP module, which not only integrates receptive fields of different scales but also improves the running speed. Compared with the drivable area decoder structure, except for the addition of an SPPF module after adjusting the channel dimension, the rest of the structure is basically the same.

[0094] Since the model of this embodiment needs to complete traffic vehicle detection, lane line detection, and drivable area segmentation at the same time, a weighted loss function corresponding to each of the three tasks is used to train the model, as shown in formula (4).

[0095] L all =λ1L det +λ2L daseg +λ3L llseg (4)

[0096] Among them, λ1, λ2, λ3 are the single task loss weight values; L det is the target detection branch loss function; L daseg is the driving area segmentation loss function; L llseg is the lane segmentation loss function; L all is the overall structure loss function of the network.

[0097] The target detection branch loss function consists of three parts: classification loss, confidence loss, and target box regression loss:

[0098] L det =α1L class +α2Lconf +α3L EIoU (5)

[0099] Among them, α1, α2, α3 are the weight values ​​of each loss function term; L class and L conf For classification loss and confidence loss, the binary cross entropy loss function is used; the target box regression loss function L EIOU Use EIOU Loss to constrain the overlap area, center point distance, width, and height difference between the predicted box and the true box.

[0100] The lane detection branch loss function consists of a cross entropy loss and a Dice loss function. The cross entropy loss function is used to obtain the loss value by comparing each pixel one by one. Since the number of lane line pixels is far less than that of background pixels, using only the cross entropy loss function will cause the loss function to tend toward background pixels. Therefore, this embodiment uses the Dice loss function to suppress the problem of sample category imbalance. The formula is:

[0101] L llseg =L CE +γ1L dice (6)

[0102]

[0103] Among them, γ1 is the weight value of Dice loss function; p i j is the number of pixels predicted as category j when the true label is category i; k is the maximum number in the semantic segmentation category label, k+1 is the total number of categories, and k=1 in this experiment.

[0104] The drivable area detection has a large number of target pixels and is less difficult to detect, so only the cross entropy is used as the loss function:

[0105] L daseg =L CE (8)

[0106] Example 2

[0107] Embodiment 2 provides a panoramic surround view assisted driving system based on machine vision, such as Figure 2 and Figure 3 As shown, including:

[0108] A shooting module, used for acquiring a plurality of first images around the special vehicle by using a fisheye camera;

[0109] Calibration module, used to obtain the distortion model of the fisheye camera;

[0110] A correction module, used to correct the first image through a distortion model using a polynomial model correction method to obtain a second image;

[0111] A transformation module, used for performing a pitch image transformation on the second image using a direct linear transformation algorithm to obtain a third image;

[0112] An image fusion module is used to eliminate the stitching gaps of all third images using a weighted average algorithm to obtain a panoramic surround image;

[0113] The target detection module is used to detect targets in the panoramic surround image based on the YOLOv5s multi-task perception algorithm, and obtain the segmentation results of the drivable area and lane lines in the panoramic surround image;

[0114] The display module is used to display the panoramic surround image and the targets in the panoramic surround image.

[0115] In a specific implementation of this embodiment, the machine vision-based panoramic surround view assisted driving system aims to eliminate all visual blind spots for the driver as much as possible during driving, and to reduce the occurrence of traffic accidents to a certain extent. The system collects images from the four directions of the front, back, left, and right of the car by installing fisheye cameras on the four directions of the car, and after a series of image processing processes, the four images are spliced ​​into a panoramic image and displayed on the central control screen of the car.

[0116] like Figure 4 As shown, it is the initial interface of the panoramic surround view assisted driving system based on machine vision in a specific implementation manner of the present invention. After the system installation and loading are completed, select the entry mode on the homepage for use.

[0117] like Figure 5(a)-Figure 5(e) As shown, when the system starts to be used, different detection categories can be selected. Cameras in different directions will be called to collect images, and target detection will be performed on the collected images in real time. The detection results will be displayed in the result box below. At the same time, if vehicles and people are detected, the results will also be displayed in the bar chart on the right.

[0118] like Figure 6 As shown, it is a schematic diagram of target detection in a specific implementation manner of the present invention, and the system is capable of performing target detection.

[0119] like Figure 7 As shown, it is the software closing interface of the specific implementation mode of the present invention. When the system is finished, the user can click the "×" button in the upper right corner to choose to exit the software or cancel the exit.

[0120] Example 3

[0121] Embodiment 3 provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned panoramic surround view assisted driving method based on machine vision is implemented.

[0122] Example 4

[0123] Embodiment 4 provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned panoramic surround view assisted driving method based on machine vision is implemented.

[0124] The memory in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0125] The method for panoramic surround view assisted driving based on machine vision disclosed in the embodiment of the present invention can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method for panoramic surround view assisted driving based on machine vision can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory, and the processor reads the information in the memory, and combines its hardware to complete the steps of the method for panoramic surround view assisted driving based on machine vision provided in the embodiment of the present invention.

[0126] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), FPGA, general purpose processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0127] It can be understood that the memory can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0128] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A panoramic surround-view assisted driving method based on machine vision, characterized in that: include: Using multiple fisheye cameras to obtain multiple first images around the special vehicle; Get the distortion model of the fisheye camera; The first image is corrected by using a polynomial model correction method through a distortion model to obtain a second image; Performing a pitch transformation on the second image using a direct linear transformation algorithm to obtain a third image; A weighted average algorithm is used to eliminate the stitching gaps of all third images to obtain a panoramic image; Perform target detection on the panoramic surround image based on the YOLOv5s multi-task perception algorithm, and obtain the drivable area segmentation results and lane line segmentation results in the panoramic surround image; The panoramic view image and the targets in the panoramic view image are displayed.

2. The method for assisted driving based on machine vision according to claim 1, characterized in that: The number of the fisheye cameras is greater than or equal to 5, and the multiple fisheye cameras are installed around the special vehicle and on the bottom of the special vehicle.

3. The method for assisted driving based on machine vision according to claim 1, characterized in that: The specific method to obtain the distortion model of the fisheye camera is: The internal parameters and distortion coefficients of the fisheye camera are solved through Zhang Zhengyou's plane calibration method, and the mapping relationship between the world coordinate system in the three-dimensional space and the image coordinate system on the two-dimensional plane is established. The radial distortion model and the tangential distortion model are established for the fisheye camera respectively, and the distortion model is obtained by combining the radial distortion model and the tangential distortion model.

4. The method for assisted driving based on machine vision according to claim 1, characterized in that: The polynomial model correction method is used to correct the first image through the distortion model, and the specific method of obtaining the second image is as follows: A distortion transformation model is established between the image points imaged by the ideal linear model and the image points of the distorted image imaged by the nonlinear imaging model. The distortion transformation model is used to solve the mapping relationship between the positions of each pixel point on the first image and the positions where they should be when imaged under the linear model, so as to restore the pixel points at the distorted position to the normal position and obtain the second image.

5. The method for assisted driving based on machine vision according to claim 1, characterized in that: The specific method of using the direct linear transformation algorithm to transform the second image into a pitch image to obtain the third image is: Solve the homography transformation matrix to establish a linear relationship between the world coordinate system and the image coordinate system to transform the second image between the perspective coordinate system and the image plane coordinate system, and transform the second image from a shooting angle with a certain angle to the ground to a bird's-eye view to obtain the third image.

6. The method for assisted driving based on machine vision according to claim 1, characterized in that: The specific method of using the weighted average algorithm to eliminate the stitching gaps of all third images and obtain the panoramic surround image is as follows: Weights are added on the basis of the average value method, and the pixel grayscale values ​​of the feature points in the overlapping area of ​​the two adjacent third images after registration are weighted averaged according to certain weights to obtain the solution result, and the solution result is used as the new grayscale value of the fused feature point to eliminate the stitching gaps of the two adjacent third images, until all the stitching gaps of the two adjacent third images are eliminated to obtain a panoramic surround image.

7. The method for assisted driving based on machine vision according to claim 1, characterized in that: In the process of target detection of panoramic surround images through YOLOv5s' multi-task perception algorithm, an image detection decoder is established. The image detection decoder includes 4 residual blocks, 3 upsampling modules and 1 spatial pyramid pooling module. The drivable area segmentation result is obtained through 4 residual blocks and 3 upsampling modules. In 1 spatial pyramid pooling module, multiple small-size convolution kernels are used to replace a single large-size convolution kernel to obtain the lane line segmentation result.

8. A panoramic view assisted driving system based on machine vision, characterized in that: include: A shooting module, used for acquiring a plurality of first images around the special vehicle using a plurality of fisheye cameras; Calibration module, used to obtain the distortion model of the fisheye camera; A correction module, used to correct the first image through a distortion model using a polynomial model correction method to obtain a second image; A transformation module, used for performing a pitch image transformation on the second image using a direct linear transformation algorithm to obtain a third image; An image fusion module is used to eliminate the stitching gaps of all third images using a weighted average algorithm to obtain a panoramic surround image; The target detection module is used to detect targets in the panoramic surround image based on the YOLOv5s multi-task perception algorithm, and obtain the segmentation results of the drivable area and lane lines in the panoramic surround image; The display module is used to display the panoramic surround image and the targets in the panoramic surround image.

9. An electronic device, characterized in that: The invention comprises a plurality of fisheye cameras, a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the plurality of fisheye cameras are used to obtain a plurality of first images around the special vehicle, and when the processor executes the program, a panoramic surround view assisted driving method based on machine vision as claimed in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the panoramic surround view assisted driving method based on machine vision as described in any one of claims 1 to 7 is implemented.

Images