A Detection Method and System Based on Visible Driving Area

Abstract

An embodiment of the present invention provides a detection method and system based on a visually drivable area. The method includes: obtaining a real-time fisheye image in a vehicle camera video stream signal, and determining a first longitude and latitude unfolded image corresponding to the real-time fisheye image according to a first preset mapping relationship, where the first preset mapping relationship is a mapping relationship from the fisheye image to the longitude and latitude unfolded image; using the first longitude and latitude unfolded image as the input of a deep network model to obtain a pixel-level classification result of the first longitude and latitude unfolded image; traversing the pixel-level classification result through a boundary search algorithm to obtain multiple boundary points of the drivable area; and determining the real distance between the vehicle and the real position corresponding to each boundary point according to a second preset mapping relationship, where the second preset mapping relationship includes a mapping relationship between each pixel point in the longitude and latitude unfolded image and the real distance between the vehicle and the real position corresponding to the pixel point. The present invention can effectively warn of potential dangers in the horizontal and vertical directions of the vehicle and achieve all-round warning.

Description

Technical Field

[0001] The present invention relates to the technical field of driving detection, and more particularly, to a method and system for detecting a visible driving area based on vision. Background Art

[0002] With the continuous development of science and technology and the improvement of living standards, users have higher and higher requirements for the safety performance of vehicles. During the driving process of a vehicle, due to the limitation of the visible range of the cab, accidents such as longitudinal rear-end collisions and lateral rubbing may occur, which pose certain safety hazards to the passengers in the vehicle and cause certain property losses. To avoid the occurrence of the above accidents, traditional methods mainly use millimeter-wave radars for detection, but they are greatly interfered by factors such as weather conditions, or use monocular cameras, but lateral hazards cannot be detected. Currently, the methods in the prior art use detection algorithms for the fisheye panoramic visible driving area, but due to the large distortion of the fisheye camera, objects imaged at the edge of the camera or relatively large objects at a certain distance from the vehicle in the real world only show a few pixels in the picture. This causes great interference to detection and segmentation algorithms, and the accuracy is not high. Summary of the Invention

[0003] This specification provides a method and system for detecting a visible driving area based on vision to overcome at least one technical problem existing in the prior art.

[0004] According to an embodiment of this specification, a method for detecting a visible driving area based on vision is provided, and the method includes:

[0005] Obtain a real-time fisheye image from a vehicle camera video stream signal, and determine a first longitude-latitude unfolded image corresponding to the real-time fisheye image according to a first preset mapping relationship, where the first preset mapping relationship is a mapping relationship from a fisheye image to a longitude-latitude unfolded image;

[0006] Use the first longitude-latitude unfolded image as the input of a deep network model to obtain a pixel-level classification result of the first longitude-latitude unfolded image;

[0007] Traverse the pixel-level classification result through a boundary search algorithm to obtain multiple boundary points of the visible driving area;

[0008] Determine the real distance between the vehicle and the real position corresponding to each boundary point according to a second preset mapping relationship, where the second preset mapping relationship includes a mapping relationship between each pixel point in the longitude-latitude unfolded image and the real distance between the vehicle and the real position corresponding to the pixel point.

[0009] Optionally, before obtaining the real-time fisheye image in the vehicle camera video stream signal and determining the first longitude-latitude unfolded image corresponding to the real-time fisheye image according to the first preset mapping relationship, the method further includes:

[0010] Obtain the fisheye image of the vehicle camera, and convert the fisheye image into a longitude-latitude unfolded image by using a longitude-latitude unfolding algorithm;

[0011] Determine the mapping relationship from the fisheye image to the longitude-latitude unfolded image according to the fisheye image and the longitude-latitude unfolded image, and generate the first preset mapping relationship.

[0012] Further optionally, the specific steps of converting the fisheye image into a longitude-latitude unfolded image by using a longitude-latitude unfolding algorithm include:

[0013] Perform normalization processing on the fisheye image, map the pixels of the fisheye image onto a 2D unit circle model, and obtain the pixel coordinates (x f , y f ) in the unit fisheye coordinate system. Among them, in the unit fisheye coordinate system, the calculation formula of the imaging height r and the included angle ф in the fisheye image is as follows:

[0014]

[0015] Among them, the relationship between the incident angle θ of the lens of the vehicle camera and the imaging height r in the fisheye image is r = f(θ);

[0016] Map the pixel coordinates (x f , y f ) from the 2D unit circle model to the unit spherical coordinates (x p , y p , z p ) of the 3D unit sphere model. Among them, the correspondence between the unit spherical coordinates (x p , y p , z p ) and the incident angle θ and the included angle ф is:

[0017]

[0018] Convert the unit spherical coordinates (x p , y p , z p ) into unit spherical longitude-latitude coordinates (longitude, latitude). Among them, the correspondence between the unit spherical longitude-latitude coordinates (longitude, latitude) and the unit spherical coordinates (x p , y p , z p) The corresponding relationship between them is:

[0019]

[0020] Map the longitude and latitude coordinates (longitude, latitude) of the unit sphere from the 3D unit sphere model to the 2D unit longitude and latitude unfolded coordinates (x e , y e ), to obtain the longitude and latitude unfolded image, where the corresponding relationship between the unit longitude and latitude unfolded coordinates (x e , y e ) and the longitude and latitude coordinates (longitude, latitude) of the unit sphere is:

[0021]

[0022] Further optionally, the 3D unit sphere model is the unit sphere model multiplied by the rotation matrix of the unit sphere coordinates and the world coordinates.

[0023] Further optionally, the deep network model adopts an encoder-decoder network structure, and the deep network model is trained through the following steps:

[0024] Construct a deep network model and obtain the longitude and latitude unfolded image as the training sample set;

[0025] Perform deep network training on the training sample set to obtain the trained deep network model.

[0026] Further optionally, before traversing the pixel-level classification result through the boundary search algorithm to obtain multiple boundary points of the drivable area, the method further includes:

[0027] Set a search source, and use the bottom edge pixels of the longitude and latitude unfolded image as the radiation source of the search source;

[0028] Set a search line, starting from the radiation source, vertically upward, and sampling at a set interval;

[0029] Search, starting from the search source, comparing the pixel-level categories of each search point one by one;

[0030] If the pixel-level category of the current search point is inconsistent with the pixel-level category of the next search point, it is determined that a jump from the vehicle body to the drivable area or from the drivable area to the obstacle area has occurred, and the boundary points that meet the boundary conditions are obtained and saved in an array, where the boundary points that meet the boundary conditions include the entry points and exit points of the drivable area;

[0031] Repeat the search for each search line until the boundary point search of the drivable area is completed;

[0032] Generate a boundary lookup table for all boundary points that meet the boundary conditions, and record the entry points of the drivable area.

[0033] Further optionally, the traversing of the pixel-level classification result by the boundary search algorithm to obtain multiple boundary points of the drivable area specifically includes:

[0034] Use the pixel points corresponding to the entry points of the drivable area in the pixel-level classification result as the starting points for the search;

[0035] During the search, starting from the starting point of the search, search upward according to the boundary lookup table;

[0036] If the pixel-level category of the current search point is inconsistent with the pixel-level category of the next search point, it is determined that a jump from the drivable area to the obstacle area has occurred, and the boundary points of the drivable area are obtained;

[0037] Traverse the entire boundary lookup table until the search for the boundary points of the drivable area is completed, and obtain multiple boundary points of the drivable area.

[0038] Further optionally, before determining the true distance between the vehicle and the true position corresponding to each boundary point according to the second preset mapping relationship, the method further includes:

[0039] Model the camera imaging model;

[0040] Under the assumption of ground level, calculate the coordinates of each pixel point of the fisheye image in the vehicle rear axle coordinate system according to the camera imaging model, and obtain the mapping relationship from the fisheye image to the vehicle rear axle coordinate system;

[0041] According to the preset mapping relationship and the mapping relationship from the fisheye image to the vehicle rear axle coordinate system, obtain the mapping relationship between the true distance between each pixel point in the longitude and latitude unfolded image and the true position corresponding to the vehicle and this pixel point, and generate the second preset mapping relationship, where the preset mapping relationship is the preset mapping relationship from the fisheye image to the longitude and latitude unfolded image or the inverse projection relationship between the longitude and latitude unfolded image and the fisheye image.

[0042] Further optionally, the camera internal parameter model of the camera imaging model is:

[0043]

[0044] where (u, v) are image coordinates, (X c , Y c , Z c ) are camera coordinates, (fx , f y , C x , C y ) is the internal parameters of the camera calibrated using Zhang Zhengyou calibration method;

[0045] The external parameter model of the camera is:

[0046]

[0047] where, (X w , Y w , Z w ) represents the coordinates of the rear axle of the vehicle, is the transformation matrix between the camera coordinates and the rear axle coordinates of the vehicle;

[0048] Under the assumption that the ground is horizontal, the Z of each of the pixel points in the rear axle coordinate system of the vehicle w is 0. Then, according to the above formulas (5) and (6), the coordinates (X w , Y w ) of the pixel point in the rear axle coordinate system of the vehicle are calculated.

[0049] According to the embodiments of the present specification, a detection system based on a visual drivable area is also provided, including:

[0050] An image acquisition module, configured to acquire a real-time fisheye image in the video stream signal of the vehicle camera;

[0051] A longitude and latitude unfolding module, configured to determine a first longitude and latitude unfolded image corresponding to the real-time fisheye image according to a first preset mapping relationship, where the first preset mapping relationship is a mapping relationship from the fisheye image to the longitude and latitude unfolded image;

[0052] A deep network inference module, configured to use the first longitude and latitude unfolded image as the input of a deep network model to obtain a pixel-level classification result of the first longitude and latitude unfolded image;

[0053] A boundary point search module, configured to traverse the pixel-level classification result through a boundary search algorithm to obtain multiple boundary points of the drivable area;

[0054] An obstacle discrimination module, configured to determine the real distance between the vehicle and the real position corresponding to each of the boundary points according to a second preset mapping relationship, where the second preset mapping relationship includes a mapping relationship between each pixel point in the longitude and latitude unfolded image and the real distance between the vehicle and the real position corresponding to the pixel point.

[0055] The beneficial effects of the embodiments of the present specification are as follows:

[0056] Based on the longitude-latitude unfolding model of the fisheye camera, after the fisheye image is unfolded in longitude and latitude, the object will not have large distortion, basically conforming to the rule of near-big and far-small, improving the accuracy of detection and segmentation algorithms. During the driving process of the vehicle, it can more effectively warn of potential dangers in the lateral and longitudinal directions of the vehicle, and then achieve 360° blind spot-free warning.

[0057] Detect obstacles in the front, rear, left, and right directions when the vehicle is driving. Compared with the detection method of the undistorted panoramic stitching image in the prior art, the warning of the detection method in this embodiment will be earlier, the braking safety distance will be longer, the detection accuracy will be higher, and less original image information will be lost. In the offline stage of the detection method in the embodiments of this specification, the mapping relationship from the fisheye image to the longitude-latitude unfolded image and the true distance from each pixel of the longitude-latitude unfolded image to the center of the vehicle's rear axle is calculated. Since the online stage is carried out on the in-vehicle chip and the hardware computing power is limited, through these two mapping relationships, the longitude-latitude unfolded image with angular deviation and the true distance from each pixel point in the longitude-latitude unfolded image to the vehicle itself can be quickly obtained, reducing the calculation time. During the driving process, a quick response can be obtained, and the frame rate of the received image can be increased, greatly improving the safety during the driving process.

[0058] The technical effects of the embodiments of this specification include:

[0059] 1. In this embodiment, the mapping relationship from the fisheye image to the longitude-latitude unfolded image with angular deviation and the mapping relationship from each pixel of the longitude-latitude unfolded image to the true distance of the center of the vehicle's rear axle are determined offline, so that in the subsequent online stage, rapid calculations can be performed using the mapping relationships, effectively solving the problem of limited hardware computing power. Through the two mapping relationships, the longitude-latitude unfolded image with angular deviation and the true distance from each pixel point in the longitude-latitude unfolded image to the vehicle itself can be quickly obtained, reducing the calculation time, enabling a quick response, and improving the safety during the driving process, which is one of the technical effects of the embodiments of this specification.

[0060] 2. In this embodiment, angular deviation is added to the longitude-latitude unfolded image after longitude-latitude unfolding, keeping the geometric relationship of parallel and perpendicular in the fisheye image consistent with that in the world coordinate system and widening the image, not only improving the accuracy of detecting distant obstacles but also further enhancing the safety of the entire system, which is one of the technical effects of the embodiments of this specification.

[0061] 3. In this embodiment, the longitude-latitude unfolded image with angular deviation is inverse-projected into the fisheye image, and its true distance from the center of the vehicle's rear axle is calculated, losing less original image information and having higher detection accuracy, which is one of the technical effects of the embodiments of this specification.

[0062] 4. In this embodiment, the boundary point search module receives the per-pixel classification result of the network and obtains the boundary line of the drivable area. The search is more comprehensive and faster, which is one of the technical effects of the embodiments of this specification. Description of the Drawings

[0063] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0064] Figure 1 Flow diagram of the detection method for the vision-based drivable area provided by the embodiments of this specification;

[0065] Figure 2 Mapping relationship diagram from the fisheye image to the longitude-latitude unfolded image in the detection method for the vision-based drivable area provided by the embodiments of this specification; where (a) is the unit fisheye coordinate, (b) is the unit spherical coordinate, (c) is the unit spherical longitude-latitude coordinate, and (d) is the unit longitude-latitude unfolded coordinate;

[0066] Figure 3 Original fisheye image in the detection method for the vision-based drivable area provided by the embodiments of this specification;

[0067] Figure 4 Longitude-latitude unfolded image in the detection method for the vision-based drivable area provided by the embodiments of this specification;

[0068] Figure 5 Angle-offset longitude-latitude unfolded image in the detection method for the vision-based drivable area provided by the embodiments of this specification;

[0069] Figure 6 Search diagram in the offline stage of the detection method for the vision-based drivable area provided by the embodiments of this specification;

[0070] Figure 7 Search diagram in the online stage of the detection method for the vision-based drivable area provided by the embodiments of this specification;

[0071] Figure 8 Coordinate schematic diagram for calculating the real distance in the detection method for the vision-based drivable area provided by the embodiments of this specification. Detailed Embodiments

[0072] Next, the technical solutions in the embodiments of this specification will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0073] It should be noted that the terms "including" and "having" in the embodiments of this specification and their any deformations are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0074] The embodiments of this specification disclose a detection method based on a visually drivable area. On the basis that a vehicle is equipped with four panoramic cameras in the front, rear, left, and right directions, the four fisheye images are unfolded into longitude and latitude, and the longitude and latitude unfolding is used as the input of a deep network model to obtain a mask image of the drivable area. Then, a boundary search is performed on the mask image to obtain ordered scatter points of the drivable area, and post-processing is performed based on the ordered scatter points of the drivable area to calculate the minimum distance between the obstacle and the vehicle, thereby judging the existence of potential risks and making effective warnings and actions. The following will be described in detail separately.

[0075] Figure 1 FIG. shows a detection method based on a visually drivable area provided according to an embodiment of this specification. The method may include the following steps:

[0076] Step 100, obtain a real-time fisheye image in the vehicle camera video stream signal, and determine a first longitude and latitude unfolded image corresponding to the real-time fisheye image according to a first preset mapping relationship.

[0077] Among them, the vehicle camera is four panoramic cameras in the front, rear, left, and right directions of the vehicle. The first preset mapping relationship is a mapping relationship from the fisheye image to the longitude and latitude unfolded image, and specifically may be a look-up table from the fisheye image to the longitude and latitude unfolded image. This first preset mapping relationship is obtained in the offline stage. The following describes the generation process of the first preset mapping relationship:

[0078] (A1) Obtain the fisheye image of the vehicle camera, and convert the fisheye image into a longitude and latitude unfolded image by using a longitude and latitude unfolding algorithm.

[0079] The vehicle camera refers to the four-way surround-view cameras around the vehicle, and the fisheye images of the four-way surround-view cameras are unfolded in longitude and latitude. The longitude-latitude unfolding algorithm mainly realizes the correction of the fisheye image based on the longitude-latitude mapping image and the equiangular fisheye projection principle. As Figure 2 shown, the fisheye image is approximated as a unit sphere, and longitude and latitude are depicted from left to right and from bottom to top, and a longitude-latitude mapping plane is established. The established longitude-latitude mapping plane takes the longitude of the sphere as the abscissa and the latitude of the sphere as the ordinate, establishes the corresponding relationship between the point P on the sphere and the point P' on the longitude-latitude mapping plane, and maps all the pixel points on the sphere to the longitude-latitude mapping plane, thereby generating a rectangular longitude-latitude mapping image. Among them, it should be noted and understood that for the fisheye image, the changes in its longitude and latitude are both from 0 to π, so the image generated by correcting with the longitude-latitude mapping method is a square.

[0080] Specifically, as Figure 2 shown, the fisheye image is normalized, and the pixels of the fisheye image are mapped to the 2D unit circle model to obtain the pixel coordinates (x f , y f ) in the unit fisheye coordinate system. Then, the pixel coordinates (x f , y f ) are mapped from the 2D unit circle model to the unit spherical coordinates (x p , y p , z p ) of the 3D unit sphere model, and then the unit spherical coordinates (x p , y p , z p ) are converted to the unit spherical longitude-latitude coordinates (longitude, latitude). Finally, the unit spherical longitude-latitude coordinates (longitude, latitude) are mapped from the 3D unit sphere model to the 2D unit longitude-latitude unfolding coordinates (x e , y e ), thereby obtaining the longitude-latitude unfolding image.

[0081] Among them, in the unit fisheye coordinate system, given the pixel coordinates (x f , y f ) on the unit circle, as Figure 2 (a) shown, the calculation formulas for the imaging height r and the included angle ф in the fisheye image are as follows:

[0082]

[0083] As Figure 2As shown in (b), denote the unit spherical parameters of any point on the spherical surface of the 3D unit sphere model as ф and θ. In the above formula (1), the relationship between the incident angle θ of the lens of the vehicle camera and the imaging height r in the fisheye image is r = f(θ). That is to say, according to the above formula (1), the unit spherical parameters of any point on the unit sphere can be obtained. The unit spherical coordinates (x p , y p , z p ) and the corresponding relationship of the incident angle θ and the included angle ф are:

[0084]

[0085] From formula (1), the unit spherical parameters ф and θ of any point on the unit sphere can be obtained. Then, according to the above formula (2), the unit spherical coordinates (x p , y p , z p ) can be obtained.

[0086] The corresponding relationship between the unit sphere longitude and latitude coordinates (longitude, latitude) and the unit spherical coordinates (x p , y p , z p ) is:

[0087]

[0088] Convert the unit spherical coordinates (x p , y p , z p ) to the unit sphere longitude and latitude coordinates (longitude, latitude). As Figure 2 (c) shows, the position of point P remains unchanged. According to point P(x p , y p , z p ), the longitude and latitude (longitude, latitude) of the coordinates at this time can be determined by the above formula (3).

[0089] The corresponding relationship between the unit longitude and latitude expansion coordinates (x e , y e ) and the unit sphere longitude and latitude coordinates (longitude, latitude) is:

[0090]

[0091] Thus, given the unit sphere longitude and latitude coordinates (longitude, latitude), the corresponding unit longitude and latitude expansion coordinates (x e , y e ) can be obtained according to the above formula (4).

[0092] Based on the above steps, for each point on a fish-eye image, through a series of coordinate transformations, it can be mapped onto the longitude-latitude unfolded image, thus completing the correction algorithm for the fish-eye image.

[0093] In the embodiments of this specification, the original fish-eye image (as Figure 3 shown) is changed into the longitude-latitude unfolded image after longitude-latitude unfolding (as Figure 4 shown), and the overall effect is as Figure 4 shown. Longitude-latitude unfolding greatly eliminates the distortion of the fish-eye camera in the longitude and latitude directions. Among them, mainly because after the distortion of the fish-eye image is processed, most of the drivable areas cannot be inferred at some edge positions. The detection method in the embodiments of this specification uses the longitude-latitude unfolded image after longitude-latitude unfolding as the input of the drivable area detection model to solve the above problems, and the unfolded longitude-latitude unfolded image will not stretch the object like the distortion Figure 1 but widen the object, retaining a lot of effective image information.

[0094] In addition, the boundary point detection algorithm in the embodiments of this specification is a vertical sampling method. To make the longitude-latitude unfolded image after longitude-latitude unfolding satisfy the position relationship in the world coordinate system, the parallel lines and perpendicular lines in the world coordinate system should also satisfy their geometric relationships on the longitude-latitude unfolded image. In another specific embodiment, in the 3D unit sphere model, the rotation matrix between the unit spherical coordinates and the world coordinates is obtained, and the 3D unit sphere model is multiplied by the rotation matrix to obtain the longitude-latitude unfolded image with an angular offset. As Figure 5 shown, the longitude-latitude unfolded image with an angular offset satisfies the position relationship in the world coordinate system, and the vertical and parallel geometric relationships in the world coordinate system are also satisfied in the longitude-latitude unfolded image with an angular offset, greatly eliminating the vanishing point, widening the image, improving the accuracy of detecting distant obstacles, and thus improving the safety of the entire system. For the depth detection model, the longitude-latitude unfolded image after longitude-latitude unfolding is superior to the undistorted image because there are only a very small number of pixel points of a distant obstacle in the undistorted image. This phenomenon is because the parallel lines not parallel to the imaging plane will intersect at a point in the undistorted image, and the intersecting point is called the vanishing point. However, the longitude-latitude unfolded image after longitude-latitude unfolding will widen the object and increase the pixel points of the obstacle. When calculating the true distance between the boundary point of the drivable area and the vehicle's rear axle coordinate system, since the longitude-latitude unfolded image widens the object, there will be more valid points when calculating the distance, which can improve the detection accuracy of potential dangers.

[0095] (A2) Determine the mapping relationship from the fish-eye image to the longitude-latitude unfolded image according to the fish-eye image and the longitude-latitude unfolded image, and generate the first preset mapping relationship.

[0096] Among them, the first preset mapping relationship can specifically be a lookup table. After obtaining the longitude-latitude unfolded image corresponding to the fisheye image, the mapping relationship from the fisheye image to the longitude-latitude unfolded image can be determined by using the fisheye image and its corresponding longitude-latitude unfolded image. The first preset mapping relationship is generated according to this mapping relationship and saved as the first lookup table from the fisheye image to the longitude-latitude unfolded image for use in the online stage. Since the processing unit in the online stage is a vehicle-mounted chip with limited hardware computing power, the corresponding longitude-latitude unfolded image can be obtained through the lookup table, which can effectively reduce the calculation difficulty, reduce the calculation time, and respond more quickly during driving.

[0097] In a specific embodiment, during driving (i.e., the online stage), real-time fisheye images in the video stream signals of the four-way surround-view cameras in the front, rear, left, and right of the vehicle are acquired, and the first lookup table from the fisheye image to the longitude-latitude unfolded image generated in the offline stage is extracted. The first longitude-latitude unfolded image corresponding to the real-time fisheye image is calculated according to this first lookup table.

[0098] Step 200: Using the first longitude-latitude unfolded image as the input of the deep network model, the pixel-level classification result of the first longitude-latitude unfolded image is obtained.

[0099] This embodiment designs a lightweight and effective semantic segmentation network for realizing the pixel-by-pixel classification of the drivable area. The deep network model adopts an encoder-decoder network structure, which is modified based on the DFANet real-time semantic segmentation network in the prior art. Specifically, the unactivated channels are deleted, so that the entire model is compressed to a very small size, but the overall segmentation accuracy is only slightly affected. And the deep network model used in the online stage is the deep network model trained in the offline stage. The training steps of the deep network model are described below:

[0100] (B1) Construct a deep network model and obtain the longitude-latitude unfolded image as the training sample set.

[0101] The longitude-latitude unfolded image is the longitude-latitude unfolded image obtained by converting the fisheye image using the longitude-latitude unfolding algorithm in the above offline stage steps. As the input of the deep network model, the output is the pixel-level classification result of the longitude-latitude unfolded image, and each pixel is identified as a drivable area and a non-drivable area.

[0102] (B2) Perform deep network training on the training sample set to obtain the trained deep network model.

[0103] Read the training sample set and perform deep network training until the training ends to obtain the trained deep network model.

[0104] In a specific embodiment, in the online stage, the first longitude and latitude unfolded image obtained in the above step 100 is used as the input of the trained deep network model, and the pixel-level classification result of the first longitude and latitude unfolded image is output.

[0105] Step 300: Traverse the pixel-level classification result through a boundary search algorithm to obtain multiple boundary points of the drivable area.

[0106] During the process of the deep network model running to output the segmentation result, according to the RGB image data of the input longitude and latitude unfolded image, the network infers the foreground including the drivable area. By traversing the foreground area, the scattered point image coordinates (u, v) of the boundary part can be obtained. The per-pixel classification result is as Figure 6 and Figure 7 shown, where the black area represents the non-drivable background part, and the white area represents the drivable area. Here, the scattered points in the boundary part mentioned in this article refer to the boundary points of the drivable area. In addition, in this article, the pixel-level classification result is also called the per-pixel classification result.

[0107] In a specific embodiment, a boundary parameter model can be constructed. The role of the boundary parameter model is to receive the per-pixel classification result of the network and use the boundary search algorithm to obtain the boundary line of the drivable area. As Figure 6 shown, the black points are the exit boundary points of the drivable area, the gray points are the body contour points, and the white points are the sampling radiation source points. Generate a boundary lookup table in the offline stage, so that this boundary lookup table can be directly used in the online stage to reduce the calculation amount and improve the response speed.

[0108] The following Figure 6 describes the steps of generating the boundary lookup table in the offline stage:

[0109] (C1) Set the search source, and use the pixels at the bottom edge of the longitude and latitude unfolded image as the radiation source of the search source.

[0110] Set the search source. In order to obtain more refined boundary points of the drivable area, the starting point of the radiation source can be every 2 pixels at the bottom edge of the picture, as Figure 6 shown, and the white points are the sampling radiation source points.

[0111] (C2) Set the search line, starting from the radiation source, vertically upward, and sample at a set interval.

[0112] Set the search line, as Figure 6 shown, starting from the sampling radiation source point, vertically upward, and sample every 2 pixels.

[0113] (C3) Search, starting from the search source, compare the pixel-level categories of each point one by one.

[0114] Search point by point starting from the search source according to the above steps, and compare the pixel-level categories between the previous and the next search points. For example, Figure 6 As shown, the gray points are the body contour points, representing the entry points of the drivable area, and the black points are the exit boundary points of the drivable area, representing the exit points of the drivable area. The gray points, the black points, and the points between them are identified as the drivable area, and the points in other areas are identified as non-drivable areas.

[0115] (C4) If the pixel-level category of the current search point is inconsistent with that of the next search point, it is determined that a jump from the body to the drivable area or from the drivable area to the obstacle area has occurred, and the boundary points that meet the boundary conditions are obtained and saved in an array. Among them, the boundary points that meet the boundary conditions include the entry points and exit points of the drivable area.

[0116] Among them, if the pixel-level category of the current search point is inconsistent with that of the next search point, a jump from the body to the drivable area or from the drivable area to the obstacle area may have occurred. For example, as Figure 6 shown, if the next search point is a gray point and the current search point is a point before the gray point, the pixel-level category of the current search point is a non-drivable area, and the pixel-level category of the next search point is a drivable area. The pixel-level categories of the two are different, and a jump from the body to the drivable area occurs, that is, the entry point of the drivable area is obtained; if the current search point is a black point and the next search point is a point after the black point, the pixel-level category of the current search point is a drivable area, and the pixel-level category of the next search point is a non-drivable area. The pixel-level categories of the two are different, and a jump from the drivable area to the obstacle area occurs, that is, the exit point of the drivable area is obtained, so that the entry point and exit point of the drivable area can be obtained.

[0117] (C5) Repeat the search for each search line until the boundary points of the drivable area are searched.

[0118] Repeat the above steps (C3)–(C4), search each search line one by one until the boundary points of the drivable area on each search line are searched, and all boundary points that meet the boundary conditions are obtained.

[0119] (C6) Generate a boundary lookup table for all boundary points that meet the boundary conditions and record the entry points of the drivable area.

[0120] All boundary points that meet the boundary conditions are obtained from the above steps, that is, the entry points and exit points of the drivable area. Accordingly, a corresponding boundary lookup table is generated, and the body contour points, that is, the entry points of the drivable area, are recorded.

[0121] To ensure the accuracy of the above-generated boundary lookup table, during the offline phase, the boundary points of the identified drivable area are verified. If the verification passes, the generated boundary lookup table is used in the subsequent online phase.

[0122] The following combines Figure 7 to elaborate on the search steps for the boundary points of the drivable area in the online phase:

[0123] (D1) Use the pixel points corresponding to the entry points of the drivable area in the pixel-level classification result as the starting points for the search.

[0124] The sampling starting point in the online phase changes relative to the offline phase. Use the body contour points as the sampling starting points. As Figure 7 shown, the gray points are the body contour points, which are the entry points of the drivable area, and the black points are the exit boundary points of the drivable area, which are the exit points of the drivable area. Using the gray points as the starting points greatly reduces the number of search points and the search time, improving the search effect.

[0125] (D2) During the search, start from the starting point of the search and search upward according to the boundary lookup table.

[0126] Extract the verified boundary lookup table obtained in the offline phase. Start from the entry point of the drivable area according to the boundary lookup table and search upward for each point one by one. As Figure 7 shown.

[0127] (D3) If the pixel-level category of the current search point is inconsistent with the pixel-level category of the next search point, it is determined that a jump from the drivable area to the obstacle area has occurred, and the boundary points of the drivable area are obtained.

[0128] As Figure 7 shown, the pixel-level categories of the entry point and the exit point of the drivable area and the search points between the two points are all the drivable area. If the pixel-level category of the current search point is inconsistent with the pixel-level category of the next search point, that is, the current search point is a black point (the exit point of the drivable area), and the pixel-level category of the next search point is the non-drivable area. At this time, it is determined that a jump from the drivable area to the obstacle area has occurred, so it can be known that the current search point is the exit point of the drivable area, and the boundary points of the drivable area are obtained.

[0129] (D4) Traverse the entire boundary lookup table until the search for the boundary points of the drivable area is completed, and multiple boundary points of the drivable area are obtained.

[0130] Traverse the entire boundary lookup table by the above steps to complete the search for the boundary points of the drivable area and obtain multiple corresponding boundary points of the drivable area.

[0131] It should be noted that in actual applications, the interval between the starting points of each search line is not limited to 2 pixels, and the interval between adjacent search points on the same search line is not limited to 2 pixels either. As long as the interval between the starting points of each search line and the interval between adjacent search points on the same search line can identify a drivable area that meets the requirements.

[0132] Step 400: Determine the real distance between the vehicle and the real position corresponding to each boundary point according to the second preset mapping relationship.

[0133] Among them, the second preset mapping relationship includes the mapping relationship between each pixel point in the longitude-latitude unfolded image and the real distance between the vehicle and the real position corresponding to this pixel point. This second preset mapping relationship is obtained in the offline stage. The following elaborates on the generation process of the second preset mapping relationship:

[0134] (E1) Model the camera imaging model.

[0135] In this embodiment, the vehicle camera is a fish-eye camera. A fish-eye camera is a short-focus ultra-wide-angle lens with a focal length of about 6 - 16 millimeters, characterized by a large field of view. The imaging model of the de-distorted fish-eye camera is equivalent to the pinhole imaging model.

[0136] In a specific embodiment, the camera internal parameter model of the camera imaging model is:

[0137]

[0138] Among them, (u, v) are image coordinates, (X c , Y c , Z c ) are camera coordinates, and (f x , f y , C x , C y ) are the camera internal parameters calibrated using the Zhang Zhengyou calibration method;

[0139] The camera external parameter model is:

[0140]

[0141] Among them, (X w , Y w , Z w ) represents the vehicle rear axle coordinates, is the transformation matrix between the camera coordinates and the vehicle rear axle coordinate system, used to describe the pose transformation relationship between the camera coordinate system and the world coordinate system.

[0142] (E2) Under the assumption that the ground is level, the coordinates of each pixel point of the fisheye image in the vehicle rear axle coordinate system are calculated according to the camera imaging model to obtain a mapping relationship from the fisheye image to the vehicle rear axle coordinate system.

[0143] Since the obtained mask image is a longitude and latitude expanded image with coordinates (u′, v′), it is necessary to reversely project the longitude and latitude expanded image into the original fisheye image to obtain the image coordinates (u, v) of the receiving boundary point, and inversely calculate the coordinates of the point in the vehicle rear axle coordinate system. The mask image described in this article refers to the longitude and latitude expanded image containing the pixel-level classification results. According to the above formula (5), the (u, v) coordinates are known, and the By solving this linear equation with two variables, we can get the values of a and b.

[0144] In addition, the above formula (6) is expanded to obtain the following formula:

[0145]

[0146] Furthermore, we get the following formula:

[0147]

[0148] Under the assumption that the ground is level, it can be known that the obstacle boundary point is located at the Z coordinate system of the vehicle rear axle. w is 0, and the values of a and b are known, the above formula (8) also becomes a set of two-variable linear equations, and according to the above formulas (5) and (6), we can get (X w ,Y w ) value, which is the coordinate of the obstacle boundary point in the world coordinate system, that is, the coordinate of the pixel point in the vehicle rear axle coordinate system. Among them, the obstacle boundary point mentioned in this article refers to the boundary point of the non-drivable area, which is also the boundary point of the drivable area (the exit point of the drivable area). Figure 8 As shown in the figure, the origin of the world coordinates in the four directions of front, back, left, and right are all marked points Ob. By adding bias to them, the real coordinates in the vehicle coordinate system with the center of the rear axle of the vehicle as the origin can be obtained. A world coordinate system is established for the images in the four directions of front, back, left, and right, with the origin as Ob, the x-axis as xb, and the y-axis as yb. The origin of the coordinates of the vehicle is the center of the rear axle of the vehicle, the positive direction of the x-axis is towards the front of the vehicle, the positive direction of the y-axis is on the left side of the vehicle and is perpendicular to the x-axis, satisfying the right-hand rule, and the z-axis is perpendicular to the ground and upward.

[0149] (E3) According to the preset mapping relationship and the mapping relationship from the fisheye image to the vehicle rear axle coordinate system, a mapping relationship between each pixel point in the longitude and latitude unfolded image and the actual distance between the vehicle and the actual position corresponding to the pixel point is obtained, and a second preset mapping relationship is generated.

[0150] Among them, the preset mapping relationship is the preset mapping relationship from the fisheye image to the longitude-latitude unfolded image or the inverse projection relationship between the longitude-latitude unfolded image and the fisheye image.

[0151] In the offline stage, the longitude-latitude unfolded image is inverse projected onto the fisheye image, the fisheye image is de-distorted, and the relative distance of each pixel point with respect to the center of the rear axle of the vehicle itself is calculated through the internal and external parameter models of the camera. According to the above formulas (5)-(8), the true distance of each pixel point on the fisheye image in the world coordinate system (the coordinate system with the upper left corner of the checkerboard as the zero point) is deduced. Finally, a bias operation is performed on the distances of the images in each direction in the world coordinate system, and thus the distance from each pixel point on the fisheye image to the center of the rear axle of the vehicle itself can be obtained.

[0152] According to the mapping relationship from the longitude-latitude unfolded image to the fisheye image, the coordinates of the fisheye image in the vehicle coordinate system can be calculated. Input the longitude-latitude unfolded image, obtain the coordinates of the fisheye image through the mapping relationship, and use the coordinates of the fisheye image as an intermediate variable to input into the model for finding the true coordinates of the fisheye camera. Thus, the mapping relationship from the longitude-latitude unfolded image to the world coordinates can be directly obtained, and a lookup table for the true distance from the longitude-latitude unfolded image to the center of the rear axle of the vehicle is generated. In the subsequent online stage, the true distance of each pixel coordinate point on the longitude-latitude unfolded image from the vehicle itself can be directly obtained by looking up the table, which improves the calculation speed and response speed.

[0153] In a specific embodiment, according to the lookup table for the true distance from the longitude-latitude unfolded image to the center of the rear axle of the vehicle generated in the above offline stage, the true distances of each boundary point of the drivable area obtained in step 300 are calculated and obtained.

[0154] As can be seen from the above, the overall structure of this method is divided into two parts: an offline stage and an online stage. Among them, in the offline stage of the embodiment of this specification, the fisheye images of the four cameras are unfolded in longitude and latitude, the mapping relationship from the fisheye image to the longitude-latitude unfolding is determined, and a lookup table from the fisheye image to the longitude-latitude unfolding is generated. The longitude-latitude unfolded image is used as the input of the deep network model, and a boundary parameter model is constructed to obtain the boundary line of the drivable area according to the output of the deep network model. And the longitude-latitude unfolded image is inverse projected onto the fisheye image, and then the camera imaging model is modeled. Under the assumption condition of the ground being horizontal, the mapping relationships of each pixel point in the rear axle coordinate system of the vehicle are calculated respectively, and a lookup table for the true distance from the longitude-latitude unfolding to the center of the rear axle of the vehicle is generated.

[0155] In the online stage of the embodiments of this specification, after the camera reads the video stream signal, the mapped image of the fish-eye image to the longitude and latitude unfolding is calculated through a lookup table of the mapping relationship from the fish-eye image to the longitude and latitude unfolding. Then, it enters the depth network model respectively to obtain the pixel-level classification result of the drivable area. Then, the boundary parameter model traverses the pixel-level classification result through the boundary search algorithm to find the boundary points of the drivable area. After obtaining the boundary points of the drivable area, the true distance of the boundary points is calculated through the lookup table of the true distance from the longitude and latitude unfolding to the vehicle's rear axle coordinate center.

[0156] In summary, this specification discloses a detection method for a visually drivable area, which detects obstacles in the front, rear, left, and right directions encountered during vehicle driving. Compared with the detection method of the de-distorted panoramic stitching image in the prior art, the early warning of the detection method in this embodiment will be earlier, the braking safety distance will be longer, the detection accuracy will be higher, and less original image information will be lost. The detection method in the embodiments of this specification calculates the mapping relationship from the fish-eye image to the longitude and latitude unfolding image and the true distance from each pixel of the longitude and latitude unfolding image to the vehicle's rear axle center in the offline stage. Since the online stage is performed on the in-vehicle chip and the hardware computing power is limited, this detection method can quickly obtain the longitude and latitude unfolding image with angular bias and the true distance of each pixel point in the longitude and latitude unfolding image from the vehicle through the two mapping relationships, reducing the calculation time. During driving, a quick response can be obtained, and the frame rate of the received image can be increased, greatly improving the safety during driving.

[0157] Corresponding to the above method embodiments, the embodiments of the present invention also provide a detection system for a visually drivable area, which is used to execute the steps of the detection method for a visually drivable area in the above embodiments. The detection system for a visually drivable area includes an image acquisition module, a longitude and latitude unfolding module, a depth network inference module, a boundary point search module, and an obstacle discrimination module.

[0158] Specifically, the image acquisition module is used to acquire the real-time fish-eye image in the vehicle camera video stream signal.

[0159] The longitude and latitude unfolding module is used to determine the first longitude and latitude unfolding image corresponding to the real-time fish-eye image according to the first preset mapping relationship, where the first preset mapping relationship is the mapping relationship from the fish-eye image to the longitude and latitude unfolding image.

[0160] The depth network inference module is used to use the first longitude and latitude unfolding image as the input of the depth network model to obtain the pixel-level classification result of the first longitude and latitude unfolding image.

[0161] The boundary point search module is used to traverse the pixel-level classification result through the boundary search algorithm to obtain multiple boundary points of the drivable area.

[0162] The obstacle discrimination module is used to determine the true distance between the vehicle and the true position corresponding to each boundary point according to the second preset mapping relationship, where the second preset mapping relationship includes the mapping relationship between each pixel point in the longitude and latitude unfolded image and the true distance between the vehicle and the true position corresponding to the pixel point.

[0163] It should be noted that the detection system based on the visual drivable area provided in the embodiments of the present invention, because it is based on the same concept as the detection method of the visual drivable area of the present invention, the technical effects brought by it are the same as those of the embodiments of the detection method of the visual drivable area of the present invention. For the parts not mentioned in this embodiment, reference can be made to the description in the embodiments of the detection method of the visual drivable area of the present invention, and details will not be repeated here.

[0164] Those of ordinary skill in the art can understand that the drawings are only schematic diagrams of one embodiment, and the modules or processes in the drawings are not necessarily essential for implementing the present invention.

[0165] Those of ordinary skill in the art can understand that the modules in the device in the embodiments can be distributed in the device in the embodiments according to the description of the embodiments, or can be correspondingly changed and located in one or more devices different from the present embodiment. The modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A detection method based on a visible drivable area, characterized in that, The method includes: Obtain a real-time fisheye image in the video stream signal of the vehicle camera, and determine a first longitude-latitude unfolded image corresponding to the real-time fisheye image according to a first preset mapping relationship, where the first preset mapping relationship is a mapping relationship from the fisheye image to the longitude-latitude unfolded image; Use the first longitude-latitude unfolded image as the input of the deep network model to obtain a pixel-level classification result of the first longitude-latitude unfolded image; Traverse the pixel-level classification result through a boundary search algorithm to obtain multiple boundary points of the drivable area; Determine the real distance between the vehicle and the real position corresponding to each boundary point according to a second preset mapping relationship, where the second preset mapping relationship includes the mapping relationship between each pixel point in the longitude-latitude unfolded image and the real distance between the vehicle and the real position corresponding to the pixel point; Among them, the generation process of the second preset mapping relationship includes: Model the camera imaging model; Under the assumption of a horizontal ground, calculate the coordinates of each pixel point of the fisheye image in the vehicle rear axle coordinate system according to the camera imaging model to obtain the mapping relationship from the fisheye image to the vehicle rear axle coordinate system; According to the preset mapping relationship and the mapping relationship from the fisheye image to the vehicle rear axle coordinate system, obtain the mapping relationship between each pixel point in the longitude-latitude unfolded image and the real distance between the vehicle and the real position corresponding to the pixel point, and generate the second preset mapping relationship, where the preset mapping relationship is the preset mapping relationship from the fisheye image to the longitude-latitude unfolded image or the inverse projection relationship between the longitude-latitude unfolded image and the fisheye image.

2. The detection method based on a visually drivable area according to claim 1, wherein Before obtaining the real-time fisheye image in the video stream signal of the vehicle camera and determining the first longitude-latitude unfolded image corresponding to the real-time fisheye image according to the first preset mapping relationship, the method further includes: Obtain the fisheye image of the vehicle camera, and convert the fisheye image into a longitude-latitude unfolded image by using a longitude-latitude unfolding algorithm; According to the fisheye image and the longitude-latitude unfolded image, determine the mapping relationship from the fisheye image to the longitude-latitude unfolded image, and generate the first preset mapping relationship.

3. The detection method based on a visually drivable area according to claim 2, wherein The specific process of converting the fisheye image into a longitude-latitude unfolded image by using the longitude-latitude unfolding algorithm includes: Normalize the fisheye image, map the pixels of the fisheye image onto a 2D unit circle model, and obtain the pixel coordinates (x f , y f ) in the unit fisheye coordinate system. Among them, in the unit fisheye coordinate system, the calculation formulas for the imaging height r and the angle ф in the fisheye image are as follows: Among them, the relationship between the incident angle θ of the lens of the vehicle camera and the imaging height r in the fisheye image is r = f(θ); Map the pixel coordinates (x f , y f ) from the 2D unit circle model to the unit spherical coordinates (x p , y p , z p ) of the 3D unit sphere model, where the correspondence between the unit spherical coordinates (x p , y p , z p ) and the incident angle θ and the included angle ф is as follows: Convert the unit spherical coordinates (x p , y p , z p ) to unit spherical longitude and latitude coordinates (longitude, latitude), where the correspondence between the unit spherical longitude and latitude coordinates (longitude, latitude) and the unit spherical coordinates (x p , y p , z p ) is as follows: Map the longitude and latitude coordinates of the unit sphere surface (longitude, latitude) from the 3D unit sphere model to the 2D unit longitude and latitude unfolded coordinates (x e , y e ) to obtain the longitude and latitude unfolded image, where the correspondence between the unit longitude and latitude unfolded coordinates (x e , y e ) and the longitude and latitude coordinates of the unit sphere surface (longitude, latitude) is as follows:

4. The detection method based on a visually drivable area according to claim 3, wherein The 3D unit sphere model is the unit sphere model multiplied by the rotation matrix of the unit spherical coordinates and the world coordinates.

5. The detection method based on a visually drivable area according to claim 2, wherein The deep network model adopts an encoder-decoder network structure, and the deep network model is trained through the following steps: Construct a deep network model and obtain the longitude-latitude unfolded image as a training sample set; Perform deep network training on the training sample set to obtain a trained deep network model.

6. The detection method based on a visually drivable area according to claim 2, wherein Before traversing the pixel-level classification result through the boundary search algorithm to obtain multiple boundary points of the drivable area, the method further includes: Set a search source, and use the bottom edge pixel of the picture of the longitude-latitude unfolded image of the search source as the radiation source of the search source; Set a search line starting from the radiation source, vertically upward, and sample at a set interval; Search, starting from the search source, and compare pixel-level categories for each search point one by one; If the pixel-level category of the current search point is inconsistent with that of the next search point, it is determined that a jump from the vehicle body to the drivable area or from the drivable area to the obstacle area has occurred, and the boundary points that meet the boundary conditions are obtained and saved to an array, where the boundary points that meet the boundary conditions include the entry point and the exit point of the drivable area; Repeat the search for each search line until the boundary points of the drivable area are searched; Generate a boundary lookup table for all boundary points that meet the boundary conditions and record the entry point of the drivable area.

7. The detection method based on a visually drivable area according to claim 6, wherein The traversing of the pixel-level classification result through the boundary search algorithm to obtain multiple boundary points of the drivable area specifically includes: Use the pixel point corresponding to the entry point of the drivable area in the pixel-level classification result as the starting point of the search; During the search, start from the starting point of the search and search vertically upward according to the boundary lookup table; If the pixel-level category of the current search point is inconsistent with that of the next search point, it is determined that a jump from the drivable area to the obstacle area has occurred, and the boundary points of the drivable area are obtained; Traverse the entire boundary lookup table until the boundary points of the drivable area are searched to obtain multiple boundary points of the drivable area.

8. The detection method based on a visually drivable area according to claim 1, wherein The internal parameter model of the camera in the camera imaging model is: Among them, (u, v) are image coordinates, and (X c , Y c , Z c ) are camera coordinates, and (f x , f y , C x , C y ) are the internal camera parameters calibrated using the Zhang Zhengyou calibration method; The external parameter model of the camera is: Among them, (X w , Y w , Z w ) represents the coordinates of the rear axle of the vehicle, is the transformation matrix between the camera coordinates and the rear axle coordinates of the vehicle; Under the assumption of ground level, the Z of each of the pixel points in the vehicle rear axle coordinate system w is 0. Then, according to the above formulas (5) and (6), the coordinates (X w , Y w ) of the pixel point in the vehicle rear axle coordinate system are calculated.

9. A detection system based on a visually drivable area, characterized in that, It includes: An image acquisition module for acquiring a real-time fisheye image in the vehicle camera video stream signal; A longitude and latitude unfolding module for determining a first longitude and latitude unfolded image corresponding to the real-time fisheye image according to a first preset mapping relationship, where the first preset mapping relationship is a mapping relationship from the fisheye image to the longitude and latitude unfolded image; A deep network inference module for using the first longitude and latitude unfolded image as the input of the deep network model to obtain the pixel-level classification result of the first longitude and latitude unfolded image; A boundary point search module for traversing the pixel-level classification result through a boundary search algorithm to obtain multiple boundary points of the drivable area; An obstacle discrimination module for determining the real distance between the vehicle and the real position corresponding to each boundary point according to a second preset mapping relationship, where the second preset mapping relationship includes the mapping relationship between each pixel point in the longitude and latitude unfolded image and the real distance between the vehicle and the real position corresponding to the pixel point; where the generation process of the second preset mapping relationship includes: Model the camera imaging model; Under the assumption of a horizontal ground, calculate the coordinates of each pixel point of the fisheye image in the vehicle rear axle coordinate system according to the camera imaging model to obtain the mapping relationship from the fisheye image to the vehicle rear axle coordinate system; According to the preset mapping relationship and the mapping relationship from the fish-eye image to the vehicle rear axle coordinate system, obtain the mapping relationship between the true distance between each pixel point in the longitude-latitude unfolded image and the true position of the vehicle corresponding to the pixel point, and generate the second preset mapping relationship, where the preset mapping relationship is the preset mapping relationship from the fish-eye image to the longitude-latitude unfolded image or the inverse projection relationship between the longitude-latitude unfolded image and the fish-eye image.

Images