Around-looking ranging method and device, readable storage medium, electronic equipment and vehicle

Abstract

The invention relates to an all-round range finding method and device, a readable storage medium, electronic equipment and a vehicle, and relates to the technical field of image processing, and the method comprises the steps: obtaining fisheye images shot by a plurality of fisheye cameras, the fisheye cameras being used for collecting the fisheye images of roads in different directions in the vehicle driving process; performing distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image; based on a perspective transformation matrix of the plurality of fisheye cameras, performing overlook transformation on the corrected image to obtain an overlook image; and determining a vehicle distance measuring line in the top view image, and identifying the distance between a vehicle body and an obstacle based on the vehicle distance measuring line. According to the technical scheme provided by the invention, the problems of large computing resource occupation and relatively low application range of look-around distance measurement can be solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a surround ranging method, device, readable storage medium, electronic equipment and vehicle. Background Art

[0002] With the rapid development of vehicle intelligence technology, vehicle ring distance measurement has become a key technology in vehicle driving, which is used to estimate the distance between the vehicle and surrounding objects. Accurate distance measurement information is of great significance for avoiding collisions, ensuring road safety, and realizing functions such as autonomous driving.

[0003] Currently, vehicles generally achieve surround ranging by running a monocular depth estimation model on a fisheye camera. However, each time a surround ranging is performed, the monocular depth estimation model needs to be repeatedly trained and run, which takes up a lot of computing resources. For some released models that are not equipped with the latest computing power platform, the computing resources are difficult to support the surround ranging application, making the surround ranging application scope relatively low. Summary of the invention

[0004] In view of this, the present application provides a surround-view ranging method, device, readable storage medium, electronic device and vehicle, which can solve the problem that surround-view ranging occupies large computing resources and has a low scope of application.

[0005] According to a first aspect of the present application, a surround ranging method is provided, comprising:

[0006] Acquire fisheye images taken by a plurality of fisheye cameras, wherein the plurality of fisheye cameras are used to collect the fisheye images of roads in different directions during the driving process of the vehicle;

[0007] Performing distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image;

[0008] Based on the perspective transformation matrices of the plurality of fisheye cameras, performing a top-view transformation on the rectified image to obtain a top-view image;

[0009] A vehicle ranging line is determined in the top view image, and a distance between a vehicle body and an obstacle is identified based on the vehicle ranging line.

[0010] According to a second aspect of the present application, a surround ranging device is provided, comprising:

[0011] An acquisition module, used to acquire fisheye images taken by multiple fisheye cameras, wherein the multiple fisheye cameras are used to collect the fisheye images of roads in different directions during the driving process of the vehicle;

[0012] A processing module, used for performing distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image;

[0013] A transformation module, configured to perform a top-view transformation on the rectified image based on the perspective transformation matrices of the plurality of fisheye cameras to obtain a top-view image;

[0014] The recognition module is used to determine the vehicle ranging line in the overhead image and recognize the distance between the vehicle body and the obstacle based on the vehicle ranging line.

[0015] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the surround ranging method described in the first aspect is implemented.

[0016] According to a fourth aspect of the present application, an electronic device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the surround ranging method described in the first aspect when executing the computer program.

[0017] According to a fifth aspect of the present application, a vehicle is provided, comprising: the electronic device as described in the fourth aspect.

[0018] By means of the above technical scheme, the present application provides a surround ranging method, device, readable storage medium, electronic device and vehicle. Compared with the current method of realizing surround ranging by running a monocular depth estimation model on a fisheye camera, the present application can perform distortion correction processing on the fisheye image after obtaining the fisheye image taken by multiple fisheye cameras to obtain a corrected image corresponding to the fisheye image; then, based on the perspective transformation matrix of multiple fisheye cameras, the corrected image is transformed into a top-down view to obtain a top-down image; the vehicle ranging line is determined in the top-down image, and the distance between the vehicle and the obstacle is identified based on the vehicle ranging line. By applying the technical scheme in the present application, it is only necessary to perform simple image processing on the fisheye image, and after determining the vehicle ranging line, the distance between the vehicle body and the obstacle is identified based on the comparison reference of the vehicle ranging line. In the ranging process, there is no need to train and run a monocular depth estimation model, which can reduce the occupation of computing resources and reduce the computing resource consumption of the vehicle, so that the surround ranging application can be applied to more released models and expand the scope of application of surround ranging.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A flowchart of a surround ranging method provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a flow chart of a surround ranging method provided by another embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of an example of joint calibration of four fisheye cameras provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of an example of image stitching of a vehicle-mounted surround view system provided by an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of an example of determining a vehicle ranging line provided by an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the structure of a surround distance measurement device provided by an embodiment of the present disclosure;

[0028] Figure 7 A schematic structural diagram of a surround ranging device provided in accordance with another embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] The surround ranging method, device, readable storage medium, electronic device and vehicle according to the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0031] In order to solve the above technical problems, the present disclosure provides a surround-view ranging method, device, readable storage medium, electronic device and vehicle, which can solve the problem that surround-view ranging occupies a large amount of computing resources and is not applicable to all released vehicle models.

[0032] like Figure 1 As shown, an embodiment of the present disclosure provides a surround ranging method, including:

[0033] Step 101: Obtain fisheye images taken by multiple fisheye cameras, wherein the multiple fisheye cameras are used to collect fisheye images of roads in different directions during vehicle driving.

[0034] The vehicle-mounted surround view system includes at least four cameras, which are respectively located at different positions of the vehicle to respectively collect images of the front, rear, left and right of the vehicle, and the camera may be a fisheye camera. In the following embodiment steps, the vehicle-mounted surround view system includes four fisheye cameras, and the four fisheye cameras (front view, rear view, left view, right view) are respectively used to collect fisheye images of the front, rear, left and right of the vehicle as an example to illustrate the technical solution in the present application, but it does not constitute a specific limitation on the technical solution in the present application.

[0035] The execution subject of the embodiment of the present disclosure may be a vehicle-mounted surround view system, specifically, a vehicle-mounted computer in the vehicle-mounted surround view system. When performing surround range measurement, the vehicle-mounted computer may obtain fisheye images taken by multiple fisheye cameras arranged on the vehicle, and then successively perform distortion correction and perspective transformation processing on the fisheye images to obtain a top-view image corresponding to the fisheye image. Finally, the vehicle ranging line is determined in the top-view image, and the distance between the vehicle and the obstacle is identified based on the vehicle ranging line.

[0036] Step 102: Perform distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image.

[0037] In specific application scenarios, fisheye cameras have a larger imaging field of view than ordinary cameras, but the images they capture are severely distorted, that is, except for the shape of the object in the center of the image, the other parts that should be straight lines are distorted to a certain extent. Therefore, the fisheye images captured by the fisheye camera need to be subjected to distortion correction before use. For example, when the on-board computer receives the front fisheye image, rear fisheye image, left fisheye image, and right fisheye image captured by the front fisheye camera, rear fisheye camera, left fisheye camera, and right fisheye camera, the front fisheye image, rear fisheye image, left fisheye image, and right fisheye image can be subjected to distortion correction respectively. Specifically, the fisheye image can be subjected to distortion correction according to the fisheye camera imaging model and the distortion correction principle, and the fisheye circular distortion image can be restored to a corrected image that meets the visual effect of the human eye. In addition, in order to facilitate the subsequent top-down transformation processing, the corrected image can be further grayed to convert the color image into a grayscale image, and no specific limitation is made to this.

[0038] Step 103: Based on the perspective transformation matrices of the multiple fisheye cameras, perform a bird's-eye view transformation on the rectified image to obtain a bird's-eye view image.

[0039] In a specific application scenario, since the front fisheye camera, rear fisheye camera, left fisheye camera and right fisheye camera are all installed at a certain angle to the ground, and the image finally displayed to the user is a 360-degree bird's-eye view of the vehicle body, the corrected fisheye image needs to be transformed into a bird's-eye view, that is, the corrected image is perspective processed to obtain its bird's-eye view.

[0040] For the embodiments of the present disclosure, the perspective transformation matrix of each fisheye camera in a plurality of fisheye cameras (such as a front-view fisheye camera, a rear-view fisheye camera, a left-view fisheye camera, and a right-view fisheye camera) can be determined first, and the perspective transformation matrix can be used to perform a top-down transformation on the corrected image at each viewing angle, thereby obtaining a front-view top-down image, a rear-view top-down image, a left-view top-down image, and a right-view top-down image.

[0041] Step 104: determine a vehicle ranging line in the overhead image, and identify the distance between the vehicle body and the obstacle based on the vehicle ranging line.

[0042] Among them, the vehicle ranging line is a scale line parallel to the vehicle body and is used to calibrate the horizontal distance between the vehicle body and the vehicle.

[0043] For the disclosed embodiment, after obtaining the top view images under each viewing angle, the top view images under multiple viewing angles can be spliced, and the gaps can be removed through the image fusion algorithm to obtain a complete surround top view image. In the final surround top view image, several vehicle ranging lines are calibrated. After the vehicle ranging lines are calibrated, the distance between the specific object and the vehicle body can be obtained by combining the human body detection, vehicle detection and ground segmentation model in the subsequent tasks, thereby realizing the surround ranging capability based on multi-channel fisheye cameras.

[0044] In summary, according to the surround-view ranging method provided by the present application, after obtaining the fisheye images taken by multiple fisheye cameras, the fisheye images can be subjected to distortion correction processing to obtain the corrected images corresponding to the fisheye images; then, based on the perspective transformation matrices of the multiple fisheye cameras, the corrected images are subjected to a top-down transformation to obtain a top-down image; the vehicle ranging line is determined in the top-down image, and the distance between the vehicle body and the obstacle is identified based on the vehicle ranging line. By applying the technical solution in the present application, it is only necessary to perform simple image processing on the fisheye images, determine the vehicle ranging line, and then identify the distance between the vehicle body and the obstacle based on the comparison reference of the vehicle ranging line. In the ranging process, there is no need to train and run the monocular depth estimation model, which can reduce the occupation of computing resources and reduce the computing resource consumption of the vehicle, so that the surround-view ranging application can be applied to more released models and expand the scope of application of the surround-view ranging.

[0045] Further, as a refinement and extension of the above embodiment, in order to fully illustrate the specific implementation process of the method of this embodiment, this embodiment provides the following Figure 2 The specific method shown includes:

[0046] Step 201: Obtain fisheye images taken by multiple fisheye cameras, wherein the multiple fisheye cameras are used to collect fisheye images of roads in different directions during the driving process of a vehicle.

[0047] For the embodiment of the present disclosure, the specific implementation process can refer to the relevant description in step 101 of the embodiment, which will not be repeated here.

[0048] Step 202: Perform distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image.

[0049] For the disclosed embodiment, in view of the relatively serious distortion in the original fisheye image, pedestrians and vehicles are severely bent in the picture, especially in the four corners of the picture, so it is necessary to perform distortion correction processing on the fisheye image. When performing distortion correction processing on the fisheye image, the camera internal parameters of the fisheye camera can be first obtained by a camera calibration method. The camera internal parameters determine the relationship between a spatial point and its corresponding point projected on the image, and may include the camera focal length, optical center, radial distortion coefficient, etc. Afterwards, the above-mentioned camera internal parameters can be used to perform distortion correction on the original distorted image obtained by the fisheye camera. Among them, the camera calibration method can adopt Zhang's calibration method. When the fisheye camera is calibrated using Zhang's calibration method, for each fisheye camera, a black and white checkerboard board (calibration board) can be used as a calibration plane, and the calibration board is used to collect fisheye images. It is usually necessary to make the calibration board appear at different positions of the picture, take multiple shots, and collect multiple (such as about 20) calibration data. After obtaining the calibration data, you can directly use the camera calibration interface provided by MATLAB or OpenCV to calibrate and obtain the internal parameters of the fisheye camera.

[0050] After obtaining the camera internal parameters corresponding to the front-view fisheye camera, rear-view fisheye camera, left-view fisheye camera and right-view fisheye camera respectively based on the above camera calibration method, when using the camera internal parameters to perform distortion correction on the original fisheye image obtained by each fisheye camera, the camera internal parameters corresponding to each fisheye camera can be passed into the cv2.fisheye.initUndistortRectifyMap function provided by OpenCV, and the mapping relationship between the fisheye image (distorted image) and the dedistorted image (ideal image) under each viewing angle can be obtained. Then, by calling the cv2.remap function to complete the image mapping, the dedistorted fisheye image under each viewing angle, that is, the corrected image (such as the front-view correction image, the rear-view correction image, the left-view correction image and the right-view correction image) can be obtained.

[0051] Accordingly, for the embodiments of the present disclosure, the steps of the embodiments may include: performing camera calibration processing on multiple fisheye cameras respectively to obtain the camera internal parameters of each fisheye camera in the multiple fisheye cameras; determining the mapping relationship between the fisheye image and the corrected image under the camera internal parameters, wherein different camera internal parameters correspond to different mapping relationships; performing image mapping on the fisheye image according to the mapping relationship to obtain the corrected image after the fisheye image is dedistorted.

[0052] Step 203: Based on the perspective transformation matrix, perspective transformation processing is performed on each pixel point in the corrected image to obtain a first overhead image corresponding to the corrected image.

[0053] Among them, perspective transformation is projection transformation, which means that if the same fisheye camera is used to shoot the same plane at positions A and B respectively, two images are generated. The transformation relationship between the two images is the process of converting one imaging picture into another imaging picture.

[0054] In a specific application scenario, before executing the steps of this embodiment, it is necessary to pre-calculate the perspective transformation matrix of the fisheye camera. When calculating the perspective transformation matrix of the fisheye camera, it can be achieved by jointly calibrating multiple fisheye cameras. Figure 3 As shown, the specific method can be to place the calibration plate on the ground around the car. When placing the calibration plate, two distances need to be determined. The first is the distance between the calibration plate and the car body, and the second is the distance between the calibration plate and the border of the top view. These two distances generally need to be determined according to the desired visual range of the top view. If you want to obtain a top view with a range of 3m around the car, you can place the calibration plate in the center of the 1.5m line, so that the calibration plate is just in the center area of ​​the top view. After determining the placement position of the calibration plate around the car body, four fisheye images containing the calibration plate can be taken. Then, the camera internal parameters obtained by calibration in step 202 of the embodiment are used for dedistortion, and then at least 4 corner points of the calibration plate are selected in each corrected image after dedistortion, and their coordinate values ​​are recorded. Finally, the expected pixel coordinate values ​​corresponding to the corner points on the top view image are obtained (generally according to the 1:1 scale relationship between the real world and the panoramic view, that is, 1 pixel represents 1cm, to obtain the expected pixel coordinates of these corner points on the top view image). In this way, we get the data pairs of coordinate values ​​before and after the transformation. By sending these coordinate pairs to the findHomography function or getPerspectiveTransform function provided by OpenCV, we can calculate the perspective transformation matrix required for the perspective transformation.

[0055] Correspondingly, for the calculation process of the perspective transformation matrix, the steps of the embodiment may include: obtaining calibration images including several calibration plate areas taken by multiple fisheye cameras; performing image mapping on the calibration images based on the mapping relationship between the fisheye images and the corrected images under the camera internal parameters to obtain the calibration corrected images after the calibration images are dedistorted; determining the coordinates of the corner points of several calibration plate areas in the calibration corrected images, and the expected pixel coordinates of the corner points on the several calibration plate areas on the standard overhead view image; and calculating the perspective transformation matrices of the multiple fisheye cameras based on the corner point coordinates and the expected pixel coordinates.

[0056] For the disclosed embodiment, it is necessary to transform the corrected fisheye image into a bird's-eye view image by means of perspective transformation processing, that is, transform the picture taken by the fisheye camera installed on the vehicle body into a picture taken by a camera above the vehicle and parallel to the ground. Specifically, the perspective transformation matrices corresponding to the four fisheye cameras and the fisheye images after dedistortion can be respectively sent to the remap function provided by OpenCV for image mapping, and finally the bird's-eye view images corresponding to the four fisheye images can be obtained, such as the front bird's-eye view image, the rear bird's-eye view image, the left bird's-eye view image, and the right bird's-eye view image.

[0057] Step 204 : performing image stitching processing on the multiple first bird's-eye view images captured by the multiple fisheye cameras to obtain a final second bird's-eye view image.

[0058] In specific application scenarios, due to the brightness of different fisheye cameras, transformation errors and other issues, the images of the overlapping areas in the multiple first top-view images will have ghosting or blur. In order to solve the above problems, the multiple first top-view images taken by the multiple fisheye cameras can be stitched, and the gaps can be removed through the image fusion algorithm to obtain a complete panoramic top-view image, that is, the final second top-view image.

[0059] Correspondingly, for the embodiments of the present disclosure, the embodiment steps may include: determining a stitching area of ​​first overhead view images corresponding to adjacent fisheye cameras among multiple fisheye cameras; calculating a stitching transparency of the first overhead view image in the stitching area; and based on the stitching transparency, performing stitching processing on the first overhead view images corresponding to adjacent fisheye cameras in the stitching area to obtain a final second overhead view image.

[0060] Among them, the adjacent fisheye cameras are two fisheye cameras arranged at adjacent positions in the vehicle-mounted surround view system, such as a front fisheye camera and a left fisheye camera, a front fisheye camera and a right fisheye camera, a rear fisheye camera and a left fisheye camera, and a rear fisheye camera and a right fisheye camera. Since there are overlapping shooting areas between adjacent fisheye cameras, in order to obtain a complete top-view image of the vehicle, it is necessary to splice the top-view images exposed by the adjacent fisheye cameras. In order to avoid obvious splicing lines due to direct splicing, which makes the image splicing relatively stiff and not beautiful, when performing the splicing processing of the adjacent fisheye cameras corresponding to the first top-view image, the image coordinate system of the adjacent fisheye cameras in the vehicle-mounted surround view system can be first constructed, and the splicing area of ​​the adjacent fisheye cameras corresponding to the first top-view image can be determined in the image coordinate system to support the splicing of the adjacent fisheye cameras corresponding to the first top-view image in the splicing area. The stitching area may be an overlapping area of ​​the first overhead images corresponding to adjacent fisheye cameras in the image coordinate system, or a polygonal image area cut by the stitching start line and the stitching end line in the overlapping area. The overlapping area may be an overlapping shooting area of ​​adjacent fisheye cameras. For example, Figure 4As shown, taking the adjacent fisheye cameras as the front fisheye camera and the right fisheye camera as an example, taking the upper right corner of the vehicle model as the coordinate origin (0,0), the front direction (the positive position of the front fisheye camera corresponding to the vehicle) as the Y axis, and the vertical body direction (the positive position of the right fisheye camera corresponding to the vehicle) as the X axis, the image coordinate system of the front fisheye camera and the right fisheye camera in the vehicle-mounted surround view system is constructed. In the image coordinate system, the solid line a can be determined as the stitching starting line of the stitching area according to the empirical coefficient, and the solid line b can be determined as the stitching end line of the stitching area. Further, the polygonal image area A cut by the coordinate system origin, the solid line a, and the solid line b can be determined as the stitching area.

[0061] In a specific application scenario, the larger the stitching area, the better the stitching effect. For this, the stitching area can be determined based on an empirical coefficient, or the overlapping area of ​​the first overhead images corresponding to adjacent fisheye cameras in the image coordinate system can be directly determined as the stitching area, and no specific limitation is made to this. In the prior art, when two images are stitched together, a stitching picture boundary (stitching line) is often determined first, and the stitching of the two pictures is achieved at the picture boundary. Due to the different environments in which different fisheye cameras are located, and the poor consistency caused by the process errors of different fisheye cameras, the pictures exposed by different fisheye cameras at the same time will have differences in brightness, color saturation, etc. If such images are directly stitched together, the result will result in a more obvious visual impact at the stitching line, resulting in a stiff and less beautiful image stitching. As for this embodiment, compared with the method of direct stitching at the stitching line, the present disclosure adopts a method of smooth stitching within the stitching area. This area stitching method can blur the stitching contour of the image, achieve a natural stitching transition of different camera data, and alleviate the problem of too strong stitching of the two rows of images caused by inconsistent brightness and poor camera calibration.

[0062] For the embodiment of the present disclosure, when calculating the splicing transparency of the first overhead image in the splicing area, the transparency variation law from the splicing starting line to the splicing end line in the splicing area can be determined, and the splicing transparency of the first overhead image corresponding to the adjacent fisheye cameras in the splicing area is calculated according to the transparency variation law. The transparency variation law includes a first transparency variation law corresponding to the first overhead image of the first fisheye camera among the adjacent fisheye cameras, and a second transparency variation law corresponding to the first overhead image of the second fisheye camera among the adjacent fisheye cameras, and the first transparency variation law and the second transparency variation law have a functional constraint relationship. The first transparency variation rule may be a transparency variation trend from 1 (completely opaque) to 0 (completely transparent) or from 0 to 1, and correspondingly, the second transparency variation rule is a transparency variation trend from 0 to 1 or from 1 to 0. The functional constraint relationship between the first transparency variation rule and the second transparency variation rule is: the transparency of the pixel point of the first overhead view image corresponding to one of the adjacent fisheye cameras in the stitching area is x according to the first transparency variation rule, and the transparency of the pixel point of the first overhead view image corresponding to the other of the adjacent fisheye cameras at the same position is 1-x according to the second transparency variation rule. The sum of the first stitching transparency and the second stitching transparency is 1. Correspondingly, the steps of the embodiment may specifically include: determining the deployment position information of adjacent fisheye cameras in the vehicle-mounted surround view system; configuring a first overhead view image corresponding to a first fisheye camera among the adjacent fisheye cameras according to the deployment position information, wherein the transparency from the splicing starting line to the splicing end line changes linearly according to a first transparency change law, and the transparency of the first overhead view image corresponding to a second fisheye camera among the adjacent fisheye cameras changes linearly from the splicing starting line to the splicing end line according to a second transparency change law, and the first transparency change law and the second transparency change law have a functional constraint relationship.

[0063] Accordingly, when the first overhead images corresponding to adjacent fisheye cameras are stitched in the stitching area based on the stitching transparency to obtain the final second overhead image, the stitching image in the stitching area can be jointly determined by the two first overhead images corresponding to the adjacent fisheye cameras. In the stitching image, as a possible implementation method, the image pixels can be completely taken from the first overhead image corresponding to one of the fisheye cameras; as a possible implementation method, they can also be completely taken from the first overhead image corresponding to another fisheye camera; as a possible implementation method, they can also be taken from the combination of the two first overhead images corresponding to the two fisheye cameras. Specifically, the adjustment of the image pixels can be achieved by adjusting the transparency of the two stitching images. Specifically, the embodiment steps may include: determining the stitching transparency of the first overhead images corresponding to the adjacent fisheye cameras in the stitching area according to the pixel position of each pixel point in the stitching area and the first transparency change law and the second transparency change law; based on the stitching transparency, stitching the first overhead images corresponding to the adjacent fisheye cameras in the stitching area.

[0064] Exemplarily, taking the adjacent fisheye camera of the front-view fisheye camera and the right-view fisheye camera as an example, the stitching transparency of each pixel point contained in the stitching area can be determined according to the pixel position of each pixel point in the stitching area. For example, for the pixel position close to the front-view fisheye camera in the stitching area, compared with the right-view fisheye camera, the front-view fisheye camera is closer and the shooting is clearer, so in the first top-view image corresponding to the front-view fisheye camera, the pixel value of this point should have a higher proportion. Specifically, in the first top-view image corresponding to the front-view fisheye camera, a larger transparency, such as 0.9, can be set for the pixel point. Correspondingly, in the first top-view image corresponding to the right-view fisheye camera, a smaller transparency, such as 0.1, can be set for the pixel point at the same pixel position. By stitching the first top-view images corresponding to the adjacent fisheye cameras and setting the stitching transparency in the stitching area: the front-view fisheye camera (0.9) and the right-view fisheye camera (0.1), the stitching processing of the pixel position in the stitching area can be completed. Correspondingly, the above-mentioned process of determining the stitching transparency and performing stitching using the stitching transparency is also performed for other pixel positions in the stitching area. After the stitching process is completed for all pixel positions in the stitching area, the stitching process of the first overhead view images corresponding to the adjacent fisheye cameras is completed. After the first overhead view images of each adjacent fisheye camera among the multiple fisheye cameras are stitched together, a complete panoramic overhead view image can be obtained, that is, the final second overhead view image is obtained.

[0065] Step 205: Generate a plurality of vehicle ranging lines parallel to the vehicle body in the second top view image at a preset distance interval, where the preset distance interval is the horizontal distance between the vehicle ranging line and the vehicle body in the second top view image.

[0066] For the embodiment of the present disclosure, after determining that the second top view image is obtained, Figure 5 As shown, in order to achieve ranging capability, a vehicle ranging line (such as a 0.5m line, a 1.0m line, a 1.5m line, etc.) parallel to the vehicle body can be set on the second top view image, wherein the vehicle ranging line can be a rectangular ranging line surrounding the vehicle body 360 degrees, and the specific number of ranging lines, the shape of the ranging lines and the distance value relative to the vehicle body can be set according to application needs, and no specific limitation is made here.

[0067] Step 206: Identify whether there is an obstacle in the second overhead image. If there is an obstacle in the second overhead image, identify the horizontal distance between each obstacle pixel point in the obstacle area and the vehicle body based on the vehicle ranging line, and determine the distance between the vehicle body and the obstacle based on the horizontal distance between each obstacle pixel point in the obstacle area and the vehicle body.

[0068] For the disclosed embodiment, after acquiring the second top view image and laying out a number of vehicle ranging lines parallel to the vehicle body on the second top view image, the preset recognition model can be used to perform image recognition on the second top view image to obtain the obstacle information contained in the second top view image. Afterwards, the obstacle area can be marked based on the obstacle information, and the horizontal distance between each obstacle pixel point and the vehicle body in the obstacle area can be identified based on the number of vehicle ranging lines as the ranging basis. Furthermore, the horizontal distance between the obstacle and the vehicle body can also be comprehensively determined based on the horizontal distances between multiple obstacle pixel points and the vehicle body. Specifically, as a possible implementation method, after identifying the horizontal distance between each obstacle pixel point and the vehicle body, the average distance between all obstacle pixel points and the vehicle body can be determined as the horizontal distance between the obstacle and the vehicle body; as a possible implementation method, the horizontal distance between the center pixel point in the obstacle area and the vehicle body can also be determined as the horizontal distance between the obstacle and the vehicle body, which is not specifically limited here.

[0069] For example, when identifying the horizontal distance between each obstacle pixel point and the vehicle body in the obstacle area, Figure 5As shown, when obstacle A is identified in the second overhead image, the horizontal distance between each obstacle pixel in obstacle A and the vehicle body can be determined based on the marked obstacle area and several parallel vehicle ranging lines outside the vehicle body. For obstacle pixels that coincide with a certain vehicle ranging line, the marked distance corresponding to the vehicle ranging line can be directly determined as the horizontal distance between the obstacle pixel and the vehicle body. For example, if obstacle pixel b coincides with the 0.5m vehicle ranging line, the horizontal distance between obstacle pixel b and the vehicle body can be determined to be 0.5m; if obstacle pixel c coincides with the 1.5m vehicle ranging line, the horizontal distance between obstacle pixel c and the vehicle body can be determined to be 1.5m. For obstacle pixels located between two adjacent vehicle ranging lines, the horizontal distance between the obstacle pixel and the vehicle body can be estimated based on the two adjacent vehicle ranging lines. For example, if the obstacle pixel point a is located between the 0.5m vehicle ranging line and the 1.5m vehicle ranging line, the horizontal distance between the obstacle pixel point a and the vehicle body can be estimated based on the 0.5m vehicle ranging line and the 1.5m vehicle ranging line, such as 1.2m, 1.25m, 1.3m...

[0070] Among them, the obstacle may be a person, roadblock, other vehicle, etc. near the vehicle body; the obstacle information may include the location coordinates of the obstacle, the types of obstacles, etc., and the types of obstacles may include people, roadblocks, other vehicles, etc.; the preset recognition model may be a hidden Markov model, a conditional random field, a neural network model, and a deep learning algorithm model for type recognition, etc., which are not specifically limited here. The second top view image containing the object to be measured is input into the preset recognition model to obtain the corresponding obstacle information. In a specific application scenario, before executing the steps of this embodiment, based on a certain number of sample images marked with obstacle type labels and obstacle location coordinates, the preset recognition model can be trained for obstacle information recognition tasks through supervised learning, or unsupervised learning, semi-supervised learning, and reinforcement learning. The corresponding training method is used until it is determined that the preset recognition model has reached a convergence state and it is determined that the preset recognition model has completed training. Then the trained preset recognition model can be directly put into the specific application of obstacle information recognition in the embodiment of the present disclosure.

[0071] In summary, according to the surround ranging method provided by the present application, after obtaining the fisheye images taken by multiple fisheye cameras, the fisheye images can be subjected to distortion correction processing to obtain the corrected images corresponding to the fisheye images; then, based on the perspective transformation matrix of the multiple fisheye cameras, the corrected images are subjected to a top-down transformation to obtain a top-down image; the vehicle ranging line is determined in the top-down image, and the distance between the vehicle and the obstacle is identified based on the vehicle ranging line. The technical solution in the present application, based on the existing hardware conditions, only needs to perform simple image processing on the fisheye images, transform the original fisheye images into a perspective taken from a plane above the vehicle body parallel to the ground, and form a top view that is more suitable for surround ranging. Finally, by setting a series of vehicle ranging lines parallel to the vehicle body around the vehicle body, the 360-degree ranging capability around the vehicle body is realized. The solution gives the vehicle the ability of surround ranging without increasing the amount of additional calculations, and this capability can be adapted to some existing models with limited computing resources, which can expand the scope of application of surround ranging. In addition, there is no need to train and run a monocular depth estimation model during the ranging process, which can reduce the occupancy of computing resources and reduce the vehicle's computing resource consumption.

[0072] Based on the above Figure 1-Figure 2 The specific implementation of the method shown in the embodiment provides a surround distance measurement device, such as Figure 6 As shown, the device includes: an acquisition module 31, a processing module 32, a transformation module 33, and an identification module 34;

[0073] An acquisition module 31 may be used to acquire fisheye images taken by multiple fisheye cameras, wherein the multiple fisheye cameras are used to collect fisheye images of roads in different directions during the driving process of the vehicle;

[0074] The processing module 32 may be used to perform distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image;

[0075] The transformation module 33 can be used to perform a top-view transformation on the rectified image based on the perspective transformation matrix of multiple fisheye cameras to obtain a top-view image;

[0076] The recognition module 34 may be used to determine the vehicle ranging line in the overhead image, and to recognize the distance between the vehicle body and the obstacle based on the vehicle ranging line.

[0077] In a specific application scenario, the processing module 32 can be specifically used to perform camera calibration processing on multiple fisheye cameras respectively to obtain the camera internal parameters of each of the multiple fisheye cameras; determine the mapping relationship between the fisheye image and the corrected image under the camera internal parameters, wherein different camera internal parameters correspond to different mapping relationships; perform image mapping on the fisheye image according to the mapping relationship to obtain a corrected image after the fisheye image is dedistorted.

[0078] In specific application scenarios, such as Figure 7 As shown, the device further includes: a determination module 35 and a calculation module 36;

[0079] A determination module 35, which can be used to determine the coordinates of the corner points of several calibration plates in the rectified image, and the expected pixel coordinates of the corner points on the standard top view image;

[0080] The calculation module 36 may be used to calculate the perspective transformation matrix corresponding to the rectified image based on the corner point coordinates and the expected pixel coordinates.

[0081] In a specific application scenario, the transformation module 33 can be used to perform perspective transformation processing on each pixel in the corrected image based on the perspective transformation matrix to obtain a first overhead view image corresponding to the corrected image; and perform image stitching processing on multiple first overhead view images under multiple fisheye cameras to obtain a final second overhead view image.

[0082] In a specific application scenario, the transformation module 33 can be specifically used to determine the stitching area of ​​the first overhead view images corresponding to adjacent fisheye cameras among multiple fisheye cameras; calculate the stitching transparency of the first overhead view image in the stitching area; and based on the stitching transparency, stitch the first overhead view images corresponding to adjacent fisheye cameras in the stitching area to obtain a final second overhead view image.

[0083] In a specific application scenario, when determining the vehicle ranging line in the overhead image, the recognition module 34 can be used to generate a plurality of vehicle ranging lines parallel to the vehicle body in the second overhead image according to a preset distance interval, where the preset distance interval is the horizontal distance between the vehicle ranging line and the vehicle body.

[0084] In a specific application scenario, when identifying the distance between the vehicle and an obstacle based on the vehicle ranging line, the identification module 34 can be used to identify whether there is an obstacle in the second overhead image; when there is an obstacle in the second overhead image, the horizontal distance between each obstacle pixel point and the vehicle body in the obstacle area is identified based on the vehicle ranging line; based on the horizontal distance between each obstacle pixel point and the vehicle body in the obstacle area, the distance between the vehicle body and the obstacle is determined.

[0085] It should be noted that for other corresponding descriptions of the functional units involved in the surround ranging device provided in this embodiment, reference can be made to Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0086] Based on the above Figure 1 and Figure 2 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0087] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0088] Based on the above Figure 1 and Figure 2 The method shown, and Figure 6 and Figure 7 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device that can be configured on the vehicle (such as an electric vehicle) side, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.

[0089] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0090] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0091] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0092] Based on the above electronic device, the embodiment of the present application further provides a vehicle, which may specifically include: the above electronic device. The vehicle may specifically be an electric car, etc.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware platforms, or by hardware. The technical solution in the present application, based on the existing hardware conditions, only needs to perform simple image processing on the fisheye image, transform the original fisheye image into a perspective taken from a plane above the vehicle body parallel to the ground, and form a top view that is more suitable for surround ranging. Finally, by setting a series of vehicle ranging lines parallel to the vehicle body around the vehicle body, the 360-degree ranging capability around the vehicle body is achieved. This solution gives the vehicle the ability to look around and measure distance without adding almost any additional calculation amount. This capability can be adapted to some existing models with limited computing resources, and the scope of application of surround ranging can be expanded. In addition, there is no need to train and run a monocular depth estimation model during the ranging process, which can reduce the occupation of computing resources and reduce the computing resource consumption of the vehicle.

[0094] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0095] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A surround ranging method, characterized in that: include: Acquire fisheye images taken by a plurality of fisheye cameras, wherein the plurality of fisheye cameras are used to collect the fisheye images of roads in different directions during the driving process of the vehicle; Performing distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image; Based on the perspective transformation matrices of the plurality of fisheye cameras, performing a top-view transformation on the rectified image to obtain a top-view image; A vehicle ranging line is determined in the top view image, and a distance between a vehicle body and an obstacle is identified based on the vehicle ranging line.

2. The method according to claim 1, characterized in that The step of performing distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image includes: Performing camera calibration processing on the multiple fisheye cameras respectively to obtain camera intrinsic parameters of each of the multiple fisheye cameras; Determine a mapping relationship between the fisheye image and the rectified image under the camera intrinsic parameters, wherein different camera intrinsic parameters correspond to different mapping relationships; Image mapping is performed on the fisheye image according to the mapping relationship to obtain a first corrected image after the fisheye image is dedistorted.

3. The method according to claim 2, characterized in that The method for calculating the perspective transformation matrix includes: Acquire calibration images containing several calibration plate areas taken by multiple fisheye cameras; Performing image mapping on the calibration image based on the mapping relationship to obtain a calibration corrected image after the calibration image is dedistorted; Determining the coordinates of the corner points of the calibration plate areas in the calibration and correction image, and the expected pixel coordinates of the corner points of the calibration plate areas on the standard top view image; Based on the corner point coordinates and the expected pixel coordinates, perspective transformation matrices of the plurality of fisheye cameras are calculated.

4. The method according to claim 1, characterized in that: The step of performing a top-view transformation on the corrected image based on the perspective transformation matrix of the plurality of fisheye cameras to obtain a top-view image comprises: Based on the perspective transformation matrix, performing perspective transformation processing on each pixel point in the corrected image to obtain a first overhead image corresponding to the corrected image; The plurality of first overhead view images captured by the plurality of fisheye cameras are subjected to image stitching processing to obtain a final second overhead view image.

5. The method according to claim 4, characterized in that The step of performing image stitching processing on the plurality of first overhead images captured by the plurality of fisheye cameras to obtain a final second overhead image comprises: Determine a stitching area of ​​the first overhead view image corresponding to adjacent fisheye cameras among the plurality of fisheye cameras; Calculate the stitching transparency of the first overhead image in the stitching area; Based on the stitching transparency, the first overhead view images corresponding to the adjacent fisheye cameras are stitched in the stitching area to obtain a final second overhead view image.

6. The method according to claim 4, characterized in that Determining the vehicle ranging line in the overhead image includes: In the second top-view image, a plurality of vehicle ranging lines parallel to the vehicle body are generated at a preset distance interval, and the preset distance interval is a horizontal distance between the vehicle ranging line and the vehicle body in the second top-view image.

7. The method according to claim 6, characterized in that The identifying the distance between the vehicle body and the obstacle based on the vehicle ranging line includes: Identify whether there is an obstacle in the second overhead image; In the case where there is an obstacle in the second top-view image, identifying the horizontal distance between each obstacle pixel point and the vehicle body in the obstacle area based on the vehicle ranging line; Based on the horizontal distance between each obstacle pixel point in the obstacle area and the vehicle body, the distance between the vehicle body and the obstacle is determined.

8. A surround distance measuring device, characterized in that: include: An acquisition module, used to acquire fisheye images taken by multiple fisheye cameras, wherein the multiple fisheye cameras are used to collect the fisheye images of roads in different directions during the driving process of the vehicle; A processing module, used for performing distortion correction processing on the fisheye image to obtain a corrected image corresponding to the fisheye image; A transformation module, configured to perform a top-view transformation on the rectified image based on the perspective transformation matrices of the plurality of fisheye cameras to obtain a top-view image; The recognition module is used to determine the vehicle ranging line in the overhead image and recognize the distance between the vehicle body and the obstacle based on the vehicle ranging line.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

11. A vehicle, characterized in that: include: The electronic device as claimed in claim 10.

Images