Distance measurement method and device based on fisheye camera, electronic equipment and storage medium

Abstract

The invention discloses a distance measurement method and device based on a fisheye camera, electronic equipment and a storage medium, and relates to the technical field of vehicles, and the main technical scheme comprises the steps: collecting a target image in a preset direction outside a vehicle; performing perspective transformation processing on the distorted target image by using a transformation matrix to obtain a target top view image corresponding to the target image; and performing distance calculation according to an image distance between a target obstacle and a vehicle boundary line in the target top view image to obtain a distance between the target obstacle and the vehicle boundary line. Compared with the prior art, distance calculation is carried out through the image distance between the target obstacle and the vehicle boundary line to obtain the distance between the target obstacle and the vehicle boundary line, so that the obtained distance can better represent the distance between the vehicle and the obstacle.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a distance measurement method and device based on a fisheye camera, an electronic device and a storage medium. Background Art

[0002] Existing monocular ranging algorithms are mainly implemented through deep learning models, which rely on complex neural networks and therefore require large computing resources in the deployment environment. This makes it difficult to deploy them in scenarios with limited computing resources and high real-time requirements. In addition, the current mainstream monocular ranging algorithms often require a large amount of data to obtain good generalization capabilities, which will require a lot of manpower and resource costs.

[0003] In addition, the distance information obtained by the current mainstream solution is the distance between the object and the camera, which easily leads to the problem of inaccurate prediction of absolute depth. Summary of the invention

[0004] The present invention provides a distance measurement method and device based on a fisheye camera, an electronic device and a storage medium, which is mainly intended to solve the problem that the distance information obtained by the current mainstream solution is the distance between the object and the camera, which easily leads to inaccurate absolute depth prediction.

[0005] According to a first aspect of the present disclosure, a distance measurement method based on a fisheye camera is provided, comprising:

[0006] Collect target images in a preset direction outside the vehicle;

[0007] Performing perspective transformation processing on the dedistorted target image using a transformation matrix to obtain a target bird's-eye view image corresponding to the target image;

[0008] The distance between the target obstacle and the vehicle boundary line is calculated according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line.

[0009] Optionally, performing distance calculation according to an image distance between a target obstacle and a vehicle boundary line in the target overhead image to obtain a distance between the target obstacle and the vehicle boundary line includes:

[0010] Calculating the target actual distance corresponding to the image distance between the target obstacle and the vehicle boundary line according to the mapping relationship between a pixel size in the target overhead image and the actual distance, wherein the image distance is the product of the number of pixels between the vehicle boundary line and the target obstacle and the pixel size;

[0011] The target actual distance is taken as the distance between the target obstacle and the vehicle boundary line.

[0012] Optionally, obtaining the distance between the target obstacle and the vehicle boundary line further includes:

[0013] Setting a preset number of distance lines in the target overhead image;

[0014] Calculate the target actual distance respectively corresponding to each of the distance lines and the vehicle boundary line according to the image distance between each of the distance lines and the vehicle boundary line and the mapping relationship;

[0015] Calibrate the actual distance of the target corresponding to each of the distance lines;

[0016] The target actual distance is used to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image.

[0017] Optionally, using the target actual distance to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image includes:

[0018] Selecting a target pixel closest to the vehicle boundary line within the target obstacle range in the target overhead image;

[0019] Determining a target distance line that coincides with the target pixel from a preset number of the distance lines;

[0020] The target actual distance corresponding to the target distance line is determined as the distance between the target obstacle and the vehicle boundary line.

[0021] Optionally, before performing perspective transformation processing on the dedistorted target image using the transformation matrix, the method includes:

[0022] Collecting a preset number of first training fisheye images based on a preset number of fisheye cameras; wherein the arrangement positions of the first calibration plates in different first training fisheye images are different;

[0023] Based on the preset number of first training fisheye images, internal parameters of each of the fisheye cameras are calibrated to obtain internal parameters of the fisheye camera; wherein the first training fisheye image is only used for parameter calibration of the fisheye camera that captures the image;

[0024] The target image is dedistorted using the internal parameters to obtain the dedistorted target image.

[0025] Optionally, the using the transformation matrix to perform perspective transformation processing on the dedistorted target image includes:

[0026] Using the fisheye camera to capture a calibration image including a second calibration plate;

[0027] Acquire actual coordinates of a preset number of corner points on the second calibration plate in the calibration image, and acquire expected coordinates on the second calibration plate in the calibration image after perspective transformation corresponding to the actual coordinates;

[0028] The transformation matrix is ​​calculated based on the mapping relationship between the actual coordinates and the expected coordinates;

[0029] The transformation matrix is ​​used to perform perspective transformation processing on the dedistorted target image to obtain a target bird's-eye view image corresponding to the target image.

[0030] Optionally, the collecting of a target image in a preset direction outside the vehicle includes:

[0031] A target image in a preset direction outside the vehicle is acquired based on an image acquisition device arranged on the vehicle.

[0032] According to a second aspect of the present disclosure, a distance measuring device based on a fisheye camera is provided, comprising:

[0033] A first acquisition unit is used to acquire a target image in a preset direction outside the vehicle;

[0034] A first processing unit is used to perform perspective transformation processing on the dedistorted target image using a transformation matrix to obtain a target bird's-eye view image corresponding to the target image;

[0035] The calculation unit is used to perform distance calculation according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line.

[0036] Optionally, the computing unit includes:

[0037] A first calculation module is used to calculate the target actual distance corresponding to the image distance between the target obstacle and the vehicle boundary line according to the mapping relationship between a pixel size in the target overhead image and the actual distance, wherein the image distance is the product of the number of pixels between the vehicle boundary line and the target obstacle and the pixel size;

[0038] A determination module is used to use the target actual distance as the distance between the target obstacle and the vehicle boundary line.

[0039] Optionally, the computing unit further includes:

[0040] A setting module, used for setting a preset number of distance lines in the target overhead image;

[0041] A second calculation module, used for calculating the target actual distance respectively corresponding to each of the distance lines and the vehicle boundary line according to the image distance between each of the distance lines and the vehicle boundary line and the mapping relationship;

[0042] A calibration module, used for calibrating the actual distance of the target corresponding to each of the distance lines;

[0043] A measuring module is used to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image using the target actual distance.

[0044] Optionally, the measuring module is further used for:

[0045] Selecting a target pixel closest to the vehicle boundary line within the target obstacle range in the target overhead image;

[0046] Determining a target distance line that coincides with the target pixel from a preset number of the distance lines;

[0047] The target actual distance corresponding to the target distance line is determined as the distance between the target obstacle and the vehicle boundary line.

[0048] Optionally, the device comprises:

[0049] A second acquisition unit is used to acquire a preset number of first training fisheye images based on a preset number of fisheye cameras; wherein the arrangement positions of the first calibration plates in different first training fisheye images are different;

[0050] A calibration unit, configured to calibrate internal parameters of each of the fisheye cameras based on the preset number of first training fisheye images, to obtain internal parameters of the fisheye camera; wherein the first training fisheye image is only used for parameter calibration of the fisheye camera that acquires the image;

[0051] The second processing unit is used to perform dedistortion processing on the target image using the internal parameters to obtain the dedistorted target image.

[0052] Optionally, the first processing unit includes:

[0053] An acquisition module, used for using the fisheye camera to capture a calibration image including a second calibration plate;

[0054] An acquisition module, used to acquire actual coordinates of a preset number of corner points on the second calibration plate in the calibration image, and acquire expected coordinates on the second calibration plate in the calibration image corresponding to the actual coordinates after perspective transformation;

[0055] A calculation module, used for calculating the transformation matrix according to the mapping relationship between the actual coordinates and the expected coordinates;

[0056] The processing module is used to perform perspective transformation processing on the dedistorted target image using the transformation matrix to obtain a target bird's-eye view image corresponding to the target image.

[0057] Optionally, the first acquisition unit is further used for:

[0058] A target image in a preset direction outside the vehicle is acquired based on an image acquisition device arranged on the vehicle.

[0059] According to a third aspect of the present disclosure, a vehicle is provided, comprising the fisheye camera-based ranging device described in the second aspect.

[0060] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0061] at least one processor; and

[0062] a memory communicatively connected to the at least one processor; wherein,

[0063] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0064] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0065] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0066] The present disclosure provides a distance measurement method and device based on a fisheye camera, an electronic device and a storage medium, which collect a target image of a preset direction outside a vehicle; use a transformation matrix to perform perspective transformation processing on the dedistorted target image to obtain a target overhead image corresponding to the target image; perform distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line. Compared with the related art, the distance measurement purpose is achieved by converting the collected target image into its corresponding target overhead image, performing distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image, and obtaining the distance between the target obstacle and the vehicle boundary line. The distance calculation is performed based on the image distance between the target obstacle and the vehicle boundary line to obtain the distance between the target obstacle and the vehicle boundary line, so that the obtained distance can better represent the distance between the vehicle and the obstacle.

[0067] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0069] Figure 1 A schematic diagram of a flow chart of a distance measurement method based on a fisheye camera provided in an embodiment of the present disclosure;

[0070] Figure 2 A schematic diagram of setting a distance line provided in an embodiment of the present disclosure;

[0071] Figure 3 A camera calibration schematic diagram provided in an embodiment of the present disclosure;

[0072] Figure 4 A schematic diagram of the structure of a distance measuring device based on a fisheye camera provided in an embodiment of the present disclosure;

[0073] Figure 5 A schematic structural diagram of another distance measuring device based on a fisheye camera provided in an embodiment of the present disclosure;

[0074] Figure 6 A schematic block diagram of an exemplary electronic device 300 provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0076] The following describes a distance measurement method and device based on a fisheye camera, an electronic device, and a storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0077] Figure 1 A schematic flow chart of a distance measurement method based on a fisheye camera provided in an embodiment of the present disclosure.

[0078] like Figure 1 As shown, the method comprises the following steps:

[0079] Step 101, capturing a target image in a preset direction outside the vehicle.

[0080] As a refinement of the above step 101, the target image is an image captured by an image acquisition device, including but not limited to the environment information of the image acquisition device, the location information of surrounding objects, etc. In order to achieve the acquisition of the target image, a fisheye camera can be used but not limited to it. In order to capture the target image of a preset direction outside the vehicle, it is necessary to set the fisheye camera at the body position of the vehicle corresponding to the preset direction. It should be understood that the preset direction can be any direction in the 360-degree direction. In order to achieve coverage of the preset direction, multiple fisheye cameras need to be set on the vehicle. The above setting of the fisheye camera on the vehicle is only an exemplary description, and this embodiment does not limit this.

[0081] In order to clearly explain the acquisition process of the target image, in combination with the above embodiment, this embodiment provides an exemplary description. When acquiring the target image in a preset direction, the corresponding fisheye camera on the vehicle is started according to the preset direction, and the fisheye camera is used to acquire the target image in the preset direction. The above acquisition process of the target image is only exemplary, and this embodiment does not limit this.

[0082] Step 102: Perform perspective transformation processing on the dedistorted target image using a transformation matrix to obtain a target bird's-eye view image corresponding to the target image.

[0083] As a refinement of the above step 102, the transformation matrix is ​​obtained by calculating the projection transformation, which represents the transformation relationship between different plane images. The target image is perspectively transformed by using the transformation matrix to obtain the target overhead image corresponding to the target image.

[0084] In order to clearly explain the perspective transformation process, this embodiment provides an exemplary explanation of this, which should be understood. The transformation matrix can realize the conversion of the image of the vertical plane into the image of the horizontal plane, and the picture information in the image will not be greatly distorted. After the target image is captured by the fisheye camera on the vehicle, the target image can be regarded as the image of the aforementioned vertical plane. The target image is perspective transformed by the transformation matrix, that is, the pixel points in the target image are perspectively transformed to the image pixel points of the corresponding horizontal plane, thereby obtaining the target overhead image corresponding to the target image. The transformation matrix is ​​the transformation coefficient required when the pixel points in the target image are perspectively transformed to the image pixel points of the corresponding horizontal plane.

[0085] Step 103 , performing distance calculation according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image, to obtain the distance between the target obstacle and the vehicle boundary line.

[0086] As a refinement of the above step 103, in order to realize the calculation of the distance between the target obstacle and the vehicle boundary line, the corresponding relationship between the size of a pixel in the target overhead image and the actual length is set, for example: the size of a pixel is 0.02 cm, and the corresponding actual length is 1 cm. The distance between the target obstacle and the vehicle boundary line is calculated by determining the number of pixels between the target obstacle and the vehicle boundary line in the target overhead image. The distance in the target overhead image is the number of pixels, and the actual distance between the target obstacle and the vehicle boundary line is the number of pixels multiplied by the ratio of the actual length to the pixel size. For example: the size of a pixel is 0.02 cm, the corresponding actual length is 1 cm, the number of pixels is 100, and the actual distance between the target obstacle and the vehicle boundary line is 5000 cm. The above data involved in determining the actual distance between the target obstacle and the vehicle boundary line is only exemplary, and this embodiment does not limit this.

[0087] The present disclosure provides a distance measurement method based on a fisheye camera, which collects a target image of a preset direction outside a vehicle; uses a transformation matrix to perform perspective transformation processing on the dedistorted target image to obtain a target overhead image corresponding to the target image; performs distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line. Compared with the related art, the distance measurement purpose is achieved by converting the collected target image into its corresponding target overhead image, performing distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image, and obtaining the distance between the target obstacle and the vehicle boundary line. The distance calculation is performed based on the image distance between the target obstacle and the vehicle boundary line to obtain the distance between the target obstacle and the vehicle boundary line, so that the obtained distance can better represent the distance between the vehicle and the obstacle.

[0088] As a refinement of an embodiment of the present disclosure, when executing step 103 to perform distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line, the following implementation method may also be adopted but is not limited to, for example: based on a mapping relationship between a pixel size and the actual distance in the target overhead image, the target actual distance corresponding to the image distance between the target obstacle and the vehicle boundary line is calculated, the image distance being the product of the number of pixels between the vehicle boundary line and the target obstacle and the pixel size; and the target actual distance is used as the distance between the target obstacle and the vehicle boundary line.

[0089] In order to clearly illustrate the execution process of the above embodiment, by setting the mapping relationship between a pixel size and the actual distance in the target overhead image, for example: the size of a pixel is 0.02 cm, and the corresponding actual distance is 1 cm, wherein 0.02 cm and the 1 cm are the mapping relationship, the distance in the target overhead image is the number of pixels, and the actual distance between the target obstacle and the vehicle boundary line is the number of pixels multiplied by the ratio of the actual distance to the pixel size, for example: the size of a pixel is 0.02 cm, and the corresponding actual distance is 1 cm, the number of pixels is 100, and the target actual distance between the target obstacle and the vehicle boundary line is 5000 cm. The 5000 cm is the measured distance between the target obstacle and the vehicle boundary line in the real scene. The above data involved in determining the actual distance between the target obstacle and the vehicle boundary line is only exemplary, and this embodiment does not limit this.

[0090] As a refinement of the above embodiment, in order to obtain the distance between the target obstacle and the vehicle boundary line, the following implementation methods can also be adopted but are not limited to, for example: setting a preset number of distance lines in the target overhead image; calculating the target actual distance corresponding to each distance line and the vehicle boundary line according to the image distance between each distance line and the vehicle boundary line and the mapping relationship; calibrating the corresponding target actual distance for each distance line; and using the target actual distance to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image.

[0091] In order to clearly illustrate the contents of the above embodiment, this embodiment provides an exemplary description, for example: in order to more intuitively display the distance between the target obstacle and the vehicle boundary line, a preset number of distance lines are set in the target overhead image, and the distance lines are reference lines for judging the distance between the target obstacle and the vehicle boundary line. The number of distance lines is a preset number, and the number of distance lines in this embodiment is not limited. According to the image distance and the mapping relationship between each distance line and the vehicle boundary line, the target actual distance corresponding to each distance line and the vehicle boundary line is calculated. The calculation process of the target actual distance is the same as the calculation process of the target actual distance mentioned above, that is, by setting the corresponding relationship between a pixel size and the actual length in the target overhead image, for example: the size of a pixel is 0.02cm, and its corresponding actual length is 1cm. By determining the number of pixels between the distance line and the vehicle boundary line in the target overhead image, the distance between the distance line and the vehicle boundary line is calculated. The distance in the target overhead image is the number of pixels, and the target actual distance between the distance line and the vehicle boundary line is the number of pixels multiplied by the ratio of the actual length to the pixel size. For example, the size of a pixel is 0.02 cm, the corresponding actual length is 1 cm, the number of pixels is 100, and the target actual distance between the distance line and the vehicle boundary line is 5000 cm. The calculated target actual distance is used to calibrate the corresponding distance line, and the distance line with a scale value is obtained. The distance line with a scale value is used to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image, that is, the corresponding distance line is determined according to the position of the target obstacle relative to the distance line, and the scale of the distance line is the distance between the target obstacle and the vehicle boundary line.

[0092] To facilitate understanding of the above distance line setting process, Figure 2 A schematic diagram of setting a distance line provided in an embodiment of the present disclosure is shown in FIG. Figure 2Shown are: vehicle body, fisheye camera and distance line with scale values.

[0093] In a possible implementation of the present disclosure, determining the distance between the target obstacle and the vehicle boundary line according to the distance line includes combining the distance line with human body detection, vehicle detection and a ground segmentation model to obtain the distance between the specific obstacle and the vehicle body.

[0094] As a refinement of the above embodiment, when using the target actual distance to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image, the following implementation methods can be adopted but are not limited to, for example: selecting the target pixel closest to the vehicle boundary line within the target obstacle range in the target overhead image; determining the target distance line that coincides with the target pixel from a preset number of distance lines; and determining the target actual distance corresponding to the target distance line as the distance between the target obstacle and the vehicle boundary line.

[0095] In order to more clearly illustrate the steps of the above embodiment, this embodiment provides an exemplary description, for example: based on the distance line, the distance between a person and a vehicle is measured. In this case, the person is the target obstacle, and the target pixel point closest to the vehicle boundary line within the range of the person's feet is selected in the target overhead image. In this scenario, the target pixel point is generally a pixel point within the heel range, and then the target distance line that coincides with the target pixel point is selected from a preset number of distance lines, and the target actual distance corresponding to the target distance line is the distance between the person and the vehicle.

[0096] As a refinement of the above embodiment, before using the transformation matrix to perform perspective transformation processing on the dedistorted target image, the method can also adopt but is not limited to the following implementation methods, for example: based on a preset number of fisheye cameras, a preset number of first training fisheye images are collected; wherein the arrangement positions of the first calibration plates in different first training fisheye images are different; based on the preset number of first training fisheye images, the internal parameters of each of the fisheye cameras are calibrated respectively to obtain the internal parameters of the fisheye camera; wherein the first training fisheye image is only used for parameter calibration of the fisheye camera that collects the image; and the target image is dedistorted using the internal parameters to obtain the dedistorted target image.

[0097] In one possible implementation of the embodiment of the present application, since the target image captured by the fisheye camera is distorted, the target image needs to be dedistorted. However, when dedistorting, the dedistortion operation needs to be performed according to the internal parameters of the fisheye camera. Therefore, the internal parameters of the fisheye camera need to be calibrated in advance. The fisheye camera can be calibrated according to the following steps:

[0098] Collecting a preset number of first training fisheye images based on a preset number of fisheye cameras; wherein the arrangement positions of the first calibration plates in different first training fisheye images are different;

[0099] When dedistorting a fisheye image taken by the same fisheye camera, only the internal parameters of the same fisheye camera can be used for dedistortion. Therefore, in practical applications, as many fisheye cameras as are used, the internal parameters of the cameras need to be calibrated. The embodiment of the present application does not limit the preset number of fisheye cameras. The internal parameters of the camera include the focal length, optical center, radial distortion coefficient, etc. Specifically, the embodiment of the present application does not limit the internal parameters of the camera.

[0100] When calibrating the internal parameters of the camera, a variety of methods can be used for calibration. In the embodiment of the present application, the internal parameters of the camera are calibrated using Zhang's calibration method. It should be noted that this description method is not a specific limitation on the specific calibration method.

[0101] In order to understand the calibration process of the camera internal parameters, Figure 3 A camera calibration schematic diagram provided for an embodiment of the present disclosure includes: a calibration plate, a vehicle body, and a fisheye camera;

[0102] See also Figure 3 According to the image information captured by each fisheye camera, when shooting, the calibration plate is placed at different positions in the image, and multiple shots are taken to obtain a preset number of first training fisheye images. For example, each fisheye camera takes 20 first training fisheye images. Specifically, the number of shots of the image information can be determined based on empirical values, which is not limited in the embodiments of the present application.

[0103] Based on the preset number of first training fisheye images, internal parameters of corresponding fisheye cameras are calibrated respectively to obtain internal parameters of the fisheye cameras.

[0104] In one implementable manner of the embodiment of the present application, calibration can be performed using a camera calibration interface such as that provided by MATLAB or OpenCV to obtain the calibrated internal parameters of the camera; the embodiment of the present application does not limit the specific calibration method.

[0105] In one implementable method of the embodiment of the present application, after the internal parameter calibration of each fisheye camera is completed, after the target image is photographed by the fisheye camera, the target image photographed by the fisheye camera is dedistorted according to the camera internal parameters of each fisheye camera to obtain a dedistorted target image; then the image output by the fisheye camera is the dedistorted target image.

[0106] As a refinement of the above embodiment, when performing perspective transformation processing on the dedistorted target image using the transformation matrix described in step 102, the method may also adopt but is not limited to the following implementation methods, for example: using the fisheye camera to capture a calibration image containing a second calibration plate; obtaining the actual coordinates of a preset number of corner points on the second calibration plate in the calibration image, and obtaining the expected coordinates on the second calibration plate in the calibration image after perspective transformation corresponding to the actual coordinates; calculating the transformation matrix according to the mapping relationship between the actual coordinates and the expected coordinates; using the transformation matrix to perform perspective transformation processing on the dedistorted target image to obtain a target overhead view image corresponding to the target image.

[0107] In one possible implementation of the embodiment of the present application, the coordinate values ​​of the four corner points of each calibration plate are determined in each calibration image. It should be understood that the present disclosure does not limit the number of corner points to four. The actual coordinate value is the position of the pixel point corresponding to the corner point in the calibration image; the expected coordinate is the coordinate value of the pixel corresponding to the corner point of the calibration plate in the target overhead image; in one possible implementation of the embodiment of the present application, when determining the expected coordinates of the corner point, since the size of the calibration plate, the distance between the calibration plate and the vehicle, and the size range of the calibration image are known, the distances of the four corners of the calibration plate from each boundary in the target overhead image can be determined, and since a pixel point in the target overhead image corresponds to the actual length of the real world, the coordinate values ​​of the four corner points of the calibration plate can be determined.

[0108] In one implementable method of an embodiment of the present application, after determining the actual coordinates and expected coordinates of each corner point, the transformation matrix required for the perspective transformation can be calculated based on the actual coordinates and expected coordinates of each corner point and the findHomography or getPerspectiveTransform service provided by opencv; for the specific method of calculating the transformation matrix, please refer to any implementation method in the prior art, and the embodiments of the present application will not be described one by one here.

[0109] In one implementable manner of the embodiment of the present application, after acquiring the dedistorted target image, a perspective transformation is performed based on each of the dedistorted target images and the transformation matrix to obtain a top view corresponding to each target image.

[0110] Projection transformation refers to the process of transforming one image into another image when the same camera is used to shoot the same plane at two positions A and B, respectively. In this case, position A is where the fisheye camera is installed on the car body, and position B is a position above the car parallel to the ground. The fisheye image needs to be transformed into a bird's-eye view with the help of perspective transformation, that is, the image taken by the fisheye camera installed on the car body is transformed into the image taken by a camera above the car parallel to the ground.

[0111] In one possible implementation of the embodiment of the present application, image mapping can be performed based on the remap function provided by opencv to obtain a target bird's-eye view image after perspective transformation.

[0112] As a refinement of the above embodiment, when executing step 101 to capture the target image in a preset direction outside the vehicle, the following implementation method can be adopted but is not limited to, for example: capturing the target image in the preset direction outside the vehicle based on an image capture device set on the vehicle.

[0113] The target image of the preset direction of the vehicle is acquired by an image acquisition device arranged on the vehicle, and the image acquisition device includes but is not limited to a fisheye camera. The target image is acquired by the image acquisition device arranged on the vehicle, so that the reuse of the image acquisition device is realized, and no new image acquisition device needs to be added, thus saving costs.

[0114] In summary, this embodiment can achieve the following effects:

[0115] 1. Performing distance calculation based on the image distance between the target obstacle and the vehicle boundary line to obtain the distance between the target obstacle and the vehicle boundary line, so that the obtained distance can better represent the distance between the vehicle and the obstacle.

[0116] 2. There is no need to introduce complex neural network structures, and there is no need to invest too much manpower and resource costs to improve the generalization ability of the model.

[0117] 3. Through a one-time calibration, the fisheye camera can be equipped with the ability to measure distance, which can solve the problem that the existing solution can only measure the distance between the object and the camera but not the distance between the object and the vehicle body.

[0118] Corresponding to the above-mentioned distance measurement method based on a fisheye camera, the present invention further proposes a distance measurement device based on a fisheye camera. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0119] Figure 4A schematic diagram of a distance measuring device based on a fisheye camera provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, including:

[0120] The first acquisition unit 21 is used to acquire a target image in a preset direction outside the vehicle;

[0121] A first processing unit 22 is used to perform perspective transformation processing on the dedistorted target image using a transformation matrix to obtain a target bird's-eye view image corresponding to the target image;

[0122] The calculation unit 23 is used to perform distance calculation according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line.

[0123] The present invention provides a distance measuring device based on a fisheye camera, which collects a target image of a preset direction outside a vehicle; uses a transformation matrix to perform perspective transformation processing on the dedistorted target image to obtain a target overhead image corresponding to the target image; performs distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line. Compared with the related art, the distance between the target obstacle and the vehicle boundary line is obtained by converting the collected target image into the corresponding target overhead image, and performing distance calculation based on the image distance between the target obstacle and the vehicle boundary line in the target overhead image, thereby achieving the purpose of distance measurement. The distance is calculated based on the image distance between the target obstacle and the vehicle boundary line to obtain the distance between the target obstacle and the vehicle boundary line, so that the obtained distance can better represent the distance between the vehicle and the obstacle.

[0124] Figure 5 A structural schematic diagram of another distance measuring device based on a fisheye camera is provided in an embodiment of the present disclosure, such as Figure 5 As shown, the calculation unit 23 includes:

[0125] A first calculation module 231 is used to calculate the target actual distance corresponding to the image distance between the target obstacle and the vehicle boundary line according to the mapping relationship between a pixel size in the target overhead image and the actual distance, wherein the image distance is the product of the number of pixels between the vehicle boundary line and the target obstacle and the pixel size;

[0126] The determination module 232 is configured to use the target actual distance as the distance between the target obstacle and the vehicle boundary line.

[0127] Furthermore, in a possible implementation of this embodiment, as Figure 5As shown, the calculation unit 23 also includes:

[0128] A setting module 233, used for setting a preset number of distance lines in the target overhead image;

[0129] A second calculation module 234 is used to calculate the target actual distance corresponding to each distance line and the vehicle boundary line according to the image distance between each distance line and the vehicle boundary line and the mapping relationship;

[0130] A calibration module 235, used for calibrating the actual distance of the target corresponding to each of the distance lines;

[0131] The measuring module 236 is configured to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image using the target actual distance.

[0132] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the measurement module 236 is also used for:

[0133] Selecting a target pixel closest to the vehicle boundary line within the target obstacle range in the target overhead image;

[0134] Determining a target distance line that coincides with the target pixel from a preset number of the distance lines;

[0135] The target actual distance corresponding to the target distance line is determined as the distance between the target obstacle and the vehicle boundary line.

[0136] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the device comprises:

[0137] A second acquisition unit 24 is used to acquire a preset number of first training fisheye images based on a preset number of fisheye cameras; wherein the arrangement positions of the first calibration plates in different first training fisheye images are different;

[0138] A calibration unit 25 is used to calibrate the internal parameters of each of the fisheye cameras based on the preset number of first training fisheye images to obtain the internal parameters of the fisheye camera; wherein the first training fisheye image is only used for parameter calibration of the fisheye camera that captures the image;

[0139] The second processing unit 26 is configured to perform a dedistortion process on the target image using the internal parameters to obtain the dedistorted target image.

[0140] Furthermore, in a possible implementation of this embodiment, as Figure 5As shown, the first processing unit 22 includes:

[0141] An acquisition module 221 is used to use the fisheye camera to capture a calibration image including a second calibration plate;

[0142] An acquisition module 222 is used to acquire actual coordinates of a preset number of corner points on the second calibration plate in the calibration image, and to acquire expected coordinates on the second calibration plate in the calibration image corresponding to the actual coordinates after perspective transformation;

[0143] A calculation module 223, configured to calculate the transformation matrix according to the mapping relationship between the actual coordinates and the expected coordinates;

[0144] The processing module 224 is used to perform perspective transformation processing on the dedistorted target image using the transformation matrix to obtain a target bird's-eye view image corresponding to the target image.

[0145] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the first acquisition unit 21 is also used for:

[0146] A target image in a preset direction outside the vehicle is acquired based on an image acquisition device arranged on the vehicle.

[0147] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principle is the same, which is no longer limited in this embodiment.

[0148] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0149] Figure 6 A schematic block diagram of an example electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0150] like Figure 6As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 302 or a computer program loaded from a storage unit 308 to a RAM (Random Access Memory) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0151] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0152] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as a distance measurement method based on a fisheye camera. For example, in some embodiments, the distance measurement method based on a fisheye camera may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the aforementioned fisheye camera-based ranging method in any other appropriate manner (for example, by means of firmware).

[0153] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0155] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0157] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0158] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0159] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0161] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A distance measurement method based on a fisheye camera, characterized in that: include: Collect target images in a preset direction outside the vehicle; Performing perspective transformation processing on the dedistorted target image using a transformation matrix to obtain a target bird's-eye view image corresponding to the target image; The distance between the target obstacle and the vehicle boundary line is calculated according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line.

2. The method according to claim 1, characterized in that The performing distance calculation according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line comprises: Calculating the target actual distance corresponding to the image distance between the target obstacle and the vehicle boundary line according to the mapping relationship between a pixel size in the target overhead image and the actual distance, wherein the image distance is the product of the number of pixels between the vehicle boundary line and the target obstacle and the pixel size; The target actual distance is taken as the distance between the target obstacle and the vehicle boundary line.

3. The method according to claim 2, characterized in that The obtaining of the distance between the target obstacle and the vehicle boundary line comprises: Setting a preset number of distance lines in the target overhead image; Calculating the target actual distance respectively corresponding to each of the distance lines and the vehicle boundary line according to the image distance between each of the distance lines and the vehicle boundary line and the mapping relationship; Calibrate the corresponding actual distance of the target for each of the distance lines; The target actual distance is used to measure the distance between the target obstacle and the vehicle boundary line in the target overhead image.

4. The method according to claim 3, characterized in that Measuring the distance between the target obstacle and the vehicle boundary line in the target overhead image using the target actual distance includes: Selecting a target pixel closest to the vehicle boundary line within the target obstacle range in the target overhead image; Determining a target distance line that coincides with the target pixel from a preset number of the distance lines; The target actual distance corresponding to the target distance line is determined as the distance between the target obstacle and the vehicle boundary line.

5. The method according to claim 1, characterized in that Before performing perspective transformation processing on the dedistorted target image using the transformation matrix, the method includes: Collecting a preset number of first training fisheye images based on a preset number of fisheye cameras; wherein the arrangement positions of the first calibration plates in different first training fisheye images are different; Based on the preset number of first training fisheye images, internal parameters of each of the fisheye cameras are calibrated to obtain internal parameters of the fisheye camera; wherein the first training fisheye image is only used for parameter calibration of the fisheye camera that captures the image; The target image is dedistorted using the internal parameters to obtain the dedistorted target image.

6. The method according to claim 5, characterized in that The using the transformation matrix to perform perspective transformation processing on the dedistorted target image comprises: Using the fisheye camera to capture a calibration image including a second calibration plate; Acquire actual coordinates of a preset number of corner points on the second calibration plate in the calibration image, and acquire expected coordinates on the second calibration plate in the calibration image after perspective transformation corresponding to the actual coordinates; The transformation matrix is ​​calculated based on the mapping relationship between the actual coordinates and the expected coordinates; The transformation matrix is ​​used to perform perspective transformation processing on the dedistorted target image to obtain a target bird's-eye view image corresponding to the target image.

7. The method according to claim 1, characterized in that The collecting of the target image in the preset direction outside the vehicle comprises: A target image in a preset direction outside the vehicle is acquired based on an image acquisition device arranged on the vehicle.

8. A distance measuring device based on a fisheye camera, characterized in that: include: A first acquisition unit is used to acquire a target image in a preset direction outside the vehicle; A first processing unit is used to perform perspective transformation processing on the dedistorted target image using a transformation matrix to obtain a target bird's-eye view image corresponding to the target image; The first calculation unit is used to perform distance calculation according to the image distance between the target obstacle and the vehicle boundary line in the target overhead image to obtain the distance between the target obstacle and the vehicle boundary line.

9. A vehicle, characterized in that: Including the distance measuring device based on fisheye camera as described in claim 8.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

12. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images