Joint calibration method of vehicle cameras, electronic device, storage medium and vehicle

Abstract

The application provides a joint calibration method of a vehicle camera, an electronic device, a storage medium and a vehicle. The method comprises: determining at least one other gray image having an overlapping part with each of the obtained multiple gray images and determining the overlapping part; constructing a gray histogram according to the gray values of each pixel for each overlapping part, obtaining a first local maximum value with the highest gray value and a second local maximum value with the lowest gray value by calculating the image gradient of the gray histogram, and converting the overlapping part into a binary image based on the first local maximum value and the second local maximum value; performing polygon detection on the binary image to determine a quadrilateral that meets a preset hyperparameter; determining a homography matrix between the gray image and the other gray image by using the multiple vertices of the quadrilateral and the multiple vertices of another quadrilateral in the same overlapping part of the other gray image, and using the homography matrix as a joint calibration result to calibrate the vehicle.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of vehicle calibration, and particularly relate to a joint calibration method of vehicle cameras, an electronic device, a storage medium and a vehicle. BACKGROUND

[0002] In related calibration methods, the entire image information of the images captured by each camera is often used for calibration, which causes the overlapping part captured by each camera to have a display problem after combination.

[0003] Specifically, in related calibration methods, when using the entire image information of the images, it is inevitable to use more of the part of the images where the feature points are more concentrated, that is, the non-overlapping part of the images captured by each camera, and less or even no feature points are used in the overlapping part, so that the overlapping part is difficult to achieve good results after combination of each image.

[0004] Therefore, there is a need for a solution that can clearly display the overlapping part after combination. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a joint calibration method of vehicle cameras, an electronic device, a storage medium and a vehicle to solve the problem of unclear display of the overlapping part captured by each camera after combination.

[0006] To achieve the above purpose, the present application provides a joint calibration method of vehicle cameras, comprising:

[0007] Based on each of the obtained plurality of grayscale images, at least one other grayscale image having an overlapping part with the grayscale image is determined, and the overlapping part is determined;

[0008] For each overlapping part, a grayscale histogram is constructed according to the grayscale values of each pixel, the first local maximum value with the highest grayscale value and the second local maximum value with the lowest grayscale value are obtained by calculating the image gradient of the grayscale histogram, and the overlapping part is converted into a binary image based on the first local maximum value and the second local maximum value;

[0009] The binary image is subjected to polygon detection, and a quadrilateral that meets a preset hyperparameter is determined from the detected plurality of polygons;

[0010] The homography matrix between the grayscale image and the other grayscale image is determined using the plurality of vertices of the quadrilateral and the plurality of vertices of another quadrilateral in the same overlapping part of the other grayscale image, and the homography matrix is used as a joint calibration result to calibrate the vehicle.

[0011] Further, constructing a gray scale histogram according to the gray scale values of the respective pixels, comprising:

[0012] Determining the number of pixels in each preset gray scale value interval based on the gray scale values of the respective pixels in the overlapping portion;

[0013] Constructing the size of the gray scale value as a first coordinate of the gray scale histogram, and arranging the respective gray scale value intervals in any order of monotonous increase or monotonous decrease of the gray scale value;

[0014] Constructing the number of pixels in the respective gray scale value intervals as a second coordinate of the gray scale histogram.

[0015] Further, obtaining a first local maximum value of the highest gray scale and a second local maximum value of the lowest gray scale by calculating the image gradient of the gray scale histogram, comprising:

[0016] First-order differentiating the gray scale histogram to obtain a plurality of local maximum values about the gray scale histogram,

[0017] In the plurality of gray scale value intervals of the gray scale histogram, determining a gray scale value interval as a first limit and a gray scale value interval as a second limit by using a preset pixel number parameter, wherein the gray scale value of the first limit is higher than the gray scale value of the second limit;

[0018] In the gray scale histogram, based on the first limit, the nearest local maximum value to the first limit in the direction of decreasing gray scale value is taken as the first local maximum value, and based on the second limit, the nearest local maximum value to the second limit in the direction of increasing gray scale value is taken as the second local maximum value.

[0019] Further, determining a gray scale value interval as a first limit and a gray scale value interval as a second limit by using a preset pixel number parameter, comprising:

[0020] Starting from the gray scale value interval of the largest gray scale value in the order of monotonous decrease of the gray scale value, the number of pixels in the gray scale value interval is accumulated one by one, and in response to the accumulated number of pixels satisfying the pixel number parameter, the gray scale value interval currently accumulated to is determined as the first limit;

[0021] Starting from the gray scale value interval of the smallest gray scale value in the order of monotonous decrease of the gray scale value, the number of pixels in the gray scale value interval is accumulated one by one, and in response to the accumulated number of pixels satisfying the pixel number parameter, the gray scale value interval currently accumulated to is determined as the second limit.

[0022] Further, converting the overlapping portion into a binary image based on the first local maximum value and the second local maximum value, comprising:

[0023] calculating a mean value of the first local maximum value and the second local maximum value;

[0024] setting a gray value of a pixel greater than or equal to the mean value to 0;

[0025] setting a gray value of a pixel lower than the mean value to 1.

[0026] Further, in the detected multiple polygons, a quadrangle meeting preset hyperparameters is determined, including:

[0027] In the multiple polygons, polygons with 4 vertices are screened to obtain multiple quadrangles;

[0028] According to preset area hyperparameters about an area of the quadrangle, hyperparameters about an angle between adjacent sides, hyperparameters about a length ratio between adjacent sides, hyperparameters about a ratio between a side length and a perimeter, and hyperparameters about a size of a center pixel of the quadrangle, one quadrangle is determined from the multiple quadrangles.

[0029] Based on the same inventive concept, the application further provides a joint calibration device of a vehicle camera, including a preprocessing module, a binary graph construction module, a quadrangle screening module, and a calibration module;

[0030] The preprocessing module is configured to determine at least one other gray graph having an overlapping part with each gray graph in the obtained multiple gray graphs, and determine the overlapping part.

[0031] The binary graph construction module is configured to, for each overlapping part, construct a gray histogram according to a gray value of each pixel, obtain a first local maximum value with a highest gray value and a second local maximum value with a lowest gray value by calculating an image gradient of the gray histogram, and convert the overlapping part into a binary graph based on the first local maximum value and the second local maximum value.

[0032] The quadrangle screening module is configured to perform polygon detection on the binary graph, and determine one quadrangle meeting preset hyperparameters from the detected multiple polygons.

[0033] The calibration module is configured to determine a homography matrix between the gray graph and the other gray graph by using multiple vertices of the quadrangle and multiple vertices of another quadrangle in a same overlapping part of the other gray graph, and use the homography matrix as a joint calibration result to calibrate the vehicle.

[0034] Based on the same inventive concept, the application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the joint calibration method of the vehicle camera according to any one of the above.

[0035] Based on the same inventive concept, the application further provides a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions for causing the computer to execute the joint calibration method of the vehicle camera.

[0036] Based on the same inventive concept, the application further provides a vehicle, comprising a joint calibration device of a vehicle camera and an electronic device, wherein the electronic device executes the joint calibration method of the vehicle camera according to any one of the above.

[0037] As can be seen from the above, the joint calibration method of the vehicle camera, the electronic device, the storage medium and the vehicle provided by the application are based on the obtained multiple gray scale maps, and the overlapping parts between the gray scale maps are comprehensively considered, and the joint calibration of the vehicle is performed for the overlapping parts, wherein the first local maximum value and the second local maximum value are determined by constructing a gray scale histogram for the overlapping parts, and the gray scale histogram is converted into a binary image based on this, and based on this, the two quadrilaterals obtained by relying on the two binary images are quadrilaterals about the overlapping part, that is, when the vertices of the two quadrilaterals are taken as feature points, all the feature points can be feature points of the overlapping part, so that the calibration of the overlapping part by using the homography matrix between the two feature points is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 The schematic diagram of the camera of the vehicle of the embodiment of the application is shown;

[0040] Figure 2 The flowchart of the joint calibration method of the vehicle camera of the embodiment of the application is shown;

[0041] Figure 3 The structure schematic diagram of the joint calibration device of the vehicle camera of the embodiment of the application is shown;

[0042] Figure 4 The structure schematic diagram of the electronic device of the embodiment of the application is shown. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0044] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application should be understood as their common meanings to those skilled in the art to which the present application pertains. The terms "first", "second" and similar terms used in the embodiments of the present application do not denote any order, quantity or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean physical or mechanical connection, but can include electrical connection, whether direct or indirect. The terms "upper", "lower", "left", "right" and the like only represent relative positional relationships, which can change accordingly when the absolute positions of the described objects change.

[0045] As described in the background section, the related joint calibration method of vehicle cameras is also difficult to meet the needs of actual production.

[0046] The applicant has found that the main problem of the related joint calibration method of vehicle cameras is that, in the production of the related panoramic image system of a vehicle, the images captured by multiple cameras of the vehicle need to be combined by relying on the homography matrix, i.e., the mapping relationship of projection, calibrated in the production of each camera.

[0047] In the related calibration method, all the image information of the images captured by each camera is often used for calibration, which makes the overlapping part captured between each camera unclear after combination.

[0048] Specifically, a large number of matching points are used to construct a least squares problem between images and calculate the optimal solution, and if the extracted feature points are concentrated in a non-overlapping part, only the non-overlapping part can be accurately projected, and the projection results of other parts are incorrect.

[0049] That is, because the extracted feature points, i.e., the corner points, are too concentrated, the information of the corner points at other positions in the image, i.e., the information of the corner points in the overlapping part, cannot be effectively utilized.

[0050] The applicant has also found in the research that, due to the related calibration method, when calibration is performed by using all image information, there is a certain error in the image information of the overlapping part of the images captured by each camera, which can cause a slight error in the mapping relationship of the projection between the overlapping parts.

[0051] Meanwhile, in the related joint calibration mode, the problem of algorithm efficiency needs to be considered. There are usually dozens of corner points on the chessboard that need to be extracted. In a complex algorithm, the extraction of a large number of corner points can consume a large amount of computing resources and reduce the calibration efficiency.

[0052] Based on this, one or more embodiments of the present application provide a joint calibration method for vehicle cameras.

[0053] In the embodiments of the present application, the vehicle as a specific example has multiple cameras. In the example shown in FIG. 1, the vehicle has four cameras, i.e., a front camera, a rear camera, a left camera, and a right camera, which capture front camera images, rear camera images, left camera images, and right camera images in four different directions, respectively. Figure 1

[0054] Among the captured images, the left camera image and the right camera image have overlapping parts with the front camera image, the front camera image and the rear camera image have overlapping parts with the left camera image, the front camera image and the rear camera image have overlapping parts with the left camera image, and the left camera image and the right camera image have overlapping parts with the rear camera image.

[0055] Further, when the four cameras of the vehicle are jointly calibrated, a calibration cloth can be laid on the ground around the vehicle to obtain better calibration effect. In the calibration cloth, the positions of the above-mentioned overlapping parts can be set as black squares, and the non-overlapping parts can be set as black and white chessboard patterns.

[0056] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0057] Reference Figure 2 The joint calibration method for vehicle cameras according to an embodiment of the present application includes the following steps.

[0058] Step S201, based on each of the obtained multiple gray scale images, at least one other gray scale image having an overlapping part with the gray scale image is determined, and the overlapping part is determined.

[0059] In the embodiments of the present application, the images captured by each camera of the vehicle can be subjected to gray scale processing. After gray scale processing, the gray scale images of each image can be obtained, and the overlapping parts between the gray scale images can be determined to perform joint calibration on the overlapping parts.

[0060] ​In the embodiment, in the process of joint calibration, at least one image captured by each camera can be first subjected to a de-distortion operation, which can be implemented using a library function such as OpenCV, which is a cross-platform computer vision and machine learning software library. Through the de-distortion operation, an image before distortion can be obtained.

[0061] Further, the de-distorted image can be subjected to grayscale processing to obtain a plurality of grayscale images.

[0062] Further, in the embodiment, taking the case of one image captured by each camera as an example, since each grayscale image is captured by a different camera, it can be considered that each grayscale image has an overlapping portion that overlaps with at least one other grayscale image.

[0063] Further, for the grayscale image, other grayscale images having overlapping portions with the grayscale image can be determined, where the other grayscale images can be one or more.

[0064] Further, for each overlapping portion in the grayscale image, other grayscale images corresponding to the overlapping portion can be determined.

[0065] In Figure 1 In the example shown, the four cameras can each capture an image. After the front image, the rear image, the left image, and the right image are subjected to de-distortion processing, the four images can be subjected to grayscale processing to obtain four grayscale images.

[0066] Further, taking the front grayscale image of the front image as an example, the other grayscale images having overlapping portions with the front grayscale image can be determined to be the left grayscale image and the right grayscale image.

[0067] As can be seen, for the front grayscale image, there are two overlapping portions, one of which corresponds to the left grayscale image, and the other of which corresponds to the right grayscale image.

[0068] In step S202, for each overlapping portion, a grayscale histogram is constructed according to the grayscale values of the pixels, the first local maximum value of the grayscale value and the second local maximum value of the grayscale value are obtained by calculating the image gradient of the grayscale histogram, and the overlapping portion is converted into a binary image based on the first local maximum value and the second local maximum value.

[0069] In the embodiment of the present application, based on the overlapping portions determined above, for each overlapping portion, the first local maximum value and the second local maximum value of the grayscale of the pixels can be determined by constructing a grayscale histogram, and then the overlapping portion can be converted into a binary image according to the first local maximum value and the second local maximum value.

[0070] In the embodiment, for each of the plurality of overlapping parts determined between the respective gray-scale images, a gray-scale histogram about the overlapping part can be constructed according to the gray-scale values of the respective pixels in the overlapping part. In the embodiment, the pixels can also be referred to as pixels simply.

[0071] Specifically, a plurality of gray-scale intervals can be set for the value range [0, 255] of the gray-scale values. In the embodiment, the gray-scale intervals can be set to 1, that is, the value range [0, 255] can be divided into 255 gray-scale intervals, and the gray-scale values between each gray-scale interval are spaced by 1.

[0072] In some other embodiments, the gray-scale intervals can also be set to 2 or 3 or other interval values.

[0073] Further, for the overlapping part, based on the determined gray-scale values of the respective pixels, the number of pixels contained in each of the above-mentioned gray-scale intervals can be counted.

[0074] Specifically, for example, for the gray-scale interval [24, 25), the number of pixels with gray-scale values in the interval can be counted.

[0075] Further, based on the above-mentioned determined gray-scale intervals and the number of pixels contained in each interval, the first coordinate and the second coordinate can be constructed for the gray-scale histogram of the gray-scale image.

[0076] Specifically, the size of the gray-scale value can be constructed as the first coordinate, that is, the value range [0, 255] of the gray-scale value can be taken as the coordinate in the first coordinate.

[0077] Based on this, the respective gray-scale intervals can be arranged in a monotonically increasing order of the gray-scale values, starting from 0 and ending at 255.

[0078] In some other embodiments, the respective gray-scale intervals can also be arranged in a monotonically decreasing order.

[0079] Further, the number of pixels can be taken as the second coordinate.

[0080] Based on this, for the above-mentioned arranged respective gray-scale intervals and based on the determined number of pixels contained in each gray-scale interval, a gray-scale histogram about the number of pixels and the gray-scale value can be obtained.

[0081] In the embodiment, the first coordinate can be the horizontal coordinate, and the second coordinate can be the vertical coordinate; in some other embodiments, the first coordinate can also be the vertical coordinate, and the second coordinate can also be the horizontal coordinate.

[0082] In the embodiment, based on the constructed gray scale histogram, the first local maximum value and the second local maximum value can be obtained by determining the image gradient of the histogram, so as to convert the gray scale image of the overlapping part into a binary image by using the first local maximum value and the second local maximum value.

[0083] Specifically, the first order differential processing can be performed on the constructed gray scale histogram, so as to obtain a plurality of local maximum values of the gray scale histogram.

[0084] Further, the pixel number parameter of the gray scale histogram can be preset, for example, when the pixel number parameter is set to 10%, it can be considered that the pixel number parameter is used to count 10% of the total pixel number.

[0085] Further, when the first local maximum value and the second local maximum value are determined, the range for obtaining the first local maximum value and the second local maximum value needs to be first determined, and based on this, the first limit and the second limit of the range need to be first determined.

[0086] Specifically, based on the constructed gray scale histogram, the pixel number in the gray scale value interval can be counted one by one in the direction of decreasing gray scale value in the order of monotonous decrease from the gray scale value interval with the largest gray scale value, and the counted pixel number in the gray scale value interval is accumulated.

[0087] Further, when the accumulated pixel number reaches the preset pixel number parameter, it is considered that the accumulation can be stopped, and the currently accumulated gray scale value interval is determined, and the gray scale value interval corresponding to the current accumulation stop is taken as the first limit.

[0088] Further, based on the constructed gray scale histogram, the pixel number in the gray scale value interval can be counted one by one in the direction of increasing gray scale value in the order of monotonous increase from the gray scale value interval with the smallest gray scale value, and the counted pixel number in the gray scale value interval is accumulated.

[0089] Further, when the accumulated pixel number reaches the preset pixel number parameter, it is considered that the accumulation can be stopped, and the currently accumulated gray scale value interval is determined, and the gray scale value interval corresponding to the current accumulation stop is taken as the second limit.

[0090] It can be seen that since the first limit is obtained by accumulating from the gray scale interval with the largest gray scale value, and the second limit is obtained by accumulating from the gray scale interval with the smallest gray scale value, the gray scale value corresponding to the first limit is greater than the gray scale value corresponding to the second limit.

[0091] Further, according to the first limit and the second limit determined above, a range formed between the first limit and the second limit can be taken as a range for obtaining the first local maximum value and the second local maximum value.

[0092] Based on this, in the gray scale histogram above, a local maximum value can be searched in a direction of decreasing gray scale from the first limit, and the local maximum value closest to the first limit is taken as the first local maximum value.

[0093] Further, in the gray scale histogram, a local maximum value can be searched in a direction of increasing gray scale from the second limit, and the local maximum value closest to the second limit is taken as the second local maximum value.

[0094] In this embodiment, based on the first local maximum value and the second local maximum value determined above, a segmentation threshold value required when performing binary graph conversion can be determined.

[0095] Specifically, since the first local maximum value is searched in a direction of decreasing gray scale from a position of maximum gray scale in the range, and the second local maximum value is searched in a direction of increasing gray scale from a position of minimum gray scale in the range, the first local maximum value can be taken as a bright part threshold value to represent a threshold value related to a part with relatively bright luminance in the gray scale graph, and the second local maximum value can be taken as a dark part threshold value to represent a threshold value related to a part with relatively dark luminance in the gray scale graph.

[0096] Further, an average value of the first local maximum value and the second local maximum value can be calculated, and the calculation result is taken as the segmentation threshold value.

[0097] Based on this, binary graph conversion can be performed on the overlapping part in the gray scale graph.

[0098] Specifically, a gray scale value of a pixel equal to or greater than the segmentation threshold value can be set to 0, and a gray scale value of a pixel less than the segmentation threshold value can be set to 1.

[0099] It can be seen that based on this, each pixel in the gray scale graph can be divided into two parts, and set to black or white according to 0 and 1.

[0100] Specifically, a pixel with a gray scale value set to 0 can be set to white, and a pixel with a gray scale value set to 1 can be set to black, based on which a binary graph about the gray scale graph can be obtained.

[0101] It can be seen that in the binary graph, each pixel is represented in white or black, and when joint calibration is performed, information about shapes and feature points in the image can be more easily obtained.

[0102] In step S203, polygon detection is performed on the binary image, and a quadrangle that meets preset hyperparameters is determined from the detected polygons.

[0103] In the embodiments of the present application, based on the obtained binary image, a quadrangle can be determined from the binary image by performing polygon detection and square detection.

[0104] Specifically, the polygon detection and polygon fitting can be performed on the binary image by directly calling the relevant detection and fitting interfaces of OpenCV, so as to smooth the burrs in the image.

[0105] Based on this, a plurality of polygons can be obtained from the binary image.

[0106] Further, a plurality of hyperparameters can be used to screen a quadrangle from the detected polygons.

[0107] Specifically, a vertex hyperparameter about the number of vertices can be set, for example, the vertex hyperparameter is set to 4, then the polygon with 4 vertices, i.e., the quadrangle, can be screened, based on which a plurality of quadrangles can be obtained.

[0108] Further, an area hyperparameter about the area of the quadrangle can be set to screen the quadrangles within a specified area.

[0109] Further, an angle hyperparameter about the angle between adjacent sides can be set to screen the quadrangles with a specified angle between adjacent sides.

[0110] Further, a length hyperparameter about the length ratio between adjacent sides can be set to screen the quadrangles with a specified length ratio between adjacent sides.

[0111] Further, a ratio hyperparameter about the ratio between the length of a side and the perimeter can be set to screen the quadrangles with a specified ratio between the length of a side and the perimeter.

[0112] Further, a pixel hyperparameter about the size of the center pixel of the quadrangle can be set to screen the quadrangles with a center pixel smaller than or larger than a specified pixel threshold.

[0113] Further, based on the screening of the above-mentioned hyperparameters, a quadrangle can be determined from the plurality of quadrangles.

[0114] In step S204, a homography matrix between the gray image and the other gray image is determined by using the plurality of vertices of the quadrangle and the plurality of vertices of another quadrangle in the same overlapping part of the other gray image, and the homography matrix is taken as a joint calibration result to calibrate the vehicle.

[0115] In the embodiments of the present application, based on a quadrangle determined for a binary image of an overlapping area, a mapping relationship, i.e., a homography matrix, between the quadrangle and a quadrangle in another binary image can be determined by using feature points passing through the quadrangle.

[0116] Specifically, based on the foregoing steps, a binary image of the overlapping part in each gray image can be determined, and a quadrangle in each binary image can be determined.

[0117] Based on this, the vertices of the quadrangle in each binary image can be extracted as feature points.

[0118] Further, since each binary image is obtained from a gray image captured by a different camera, the feature points extracted from each binary image have camera coordinates of the corresponding camera.

[0119] Further, for the overlapping part between two binary images, a homography matrix can be calculated by using the camera coordinates of the feature points in the two binary images.

[0120] The homography matrix specifically corresponds to a mapping relationship between the projection of the two binary images.

[0121] Further, the determined homography matrix can be taken as a joint calibration result, and can be calibrated in the two cameras corresponding to the two binary images to complete the calibration of the panoramic image system of the vehicle.

[0122] It can be seen that the joint calibration method of the vehicle cameras in the embodiments of the present application is based on the obtained plurality of gray images, and comprehensively considers the overlapping parts between the gray images to perform joint calibration of the vehicle for the overlapping parts. By constructing a gray histogram for the overlapping part to determine a first local maximum value and a second local maximum value, and by converting the gray histogram into a binary image based on this, two quadrangles obtained from two binary images are quadrangles about the overlapping part, that is, when the vertices of the two quadrangles are taken as feature points, all the feature points can be feature points of the overlapping part, so that the calibration of the overlapping part by using a homography matrix between two feature points is more accurate.

[0123] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server, etc. The method of the embodiments can also be applied to a distributed scenario, and be completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices can interact with each other to complete the method.

[0124] It should be noted that some embodiments of the present application are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0125] Based on the same inventive concept, the embodiments of the present application also provide a vehicle camera joint calibration device corresponding to any of the above-mentioned embodiment methods.

[0126] Reference Figure 3 , the vehicle camera joint calibration device comprises a preprocessing module 301, a binary graph construction module 302, a quadrilateral screening module 303, and a calibration module 304;

[0127] The preprocessing module 301 is configured to determine at least one other gray image having an overlapping part with each gray image in the obtained multiple gray images, and determine the overlapping part based on the gray image.

[0128] The binary graph construction module 302 is configured to, for each overlapping part, construct a gray histogram according to the gray values of each pixel, obtain a first local maximum value with the highest gray value and a second local maximum value with the lowest gray value by calculating the image gradient of the gray histogram, and convert the overlapping part into a binary graph based on the first local maximum value and the second local maximum value.

[0129] The quadrilateral screening module 303 is configured to perform polygon detection on the binary graph, and determine a quadrilateral that meets a preset hyperparameter from the detected multiple polygons.

[0130] The calibration module 304 is configured to determine a homography matrix between the gray image and the other gray image by using multiple vertices of the quadrilateral and multiple vertices of another quadrilateral in the same overlapping part of the other gray image, and use the homography matrix as a joint calibration result to calibrate the vehicle.

[0131] As an optional embodiment, the binary image construction module 302 is specifically configured to:

[0132] determine the number of pixels in each preset gray value interval based on the gray value of each pixel in the overlapping part;

[0133] construct the size of the gray value as the first coordinate of the gray histogram, and arrange the gray value intervals in any order of monotonous increase or monotonous decrease of the gray value;

[0134] construct the number of pixels in the gray value interval as the second coordinate of the gray histogram.

[0135] Further, first-order differential processing is performed on the gray histogram to obtain a plurality of local maximum values of the gray histogram,

[0136] in the plurality of gray value intervals of the gray histogram, a gray value interval as a first boundary and a gray value interval as a second boundary are determined by using a preset pixel number parameter, wherein the gray value of the first boundary is higher than the gray value of the second boundary;

[0137] in the gray histogram, based on the first boundary, the local maximum value closest to the first boundary in the direction of decreasing gray value is taken as a first local maximum value, and based on the second boundary, the local maximum value closest to the second boundary in the direction of increasing gray value is taken as a second local maximum value.

[0138] wherein the gray value interval as the first boundary and the gray value interval as the second boundary are determined by using a preset pixel number parameter, comprising:

[0139] starting from the gray value interval with the largest gray value in the order of monotonous decrease of the gray value, the number of pixels in the gray value interval is accumulated one by one, and in response to the accumulated number of pixels satisfying the pixel number parameter, the gray value interval currently accumulated to is determined as the first boundary;

[0140] starting from the gray value interval with the smallest gray value in the order of monotonous decrease of the gray value, the number of pixels in the gray value interval is accumulated one by one, and in response to the accumulated number of pixels satisfying the pixel number parameter, the gray value interval currently accumulated to is determined as the second boundary.

[0141] Further, the mean value of the first local maximum value and the second local maximum value is calculated;

[0142] the gray value of the pixel greater than or equal to the mean value is set to 0;

[0143] the gray value of the pixel lower than the mean value is set to 1.

[0144] As an optional embodiment, the quadrilateral screening module 303 is specifically configured to:

[0145] In the plurality of polygons, polygons with 4 vertices are screened to obtain a plurality of quadrilaterals.

[0146] According to a preset area hyperparameter about the area of the quadrilateral, a hyperparameter about the included angle between adjacent sides, a hyperparameter about the length ratio between adjacent sides, a hyperparameter about the ratio between the length of the side and the perimeter, and a hyperparameter about the size of the center pixel of the quadrilateral, a quadrilateral is determined from the plurality of quadrilaterals.

[0147] For the convenience of description, the above apparatus is described in various modules according to functions. Of course, in the implementation of the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0148] The apparatus of the above embodiments is used to implement the joint calibration method of the vehicle camera in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.

[0149] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the embodiments of the present application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the joint calibration method of the vehicle camera as described in any of the above embodiments.

[0150] Figure 4 A more specific hardware structure schematic diagram of an electronic device provided by the present embodiment is shown, which can include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040 and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030 and the communication interface 1040 are connected to each other through the bus 1050 for communication connection within the device.

[0151] The processor 1010 can be implemented in the form of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0152] The memory 1020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0153] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0154] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or through a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0155] The bus 1050 includes a channel for transmitting information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0156] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include the components necessary to implement the embodiments of the present application, and does not have to include all the components shown in the figure.

[0157] The device of the above embodiments is used to implement the joint calibration method of the vehicle camera in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.

[0158] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a vehicle, which includes a joint calibration device of a vehicle camera and an electronic device, and the electronic device executes the joint calibration method of the vehicle camera according to any of the above embodiments.

[0159] Based on the same inventive concept, the application also provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the joint calibration method of a vehicle camera according to any of the above embodiments.

[0160] The computer readable medium of the embodiments can include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0161] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the joint calibration method of a vehicle camera according to any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not described here.

[0162] Those skilled in the art should understand that the above discussion of any of the embodiments is only exemplary and is not intended to imply that the scope (including claims) of the application is limited to these examples; the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the embodiments of the application as described above. In order to be brief, they are not provided in detail.

[0163] Additionally, to simplify the description and discussion, and so as not to obscure the embodiments of the application being presented, the well-known power / ground connections of integrated circuits (ICs) and other components can or can not be shown in the provided figures. Furthermore, devices can be shown in block diagram form in order to avoid obscuring the embodiments of the application being presented, and also in view of the fact that details of implementation of the devices shown can vary depending on the platform within which the embodiments of the application are to be implemented (i.e., these details should be understood as being well within the purview of one of ordinary skill in the art). Where specific details of particular implementations are set forth in order to describe example embodiments of the application, it can be understood, however, that the embodiments of the application can be practiced without these specific details. In other instances, well-known methods, procedures and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the application.

[0164] While the application has been described in connection with specific embodiments thereof, it will be understood that many modifications, substitutions and changes can be made by those skilled in the art to the specific embodiments without departing from the spirit or scope of the application. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0165] It is intended to cover all alternatives, modifications and variations of this application falling within the scope of the appended claims. Accordingly, all such changes are intended to be included within the scope of the application as claimed.

Claims

1. A joint calibration method of vehicle cameras, characterized by, The method comprises: based on each of the plurality of acquired grayscale images, determining at least one other grayscale image having an overlapping part with the grayscale image, and determining the overlapping part; for each overlapping part, constructing a grayscale histogram according to the grayscale values of the respective pixels, obtaining a first local maximum grayscale value and a second local maximum grayscale value by calculating the image gradient of the grayscale histogram, and converting the overlapping part into a binary image based on the first local maximum grayscale value and the second local maximum grayscale value; wherein the first local maximum grayscale value is obtained by searching in the direction of decreasing grayscale from the position of the maximum grayscale in the range, and the second local maximum grayscale value is obtained by searching in the direction of increasing grayscale from the position of the minimum grayscale in the range; performing polygon detection on the binary image, and determining a quadrangle that meets preset hyperparameters from the plurality of detected polygons; determining a homography matrix between the grayscale image and the other grayscale image using the plurality of vertices of the quadrangle and the plurality of vertices of another quadrangle in the same overlapping part of the other grayscale image, and using the homography matrix as a joint calibration result to calibrate the vehicle; the step of determining a quadrangle that meets preset hyperparameters from the plurality of detected polygons comprises: selecting polygons with four vertices from the plurality of polygons to obtain a plurality of quadrangles; determining a quadrangle from the plurality of quadrangles according to preset area hyperparameters about the area of the quadrangle, angle hyperparameters about the included angle between adjacent sides, length hyperparameters about the length ratio between adjacent sides, proportion hyperparameters about the proportion between the side length and the perimeter, and pixel hyperparameters about the size of the center pixel of the quadrangle.

2. The method of claim 1, wherein, the step of constructing a grayscale histogram according to the grayscale values of the respective pixels comprises: determining the number of pixels in each preset grayscale value interval based on the grayscale values of the respective pixels in the overlapping part; constructing the size of the grayscale value as the first coordinate of the grayscale histogram, and arranging the respective grayscale value intervals in any order of monotonically increasing or monotonically decreasing grayscale value; constructing the number of pixels in the respective grayscale value intervals as the second coordinate of the grayscale histogram.

3. The method of claim 2, wherein, the step of obtaining a first local maximum grayscale value and a second local maximum grayscale value by calculating the image gradient of the grayscale histogram comprises: performing first-order differential processing on the grayscale histogram to obtain a plurality of local maximum values about the grayscale histogram, determining a grayscale value interval as a first boundary and a grayscale value interval as a second boundary using a preset pixel quantity parameter from the plurality of grayscale value intervals of the grayscale histogram, wherein the grayscale value of the first boundary is higher than the grayscale value of the second boundary; in the grayscale histogram, based on the first boundary, the local maximum value closest to the first boundary in the direction of decreasing grayscale value is taken as the first local maximum value, and based on the second boundary, the local maximum value closest to the second boundary in the direction of increasing grayscale value is taken as the second local maximum value.

4. The method of claim 3, wherein, The determining, by using the preset pixel number parameter, of a gray value interval as a first limit and a gray value interval as a second limit comprises: starting from the gray value interval with the largest gray value in a monotonically decreasing order of gray value, accumulating the number of pixels in the gray value interval one by one, and determining the gray value interval to which the current accumulation is made as the first limit in response to the accumulated number of pixels satisfying the pixel number parameter; starting from the gray value interval with the smallest gray value in a monotonically decreasing order of gray value, accumulating the number of pixels in the gray value interval one by one, and determining the gray value interval to which the current accumulation is made as the second limit in response to the accumulated number of pixels satisfying the pixel number parameter.

5. The method of claim 1, wherein, The converting, based on the first local maximum value and the second local maximum value, of the overlapping part into a binary graph comprises: calculating the mean value of the first local maximum value and the second local maximum value; setting the gray value of a pixel greater than or equal to the mean value to 0; setting the gray value of a pixel lower than the mean value to 1.

6. A joint calibration device of vehicle cameras, characterized by It comprises: a preprocessing module, a binary graph construction module, a quadrilateral screening module, and a calibration module; The preprocessing module is configured to determine, based on each of the obtained plurality of gray scale graphs, at least one other gray scale graph having an overlapping part with the gray scale graph, and determine the overlapping part. The binary graph construction module is configured to, for each overlapping part, construct a gray scale histogram according to the gray scale value of each pixel, obtain a first local maximum value with the highest gray scale value and a second local maximum value with the lowest gray scale value by calculating the image gradient of the gray scale histogram, and convert the overlapping part into a binary graph based on the first local maximum value and the second local maximum value; wherein the first local maximum value is obtained by searching from the position with the largest gray scale in the range to the direction in which the gray scale decreases, and the second local maximum value is obtained by searching from the position with the smallest gray scale in the range to the direction in which the gray scale increases. The quadrilateral screening module is configured to perform polygon detection on the binary graph, and determine a quadrilateral that meets preset hyperparameters from the plurality of polygons detected. The calibration module is configured to determine a homography matrix between the gray scale graph and the other gray scale graph by using the plurality of vertices of the quadrilateral and the plurality of vertices of another quadrilateral in the same overlapping part of the other gray scale graph, and use the homography matrix as a joint calibration result to calibrate the vehicle. The quadrilateral screening module is specifically configured to: screen polygons with four vertices from the plurality of polygons to obtain a plurality of quadrilaterals; determine a quadrilateral from the plurality of quadrilaterals according to preset area hyperparameters about the area of the quadrilateral, angle hyperparameters about the included angle between adjacent sides, length hyperparameters about the length ratio between adjacent sides, proportion hyperparameters about the proportion between the length of a side and the perimeter, and pixel hyperparameters about the size of the center pixel of the quadrilateral.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein, The processor executes the computer program to implement the method of any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions for causing a computer to perform the method according to any one of claims 1 to 5.

9. A vehicle characterized by comprising: A joint calibration device comprising a vehicle camera as claimed in claim 6 or an electronic device as claimed in claim 7.

Images