External parameter dynamic self-correction method, system and storage medium suitable for binocular camera
By calculating the linear fit of the parallax map and height statistics map, the external parameters of the binocular camera are dynamically updated, solving the problem of external parameters failure caused by vehicle sway and improving the reliability and accuracy of autonomous driving.
Patent Information
- Application Number
- CN202311258114.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-09-27
AI Technical Summary
During the vehicle driving, the external parameters of the binocular camera fail due to posture changes caused by vehicle shaking, resulting in misdetection of obstacles and false alarms of ADAS system, affecting the reliability of autonomous driving.
By calculating the disparity map of the currently acquired image, convert it into a point cloud and generating a height raster map, counting data with the same height value, generating a height statistical map, and using linear fit to determine the external parameter angle compensation value, update the initial external parameter, Realize dynamic self-correction of binocular cameras.
Ensure that the posture between the binocular camera and the earth matches the external parameters, avoid misdetecting obstacles and false alarms of ADAS systems, and improve the reliability and accuracy of autonomous driving.
Smart Images

Figure CN118314214B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of binocular cameras, and in particular to an external parameter dynamic self-correction method applicable to a binocular camera, an external parameter dynamic self-correction system applicable to a binocular camera, and a computer-readable storage medium. Background Art
[0002] In autonomous driving technology, binocular cameras are important visual sensors for vehicles, which can provide left and right views for vehicles, so that vehicles can realize autonomous driving functions such as obstacle detection, lane line recognition, and vehicle distance detection through image recognition, binocular parallax and other technologies. Therefore, it is necessary to calibrate the external parameters of the binocular cameras installed on the vehicle to ensure that the vehicle can obtain accurate parallax (distance) information and improve the accuracy and reliability of the vehicle's autonomous driving function.
[0003] In the prior art, the external parameter calibration of the binocular camera is usually carried out on flat ground (horizontal ground) in order to establish the relative posture correspondence between the binocular camera and the ground and calibrate the external parameters of the binocular camera. However, when the vehicle passes through some special roads, such as speed bumps, slopes, potholes, etc., the relative posture between the binocular camera and the ground will change, that is, the previously calibrated external parameters are no longer applicable to the current driving state, resulting in unexpected situations such as false detection and missed detection of the vehicle's automatic driving function.
[0004] Especially for large trucks and large trucks that often travel on unstructured roads, due to the unevenness of the roads, these vehicles often shake while driving. When the vehicle passes over a bump or drives out of a pothole, there is a possibility that the normal road surface ahead will be identified as an obstacle, triggering a false alarm from the vehicle's ADAS system, and even causing the vehicle to stop suddenly, resulting in a safety accident. Summary of the invention
[0005] A technical problem to be solved by this application is: how to correct the external parameters of the binocular camera when the vehicle is in motion.
[0006] To solve the above technical problems, the first aspect of an embodiment of the present application provides an extrinsic parameter dynamic self-correction method suitable for a binocular camera, the method comprising: based on the initial extrinsic parameters, calculating the disparity map of the current acquired image, and converting the disparity map into a point cloud, thereby generating a height grid map; based on a preset direction, counting height data with the same height value in the height grid map, and generating a height statistical map based on the data statistical results; based on the height statistical map, determining the extrinsic parameter angle compensation value by linear fitting; and updating the initial extrinsic parameter based on the extrinsic parameter angle compensation value.
[0007] In some embodiments, the preset direction includes the longitudinal direction of the height grid map, the height statistical map includes a first height statistical map, based on the preset direction, the height data with the same height value in the height grid map are counted, and based on the data statistical results, the height statistical map is generated, specifically including: generating a first blank form, wherein the horizontal header of the first blank form is determined by the height value of the height data in the height grid map, and the vertical header of the first blank form is determined by the number of rows of the height grid map; based on the height value of the height data in the height grid map, the first statistical number of each height value in the height grid map is counted row by row, and the first statistical number of each height value is filled into the first blank form, recorded as the first statistical form; based on the first statistical form filled with data, a first height statistical map is generated.
[0008] In some embodiments, the extrinsic parameter angle compensation value includes a first extrinsic parameter angle compensation value, and the extrinsic parameter angle compensation value is determined based on a height statistical graph by a linear fitting method, specifically including: generating a fitting straight line based on each coordinate point in the first height statistical graph by a linear fitting method; calculating a first slope of the fitting straight line; and calculating a first extrinsic parameter angle compensation value based on the first slope, wherein the first extrinsic parameter angle compensation value is used to compensate for the pitch angle of the binocular camera.
[0009] In some embodiments, the preset direction includes the horizontal direction of the height grid map, the height statistical map includes a second height statistical map, based on the preset direction, the height data with the same height value in the height grid map are counted, and based on the data statistical results, the height statistical map is generated, specifically including: generating a second blank form, wherein the vertical header of the second blank form is determined by the height value of the height data in the height grid map, and the horizontal header of the second blank form is determined by the number of columns of the height grid map; based on the height value of the height data in the height grid map, the second statistical number of each height value in the height grid map is counted column by column, and the second statistical number of each height value is filled into the second blank form, recorded as a second statistical form; based on the second statistical form filled with data, a second height statistical map is generated.
[0010] In some embodiments, the extrinsic parameter angle compensation value includes a second extrinsic parameter angle compensation value, and the extrinsic parameter angle compensation value is determined based on the height statistical graph by linear fitting, specifically including: generating a fitting straight line based on each coordinate point in the second height statistical graph by linear fitting; calculating the second slope of the fitting straight line; based on the second slope, calculating the second extrinsic parameter angle compensation value, wherein the second extrinsic parameter angle compensation value is used to compensate for the roll angle of the binocular camera.
[0011] In some embodiments, the method further includes: determining in turn whether each height data in the height grid map is greater than a preset height threshold, and if so, updating the height data to a preset height, wherein the value of the preset height is set manually; if not, retaining the height data.
[0012] The second aspect of the embodiment of the present application provides an external parameter dynamic self-correction system suitable for a binocular camera, the system comprising: a first processing module, the first processing module is configured to calculate a disparity map of a current acquired image based on initial external parameters, and convert the disparity map into a point cloud, and then generate a height grid map; a second processing module, the second processing module is configured to count height data with the same height value in the height grid map based on a preset direction, and generate a height statistical map based on the data statistical results; a third processing module, the third processing module is configured to determine an external parameter angle compensation value based on the height statistical map by linear fitting; and a fourth processing module, the fourth processing module is configured to update the initial external parameter based on the external parameter angle compensation value.
[0013] In some embodiments, the preset direction includes the longitudinal direction of the height grid map, the height statistical map includes a first height statistical map, and the second processing module is configured to: generate a first blank form, wherein the horizontal header of the first blank form is determined by the height value of the height data in the height grid map, and the vertical header of the first blank form is determined by the number of rows of the height grid map; based on the height value of the height data in the height grid map, count the first statistical number of each height value in the height grid map row by row, and fill the first statistical number of each height value into the first blank form, recorded as a first statistical form; based on the first statistical form filled with data, generate a first height statistical map.
[0014] In some embodiments, the extrinsic parameter angle compensation value includes a first extrinsic parameter angle compensation value, and the third processing module is configured to: generate a fitting straight line based on each coordinate point in the first height statistical graph by linear fitting; calculate the first slope of the fitting straight line; and calculate the first extrinsic parameter angle compensation value based on the first slope, wherein the first extrinsic parameter angle compensation value is used to compensate for the pitch angle of the binocular camera.
[0015] A third aspect of an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any method in the technical solution of the first aspect is implemented.
[0016] Through the above technical scheme, the present application provides a method, system and storage medium for dynamic self-correction of external parameters for a binocular camera. Based on the height grid map obtained by converting the disparity map, the height data with the same height value are counted in a preset direction to obtain a height statistical map, and then a linear fitting is performed on the height statistical map. The slope of the fitting straight line is used to determine the external parameter angle compensation value, so as to correct the external parameters of the binocular camera when the vehicle is driving, ensure that the posture between the binocular camera and the ground remains matched with the external parameters, so as to avoid false detection of obstacles caused by shaking of the vehicle during driving and false alarm of the ADAS system, thereby improving the reliability of autonomous driving.
[0017] The technical solution in the present application, in order to ensure the accuracy of the calculation of the extrinsic parameter angle compensation value, performs height statistics on the longitudinal and transverse directions of the height grid map, and performs linear fitting to determine the corresponding extrinsic parameter angle compensation values, which can achieve compensation for the pitch angle and roll angle respectively, further improving the reliability and accuracy of the binocular camera extrinsic correction. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flow chart of an external parameter dynamic self-correction method applicable to a binocular camera disclosed in an embodiment of the present application;
[0020] Figure 2 is a disparity map of a flat ground disclosed in an embodiment of the present application;
[0021] Figure 3 is a height statistical diagram of a flat ground along the vertical direction disclosed in an embodiment of the present application;
[0022] Figure 4 is a height statistical diagram of a flat ground along the horizontal direction disclosed in the embodiment of the present application;
[0023] Figure 5 is a disparity map of downhill ground disclosed in the embodiment of the present application;
[0024] Figure 6 is a height statistical diagram of the downhill ground along the vertical direction disclosed in the embodiment of the present application;
[0025] Figure 7 is a schematic diagram of generating a first height statistical graph disclosed in an embodiment of the present application;
[0026] Figure 8 is a schematic diagram of a fitting straight line disclosed in an embodiment of the present application;
[0027] Fig. 9 is a schematic diagram of generating a second height statistical graph disclosed in an embodiment of the present application;
[0028] Fig.10 is a schematic block diagram of an external parameter dynamic self-correction system applicable to a binocular camera disclosed in an embodiment of the present application;
[0029] Fig.11is a schematic diagram of an image captured before self-correction disclosed in an embodiment of the present application;
[0030] Fig.12 It is a schematic diagram of a height statistical diagram of a self-correcting front horizontal direction disclosed in an embodiment of the present application;
[0031] Fig.13 is a schematic diagram of a self-corrected captured image disclosed in an embodiment of the present application;
[0032] Fig.14 It is a schematic diagram of a height statistics diagram along the horizontal direction after self-correction disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0033] The following is a further detailed description of the implementation methods of the present application in conjunction with the accompanying drawings and examples. The detailed descriptions of the following examples and the accompanying drawings are used to exemplarily illustrate the principles of the present application, but cannot be used to limit the scope of the present application. The present application can be implemented in many different forms and is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.
[0034] The present application provides these embodiments to make the present application thorough and complete, and to fully express the scope of the present application to those skilled in the art. It should be noted that unless otherwise specifically stated, the relative arrangement / sequence of the parts and steps, the composition of the materials, the numerical expressions and the numerical values set forth in these embodiments should be interpreted as being merely exemplary, rather than as limiting.
[0035] It should be noted that, in the description of this application, unless otherwise specified, the meaning of "multiple" is greater than or equal to two; the terms "upper", "lower", "left", "right", "inner", "outer", etc., indicating the orientation or positional relationship, are only for the convenience of describing the simplified description of this application, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.
[0036] In addition, the words "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different parts. "Vertical" does not mean vertical in the strict sense, but is within the tolerance range. "Parallel" does not mean parallel in the strict sense, but is within the tolerance range. "Include" or "comprising" and similar words mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of including other elements.
[0037] It should also be noted that in the description of this application, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances. When a specific device is described as being located between a first device and a second device, there may or may not be an intermediate device between the specific device and the first device or the second device.
[0038] All terms used in this application have the same meaning as those understood by those of ordinary skill in the art to which this application belongs, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries, such as general dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined herein.
[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0040] Embodiment 1:
[0041] like Figure 1 As shown, an embodiment of the present application provides an external parameter dynamic self-correction method suitable for a binocular camera, the method comprising: step 1: based on the initial external parameter, calculating the disparity map of the current acquired image, and converting the disparity map into a point cloud, and then generating a height grid map; step 2: based on a preset direction, counting the height data with the same height value in the height grid map, and generating a height statistical map based on the data statistical results; step 3: based on the height statistical map, using a linear fitting method to determine the external parameter angle compensation value; step 4: based on the external parameter angle compensation value, updating the initial external parameter.
[0042] Specifically, for a vehicle equipped with a binocular camera, those skilled in the art can understand that the installed binocular camera is in a calibrated state, and its related parameters (such as the initial external parameters of the binocular camera) are accurate values. It can normally acquire images within the field of view and perform related image processing to realize the automatic driving function.
[0043] This embodiment only deals with the situation where the relative posture between the binocular camera and the ground changes due to the shaking of the vehicle during driving. Therefore, the binocular camera and the vehicle on which the binocular camera is installed are set as a rigid body.
[0044] This embodiment establishes the following world coordinate system, which complies with the right-hand rule:
[0045] 1) The origin W of the world coordinate system is the projection point of the origin of the binocular camera coordinate system on the ground;
[0046] 2) The positive direction of the WY axis is vertically downward;
[0047] 3) The plane where the WX axis and the WZ axis are located is a horizontal plane, wherein the WX axis faces right and the WZ axis faces forward.
[0048] During the driving process, the binocular camera can obtain the image in front of the vehicle in the field of view in real time. Based on the calibrated initial external parameters and the left and right images (currently acquired images), the corresponding disparity map within the preset range can be obtained. Then, through the point cloud conversion method, the disparity map can be converted into a point cloud map, which consists of multiple three-dimensional point cloud coordinate points.
[0049] It should be noted that the preset range for obtaining the disparity map in this embodiment can be set and adjusted according to actual needs.
[0050] Take any point cloud coordinate point (x a ,y a ,z a ), it is converted into a height grid map by projection. The conversion process is as follows:
[0051] 1) Its x-axis coordinate is used as the horizontal coordinate of the corresponding coordinate point in the height grid map;
[0052] 2) taking its z-axis coordinate as the longitudinal coordinate of the corresponding coordinate point in the height grid map;
[0053] 3) The y-axis coordinate is used as the value of the corresponding coordinate point in the height grid map, that is, the height data.
[0054] Therefore, the above point cloud coordinate point (x a ,y a ,z a ) is converted into coordinate points in the height grid map as follows:
[0055]
[0056] It should be noted that in the process of converting to the height grid map, the actual size represented by the pixel point can be set according to actual needs, such as one pixel represents 5*5mm, 10*10mm, etc., and all point clouds are traversed and mapped to the corresponding pixel coordinates to generate a height grid map.
[0057] Therefore, the height grid map in this embodiment includes position coordinates and height data corresponding to the position coordinates. The height data can be the lowest value of the height corresponding to the point cloud within the real space range, or a calculated value obtained through calculation.
[0058] Afterwards, the number of height data with the same height value in the height grid map is counted along the horizontal and vertical directions of the height grid map to obtain a height statistics map; and then the relative posture change between the front ground and the binocular camera can be obtained by linear fitting.
[0059] like Figures 2 to 4 As shown in the figure, the disparity map of the flat ground obtained by the binocular camera can reflect the ground and the objects on the ground. After the height statistics map is generated, the statistical results can be fitted into a straight line in both the horizontal and vertical directions. Therefore, based on the height statistics map, it can be determined that the ground in the current state is a flat ground.
[0060] Another example Figure 5 and Figure 6 As shown, taking the downhill bridge deck as an example, the disparity map can only reflect the collected disparity data of the bridge deck, and cannot reflect the inclination of the bridge deck; after generating the height statistics map, the statistical results show a tilt trend, so a slant line is obtained by fitting.
[0061] Through analysis, it can be seen that the slope of the oblique line corresponds to the relative posture change between the front ground and the binocular camera. The pitch angle Pitch and roll angle Roll of the binocular camera can be compensated to ensure the accuracy of the binocular camera's external parameters and achieve self-correction.
[0062] It should be noted that when performing external parameter self-correction, the pitch angle Pitch can be corrected first, and then the roll angle Roll can be corrected.
[0063] Through further research, it was found that when statistics are performed based on the longitudinal direction of the height grid map, the resulting height statistical map can be used to compensate for the pitch angle Pitch of the binocular camera; when statistics are performed based on the transverse direction of the height grid map, the resulting height statistical map can be used to compensate for the roll angle Roll of the binocular camera.
[0064] In some embodiments, the preset direction includes the longitudinal direction of the height grid map, and the height statistical map includes a first height statistical map. In step 2, based on the preset direction, height data with the same height value in the height grid map are counted, and based on the data statistical results, a height statistical map is generated, specifically including: generating a first blank form, wherein the horizontal header of the first blank form is determined by the height value of the height data in the height grid map, and the vertical header of the first blank form is determined by the number of rows of the height grid map; based on the height value of the height data in the height grid map, the first statistical number of each height value in the height grid map is counted row by row, and the first statistical number of each height value is filled into the first blank form, recorded as a first statistical form; based on the first statistical form filled with data, a first height statistical map is generated.
[0065] Specifically, Figure 7 As shown, a blank form is first generated, in which the horizontal header represents the height value and the vertical header represents the number of rows. The value of the horizontal header can be manually defined according to actual needs.
[0066] Then, the height data in the height grid map (as shown in the table on the left) is counted. The data in row 0 is used as an example. The height data in row 0 are: 0, -2, 0, 1, -1, and the corresponding height values are -2, -1, 0, 1, and the corresponding statistical numbers are 1, 1, 2, 1.
[0067] Therefore, the number of statistics in row 0 is filled into row 0 of the first blank form, and then the height data of row 1 of the height grid map is counted, and the process is repeated row by row until the data statistics of all rows are completed. The first blank form filled with the corresponding data is used as the first statistical form, as shown in the table on the right. Finally, based on the first statistical form, the first height statistical graph is generated by chart conversion.
[0068] Through the above process in this embodiment, the height grid map obtained by converting the disparity map is statistically analyzed along the longitudinal direction to generate a first height statistical map, which helps to intuitively reflect the inclination of the road ahead, and then compensate for the pitch angle Pitch of the binocular camera to ensure the accuracy of the binocular camera external parameters.
[0069] In some embodiments, in step 3, the extrinsic parameter angle compensation value includes a first extrinsic parameter angle compensation value, and the extrinsic parameter angle compensation value is determined based on the height statistical graph by linear fitting, specifically including: generating a fitting straight line by linear fitting based on each coordinate point in the first height statistical graph; calculating a first slope of the fitting straight line; and calculating a first extrinsic parameter angle compensation value based on the first slope, wherein the first extrinsic parameter angle compensation value is used to compensate for the pitch angle of the binocular camera.
[0070] Specifically, the linear fitting method adopted in this embodiment may be one of the least squares method, the RANSAC method, the Hough method, or other linear fitting methods.
[0071] When selecting the coordinate points used for fitting, you can select the coordinates corresponding to the maximum statistical number, such as the maximum statistical number in the 0th row is 2, and its corresponding coordinate is (0,0); you can also calculate the weighted coordinates corresponding to each coordinate point in the current row in a weighted manner, where the value of the weighted weight can be set based on the value of the height data.
[0072] After getting the fitting straight line, Figure 8As shown, with the upper left corner as the origin, the right as the x-axis, and the bottom as the y-axis, the equation corresponding to the current fitting line can be calculated, such as: y = k1x + b. At this time, the calculation formula for the first external parameter angle compensation value is:
[0073] fDiffPitch=arctan(1 / k1)
[0074] Wherein, fDiffPitch is the first external parameter angle compensation value, and k1 is the first slope.
[0075] Those skilled in the art can understand that when k1=0, that is, the fitting line is vertical, at this time, the height above the ground is 0, and the vehicle is traveling on a flat ground, and no external parameter correction is required.
[0076] In some embodiments, the preset direction includes the horizontal direction of the height grid map, and the height statistical map includes a second height statistical map. In step 2, based on the preset direction, height data with the same height value in the height grid map are counted, and based on the data statistical results, a height statistical map is generated, specifically including: generating a second blank form, wherein the vertical header of the second blank form is determined by the height value of the height data in the height grid map, and the horizontal header of the second blank form is determined by the number of columns of the height grid map; based on the height value of the height data in the height grid map, the second statistical number of each height value in the height grid map is counted column by column, and the second statistical number of each height value is filled into the second blank form, recorded as a second statistical form; based on the second statistical form filled with data, a second height statistical map is generated.
[0077] like Fig. 9 As shown, in order to compensate for the roll angle Roll of the binocular camera, on the basis of the above embodiment, the statistical direction of the height grid map is adjusted, statistics are performed along the horizontal direction, and the second statistical number of height values of each column in the height grid map is counted to generate a second height statistical map.
[0078] In some embodiments, in step 3, the extrinsic parameter angle compensation value includes a second extrinsic parameter angle compensation value, and the extrinsic parameter angle compensation value is determined based on the height statistical graph by linear fitting, specifically including: generating a fitting straight line by linear fitting based on each coordinate point in the second height statistical graph; calculating a second slope of the fitting straight line; and calculating a second extrinsic parameter angle compensation value based on the second slope, wherein the second extrinsic parameter angle compensation value is used to compensate for the roll angle of the binocular camera.
[0079] Specifically, the calculation formula of the second external parameter angle compensation value is:
[0080] fDiffRoll = arctan(1 / k2)
[0081] Where, fDiffRoll is the second external parameter angle compensation value. K2 is the second slope.
[0082] In the initial external parameter rotation matrix R of the binocular camera, fDiffPitch is compensated to the original Pitch, and fDiffRoll is compensated to the original Roll, that is, the real rotation matrix R at this moment is obtained. The corresponding calculation formula is:
[0083]
[0084] In some embodiments, in order to ensure the validity of the data in the height grid map and filter out noise data, after step 1, it also includes: judging in turn whether each height data in the height grid map is greater than a preset height threshold, if so, updating the height data to a preset height, wherein the value of the preset height is set manually; if not, retaining the height data.
[0085] For example, set the preset height threshold to [-10,10], such as Figure 7 or Fig. 9 As shown, the height data "120" in the height grid is greater than the preset height threshold, so it can be determined that the height data "120" is noise data and needs to be filtered out. At this time, the height data "120" can be directly updated to the preset height 0, or updated to other preset heights.
[0086] It should be noted that, for height data greater than a preset height threshold, the statistical process in step 2 may be omitted, and the specific process will not be described in detail.
[0087] Embodiment 2:
[0088] like Fig.10 As shown, the embodiment of the present application also provides an external parameter dynamic self-correction system 100 suitable for a binocular camera, and the system 100 includes: a first processing module 10, which is configured to calculate the disparity map of the current acquired image based on the initial external parameter, and convert the disparity map into a point cloud, and then generate a height grid map; a second processing module 20, which is configured to count the height data with the same height value in the height grid map based on a preset direction, and generate a height statistical map based on the data statistical results; a third processing module 30, which is configured to determine the external parameter angle compensation value based on the height statistical map by linear fitting; a fourth processing module 40, which is configured to update the initial external parameter based on the external parameter angle compensation value.
[0089] Specifically, during the driving process of the vehicle, the binocular camera can obtain the image in front of the vehicle within the field of view in real time, and based on the calibrated initial external parameters and the left and right images (currently acquired images), the corresponding disparity map can be obtained. Then, through the point cloud conversion method, the disparity map can be converted into a point cloud map, which is composed of multiple three-dimensional point cloud coordinate points.
[0090] Take any point cloud coordinate point (x a ,y a ,z a ), it is converted into a height grid map by projection. The conversion process is as follows:
[0091] 1) Its x-axis coordinate is used as the horizontal coordinate of the corresponding coordinate point in the height grid map;
[0092] 2) taking its z-axis coordinate as the longitudinal coordinate of the corresponding coordinate point in the height grid map;
[0093] 3) The y-axis coordinate is used as the value of the corresponding coordinate point in the height grid map, that is, the height data.
[0094] Therefore, the above point cloud coordinate point (x a ,y a ,z a ) is converted into coordinate points in the height grid map as follows:
[0095]
[0096] It should be noted that in the process of converting to the height grid map, the actual size represented by the pixel point can be set according to actual needs, such as one pixel represents 5*5mm, 10*10mm, etc., and all point clouds are traversed and mapped to the corresponding pixel coordinates to generate a height grid map.
[0097] Therefore, the height grid map in this embodiment includes position coordinates and height data corresponding to the position coordinates. The height data can be the lowest value of the height corresponding to the point cloud within the real space range, or a calculated value obtained through calculation.
[0098] Afterwards, the number of height data with the same height value in the height grid map is counted along the horizontal and vertical directions of the height grid map to obtain a height statistics map; then, the relative posture change between the front ground and the binocular camera can be obtained by linear fitting. The current fitting result corresponds to a diagonal line.
[0099] Through analysis, it can be seen that the slope of the oblique line corresponds to the relative posture change between the front ground and the binocular camera. The pitch angle and roll angle of the binocular camera can be compensated to ensure the accuracy of the external parameters of the binocular camera and achieve self-correction.
[0100] When statistics are performed based on the longitudinal direction of the height grid map, the obtained height statistics map can be used to compensate for the pitch angle Pitch of the binocular camera; when statistics are performed based on the transverse direction of the height grid map, the obtained height statistics map can be used to compensate for the roll angle Roll of the binocular camera.
[0101] In some embodiments, the preset direction includes the longitudinal direction of the height grid map, the height statistical map includes a first height statistical map, and when the second processing module 20 generates the height statistical map, it is specifically configured to include: generating a first blank form, wherein the horizontal header of the first blank form is determined by the height value of the height data in the height grid map, and the vertical header of the first blank form is determined by the number of rows of the height grid map; based on the height value of the height data in the height grid map, counting the first statistical number of each height value in the height grid map row by row, and filling the first statistical number of each height value into the first blank form, recorded as a first statistical form; based on the first statistical form filled with data, generating a first height statistical map.
[0102] In some embodiments, the extrinsic parameter angle compensation value includes a first extrinsic parameter angle compensation value. When the third processing module 30 determines the extrinsic parameter angle compensation value, it is specifically configured as follows: based on each coordinate point in the first height statistical graph, a fitting straight line is generated by a linear fitting method; a first slope of the fitting straight line is calculated; based on the first slope, a first extrinsic parameter angle compensation value is calculated, wherein the first extrinsic parameter angle compensation value is used to compensate for the pitch angle of the binocular camera.
[0103] In some embodiments, the preset direction includes the horizontal direction of the height grid map, the height statistical map includes a second height statistical map, and when the second processing module 20 generates the height statistical map, it is specifically configured to: generate a second blank form, wherein the vertical header of the second blank form is determined by the height value of the height data in the height grid map, and the horizontal header of the second blank form is determined by the number of columns of the height grid map; based on the height value of the height data in the height grid map, count the second statistical number of each height value in the height grid map column by column, and fill the second statistical number of each height value into the second blank form, recorded as a second statistical form; based on the second statistical form filled with data, generate a second height statistical map.
[0104] In some embodiments, the extrinsic parameter angle compensation value includes a second extrinsic parameter angle compensation value. When the third processing module 30 determines the extrinsic parameter angle compensation value, it is specifically configured to: generate a fitting straight line based on each coordinate point in the second height statistical graph by linear fitting; calculate the second slope of the fitting straight line; and calculate the second extrinsic parameter angle compensation value based on the second slope, wherein the second extrinsic parameter angle compensation value is used to compensate for the roll angle of the binocular camera.
[0105] like Figures 11 to 14As shown, this embodiment is described by taking an example where a wheel on the front side of a vehicle drives onto a raised step on the road surface.
[0106] When a vehicle drives onto a raised step on the road, the vehicle will tilt. If the binocular camera on the vehicle is not corrected for external parameters, the image obtained will be as follows: Fig.11 As shown, although the obstacle ahead can be detected at this time, its position is deviated and there is a possibility of collision.
[0107] Through the above process in this embodiment, a second height statistical graph can be obtained, such as Fig.12 As shown in , the corresponding fitting straight line is a slant line, based on which the second external parameter angle compensation value of the rolling angle can be calculated to self-correct the external parameters of the binocular camera. Based on the self-corrected external parameters, in the current state of the vehicle, the image obtained is as follows Fig.13 As shown, correspondingly, the second height statistics diagram is as follows Fig.14 At this time, the position of the obstacle ahead can be accurately obtained, which helps to ensure the safe driving of the vehicle.
[0108] It should be noted that the self-correction process of the front wheel of the vehicle when it falls into a pothole in this embodiment is similar to the process of driving onto a bump on the road surface, and will not be described in detail.
[0109] In some embodiments, the first processing module 10 is further configured to: determine in turn whether each height data in the height grid map is greater than a preset height threshold; if so, update the height data to a preset height, wherein the value of the preset height is set manually; if not, retain the height data.
[0110] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the stored computer program is executed by a processor, the method steps in any of the above embodiments are implemented.
[0111] Specifically, the computer program includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method steps in any of the above embodiments.
[0112] The above-mentioned computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, etc.
[0113] The above-mentioned processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0114] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.
[0115] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.
[0116] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.
[0117] Those skilled in the art should understand that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in each embodiment may be combined in any manner.
Claims
1. A dynamic self-correction method for external parameters of a binocular camera, characterized in that: The method includes: Based on the initial external parameters, a disparity map of the current captured image is calculated, and the disparity map is converted into a point cloud to generate a height grid map; Based on a preset direction, counting height data with the same height value in the height grid map, and generating a height statistical map based on the data statistical results; Based on the height statistics graph, a linear fitting method is used to determine an external parameter angle compensation value; Based on the external parameter angle compensation value, updating the initial external parameter, The preset direction is the longitudinal direction of the height grid image and / or the transverse direction of the height grid image. When the preset direction is the longitudinal direction of the height grid map, the height statistical map includes a first height statistical map, and the external parameter angle compensation value includes a first external parameter angle compensation value. The method of counting the height data with the same height value in the height grid map based on the preset direction, and generating a height statistical map based on the data statistical results, specifically includes: Generate a first blank form, wherein a horizontal header of the first blank form is determined by a height value of the height data in the height grid map, and a vertical header of the first blank form is determined by the number of rows of the height grid map; Based on the height values of the height data in the height grid map, first statistical numbers of each height value in the height grid map are counted row by row, and the first statistical numbers of each height value are filled into the first blank form, which is recorded as a first statistical form; Generate the first height statistical graph based on the first statistical form filled with data; Determining the external parameter angle compensation value based on the height statistical graph by linear fitting specifically includes: Based on each coordinate point in the first height statistical graph, a fitting straight line is generated by using a linear fitting method; Calculating a first slope of the fitted straight line; Based on the first slope, calculating the first extrinsic parameter angle compensation value, wherein the first extrinsic parameter angle compensation value is used to compensate for the pitch angle of the binocular camera; When the preset direction is the horizontal direction of the height grid map, the height statistical map includes a second height statistical map, and the external parameter angle compensation value includes a second external parameter angle compensation value. The method of counting the height data with the same height value in the height grid map based on the preset direction, and generating a height statistical map based on the data statistical results, specifically includes: Generate a second blank form, wherein the vertical header of the second blank form is determined by the height value of the height data in the height grid map, and the horizontal header of the second blank form is determined by the number of columns of the height grid map; Based on the height values of the height data in the height grid map, second statistical numbers of each height value in the height grid map are counted column by column, and the second statistical numbers of each height value are filled into the second blank form, which is recorded as a second statistical form; Based on the second statistical form filled with data, generating the second height statistical graph; Determining the external parameter angle compensation value based on the height statistical graph by linear fitting specifically includes: Based on each coordinate point in the second height statistical graph, a fitting straight line is generated by using a linear fitting method; Calculating a second slope of the fitted straight line; Based on the second slope, the second extrinsic parameter angle compensation value is calculated, wherein the second extrinsic parameter angle compensation value is used to compensate for the rolling angle of the binocular camera.
2. The external parameter dynamic self-correction method suitable for a binocular camera according to claim 1, characterized in that: The method further comprises: Determine in sequence whether each height data in the height grid image is greater than a preset height threshold, If yes, update the height data to a preset height, wherein the value of the preset height is manually set; If not, retain the altitude data.
3. An external parameter dynamic self-correction system suitable for a binocular camera, characterized in that: The system includes: A first processing module, wherein the first processing module is configured to calculate a disparity map of a current captured image based on an initial external parameter, and convert the disparity map into a point cloud, thereby generating a height grid map; A second processing module, the second processing module is configured to count the height data with the same height value in the height grid map based on a preset direction, and generate a height statistical map based on the data statistical results; A third processing module, wherein the third processing module is configured to determine an external parameter angle compensation value by linear fitting based on the height statistical graph; a fourth processing module, wherein the fourth processing module is configured to update the initial external parameter based on the external parameter angle compensation value, The preset direction is the longitudinal direction of the height grid image and / or the transverse direction of the height grid image. When the preset direction is the longitudinal direction of the height grid map, the height statistical map includes a first height statistical map, and the external parameter angle compensation value includes a first external parameter angle compensation value, The second processing module is configured as follows: Generate a first blank form, wherein a horizontal header of the first blank form is determined by a height value of the height data in the height grid map, and a vertical header of the first blank form is determined by the number of rows of the height grid map; Based on the height values of the height data in the height grid map, first statistical numbers of each height value in the height grid map are counted row by row, and the first statistical numbers of each height value are filled into the first blank form, which is recorded as a first statistical form; Generate the first height statistical graph based on the first statistical form filled with data; The third processing module is configured as follows: Based on each coordinate point in the first height statistical graph, a fitting straight line is generated by using a linear fitting method; Calculating a first slope of the fitted straight line; Based on the first slope, calculating the first extrinsic parameter angle compensation value, wherein the first extrinsic parameter angle compensation value is used to compensate for the pitch angle of the binocular camera; When the preset direction is the horizontal direction of the height grid map, the height statistical map includes a second height statistical map, and the external parameter angle compensation value includes a second external parameter angle compensation value. The second processing module is configured as follows: The method of counting the height data with the same height value in the height grid map based on the preset direction, and generating a height statistical map based on the data statistical results, specifically includes: Generate a second blank form, wherein the vertical header of the second blank form is determined by the height value of the height data in the height grid map, and the horizontal header of the second blank form is determined by the number of columns of the height grid map; Based on the height values of the height data in the height grid map, second statistical numbers of each height value in the height grid map are counted column by column, and the second statistical numbers of each height value are filled into the second blank form, which is recorded as a second statistical form; Based on the second statistical form filled with data, generating the second height statistical graph; The third processing module is configured as follows: Based on each coordinate point in the second height statistical graph, a fitting straight line is generated by using a linear fitting method; Calculating a second slope of the fitted straight line; Based on the second slope, the second extrinsic parameter angle compensation value is calculated, wherein the second extrinsic parameter angle compensation value is used to compensate for the rolling angle of the binocular camera.
4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.
