Parameter calibration method and device and vehicle

Abstract

The invention relates to the technical field of computer vision, and discloses a parameter calibration method and device and a vehicle. The parameter calibration method comprises the following steps: the electronic equipment can input a first image and a second image into a neural network optical flow model to obtain a dense optical flow; next, the electronic device can sparise the dense optical flow based on the down-sampling operation to obtain a sparse optical flow, and process a first optical flow value corresponding to each optical flow in the sparse optical flow, so that each second optical flow value obtained after processing can be matched with the resolution of the first image. And next, determining each second pixel point matched with each corresponding first pixel point in the second image based on each second optical flow value, and determining the optimal external parameter of the camera module based on the re-projection errors of each first pixel point and each matched second pixel point under different external parameters. Therefore, the accuracy of the obtained external parameters can be improved under the condition of reducing the consumption of computing resources.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly relates to a parameter calibration method, device, and vehicle. Background Art

[0002] In an autonomous driving system, an electronic device such as an in-vehicle computer can convert the image information captured by a camera module into three-dimensional space information in the vehicle coordinate system through the pose (i.e., position and attitude) relationship (which can be described as "external parameters") between the camera module coordinate system (the coordinate origin can be the center of the camera module) and the vehicle coordinate system (the coordinate origin can be the center of the vehicle), so as to achieve accurate environmental perception and autonomous driving decision-making.

[0003] Among them, with the use of the vehicle, due to factors such as the aging and deformation of the vehicle itself or the aging and deformation of the camera module bracket resulting in the tilt of the position of the camera module, the relative pose between the camera module and the vehicle center will change, and thus the external parameters between the camera module coordinate system and the vehicle coordinate system will also change. Therefore, how to adjust the external parameters between the camera module coordinate system and the vehicle coordinate system to adapt to various changing situations is an urgent problem to be solved currently. Summary of the Invention

[0004] To solve the above problems, this application provides a parameter calibration method, device, and vehicle, which can improve the accuracy of the obtained external parameters while reducing the consumption of computing resources.

[0005] In a first aspect, this application provides a parameter calibration method, which is applied to an electronic device and includes: obtaining a first image and a second image captured by a camera module; inputting the first image and the second image into a neural network optical flow model to obtain a dense optical flow corresponding to the first image, and each optical flow in the dense optical flow is used to indicate the motion state of the corresponding pixel point between the first image and the second image; performing a downsampling operation on the dense optical flow to obtain a sparse optical flow; adjusting each first optical flow value corresponding to the optical flows in the sparse optical flow to a second optical flow value matching the resolution of the first image, and each second optical flow value corresponds to a plurality of first pixel points in the first image; determining, based on each second optical flow value, each second pixel point in the second image that matches the corresponding first pixel point; and determining a first external parameter of the camera module based on the reprojection errors of each first pixel point and the corresponding second pixel points that match under different external parameters.

[0006] In some embodiments, a deep learning-based neural network (NN) optical flow model (e.g., the Flownet NN model) can perform feature matching on multiple pixel points in two frames of images respectively to determine the matching pixel points, and determine the corresponding optical flow based on the position differences of each pair of matching pixel points in the two frames of images to obtain a dense optical flow. Therefore, after the electronic device acquires the first image and the second image to be processed, it can input the two frames of images into the NN optical flow model to obtain a dense optical flow. Then, the electronic device can sparsify the dense optical flow based on a downsampling operation to obtain a sparse optical flow. For example, if the downsampling step size is k, it means that one optical flow can be selected every k optical flows in the dense optical flow. Among them, the optical flow value can represent the number of pixel points between position 1 and another position 2. If some pixel points between position 1 and another position 2 in the optical flow map are deleted through the downsampling operation, the number of pixel points between position 1 and another position 2 will decrease, and thus the optical flow value will also decrease. Therefore, the electronic device also needs to increase the first optical flow value corresponding to each optical flow in the sparse optical flow so that each second optical flow value obtained after the increase can match the resolution of the first image. Next, the electronic device can determine the second pixel points in the second image that match the first pixel points corresponding to each second optical flow value based on each second optical flow value. Then, during the process of adjusting the external parameters of the camera module, the electronic device can monitor the average value of the absolute values of the reprojection errors corresponding to each pair of matching pixel points in real time. Finally, when the electronic device monitors that the average value of the absolute values of the reprojection errors is less than a preset threshold, the electronic device can use the external parameters of the camera module corresponding to this set of reprojection errors as the optimal external parameters of the camera module (i.e., the first external parameters).

[0007] Thus, based on the above parameter calibration method, it is possible to improve the accuracy of optical flow tracking for pixel points while reducing the consumption of computing resources, and then determine the optimal external parameters of the camera model by monitoring the reprojection errors of each pair of matching pixel points.

[0008] In a possible implementation of the first aspect above, determining the first external parameters of the camera module based on the reprojection errors of each first pixel point and the matching second pixel points under different external parameters includes: when the camera module is under different external parameters, calculating the reprojection errors corresponding to each first pixel point and the matching second pixel points respectively to obtain multiple sets of reprojection errors, where the multiple sets of reprojection errors include the first set of reprojection errors; when the average value of the absolute values of the reprojection errors in the first set of reprojection errors is less than a preset threshold, determining the external parameters corresponding to the first set of reprojection errors as the first external parameters.

[0009] In some embodiments, after the electronic device determines the corresponding first pixel points and the matching second pixel points based on the respective second optical flow values, the electronic device can adjust the external parameters of the camera module multiple times. Moreover, each time the external parameters of the camera module are adjusted, the electronic device can record a set of reprojection errors corresponding to the current external parameters of the camera module (including the reprojection errors corresponding to each pair of matching pixel points). In this way, after the electronic device adjusts the external parameters of the camera module multiple times, it can obtain multiple sets of reprojection errors, and can also calculate the average value of the absolute values of the respective reprojection errors in each set of reprojection errors. Among them, if the electronic device determines that the average value of the absolute values of the respective reprojection errors in the first set of reprojection errors is less than a preset threshold, the electronic device can determine that the external parameters corresponding to the first set of reprojection errors are the optimal external parameters of the camera module; or, if the electronic device determines that the average value of the absolute values of the respective reprojection errors in the first set of reprojection errors is less than the average values corresponding to other sets of reprojection errors, the electronic device can also determine that the external parameters corresponding to the first set of reprojection errors are the optimal external parameters of the camera module. In this way, the electronic device can determine the first external parameters based on the pairs of matching pixel points determined by optical flow tracking.

[0010] In a possible implementation of the foregoing first aspect, the foregoing method further includes: determining multiple first external parameters of the camera module based on the reprojection errors of each pair of matching pixel points in multiple sets of two-frame images under different external parameters; determining the distribution frequency of each first external parameter among the multiple first external parameters through a histogram; and adjusting the external parameters of the camera module to the first external parameter corresponding to the maximum distribution frequency.

[0011] In some embodiments, the electronic device can also determine multiple first external parameters based on the reprojection errors of each pair of matching pixel points in multiple sets of two-frame images under different external parameters. Then, the electronic device can observe the distribution frequency of each first external parameter among the multiple first external parameters through methods such as histograms or convergence, and use the first external parameter corresponding to the maximum distribution frequency as the optimal external parameters of the camera module. In this way, through multiple operations, the obtained optimal module external parameters can be made more accurate, avoiding contingency.

[0012] In a possible implementation of the foregoing first aspect, the foregoing method further includes: calculating the reprojection errors of each first pixel point and the matching second pixel points under different external parameters based on the odometer information of the vehicle corresponding to the time period when the camera module captures the first image and the second image; wherein, the yaw angular velocity in the odometer information is less than the angular velocity threshold, and the pitch angle in the odometer information is less than the angle threshold.

[0013] In some embodiments, the electronic device may calculate reprojection errors of pairs of matching pixel points in multiple sets of two frames of images under different extrinsic parameters based on the rotation matrix and translation vector in the vehicle odometer information corresponding to each frame of image captured by the camera module. Among them, due to the large errors of the rotation matrix and translation vector obtained when the vehicle is bumping, jittering, swaying, etc., the determined reprojection errors and the errors of the extrinsic parameters of the camera module are also large. Therefore, if the yaw angular velocity in the odometer information corresponding to the vehicle is greater than the angular velocity threshold or the pitch angle is greater than the angle threshold during the time period of capturing a certain two frames of images, the electronic device may determine that the vehicle is in stages such as on a slope, accelerating rapidly, or decelerating rapidly at this time, and the electronic device may discard the two frames of images corresponding to the odometer information and re-determine the extrinsic parameters of the camera module based on other two frames of images. In this way, the accuracy of the determined extrinsic parameters of the camera module can be higher.

[0014] In a possible implementation of the first aspect above, the first image is the area of the road surface in the third image captured by the camera module, and the second image is the area of the road surface in the fourth image captured by the camera module.

[0015] In some embodiments, to save computing resources, the electronic device may extract a partial area from the previous complete image (i.e., the third image) captured by the camera module as the first image, and extract a partial area from the next complete image (i.e., the fourth image) captured by the camera module as the second image, and then input the first image and the second image into the NN optical flow model to obtain a dense optical flow. Among them, the vehicle may overtake or pass pedestrians, buildings, etc. during driving, resulting in non-road surface feature points such as the vehicle, pedestrians, and buildings may not appear in the two frames of images captured of the front at the same time. Therefore, in the embodiments of the present application, the areas intercepted by the electronic device in the third image and the fourth image may both be the areas of the road surface, so as to subsequently track only the road surface feature points.

[0016] In some embodiments, the full-image resolution of the camera module may be the first resolution, and the image resolutions corresponding to the third image and the fourth image may both be the second resolution, and the second resolution is less than the first resolution. The full-image resolution may refer to: the maximum resolution of the image that the camera module can capture without adjusting the camera module. For example, the first resolution may be 3840×2160 (width×height), and the second resolution may be 1920×1080. In this way, when the electronic device processes low-resolution images, it can reduce resource consumption and improve processing efficiency.

[0017] In a possible implementation of the first aspect, the downsampling operation is performed on the dense optical flow to obtain the sparse optical flow, including: calculating the displacement corresponding to each optical flow in the dense optical flow; deleting the optical flow whose displacement is less than a preset displacement threshold in the dense optical flow to obtain the deleted dense optical flow; and performing a downsampling operation on the deleted dense optical flow to obtain the sparse optical flow.

[0018] In some embodiments, since the NN optical flow model has poor optical flow tracking accuracy for targets with small displacements, the electronic device may further eliminate optical flow values ​​with small changes in feature point displacements between frames, thereby improving the accuracy of parameter calibration.

[0019] In a possible implementation of the first aspect above, the above-mentioned determining, based on each second optical flow value, each second pixel point that matches the corresponding first pixel point in the second image includes: identifying road feature points and non-road feature points in the first image based on semantic segmentation; deleting the second optical flow values ​​corresponding to the non-road feature points identified by semantic segmentation; if there are non-road feature points in a preset surrounding area of ​​the identified first road feature point, deleting the first road feature point and the second optical flow value corresponding to the first road feature point to obtain the deleted second optical flow value; based on the deleted second optical flow values, determining each second pixel point that matches the corresponding first pixel point in the second image.

[0020] In some embodiments, a vehicle may overtake a vehicle or pass a pedestrian or a building during driving, so that non-road feature points such as vehicles, pedestrians, and buildings may not appear in both the first image and the second image taken of the front. Therefore, the electronic device can use a semantic segmentation algorithm to match only the road feature points in the first image, thereby improving the accuracy of parameter calibration.

[0021] In a possible implementation of the first aspect above, the first extrinsic parameter of the camera module is determined based on the reprojection error of each first pixel point and the matching second pixel points under different extrinsic parameters, including: deleting the erroneous optical flow values ​​in each second optical flow value based on a random sampling consistency algorithm; deleting the first pixel points and the second pixel points corresponding to the erroneous optical flow values ​​in the first image and the second image to obtain a modified first image and a modified second image; in the modified first image and the modified second image, the first extrinsic parameter of the camera module is determined based on the reprojection error of each first pixel point and the matching second pixel points under different extrinsic parameters.

[0022] In some embodiments, the electronic device may also use the random sample consensus (Ransac) algorithm to eliminate the incorrect tracking points and their corresponding pixel points. In this way, the electronic device can determine the first external parameters of the camera module based on the reprojection errors of each first pixel point and each matched second pixel point under different external parameters in the modified first image and the modified second image.

[0023] In other embodiments, if the number of remaining optical flow points is too small after eliminating the incorrect optical flow, it is considered that this frame of image has no reference significance and can be directly discarded, and the external parameters of the camera module are determined again based on other images.

[0024] In this way, through the parameter calibration method provided by this application, the electronic device can determine the external parameters of the camera module only through the camera module and vehicle odometer information without other sensors. Moreover, the above parameter calibration method is applicable to the driving scenario, saving computing resources and having high accuracy, and also having good adaptability to the bumps, jitters, and swings of the vehicle.

[0025] In a second aspect, this application provides an electronic device, including: a memory and a processor, the memory is coupled to the processor; the memory is used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, the electronic device executes the parameter calibration method mentioned in this application.

[0026] In a third aspect, this application provides a vehicle, which includes the electronic device mentioned in the second aspect above.

[0027] For the beneficial effects of the second aspect to the third aspect above, reference can be made to the relevant descriptions in the first aspect and various possible implementations of the first aspect, and details are not described here. Description of the Drawings

[0028] Figure 1 According to some embodiments of this application, a schematic diagram of a scene photographed by a camera module is shown;

[0029] Figure 2 According to some embodiments of this application, a schematic diagram of coordinate system mapping is shown;

[0030] Figure 3 According to some embodiments of this application, a schematic diagram of optical flow is shown;

[0031] Figure 4 According to some embodiments of this application, a schematic diagram of downsampling dense optical flow is shown;

[0032] Figure 5 According to some embodiments of this application, a schematic diagram of image parsing and truncation is shown;

[0033] Figure 6 According to some embodiments of the present application, a flowchart of a parameter calibration method is shown;

[0034] Figure 7 According to some embodiments of the present application, a schematic diagram of semantic segmentation of an image is shown;

[0035] Figure 8 According to some embodiments of the present application, a specific flowchart of a parameter calibration method is shown;

[0036] Figure 9 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown. Detailed implementation manners

[0037] Illustrative embodiments of the present application include, but are not limited to, a parameter calibration method, device, and vehicle.

[0038] To more clearly understand the solution of the present application, first, terms related to the relevant fields involved in the present application are explained.

[0039] 1. Extrinsic parameters of the camera module

[0040] The extrinsic parameters of the camera module represent the pose relationship between the camera module coordinate system and the vehicle coordinate system, and can be used to convert the image information captured by the camera module into three-dimensional space information in the vehicle coordinate system. For example, as shown in the reference Figure 1 The camera module in vehicle 101 can send the captured road image 102 to an electronic device such as the in-vehicle computer of vehicle 101. The in-vehicle computer can map the road image 102 from the coordinate system where the camera module is located to the coordinate system where vehicle 101 is located through the extrinsic parameters of the camera module, so that the in-vehicle computer can more accurately determine spatial data such as the distance between obstacles (such as pedestrians, vehicles ahead, etc.) and vehicle 101, and further can determine accurate autonomous driving decisions, such as deceleration, constant speed, etc. autonomous driving decisions.

[0041] Among them, the extrinsic parameters of the camera module can include the rotation matrix and the translation vector The rotation matrix represents the rotation degree of the coordinate system where the camera module is located relative to the vehicle coordinate system, and the translation vector represents the translation degree of the coordinate system where the camera module is located relative to the vehicle coordinate system.

[0042] 2. Homography (H) matrix

[0043] The homography matrix is a geometric transformation matrix widely used in image processing and computer vision, which can be used to map image feature points from one coordinate system to another. For example, referring to Figure 2 As shown, the camera module inside the vehicle can continuously capture two frames of images (e.g., frame a and frame b). If the coordinate value of feature point A in the coordinate system of frame a is p1(x1, y1), then the coordinate value p2'(x2', y2') of this feature point A in the coordinate system of frame b can be determined by the H matrix as H×p1.

[0044] Among them, compared with the pose of the camera module when capturing the previous frame of image (e.g., frame a), if the parameters corresponding to the rotation degree and translation degree of the camera module when capturing the next frame of image (e.g., frame b) are the rotation matrix R cam and the translation vector S cam respectively, then the electronic device can calculate the above H matrix based on this rotation matrix R cam and the translation vector S cam . Specifically, the calculation process of the H matrix can refer to the formula (I) shown as follows:

[0045]

[0046] Among them, in the above formula (I), H is the homography matrix, which can be used to map feature points from the coordinate system of frame a to the coordinate system of frame b. K is the internal parameter matrix corresponding to the internal parameters (focal length, distortion, etc.) of the camera module, and K -1 is the inverse matrix of the internal parameter matrix K. R cam is the rotation matrix corresponding to the rotation degree of the camera module when capturing two frames of images; S cam is the translation vector corresponding to the translation degree of the camera module when capturing two frames of images; d is the distance from the camera module to the ground. n is the ground normal vector, and n T refers to the transpose matrix of the normal vector n, that is, n T means transposing the data in the i-th row and k-th column of the normal vector n into the data in the k-th row and i-th column.

[0047] Among them, in the above formula (I) for calculating the H matrix, since the K matrix is determined when the camera module is assembled or manufactured and usually remains unchanged, therefore, K and K -1 are fixed values. In addition, in the above formula (I), the values of R cam , S cam , n, and d are all related to the external parameters of the camera module. Specifically, referring to the following formula (II) to formula (V), the values of R cam , S cam , n, and d can be determined respectively.

[0048]

[0049] In the above formula (II), R cam is the rotation matrix corresponding to the rotation degree when the camera module captures two frames of images. is the rotation matrix in the extrinsic parameters of the camera module, representing the rotation degree of the coordinate system where the camera module is located relative to the vehicle coordinate system; can be the rotation matrix in the extrinsic parameters of the camera module which is the inverse matrix of, representing the rotation degree of the vehicle coordinate system relative to the coordinate system where the camera module is located. R vcs is the rotation matrix corresponding to the rotation degree of the vehicle when the camera module captures two frames of images, representing the rotational motion state of the vehicle during driving (the driving time is the interval period for capturing two frames of images). Among them, the in-vehicle computer can determine the rotation matrix R by obtaining the rotation data of the vehicle stored in the vehicle odometry (odometry, odo) information vcs . And the vehicle odo information is provided by functional modules such as the in-vehicle inertial measurement unit, wheel speed sensor, and global navigation satellite system, and is used to reflect the translational, rotational, and other motion states of the vehicle during driving.

[0050] In the above formula (III), S cam is the translation vector corresponding to the rotation degree when the camera module captures two frames of images. is the rotation matrix in the extrinsic parameters of the camera module; is the translation vector in the extrinsic parameters of the camera module. R vcs is the rotation matrix corresponding to the rotation degree of the vehicle when the camera module captures two frames of images; S vcs is the translation matrix corresponding to the translation degree of the vehicle when the camera module captures two frames of images, representing the translational motion state of the vehicle during driving (the driving time is the interval period for capturing two frames of images). Among them, the in-vehicle computer can obtain the rotation data and translation data of the vehicle in the vehicle odo information, so as to determine the rotation matrix R vcs and the translation vector S vcs .

[0051] In the above formula (IV), n is the ground normal vector; is the rotation matrix in the extrinsic parameters of the camera module; [0, 0, 1] T represents the transpose matrix of the vector [0, 0, 1].

[0052] In the above formula (V), d is the distance from the camera module to the ground; is the translation vector in the extrinsic parameters of the camera module; is the translation vector in the third element.

[0053] Based on the above formulas (I) - (V), it can be seen that the H matrix is related to the values of R cam , S cam , n, and d, and the values of R cam , S cam , n, and d are all related to the extrinsic parameters of the camera module (rotation matrix or translation vector ). Therefore, if the accuracy of the extrinsic parameters of the camera module is higher, the accuracy of R cam , S cam , n, and d calculated based on the extrinsic parameters of the camera module will be higher, and further, the accuracy of the H matrix calculated based on R cam , S cam , n, and d will also be higher.

[0054] 2. Reprojection Error

[0055] The reprojection error can be the difference between the actual coordinate value of a feature point in a coordinate system and the coordinate value of this point obtained through calculation. For example, the camera module inside the vehicle can continuously capture two frames of images (such as frame a and frame b). If the coordinate value of feature point A in the coordinate system of frame a is p1(x1, y1), then the coordinate value p2'(x2', y2') of this feature point A in the coordinate system of frame b determined by the H matrix can be H×p1. Among them, if the actual coordinate value of this feature point A in the coordinate system of frame b is p2(x2, y2), then the difference between p2 and p2' (that is, the difference between p2 and H×p1) is the reprojection error. Specifically, the calculation formula of the reprojection error e can refer to formula (VI) as follows:

[0056] e = p2 - H×p1 (VI)

[0057] In the above formula (VI), e is the reprojection error; p2 is the actual coordinate value of the feature point in the coordinate system of the latter frame of image; H is the homography matrix; p1 is the actual coordinate value of the feature point in the coordinate system of the previous frame of image; H×p1 is the coordinate value p2' of the feature point in the coordinate system of the latter frame of image determined by the H matrix.

[0058] Among them, if the accuracy of the H matrix is relatively high, then the coordinate value p2' of the feature point in the coordinate system of the latter frame of image calculated by H×p1 is the same as the actual coordinate value p2. That is, the higher the accuracy of the H matrix, the smaller the absolute value of the reprojection error e. For example, the reprojection error e can be 0.

[0059] In summary, the higher the accuracy of the extrinsic parameters of the camera module, the higher the accuracy of the H matrix, and thus the smaller the absolute value of the reprojection error e. Therefore, the electronic device can improve the accuracy of the extrinsic parameters of the camera module by adjusting the reprojection error e, that is, the electronic device can calibrate the extrinsic parameters of the camera module by adjusting the reprojection error.

[0060] The method for calibrating the extrinsic parameters of the camera module by adjusting the reprojection error will be introduced below.

[0061] When the vehicle leaves the factory, the initial extrinsic parameters between the camera module coordinate system and the vehicle coordinate system are usually stored in the in-vehicle computer, so that the in-vehicle computer can process image information based on the initial extrinsic parameters during the vehicle driving process. Then, as the relative pose between the camera module and the vehicle center changes, for example, the vehicle ages and deforms, the extrinsic parameters between the camera module coordinate system and the vehicle coordinate system also change. The in-vehicle computer cannot accurately map the images captured by the camera module into the vehicle coordinate system based on the initial extrinsic parameters, and thus cannot accurately determine the distance between the obstacle and the vehicle, reducing the safety of vehicle driving.

[0062] However, as mentioned above, the higher the accuracy of the extrinsic parameters of the camera module, the higher the accuracy of the H matrix, and thus the smaller the absolute value of the reprojection error e. That is to say, when the electronic device adjusts the extrinsic parameters of the camera module (rotation matrix or translation vector ), it can only monitor the magnitude of the absolute value of the reprojection error e. When the electronic device determines that the absolute value of the reprojection error e meets the preset condition (such as, less than the preset threshold), it can determine that the extrinsic parameters of the camera module corresponding to the reprojection error e are the optimal extrinsic parameters of the camera module. For example, when the electronic device adjusts the rotation matrix of the extrinsic parameters of the camera module to R1, it can calculate that the absolute value of the corresponding reprojection error e is e1. Next, when the electronic device adjusts the rotation matrix of the extrinsic parameters of the camera module to R2, it can calculate that the absolute value of the corresponding reprojection error e is e2. Among them, if the electronic device determines that e2 < e1, it can indicate that the accuracy when the rotation matrix is R2 is higher than the accuracy when the rotation matrix is R1. In this way, when the electronic device determines the optimal reprojection error e, it can determine the corresponding optimal extrinsic parameters of the camera module.

[0063] Among them, as shown in the above formula (VI), when calculating the reprojection error, it is necessary to determine the actual coordinate value p1 of the feature point (e.g., feature point A) in the coordinate system of the previous frame image, and determine the actual coordinate value p2 of the same feature point (e.g., still feature point A) in the coordinate system of the next frame image. In this way, the reprojection error can be calculated based on the actual coordinate value p1, the actual coordinate value p2, and the H matrix. Therefore, the electronic device can track the feature points to avoid the situation where the actual coordinate value p1 and the actual coordinate value p2 belong to different feature points, so that the reprojection error can be calculated based on the actual coordinate values of the same feature point in the coordinate systems of two frame images respectively.

[0064] It should be understood that the motion state of pixel points in two consecutive frame images can usually be described by "optical flow". After obtaining the optical flow value corresponding to a certain pixel point in the previous frame image, the corresponding pixel point can be tracked in the next frame image based on the optical flow. For example, Figure 3 Fig. 301 shows an optical flow map composed of optical flows corresponding to pixel points in an image. Among them, in the optical flow map 301, each optical flow can be represented by a vector, and the starting point of the vector (i.e., the circle) represents the position of the corresponding pixel point in the previous frame image, and the end point of the vector (i.e., the arrow) represents the position of the corresponding pixel point in the next frame image. Among them, the direction pointed by the vector is the motion direction of the pixel point, and the number of horizontal and vertical pixel points passed by the starting point and the end point of the vector (which can be expressed as (x, y)) is the optical flow value corresponding to the pixel point. After the electronic device determines the optical flow corresponding to a certain pixel point in the previous frame image, it can determine the position of the pixel point in the next frame image. For example, if the optical flow value of the optical flow corresponding to pixel point A in the previous frame image is (x1, y1), when stacking and overlapping the previous frame and the next frame images, it means that at the same position in the next frame, move x pixel points in the horizontal direction and y pixel points in the vertical direction, and the corresponding matching pixel point A' can be found in the next frame. In addition, optical flow can include dense optical flow and sparse optical flow. Dense optical flow refers to determining the optical flows corresponding to multiple pixel points in the previous frame image to form a complete optical flow field; sparse optical flow refers to determining the optical flows corresponding to a small number of pixel points in the previous frame image.

[0065] Therefore, the present application provides a parameter calibration method. In the present application, after the electronic device obtains the first image and the second image, the two frames of images can be input into the NN optical flow model to obtain the dense optical flow. Among them, the NN optical flow model based on deep learning (such as, the Flownet NN model) can perform feature matching on multiple pixel points in the two frames of images respectively to determine the matching pixel points, and determine the corresponding optical flow based on the position differences of each pair of matching pixel points in the two frames of images to obtain the dense optical flow. Next, the electronic device can sparsify the dense optical flow based on the downsampling operation to obtain the sparse optical flow. For example, if the downsampling step size is k, it means that one optical flow can be selected every k optical flows in the dense optical flow. Among them, the optical flow value can represent the number of pixel points in the middle interval from position 1 to another position 2. If some pixel points in the middle interval from position 1 to another position 2 in the optical flow map are deleted through the downsampling operation, the number of pixel points in the middle interval from position 1 to another position 2 will be reduced, and thus the optical flow value will also be reduced. Therefore, the electronic device also needs to increase the first optical flow value corresponding to each optical flow in the sparse optical flow so that each second optical flow value obtained after the increase can match the resolution of the first image. Next, the electronic device can determine the second pixel points in the second image that match the first pixel points corresponding to each second optical flow value based on each second optical flow value. Then, during the process of adjusting the external parameters of the camera module, the electronic device can monitor the average value of the absolute values of the reprojection errors corresponding to each pair of matching pixel points in real time. Finally, when the electronic device monitors that the average value of the absolute values of the reprojection errors is less than the preset threshold, the electronic device can use the external parameters of the camera module corresponding to this set of reprojection errors as the optimal external parameters of the camera module (as the first external parameter).

[0066] In this way, based on the above method, it is possible to improve the accuracy of optical flow tracking of pixel points while reducing the consumption of computing resources, and then determine the optimal external parameters of the camera model by monitoring the reprojection errors of each pair of matching pixel points.

[0067] It can be understood that after the first image and the second image are input into the NN optical flow model, the dense optical flow corresponding to the first image obtained is represented by an optical flow map. And in the optical flow map, if some optical flows in the middle interval from position 1 to another position 2 in the optical flow map are deleted through the downsampling operation, the number of pixel points in the middle interval from position 1 to another position 2 will be reduced, and thus the optical flow value will also become smaller. For example, in Figure 4Among them, the optical flow map 401 can be an optical flow map corresponding to dense optical flow. Among them, in the optical flow map 401 corresponding to dense optical flow, the point p1 can represent the position of the pixel point A in the first image, and the point p2 can represent the position of the matching pixel point A' in the second image. Among them, the vector between p1 and p2 can represent the optical flow corresponding to the pixel point A, and the number of pixel points x passed by the vector in the horizontal direction is the horizontal component of the optical flow value, and the number of pixel points y passed by the vector in the vertical direction is the vertical component of the optical flow value. That is, the optical flow value corresponding to the pixel point A can be (x, y). However, if the dense optical flow is changed to sparse optical flow through downsampling operation, the optical flow map 402 corresponding to the sparse optical flow will be obtained. Compared with the optical flow map 401, in the optical flow map 402, since some optical flows are deleted, the number of pixel points between the p1 and p2 vectors will be reduced, and thus the optical flow value used to describe the vector displacement will also become smaller. Therefore, after the electronic device obtains the sparse optical flow through the downsampling operation, it can multiply the first optical flow value corresponding to each optical flow in the sparse optical flow by a certain value (which can be described as a "scale factor") so as to amplify the first optical flow value to a second optical flow value matching the resolution of the first image.

[0068] Among them, in some embodiments, in order to reduce the computational amount, the NN optical flow model may perform downsampling processing on the input high-resolution first image and second image to obtain a low-resolution first image and second image. However, the NN optical flow map will determine the dense optical flow corresponding to the low-resolution first image. For example, if the resolutions of the input high-resolution first image and second image are 1216×192 (width×height), the dimension of the optical flow map corresponding to the dense optical flow output by the NN optical flow model is 608×96. It can be seen that the optical flow values of the optical flows in the dense optical flow do not correspond to the original resolution of the first image. Therefore, when the electronic device multiplies the first optical flow value corresponding to each optical flow in the sparse optical flow by the scale factor so as to amplify the first optical flow value to a second optical flow value matching the resolution of the first image, it does not mean directly amplifying the first optical flow value to the corresponding optical flow value in the dense optical flow.

[0069] Among them, the scale factor can be determined through experiments. For example, in a certain experiment, the ratio of a certain first optical flow value to the corresponding optical flow value in the dense optical flow can be determined, and then based on this ratio and the ratio of the dense optical flow map to the resolution of the first image, a scale factor can be determined. Then, after determining multiple scale factors through multiple groups of experiments, the average value of the multiple scale factors or the scale factor with the most occurrences can be used as the target scale factor. The present application does not limit the determination method of the scale factor.

[0070] In some embodiments, after the electronic device determines the corresponding first pixel points and the matched second pixel points based on the respective second optical flow values, the electronic device can adjust the external parameters of the camera module multiple times. Moreover, each time the external parameters of the camera module are adjusted, the electronic device can record a set of reprojection errors corresponding to the current external parameters of the camera module (including the reprojection errors corresponding to each pair of matched pixel points). In this way, after the electronic device adjusts the external parameters of the camera module multiple times, it can obtain multiple sets of reprojection errors, and can also calculate the average value of the absolute values of the respective reprojection errors in each set of reprojection errors. Among them, if the electronic device determines that the average value of the absolute values of the respective reprojection errors in the first set of reprojection errors is less than a preset threshold, the electronic device can determine that the external parameters corresponding to the first set of reprojection errors are the optimal external parameters of the camera module; or, if the electronic device determines that the average value of the absolute values of the respective reprojection errors in the first set of reprojection errors is less than the average values corresponding to other sets of reprojection errors, the electronic device can also determine that the external parameters corresponding to the first set of reprojection errors are the optimal external parameters of the camera module. In this way, the electronic device can determine the optimal external parameters of the camera module (as the first external parameters) based on each pair of matched pixel points determined by optical flow tracking.

[0071] In addition, in some embodiments, the electronic device can also repeat the above operation process multiple times. Specifically, after the electronic device determines the first external parameters of the camera module through the first image and the second image, the electronic device can obtain two other frames of images again. Then, by repeating the above operation process (obtaining dense optical flow, obtaining sparse optical flow, determining matching feature points, etc.), the reprojection errors of each pair of matched pixel points in the two other frames of images under different external parameters are obtained, so as to obtain the first external parameters corresponding to the two other frames of images. In addition, the electronic device can also determine multiple first external parameters based on the reprojection errors of each pair of matched pixel points in multiple sets of two frames of images under different external parameters. Then, the electronic device can observe the distribution frequency of each first external parameter among the above multiple first external parameters through methods such as histograms or convergence, and use the first external parameter corresponding to the maximum distribution frequency as the optimal external parameters of the camera module. In this way, through multiple operations, the obtained optimal module external parameters can be made more accurate, avoiding contingency.

[0072] In some embodiments, to save computing resources, the electronic device may extract a partial area from the previous complete image captured by the camera module (as an example of the third image) as the first image, and extract a partial area from the subsequent complete image captured by the camera module (as an example of the fourth image) as the second image, and then input the first image and the second image into the NN optical flow model to obtain a dense optical flow. Among them, during the driving process of the vehicle, it may overtake or pass pedestrians, buildings, etc., resulting in non-road feature points such as vehicles, pedestrians, and buildings may not appear in the two images captured for the front at the same time. Therefore, in the embodiments of the present application, the areas intercepted by the electronic device in the third image and the fourth image can both be the areas where the road surface is located, so as to subsequently track only the road surface feature points. For example, referring to Figure 5 As shown, if the previous complete image captured by the camera module is Image 501, the electronic device can intercept the area where the road surface is located to obtain the first image 502. In this way, inputting a partial area of the complete image into the NN optical flow model to obtain a dense optical flow can further reduce the consumption of computing resources.

[0073] In some other embodiments, the resolutions of the two complete images (the third image and the fourth image) captured by the camera module can also be less than the full-image resolution of the camera module. That is to say, without adjusting the camera module, the maximum resolution of the images that the camera module can capture (that is, the full-image resolution) can be the first resolution. However, the electronic device can adjust the resolution setting of the camera module so that the camera module captures the third image and the fourth image at the second resolution. That is, the image resolutions of the third image and the fourth image can both be the second resolution; and, the second resolution is less than the first resolution. For example, the second resolution can be half of the first resolution. For example, the first resolution can be 3840×2160 (width×height), then the second resolution can be 1920×1080. In this way, when the electronic device processes low-resolution images, it can reduce resource consumption and improve processing efficiency.

[0074] It can be understood that the above parameter calibration method of the present application can be applied to any electronic device. The electronic device includes, but is not limited to, a mobile station (MS), a mobile terminal (MT), etc. For example, the electronic device can be an in-vehicle computer, a computer, a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, etc. The specific form of the electronic device is not limited in the embodiments of the present application.

[0075] Based on Figure 6 the following schematic flow diagram, a brief introduction to the parameter calibration method mentioned in the embodiments of the present application will be given. Among them, the parameter calibration method can be applied to an electronic device, such as any electronic device mentioned above, such as an in-vehicle computer. As Figure 6 shown, specifically, the method is as follows:

[0076] S601: Obtain a first image and a second image captured by the camera module.

[0077] In some embodiments, the first image and the second image can be two complete images continuously captured by the camera module inside the vehicle.

[0078] In some other embodiments, the first image and the second image can also be the road surface areas in two complete images continuously captured by the camera module inside the vehicle.

[0079] S602: Input the first image and the second image into the NN optical flow model to obtain the dense optical flow corresponding to the first image, where each optical flow in the dense optical flow is used to indicate the motion state of the corresponding pixel point between the first image and the second image.

[0080] In some embodiments, the NN optical flow model is a deep learning-based optical flow model that can perform feature matching on multiple pixel points in two frames of images to determine the matching pixel points, and determine the corresponding optical flow based on the position differences of each pair of matching pixel points in the two frames of images to obtain a dense optical flow. After the first image and the second image are input into the NN optical flow model, the electronic device can obtain an optical flow map representing the dense optical flow corresponding to the first image. For example, in the above Figure 4 after the first image and the second image are input into the NN optical flow model, the electronic device can obtain the optical flow map 401 corresponding to the dense optical flow of the first image.

[0081] S603: Perform a downsampling operation on the dense optical flow to obtain a sparse optical flow.

[0082] In some embodiments, performing a downsampling operation on the dense optical flow may be to delete some of the optical flows in the dense optical flow. For example, in the above Figure 4 the electronic device can perform a downsampling operation on the dense optical flow in the optical flow map 401 to obtain the sparse optical flow shown in the optical flow map 402. Among them, the step size of the downsampling operation can be set arbitrarily. For example, if the step size of the downsampling operation is k, it can mean that in the dense optical flow, one optical flow is obtained every k optical flows horizontally, and one optical flow value is obtained every k optical flow values vertically.

[0083] In other embodiments, since the NN optical flow model has poor optical flow tracking accuracy for small displacement targets, after the electronic device obtains the dense optical flow through the above S602, it can also calculate the displacements corresponding to the optical flows in the dense optical flow. For example, if the optical flow value corresponding to an optical flow is (1, 2), the corresponding displacement can be √5. Then, the electronic device can delete the optical flows with displacements less than the preset displacement threshold, and thus perform a downsampling operation on the dense optical flow after deleting some optical flows to obtain a sparse optical flow. In this way, the optical flows with lower accuracy in the dense optical flow obtained by the NN optical flow model can be deleted, thereby improving the accuracy of parameter calibration.

[0084] S604: Adjust each first optical flow value in the sparse optical flow to a second optical flow value that matches the resolution of the first image, where each second optical flow value corresponds to multiple first pixel points in the first image.

[0085] In some embodiments, each optical flow can be represented by an optical flow map. After downsampling the dense optical flow to obtain the sparse optical flow, that is, downsampling the optical flow map corresponding to the dense optical flow to obtain the optical flow map corresponding to the sparse optical flow. Among them, when performing the downsampling operation on the optical flow map corresponding to the dense optical flow, the optical flow value corresponding to the optical flow will be reduced. Therefore, after the electronic device obtains the sparse optical flow through the downsampling operation, it can multiply the first optical flow value corresponding to each optical flow in the sparse optical flow by a certain value, so as to amplify the first optical flow value to the second optical flow value that matches the original resolution.

[0086] S605: Based on each second optical flow value, determine each second pixel point that matches the corresponding first pixel point in the second image.

[0087] In some embodiments, after obtaining the second optical flow value corresponding to the first pixel point, the corresponding second pixel point can be tracked in the second image based on the second optical flow value. For example, if the second optical flow value corresponding to the first pixel point is (x, y), it means moving x pixel points in the horizontal direction and y pixel points in the vertical direction at the same position in the second image, and the second pixel point can be found in the second image. In this way, based on each second optical flow value, multiple pairs of matching pixel points between the first image and the second image can be determined.

[0088] In some embodiments, during the driving process of the vehicle, it may overtake or pass pedestrians, buildings, etc., resulting in non-road surface feature points such as vehicles, pedestrians, and buildings may not appear in both the first image and the second image captured of the front at the same time. Therefore, the electronic device can only match the road surface feature points in the first image. Specifically, the electronic device can first identify the road surface feature points and non-road surface feature points in the first image through a semantic segmentation algorithm, and delete the identified non-road surface feature points and the second optical flow values corresponding to the non-road surface feature points. In addition, if there are non-road surface feature points in the preset surrounding area of a certain identified first road surface feature point, it is also necessary to delete the first road surface feature point and the second optical flow value corresponding to the first road surface feature point to obtain the second optical flow value after deletion. Next, the electronic device can determine each second pixel point that matches the corresponding first pixel point in the second image based on each second optical flow value after deletion

[0089] For example, in Figure 7Among them, the electronic device can perform semantic segmentation on the image 701 (an example of the first image mentioned in this application), so as to identify the road feature points and non-road feature points in the image 701 (such as vehicle feature points, building feature points, etc.), and then delete the non-road feature points and the corresponding second optical flow values of the non-road feature points. Among them, after the electronic device identifies the first road feature point A through the semantic segmentation algorithm, it can also determine whether all the points in the preset surrounding area 702 where the first road feature point A is located are road feature points. If there are non-road feature points in the preset surrounding area 702, it is also necessary to delete the first road feature point A and the second optical flow value corresponding to the first road feature point A. In this way, the electronic device can determine the corresponding second pixel points in the second image that match each first pixel point based on the deleted second optical flow values.

[0090] In this way, the electronic device can only perform tracking and matching on road feature points, thereby improving the matching accuracy and the calculation accuracy of the camera module parameters.

[0091] S606: Determine the first external parameter of the camera module based on the reprojection errors of each first pixel point and the corresponding second pixel points under different external parameters.

[0092] In some embodiments, referring to the formulas (1) to (6) above, after determining the matching pixel points, the electronic device can continuously adjust the external parameters of the camera module to obtain the corresponding H matrix. Among them, each time the external parameters of the camera module are adjusted, the electronic device can determine a set of reprojection errors through p2 - H×p1 (p1 is the coordinate of each first pixel point, and p2 is the coordinate of the corresponding second pixel point). Then, if the average value of the absolute values of the first set of reprojection errors is less than the preset threshold or this average value is the smallest, the electronic device can use the external parameter corresponding to the first set of reprojection errors as the target external parameter of the camera module.

[0093] In addition, in some other embodiments, referring to the formulas (1) to (6) above, it can be seen that the electronic device can be based on the rotation matrix R vcs and the translation vector S vcs in the vehicle odo information corresponding to each frame of image captured by the camera module, calculate the reprojection errors of each pair of matching pixel points in multiple pairs of two frames of images under different external parameters. Among them, due to the vehicle being in a bumpy, jittery, swaying, etc. situation, the obtained rotation matrix R vcs and the translation vector S vcsThe error is relatively large, which in turn makes the determined reprojection error and the error of the external parameters of the camera module relatively large. Therefore, if the yaw angular velocity in the odo information corresponding to the vehicle is greater than the angular velocity threshold or the pitch angle is greater than the angle threshold during the time period when two frames of images are captured, the electronic device can determine that the vehicle is in a slope, rapid acceleration, rapid deceleration, etc. stage at this time. The electronic device can discard the two frames of images corresponding to the odo information and re-determine the external parameters of the camera module based on other two frames of images. That is to say, when the electronic device calculates the reprojection errors of each first pixel point and the matching second pixel points under different external parameters based on the vehicle odo information corresponding to the time period when the camera module captures the first image and the second image, the yaw angular velocity in the odo information needs to be less than the angular velocity threshold, and the pitch angle in the vehicle odo information needs to be less than the angle threshold. In this way, the accuracy of the determined external parameters of the camera module can be higher.

[0094] In some other embodiments, the electronic device can also use the random sample consensus (Ransac) algorithm to eliminate the incorrect tracking points and their corresponding pixel points. That is, the electronic device can delete the incorrect optical flow values in each second optical flow value based on the Ransac algorithm, and delete the first pixel point and the second pixel point corresponding to the incorrect optical flow value in the first image and the second image. In this way, the electronic device can determine the first external parameter of the camera module based on the reprojection errors of each first pixel point and the matching second pixel points under different external parameters in the modified first image and the modified second image.

[0095] Among them, the Ransac algorithm is an iterative method that repeats multiple operations. Specifically, each iteration process can include the following steps: (1) The electronic device can randomly select multiple pairs of matching pixel points from each pair of matching pixel points corresponding to each second optical flow value as the reference set. (2) The electronic device can calculate the fundamental matrix from the first image mapped to the second image based on each pair of matching pixel points in the reference set. (3) The electronic device maps other first pixel points in the first image (e.g., the first pixel point A and the matching second pixel point is A') to the second image through the fundamental matrix. If the coordinates after mapping are significantly different from the coordinates of the matching pixel points obtained by tracking through the second optical flow value, e.g., the coordinates of the second pixel point A' obtained by tracking through the second optical flow value are significantly different from the coordinates of the first pixel point A after mapping through the fundamental matrix, the electronic device can delete the incorrect second optical flow value and the corresponding first pixel point and second pixel point. (4) After deleting each incorrect optical flow value, an iteration process can be completed, and the number of remaining pixel points is recorded. Similarly, the electronic device can repeatedly execute the above processes (1) to (4), and if the number of remaining pixel points is the largest or greater than the number threshold after a certain iteration operation, the operation result corresponding to this operation can be used as the final result.

[0096] In some other embodiments, if there are too few optical flow points remaining after removing the incorrect optical flow, it is considered that this frame of image has no reference significance and can be directly discarded, and the external parameters of the camera module are re-determined based on other images.

[0097] In this way, the above parameter calibration method for calibrating the external parameters of the camera module enables the electronic device to determine the external parameters of the camera module only through the camera module and vehicle odo information without other sensors. Moreover, the above parameter calibration method is applicable to the driving scenario, saving computing resources and having a high accuracy, and also having a good adaptability to the bumps, jitters, and swings of the vehicle.

[0098] Next, based on Figure 8 the following flow schematic diagram, the parameter calibration method mentioned in the embodiments of the present application will be introduced in detail. Among them, this parameter calibration method can be applied to an electronic device, such as any electronic device mentioned above, such as an in-vehicle computer. As Figure 8 shown, specifically, the method is as follows:

[0099] S801: Obtain the inter-frame optical flow based on two adjacent frames of images.

[0100] In some embodiments, two adjacent frames of images may be a first image and a second image. Obtaining the inter-frame optical flow may include: inputting the first image and the second image into an NN optical flow model to obtain the dense optical flow corresponding to the first image; performing a downsampling operation on the dense optical flow to obtain a sparse optical flow; and adjusting each first optical flow value in the sparse optical flow to a second optical flow value that matches the resolution of the first image. The specific process of obtaining the inter-frame optical flow may refer to S602 to S604 described above, and will not be elaborated here.

[0101] S802: Based on a semantic segmentation algorithm, eliminate the non-road surface feature points and corresponding optical flow of the first image.

[0102] In some embodiments, when the vehicle is driving, it may overtake or pass pedestrians, buildings, etc., resulting in non-road surface feature points such as vehicles, pedestrians, and buildings not being able to appear in the first image and the second image captured of the front at the same time. Therefore, the electronic device can only match the road surface feature points in the first image. Specifically, the electronic device can perform semantic segmentation on the first image to identify the road surface feature points and non-road surface feature points (such as vehicle feature points, building feature points, etc.) in the first image, and delete the identified non-road surface feature points and the second optical flow values corresponding to the non-road surface feature points.

[0103] In addition, in some other embodiments, if there are non-road surface feature points in the preset surrounding area of a certain first road surface feature point identified by the semantic segmentation algorithm, it is also necessary to delete the first road surface feature point and the second optical flow value corresponding to the first road surface feature point to obtain the deleted second optical flow value.

[0104] S803: Based on the inter-frame optical flow, perform optical flow tracking on the inter-frame pixel points, and eliminate the incorrect optical flow and its corresponding pixel points.

[0105] In some embodiments, after the electronic device eliminates some optical flow through S802, the electronic device can determine the second pixel points in the second image that match the corresponding first pixel points based on the remaining second optical flow values.

[0106] In some embodiments, the electronic device can also delete the incorrect optical flow values in each second optical flow value based on the Ransac algorithm, and delete the first pixel points and second pixel points corresponding to the incorrect optical flow value in the first image and the second image.

[0107] S804: Estimate the H matrix by adjusting the reprojection error to optimize the external parameters of the camera module.

[0108] In some embodiments, as shown in the above formulas (1) to (6), after determining the matching pixel points, the electronic device can continuously adjust the external parameters of the camera module to obtain the corresponding H matrix. Among them, each time the external parameters of the camera module are adjusted, the electronic device can determine a set of reprojection errors through p2 - H×p1 (where p1 is the coordinate of each first pixel point and p2 is the coordinate of the matching second pixel point). Then, if the average value of the absolute values of the first set of reprojection errors is less than a preset threshold or this average value is the smallest, the electronic device can use the external parameters corresponding to the first set of reprojection errors as the optimized external parameters of the camera module.

[0109] In other embodiments, if at the moment of taking a certain image, the yaw angular velocity in the odo information corresponding to the vehicle is greater than the angular velocity threshold or the pitch angle is greater than the angle threshold, the electronic device can determine that the vehicle is in stages such as on a slope, accelerating rapidly, or decelerating rapidly at this time, and the electronic device can discard the two frames of images corresponding to the odo information and re - determine the external parameters of the camera module based on other two frames of images. That is to say, when the electronic device calculates the reprojection errors of each first pixel point and the matching second pixel points under different external parameters based on the vehicle odo information corresponding to the time period when the camera module captures the first image and the second image, the yaw angular velocity in the odo information needs to be less than the angular velocity threshold, and the pitch angle in the vehicle odo information needs to be less than the angle threshold.

[0110] S805: Perform histogram filtering or convergence judgment on the optimization results of multiple camera module external parameters.

[0111] In some embodiments, the above S801 - S804 can also be repeated multiple times to obtain multiple optimization results of the camera module external parameters. Then, the electronic device can observe the distribution frequency of each optimization result among the above - mentioned multiple optimization results through methods such as histogram or convergence, and use the optimization result corresponding to the maximum distribution frequency as the best camera module external parameters. In this way, through multiple operations, the obtained best module external parameters can be made more accurate, avoiding contingency.

[0112] S806: Determine the best camera module external parameters.

[0113] In some embodiments, after the electronic device observes the distribution frequency of each optimization result among multiple optimization results through methods such as histogram or convergence in the above S806, it can use the optimization result corresponding to the maximum distribution frequency as the best camera module external parameters (i.e., the first external parameters).

[0114] Thus, the above-mentioned parameter calibration method for calibrating the external parameters of the camera module enables the electronic device to determine the external parameters of the camera module only through the camera module and vehicle odo information, without the need for other sensors. Moreover, the above-mentioned parameter calibration method is applicable to driving scenarios, saving computing resources and having a high accuracy, as well as having a good adaptability to the bumps, jitters, and swaying of the vehicle.

[0115] In some embodiments, the present application further provides a computer program product, where the computer program product includes computer instructions. When the computer instructions run on the electronic device, the electronic device is caused to execute the parameter calibration method mentioned in the present application.

[0116] In other embodiments, the present application further provides an electronic device, which includes a memory and a processor, and the memory is coupled to the processor. Among them, the memory is used to store computer program code / instructions, and when the computer program code / instructions are executed by the processor, the electronic device can be caused to execute the parameter calibration method mentioned in the present application.

[0117] In addition, the present application further provides a vehicle, which includes the above-mentioned electronic device.

[0118] Next, in conjunction with Figure 9 as shown, an exemplary schematic diagram of the hardware structure of the electronic device 1200 according to an embodiment of the present application is described. As Figure 9 shown, the electronic device 1200 may include one or more processors 1202, a system control logic 1201 connected to at least one of the processors 1202, a system memory 1205 connected to the system control logic 1201, a memory 1203 connected to the system control logic 1201, and a network interface 1208 connected to the system control logic 1201.

[0119] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a limitation on the only implementable manner of the electronic device 1200. In other embodiments of the present application, the electronic device 1200 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0120] The processor 1202 may include one or more single-core or multi-core processors. In some embodiments, the processor 1202 may include any combination of a general-purpose processor and a dedicated processor (e.g., an application processor, a baseband processor, etc.). It can be understood that in the embodiments of the present application, the processor 1202 may be configured to execute the executable instructions 1204 stored in the memory 1203 to implement the parameter calibration method of the embodiments of the present application. When at least one of the processors 1202 executes the instructions, the electronic device 1200 implements the parameter calibration method of the embodiments of the present application.

[0121] The system control logic 1201 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1202 and / or any suitable device or component communicating with the system control logic 1201. The system control logic 1201 may include one or more memory controllers to provide an interface connected to the system memory 1205. The system memory 1205 may be used to load and store data and / or instructions. In some embodiments, the system memory 1205 of the electronic device 1200 may include any suitable volatile memory, such as a suitable dynamic random access memory.

[0122] The memory 1203 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the memory 1203 may include any suitable volatile memory and / or any suitable non-volatile storage device. For example, the memory 1203 may include: a random access memory (RAM) and / or a cache storage unit, and may further include a read-only memory (ROM).

[0123] The memory 1203 may include a part of the storage resources installed on the device of the electronic device 1200, or it may be accessible by the device but not necessarily part of the device. For example, the memory 1203 may be accessed via the network interface 1208 through a network.

[0124] In particular, the system memory 1205 and the memory 1203 may respectively include: a temporary copy and a permanent copy of the instructions 1206 and a temporary copy and a permanent copy of the instructions 1204. The instructions 1206 may include: when executed by at least one of the processors 1202, causing the electronic device 1200 to implement the parameter calibration method of the embodiments of the present application. In some embodiments, the instructions 1206, hardware, firmware, and / or its software components may alternatively be disposed in the system control logic 1201, the network interface 1208, and / or the processor 1202.

[0125] The network interface 1208 may include a transceiver for providing a radio interface for the electronic device 1200, and further for communicating with any other suitable devices (such as a front-end module, an antenna, etc.) via one or more networks. In some embodiments, the network interface 1208 may be integrated with other components of the electronic device 1200. For example, the network interface 1208 may be integrated with at least one of the processor 1202, the system memory 1205, the memory 1203, and a firmware device (not shown) having instructions.

[0126] The network interface 1208 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1208 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0127] The electronic device 1200 may further include: an input / output (I / O) device 1207. The I / O device 1207 may include a user interface that enables a user to interact with the electronic device 1200; the design of the peripheral component interface enables peripheral components to also interact with the electronic device 1200. In some embodiments, the electronic device 1200 further includes sensors for determining at least one of environmental conditions and location information related to the electronic device 1200.

[0128] In some embodiments, the user interface may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash), and a keyboard.

[0129] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0130] In some embodiments, the sensors may include, but are not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be a part of the network interface 1208 or interact with the network interface 1208 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0131] The embodiments disclosed in the present application may be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memories and / or storage elements), at least one input device, and at least one output device.

[0132] Program code can be applied to the input instructions to perform the various functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0133] The program code can be implemented in a high-level procedural language or an object-oriented programming language in order to communicate with the processing system. When needed, the program code can also be implemented in assembly language or machine language. In fact, the mechanisms described in this application are not limited to the scope of any particular programming language. In any case, the language can be a compiled language or an interpreted language.

[0134] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions can be distributed via a network or via other computer-readable media. Thus, a machine-readable medium can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to, a floppy disk, a compact disc, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), a magnetic card or an optical card, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) in electrical, optical, acoustic, or other forms using the Internet. Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0135] In the drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or ordering may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Additionally, the inclusion of a structural or method feature in a particular figure does not imply that such a feature is required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0136] It should be noted that each unit / module mentioned in the device embodiments of the present application is a logical unit / module. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or can be implemented as a combination of multiple physical units / module. The physical implementation manner of these logical units / modules themselves is not the most important. The combination of the functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above device embodiments.

[0137] It should be noted that in the examples and descriptions of the present application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one" does not exclude the existence of another identical element in the process, method, article or device including the said element.

[0138] Although the present application has been illustrated and described by referring to some preferred embodiments of the present application, those of ordinary skill in the art should understand that various changes can be made in form and detail without departing from the scope of the present application.

Claims

1. A parameter calibration method, applied to electronic equipment, characterized in that: The method comprises: Acquire a first image and a second image captured by the camera module; Inputting the first image and the second image into a neural network optical flow model to obtain a dense optical flow corresponding to the first image, wherein each optical flow in the dense optical flow is used to indicate a motion state of a corresponding pixel point between the first image and the second image; Performing a downsampling operation on the dense optical flow to obtain a sparse optical flow; Adjusting a first optical flow value corresponding to each optical flow in the sparse optical flow to a second optical flow value matching a resolution of the first image, wherein each second optical flow value corresponds to a plurality of first pixel points in the first image; Based on the second optical flow values, determining second pixel points in the second image that match the corresponding first pixel points; Based on the reprojection errors of each first pixel point and each matching second pixel point under different external parameters, a first external parameter of the camera module is determined.

2. The method according to claim 1, characterized in that The determining of the first extrinsic parameter of the camera module based on the reprojection errors of each first pixel point and each matching second pixel point under different extrinsic parameters includes: When the camera module is under different external parameters, respectively calculating the reprojection errors corresponding to each first pixel point and each matching second pixel point to obtain multiple groups of reprojection errors, wherein the multiple groups of reprojection errors include the first group of reprojection errors; When an average of absolute values ​​of the reprojection errors in the first group of reprojection errors is less than a preset threshold, the extrinsic parameter corresponding to the first group of reprojection errors is determined to be the first extrinsic parameter.

3. The method according to claim 2, characterized in that The method further comprises: Determine a plurality of first extrinsic parameters of the camera module based on reprojection errors of matching pixel points in a plurality of groups of two frames of images under different extrinsic parameters; Determining the distribution frequency of each first external parameter among the plurality of first external parameters by means of a histogram; The external parameter of the camera module is adjusted to be a first external parameter corresponding to the maximum distribution frequency.

4. The method according to claim 1, characterized in that: The method further comprises: Based on the odometer information of the vehicle corresponding to the time period when the camera module takes the first image and the second image, the reprojection error of each first pixel point and each matching second pixel point under different external parameters is calculated; wherein, The yaw angle velocity in the odometer information is less than an angular velocity threshold, and the pitch angle in the odometer information is less than an angle threshold.

5. The method according to claim 1, characterized in that The first image is the area where the road surface is located in the third image taken by the camera module, and the second image is the area where the road surface is located in the fourth image taken by the camera module.

6. The method according to claim 1, characterized in that The performing a downsampling operation on the dense optical flow to obtain a sparse optical flow includes: Calculating the displacement corresponding to each optical flow in the dense optical flow; In the dense optical flow, the optical flow whose displacement is less than a preset displacement threshold is deleted to obtain a dense optical flow after deletion; A downsampling operation is performed on the deleted dense optical flow to obtain the sparse optical flow.

7. The method according to claim 6, characterized in that The determining, based on the second optical flow values, second pixel points that match the corresponding first pixel points in the second image includes: Identifying road feature points and non-road feature points in the first image based on semantic segmentation; Deleting the non-road feature points identified by semantic segmentation and the second optical flow values ​​corresponding to the non-road feature points; If there are non-road feature points in the preset surrounding area of ​​the identified first road feature point, deleting the first road feature point and the second optical flow value corresponding to the first road feature point to obtain the deleted second optical flow value; Based on the deleted second optical flow values, second pixel points that match the corresponding first pixel points in the second image are determined.

8. The method according to claim 7, characterized in that The determining of the first extrinsic parameter of the camera module based on the reprojection errors of each first pixel point and each matching second pixel point under different extrinsic parameters includes: Based on the random sampling consistency algorithm, the erroneous optical flow values ​​in each second optical flow value are deleted; In the first image and the second image, deleting the first pixel point and the second pixel point corresponding to the erroneous optical flow value to obtain a modified first image and a modified second image; In the modified first image and the modified second image, a first extrinsic parameter of the camera module is determined based on a reprojection error between each first pixel point and each matching second pixel point under different extrinsic parameters.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is coupled to the processor; the memory is used to store computer program codes / instructions; when the computer program codes / instructions are executed by the processor, the electronic device executes the parameter calibration method described in any one of claims 1 to 8.

10. A vehicle, characterized in that: The vehicle includes the electronic device according to claim 9.

Images