A camera posture automatic adjustment method and system

Abstract

The present invention provides a method and system for automatically adjusting the position and posture of a camera, which belongs to the field of computer vision technology, and includes: using a deep learning algorithm and a traditional machine learning algorithm to obtain matching points between a camera-taken picture and a reference picture, and filtering out erroneous matching points; obtaining camera internal parameters through camera calibration, and using epipolar geometry and triangulation to obtain the three-dimensional coordinates of matching points; using a direct linear transformation algorithm to obtain the position and posture of the camera, thereby adjusting the camera posture; and finally using an identifier on the picture to adjust the error so that the error is within a controllable range. The present invention adopts a camera adaptive adjustment algorithm, which is simple to implement, not easily affected by the external environment, and has high execution efficiency, and can greatly improve the accuracy of camera calibration.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for automatically adjusting camera posture. Background Art

[0002] In the process of hydrological monitoring through video imaging, the camera is often affected by complex environmental factors and causes jitter, such as geological vibration, strong wind and other special situations. As a result, the captured images are mismatched in the matching of the same-name points in consecutive frames, resulting in errors in the final calculation.

[0003] Therefore, an automatic adjustment method of camera pose that is not affected by the environment is proposed to achieve accurate calculation of hydrological monitoring. Summary of the invention

[0004] The present invention provides a method and system for automatically adjusting the position and posture of a camera, which are used to solve the defects existing in the camera calibration in water level monitoring in the prior art.

[0005] In a first aspect, the present invention provides a method for automatically adjusting a camera posture, comprising:

[0006] Obtain a reference image and a captured image of the camera to be calibrated, and use a deep learning algorithm and a feature matching algorithm to extract initial feature matching points of the reference image and the captured image;

[0007] Using an outlier removal algorithm to remove outlier matching points from the initial feature matching points to obtain accurate matching points;

[0008] The RAC-based calibration method is used to obtain the intrinsic parameters of the camera to be calibrated;

[0009] Based on the intrinsic parameters of the camera to be calibrated, the camera motion is restored by solving the epipolar geometry to obtain the initial pose change of the camera to be calibrated, and the two-dimensional coordinates and three-dimensional coordinates of the precise matching points are obtained by using the triangulation method;

[0010] Using the DLT algorithm, the two-dimensional coordinates and the three-dimensional coordinates are calculated to obtain the camera pose;

[0011] The camera to be calibrated is adjusted using the camera posture, and the deviation value of the corresponding identifier in the captured image and the reference image is calculated. If the deviation value is within a preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed.

[0012] According to a camera posture automatic adjustment method provided by the present invention, a deep learning algorithm and a feature matching algorithm are used to extract initial feature matching points of the reference image and the collected image, including:

[0013] Using a VGG convolutional neural network to extract features from the reference image and the collected image, respectively, to obtain a reference feature map and a collected feature map;

[0014] A deep nearest neighbor search (DNNS) algorithm is used to extract matching points from the benchmark feature map and the acquisition feature map, the acquisition feature map is searched, the L2 distance between each element in the benchmark feature map and the acquisition feature map is obtained, and the initial feature matching point is determined according to the ratio of the best match to the second best match corresponding to the L2 distance.

[0015] According to a camera posture automatic adjustment method provided by the present invention, an abnormal point removal algorithm is used to remove abnormal matching points in the initial feature matching points to obtain accurate matching points, including:

[0016] Calculating the x-axis deviation value, y-axis deviation value, x-axis deflection angle, and y-axis deflection angle of the initial feature matching point, and removing the initial feature matching points that are greater than the deviation value threshold or greater than the deflection angle threshold;

[0017] Calculate the mean and standard deviation corresponding to the x-axis deviation value, y-axis deviation value, x-axis deflection angle, and y-axis deflection angle respectively, determine the minimum value by subtracting 3 times the standard deviation from the mean, determine the maximum value by adding 3 times the standard deviation to the mean, remove feature matching points that are less than the minimum value or greater than the maximum value, and obtain the precise matching point.

[0018] According to a camera posture automatic adjustment method provided by the present invention, a RAC calibration method is used to obtain the intrinsic parameters of the camera to be calibrated, including:

[0019] Get the conversion relationship between the camera coordinate system and the world coordinate system:

[0020]

[0021]

[0022] Where (x, y) is the pixel coordinate in the camera, (X, Y, Z) is the real world coordinate, K is the camera intrinsic parameter matrix, (Rt) is the extrinsic parameter matrix, R is the rotation matrix describing the direction of the camera, t is the three-dimensional translation vector describing the center position of the camera, f is the focal length, s is the distortion coefficient, and (u0, v0) is the coordinate of the center point of the image;

[0023] Solve the transformation from ideal image coordinates to digital image coordinates based on radial deviation:

[0024]

[0025] Among them, (u, v) represents the coordinates of any point in the image coordinate system, (x, y) is the pixel coordinates in the camera after distortion correction, (u c,v c ) represents the coordinates of the distortion center, s1 is the first-order distortion coefficient;

[0026] Perform radial consistency constraints. If there exists (u c ,v c ), (u,v) and (x,y) are collinear, then:

[0027]

[0028] Taking the center point of the image (u0, v0) as the distortion center coordinate, we get:

[0029]

[0030] Solve the camera's external parameter rotation matrix R and the translation vectors t1 and t2 in the x-axis and y-axis directions, expand R and t, and get:

[0031]

[0032]

[0033]

[0034] From the radial consistency constraint we get:

[0035]

[0036] From at least 7 sets of exact matching points, we get:

[0037] M0=(m1,m2,m3,m4,m5,m6,m7,m8)≈(sr1,sr2,sr3,st1,r4,r5,r6,t2);

[0038] M0 divided by Get a set of solutions (sr1, sr2, sr3, st1, r4, r5, r6, t2);

[0039] Depend on Get s and t1;

[0040] (r7, r8, r9) = (r1, r2, r3) × (r4, r5, r6) or (r7, r8, r9) = (r4, r5, r6) × (r1, r2, r3);

[0041] Let s = 0 as the initial value, then the expressions of x and y are:

[0042] x-u0=u-u0,y-v0=v-v0

[0043] Substituting the obtained values ​​of R, t1 and t2 into the equation:

[0044] (u-u0)(r7X+r8Y+r9Z+t3)=fs(r1X+r2Y+r3Z+t1)

[0045] (v-v0)(r7X+r8Y+r9Z+t3)=f(r4X+r5Y+r6Z+t2)

[0046] Find f and t3;

[0047] Taking f, t3 and s=0 as initial values, perform linear optimization on the following formula to obtain the true values ​​of f, t3 and s:

[0048]

[0049] According to a camera posture automatic adjustment method provided by the present invention, based on the intrinsic parameters of the camera to be calibrated, the camera motion is restored by solving the epipolar geometry to obtain the initial posture change of the camera to be calibrated, and the two-dimensional coordinates and three-dimensional coordinates of the precise matching points are obtained by using the triangulation method, including:

[0050] Determine the augmented matrix [R|t] based on the rotation matrix R and the three-dimensional translation vector t:

[0051]

[0052] The two-dimensional coordinates and three-dimensional coordinates of the exact matching point are obtained by conversion:

[0053]

[0054] Among them, (X, Y, Z) are three-dimensional coordinates and (u1, v1) are two-dimensional coordinates.

[0055] According to a camera posture automatic adjustment method provided by the present invention, a DLT algorithm is used to calculate the two-dimensional coordinates and the three-dimensional coordinates to obtain the camera posture, including:

[0056] From the two-dimensional coordinates and the three-dimensional coordinates, we get:

[0057]

[0058] Define the row vector of T as follows:

[0059] t1=(t1,t2,t3,t4) T ,t2=(t5,t6,t7,t8) T ,t3=(t9,t 10 ,t 11 ,t 12 ) T

[0060] Substituting the row vector of T into the equation:

[0061]

[0062] Based on N feature points, we get the linear equation system:

[0063]

[0064] Solve the linear equations to obtain the camera pose.

[0065] According to a camera posture automatic adjustment method provided by the present invention, the camera to be calibrated is adjusted using the camera posture, and the deviation value of the corresponding identifier in the captured image and the reference image is calculated. If the deviation value is within a preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed, including:

[0066] Calculating the gradient strength and gradient direction of each pixel in the acquired image and the reference image, clustering the gradient strength and the gradient direction, and clustering pixels that may form a straight line;

[0067] Using the weighted least square method, using the gradient strength as the weight of each point in the straight line, fitting the straight line equation to obtain the detection line segment;

[0068] The detected line segments are grouped, wherein the distance between the end point of any edge and the start point of any other edge is less than a preset distance threshold, and the connected line segments are in a counterclockwise direction, and the directed line segments and candidate line segments in the line segment tree formed after grouping are used to obtain the identifier quadrilateral structure feature;

[0069] Detecting the corner points of the quadrilateral structural features of the identifier, extracting the brightness information in the squares around the corner points, encoding the brightness information, mapping it to a label to obtain a displacement identifier ID, and obtaining an identifier matching point from the corner points and the displacement identifier ID;

[0070] Based on the identifier matching points, the identifiers in the collected image and the reference image are respectively calibrated, and deviation values ​​of the identifiers are calculated;

[0071] If it is determined that the deviation value is less than the preset deviation threshold, the calibration adjustment process is terminated, otherwise the calibration adjustment process is executed again.

[0072] In a second aspect, the present invention further provides a camera posture automatic adjustment system, comprising:

[0073] A matching module is used to obtain a reference image and a captured image of the camera to be calibrated, and to extract initial feature matching points of the reference image and the captured image using a deep learning algorithm and a feature matching algorithm;

[0074] A removal module, used to remove abnormal matching points from the initial feature matching points by using an abnormal point removal algorithm to obtain accurate matching points;

[0075] The calibration module is used to obtain the intrinsic parameters of the camera to be calibrated using the RAC calibration method;

[0076] A measurement module, for restoring the camera motion by solving the epipolar geometry based on the intrinsic parameters of the camera to be calibrated, obtaining the initial posture change of the camera to be calibrated, and obtaining the two-dimensional coordinates and three-dimensional coordinates of the precise matching points by using a triangulation method;

[0077] A calculation module, used to calculate the two-dimensional coordinates and the three-dimensional coordinates by using a DLT algorithm to obtain a camera pose;

[0078] The adjustment module is used to adjust the camera to be calibrated using the camera posture, calculate the deviation value of the corresponding identifier in the acquired image and the reference image, and terminate the calibration adjustment process if the deviation value is within a preset deviation range, otherwise re-execute the calibration adjustment process.

[0079] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for automatically adjusting the camera posture as described above is implemented.

[0080] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for automatically adjusting the camera posture.

[0081] The camera posture automatic adjustment method and system provided by the present invention adopt a camera adaptive adjustment algorithm, which is simple to implement, not easily affected by the external environment, has high execution efficiency, and can greatly improve the accuracy of camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0083] Figure 1 It is one of the flow charts of the method for automatic adjustment of camera posture provided by the present invention;

[0084] Figure 2This is the second flow chart of the method for automatically adjusting the camera posture provided by the present invention;

[0085] Figure 3 It is a structural schematic diagram of the camera posture automatic adjustment system provided by the present invention;

[0086] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Figure 1 is one of the flow charts of the method for automatically adjusting the camera posture provided by an embodiment of the present invention, such as Figure 1 As shown, including:

[0089] Step 100: Obtain a reference image and a captured image of the camera to be calibrated, and use a deep learning algorithm and a feature matching algorithm to extract initial feature matching points of the reference image and the captured image;

[0090] Step 200: using an outlier removal algorithm to remove outlier matching points from the initial feature matching points to obtain accurate matching points;

[0091] Step 300: Acquire the intrinsic parameters of the camera to be calibrated using a RAC-based calibration method;

[0092] Step 400: Based on the intrinsic parameters of the camera to be calibrated, the camera motion is restored by solving the epipolar geometry to obtain the initial pose change of the camera to be calibrated, and the two-dimensional coordinates and three-dimensional coordinates of the precise matching points are obtained by using the triangulation method;

[0093] Step 500: using the DLT algorithm to calculate the two-dimensional coordinates and the three-dimensional coordinates to obtain the camera pose;

[0094] Step 600: Use the camera posture to adjust the camera to be calibrated, calculate the deviation value of the corresponding identifier in the captured image and the reference image, and if the deviation value is within a preset deviation range, end the calibration adjustment process, otherwise re-execute the calibration adjustment process.

[0095] Specifically, the camera posture automatic adjustment method proposed in the embodiment of the present invention has the following overall logic flow:

[0096] First, a reference image is defined as a reference. The camera to be calibrated will collect a certain number of images for comparison. The deep learning method is used to extract the features of the image to be matched, so as to further match the matching feature points, and the outlier removal algorithm is used to eliminate the feature points that are mismatched. Then, the radial alignment constraint (RAC) calibration method is used to calibrate the camera to obtain the camera internal parameters. The three-dimensional coordinates of the matching points are obtained by using epipolar geometry and triangulation. Based on the obtained three-dimensional coordinates, the direct linear transformation (DLT) algorithm is used to calculate the above three-dimensional coordinates, so as to obtain the camera pose and adjust the camera. Finally, for the identifiers corresponding to the collected image and the reference image, the coordinates of the identifiers in their respective images are calibrated, and the irregular quadrilaterals in the image are obtained by using the traditional machine learning method. The irregular quadrilaterals are further classified by the deep learning method, so as to filter out the identifiers that do not meet the conditions, and calculate the deviations between the remaining identifiers that meet the conditions. If the deviation is less than the set threshold, it is considered that the calibration has met the requirements and the entire calibration adjustment process is stopped. Otherwise, the calibration adjustment process is re-entered until the deviation requirements are met.

[0097] The present invention adopts a camera adaptive adjustment algorithm, which has the advantages of simple implementation, not being easily affected by the external environment, high execution efficiency, and being able to greatly improve the accuracy of camera calibration.

[0098] Based on the above embodiment, the deep learning algorithm and the feature matching algorithm are used to extract the initial feature matching points of the reference image and the collected image, including:

[0099] Using a VGG convolutional neural network to extract features from the reference image and the collected image, respectively, to obtain a reference feature map and a collected feature map;

[0100] A deep nearest neighbor search (DNNS) algorithm is used to extract matching points from the benchmark feature map and the acquisition feature map, the acquisition feature map is searched, the L2 distance between each element in the benchmark feature map and the acquisition feature map is obtained, and the initial feature matching point is determined according to the ratio of the best match to the second best match corresponding to the L2 distance.

[0101] Specifically, in the embodiment of the present invention, the reference image to be calibrated is set as imageA, and the camera to be calibrated performs image acquisition, and the acquired image is imageB.

[0102] The VGG convolutional neural network is used to extract features from the calibration and camera captured images, and the feature maps FA and FB corresponding to imageA and imageB are obtained respectively. The subsequent feature point matching is to directly calculate the feature maps FA and FB.

[0103] For the calculation of matching feature points, the embodiment of the present invention uses a deep nearest neighbor search (DNNS) algorithm to extract the best matching point from the feature map. DNNS searches FB to find the best match for each element of FA. Potential matches are defined as the nearest neighbors represented by L2 distance. L2 distance generally refers to the Euclidean distance between two points in m-dimensional space, also known as Euclidean distance. Here, for a point pA in the feature map FA, if the ratio of the distance to the best match pB and the second best match is lower than a given threshold, then the points pA and pB are considered to be matched.

[0104] It should be noted that the embodiment of the present invention also performs hierarchical optimization on each layer of feature maps output by the VGG network to improve the accuracy and robustness of matching.

[0105] Based on the above embodiment, an outlier removal algorithm is used to remove outlier matching points in the initial feature matching points to obtain accurate matching points, including:

[0106] Calculating the x-axis deviation value, y-axis deviation value, x-axis deflection angle, and y-axis deflection angle of the initial feature matching point, and removing the initial feature matching points that are greater than the deviation value threshold or greater than the deflection angle threshold;

[0107] Calculate the mean and standard deviation corresponding to the x-axis deviation value, y-axis deviation value, x-axis deflection angle, and y-axis deflection angle respectively, determine the minimum value by subtracting 3 times the standard deviation from the mean, determine the maximum value by adding 3 times the standard deviation to the mean, remove feature matching points that are less than the minimum value or greater than the maximum value, and obtain the precise matching point.

[0108] Specifically, since the obtained feature matching points may have incorrect matching, which will affect the calculation of the camera intrinsic parameters and cause errors in the camera pose, the outlier processing method is used to eliminate the incorrect matching points and obtain the final accurate matching points.

[0109] In the embodiment of the present invention, the deviation values ​​of the matching points in the x-axis direction and the y-axis direction and the angle formed are first calculated as dx, dy and angle respectively, and thresholds are set respectively. Matching points greater than the threshold conditions are regarded as abnormal matching points and filtered out.

[0110] Then, the mean and variance of dx, dy and angle are calculated respectively, and the minimum and maximum values ​​are calculated by formula (1). The matching points that do not fall within the range are regarded as abnormal matching points and filtered out.

[0111]

[0112] Among them, min is the minimum value, max is the maximum value, mean is the mean, and std is the standard deviation.

[0113] Based on the above embodiment, the RAC calibration method is used to obtain the intrinsic parameters of the camera to be calibrated, including:

[0114] Specifically, first determine the conversion relationship between the camera coordinate system and the world coordinate system:

[0115]

[0116]

[0117] Among them, (x, y) is the pixel coordinate in the camera, (X, Y, Z) is the real world coordinate, K is the camera intrinsic parameter matrix, (Rt) is the extrinsic parameter matrix, R is the rotation matrix describing the direction of the camera, t is the three-dimensional translation vector describing the center position of the camera, f is the focal length, s is the distortion coefficient, and (u0, v0) is the coordinate of the center point of the picture.

[0118] The ideal image coordinates are transformed into digital image coordinates (only radial deviation is considered in the embodiment of the present invention). The radial distortion satisfies the following equation, and the radial distortion coefficient only takes the first order:

[0119]

[0120] Among them, (u,v) represents the coordinates of any point in the image coordinate system, (x,y) is the pixel coordinates in the camera after distortion correction, (u2,v c ) represents the coordinates of the distortion center, s1 is the first-order distortion coefficient, and the distortion caused by camera shooting is corrected by subtracting the coefficient.

[0121] Then, radial consistency constraints are performed. If there exists (u c ,v c ), (u,v) and (x,y) are collinear, then:

[0122]

[0123] Taking the center point of the image (u0, v0) as the distortion center coordinate, we get:

[0124]

[0125] Solve the camera's external parameter rotation matrix R and the translation vectors t1 and t2 in the x-axis and y-axis directions, expand R and t, and get:

[0126]

[0127]

[0128]

[0129] Then from the previous radial consistency constraint we get:

[0130]

[0131] From at least 7 sets of matching points on the reference image and the collected image, a set of solutions can be obtained:

[0132] M0=(m1,m2,m3,m4,m5,m6,m7,m8)≈(sr1,sr2,sr3,st1,r4,r5,r6,t2);

[0133] M0 divided by Get a set of solutions (sr1, sr2, sr3, st1, r4, r5, r6, t2);

[0134] Depend on Get s and t1;

[0135] (r7, r8, r9) = (r1, r2, r3) × (r4, r5, r6) or (r7, r8, r9) = (r4, r5, r6) × (r1, r2, r3)

[0136] Further solve the effective focal length f, the translation t3 in the z direction and the distortion coefficient s:

[0137] Let s = 0 as the initial value, then the expressions of x and y are:

[0138] x-u0=u-u0,y-v0=v-v0 (11) Then, from the expressions of x and y, we can substitute the values ​​of R, t1, and t2 obtained in the first step into the equation:

[0139]

[0140] Thus, f and t3 can be obtained. Taking the obtained f, t3 and s=0 as the initial values, the following equation is linearly optimized to estimate the true values ​​of t3, f and s:

[0141]

[0142] Based on the above embodiment, based on the intrinsic parameters of the camera to be calibrated, the camera motion is restored by solving the epipolar geometry to obtain the initial pose change of the camera to be calibrated, and the two-dimensional coordinates and three-dimensional coordinates of the precise matching points are obtained by using the triangulation method, including:

[0143] Specifically, the embodiment of the present invention solves the epipolar geometry to restore the camera motion, obtains the initial rotation matrix R and translation matrix t through the camera intrinsic parameter K obtained by calibration, and obtains the initial position and posture change of the camera.

[0144] The triangulation method is used to obtain the two-dimensional and three-dimensional coordinates of the corresponding matching points, in preparation for solving the direct linear transformation algorithm for optimizing the camera pose.

[0145] The 2D and 3D coordinates of the matching points are obtained through epipolar geometry and triangulation methods, and then the direct linear transformation DLT algorithm is used to solve the final required camera pose. The augmented matrix [R|t] is defined as a 3×4 matrix, and the expansion is as follows:

[0146]

[0147] The three-dimensional coordinates and two-dimensional coordinates of the matching point pair can be transformed through the rotation matrix and translation matrix, which are expanded as follows:

[0148]

[0149] Among them, (X, Y, Z) are three-dimensional coordinates and (u1, v1) are two-dimensional coordinates.

[0150] Based on the above embodiment, the DLT algorithm is used to calculate the two-dimensional coordinates and the three-dimensional coordinates to obtain the camera pose, including:

[0151] Specifically, after obtaining the 2D and 3D coordinates of the matching points, equation (15) is processed and s is eliminated using the last line to obtain:

[0152]

[0153] To simplify the representation, the row vector of T is defined as follows:

[0154] t1=(t1,t2,t3,t4) T ,t2=(t5,t6,t7,t8) T ,t3=(t9,t 10 ,t 11 ,t 12 ) T (17)

[0155] Substituting formula (17) into formula (16) yields formula (18);

[0156]

[0157] Where t is the vector to be found, and it can be seen that each feature point provides two linear constraints on t.

[0158] Assuming there are N feature points in total, list the linear equation:

[0159]

[0160] The camera pose is obtained by solving the equations and adjusting the camera pose.

[0161] Based on the above embodiment, the camera to be calibrated is adjusted using the camera posture, and the deviation value of the corresponding identifier in the captured image and the reference image is calculated. If the deviation value is within the preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed, including:

[0162] Specifically, after completing the camera posture adjustment, the embodiment of the present invention also performs deviation verification through identifiers on respective pictures.

[0163] First, the gradient strength and gradient direction of each pixel in the entire image are calculated, and then the gradient strength and gradient direction are clustered so that pixels that may form a straight line are clustered together.

[0164] After completing the clustering of all points in the image, the weighted least squares method is used to fit the straight line equation, where the weight of each point in the straight line equation is the gradient strength of the point, thereby obtaining the detection line.

[0165] After all the line segments are detected, they are grouped. It can be understood that each group of line segments is a candidate that can form a quadrilateral. The principles followed by grouping include: the distance between the end point of the previous edge and the start point of the next edge should be less than a threshold, and the connected line segments should form a counterclockwise direction. After the grouping is successful, all the line segments will form a tree. The first layer of the tree is all directed line segments, and the nodes from the second layer to the last layer are the same group of candidate line segments. The depth-first search is applied to traverse the entire tree. If the last edge and the first edge form a closed loop when the depth of the tree is 4, it means that they meet the requirements of the identifier quadrilateral, thereby obtaining the quadrilateral structure features that may form the identifier.

[0166] Furthermore, the corner points of the quadrilateral are detected, and the brightness information is extracted from the squares around each corner point. The brightness value in the square is determined by comparing the pixel values ​​in the square. The brightness information is encoded and finally mapped to the displacement identification ID of the tag. In this way, the corner points and ID information of the identifier in the image are obtained, so as to match the matching points of the reference image and the collected image.

[0167] Finally, the identifiers in the reference image imageA and the camera captured image imageB are matched to obtain the deviation value of the identifier. If it is determined that the deviation value is less than the preset deviation threshold, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed until the deviation requirement is met.

[0168] Compared with traditional water level monitoring that requires reliance on the external environment for reference calibration, the method used in the present invention to calibrate the camera and comparison pictures and then adjust the camera posture solves the problem of camera shake caused by environmental influences, thereby causing deviations in the collected pictures. It has the advantages of simple operation, low implementation cost and high execution efficiency.

[0169] The camera posture automatic adjustment system provided by the present invention is described below. The camera posture automatic adjustment system described below and the camera posture automatic adjustment method described above can refer to each other.

[0170] Figure 3 is a schematic diagram of the structure of the camera posture automatic adjustment system provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a matching module 31, a removal module 32, a calibration module 33, a measurement module 34 and a calculation module 35, wherein:

[0171] The matching module 31 is used to obtain a reference image and a captured image of the camera to be calibrated, and to extract initial feature matching points of the reference image and the captured image by using a deep learning algorithm and a feature matching algorithm; the removal module 32 is used to remove abnormal matching points in the initial feature matching points by using an abnormal point removal algorithm to obtain precise matching points; the calibration module 33 is used to obtain the intrinsic parameters of the camera to be calibrated by using a RAC calibration method; the measurement module 34 is used to restore the camera motion by solving the epipolar geometry based on the intrinsic parameters of the camera to be calibrated, obtain the initial posture change of the camera to be calibrated, and obtain the two-dimensional coordinates and three-dimensional coordinates of the precise matching points by using a triangulation method; the calculation module 35 is used to calculate the two-dimensional coordinates and the three-dimensional coordinates by using a DLT algorithm to obtain the camera posture; the adjustment module 36 is used to adjust the camera to be calibrated by using the camera posture, and calculate the deviation value of the corresponding identifier in the captured image and the reference image. If the deviation value is within a preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed.

[0172] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the camera posture automatic adjustment method, which includes: obtaining a reference picture and a captured picture of the camera to be calibrated, and using a deep learning algorithm and a feature matching algorithm to extract the initial feature matching points of the reference picture and the captured picture; using an outlier removal algorithm to remove abnormal matching points in the initial feature matching points to obtain precise matching points; using a RAC-based calibration method to obtain the internal parameters of the camera to be calibrated; based on the internal parameters of the camera to be calibrated, the epipolar geometry solution is used to restore the camera motion to obtain the initial posture change of the camera to be calibrated, and the triangulation method is used to obtain the two-dimensional coordinates and three-dimensional coordinates of the precise matching points; using the DLT algorithm to calculate the two-dimensional coordinates and the three-dimensional coordinates to obtain the camera posture; using the camera posture to adjust the camera to be calibrated, and calculating the deviation value of the corresponding identifier in the captured picture and the reference picture. If the deviation value is within a preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed.

[0173] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0174] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the camera posture automatic adjustment method provided by the above-mentioned methods, the method comprising: obtaining a reference image and a captured image of the camera to be calibrated, and extracting initial feature matching points of the reference image and the captured image using a deep learning algorithm and a feature matching algorithm; removing abnormal matching points from the initial feature matching points using an abnormal point removal algorithm to obtain precise matching points; obtaining intrinsic parameters of the camera to be calibrated using a RAC-based calibration method; based on the intrinsic parameters of the camera to be calibrated, restoring camera motion by epipolar geometry solution to obtain the initial posture change of the camera to be calibrated, and obtaining the two-dimensional coordinates and three-dimensional coordinates of the precise matching points using a triangulation method; calculating the two-dimensional coordinates and the three-dimensional coordinates using a DLT algorithm to obtain the camera posture; adjusting the camera to be calibrated using the camera posture, and calculating the deviation value of the corresponding identifier in the captured image and the reference image, and if the deviation value is within a preset deviation range, terminating the calibration adjustment process, otherwise re-executing the calibration adjustment process.

[0175] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically adjusting camera posture, characterized in that: include: Obtain a reference image and a captured image of the camera to be calibrated, and use a deep learning algorithm and a feature matching algorithm to extract initial feature matching points of the reference image and the captured image; Using an outlier removal algorithm to remove outlier matching points from the initial feature matching points to obtain accurate matching points; The radial arrangement constraint (RAC) calibration method is used to obtain the intrinsic parameters of the camera to be calibrated. Based on the intrinsic parameters of the camera to be calibrated, the camera motion is restored by solving the epipolar geometry to obtain the initial pose change of the camera to be calibrated, and the two-dimensional coordinates and three-dimensional coordinates of the precise matching points are obtained by using the triangulation method; Using a direct linear transformation (DLT) algorithm to calculate the two-dimensional coordinates and the three-dimensional coordinates to obtain a camera pose; The camera to be calibrated is adjusted using the camera posture, and the deviation value of the corresponding identifier in the captured image and the reference image is calculated. If the deviation value is within a preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed; The camera to be calibrated is adjusted using the camera posture, and the deviation value of the corresponding identifier in the captured image and the reference image is calculated. If the deviation value is within a preset deviation range, the calibration adjustment process is terminated, otherwise the calibration adjustment process is re-executed, including: Calculating the gradient strength and gradient direction of each pixel in the acquired image and the reference image, clustering the gradient strength and the gradient direction, and clustering pixels that may form a straight line; The weighted least square method is adopted, the gradient strength is used as the weight of each point in the straight line, and the straight line equation is fitted to obtain the detection line segment; The detected line segments are grouped, wherein the distance between the end point of any edge and the start point of any other edge is less than a preset distance threshold, and the connected line segments are in a counterclockwise direction, and the directed line segments and candidate line segments in the line segment tree formed after grouping are used to obtain the identifier quadrilateral structure feature; Detecting the corner points of the quadrilateral structural features of the identifier, extracting the brightness information in the squares around the corner points, encoding the brightness information, mapping it to a label to obtain a displacement identifier ID, and obtaining an identifier matching point from the corner points and the displacement identifier ID; Based on the identifier matching points, the identifiers in the collected image and the reference image are respectively calibrated, and deviation values ​​of the identifiers are calculated; If it is determined that the deviation value is less than the preset deviation threshold, the calibration adjustment process is terminated, otherwise the calibration adjustment process is executed again.

2. The camera posture automatic adjustment method according to claim 1, characterized in that: The initial feature matching points of the reference image and the collected image are extracted using a deep learning algorithm and a feature matching algorithm, including: Using a VGG convolutional neural network to extract features from the reference image and the collected image, respectively, to obtain a reference feature map and a collected feature map; A deep nearest neighbor search (DNNS) algorithm is used to extract matching points from the benchmark feature map and the acquisition feature map, the acquisition feature map is searched, the L2 distance between each element in the benchmark feature map and the acquisition feature map is obtained, and the initial feature matching point is determined according to the ratio of the best match to the second best match corresponding to the L2 distance.

3. The camera posture automatic adjustment method according to claim 1, characterized in that: An outlier point removal algorithm is used to remove outlier matching points from the initial feature matching points to obtain accurate matching points, including: Calculating the x-axis deviation value, y-axis deviation value, x-axis deflection angle, and y-axis deflection angle of the initial feature matching point, and removing the initial feature matching points that are greater than the deviation value threshold or greater than the deflection angle threshold; Calculate the mean and standard deviation corresponding to the x-axis deviation value, y-axis deviation value, x-axis deflection angle, and y-axis deflection angle respectively, determine the minimum value by subtracting 3 times the standard deviation from the mean, determine the maximum value by adding 3 times the standard deviation to the mean, remove feature matching points that are less than the minimum value or greater than the maximum value, and obtain the precise matching point.

4. The camera posture automatic adjustment method according to claim 1, characterized in that: The RAC calibration method is used to obtain the intrinsic parameters of the camera to be calibrated, including: Get the conversion relationship between the camera coordinate system and the world coordinate system: Where (x, y) is the pixel coordinate in the camera, (X, Y, Z) is the real world coordinate, K is the camera intrinsic parameter matrix, (Rt) is the extrinsic parameter matrix, R is the rotation matrix describing the direction of the camera, t is the three-dimensional translation vector describing the center position of the camera, f is the focal length, s is the distortion coefficient, and (u0, v0) is the coordinate of the center point of the image; Solve the transformation from ideal image coordinates to digital image coordinates based on radial deviation: Among them, (u, v) represents the coordinates of any point in the image coordinate system, (x, y) is the pixel coordinates in the camera after distortion correction, (u c ,v c ) represents the coordinates of the distortion center, s1 is the first-order distortion coefficient; Perform radial consistency constraints. If there exists (u c ,v c ), (u,v) and (x,y) are collinear, then: Taking the center point of the image (u0, v0) as the distortion center coordinate, we get: Solve the camera's external parameter rotation matrix R and the translation vectors t1 and t2 in the x-axis and y-axis directions, expand R and t, and get: From the radial consistency constraint we get: From at least 7 sets of exact matching points, we get: M0=(m1,m2,m3,m4,m5,m6,m7,m8)≈(sr1,sr2,sr3,st1,r4,r5,r6,t2); M0 divided by Get a set of solutions (sr1, sr2, sr3, st1, r4, r5, r6, t2); Depend on Get s and t1; (r7, r8, r9) = (r1, r2, r3) × (r4, r5, r6) or (r7, r8, r9) = (r4, r5, r6) × (r1, r2, r3); Let s = 0 as the initial value, then the expressions of x and y are: x-u0=u-u0,y-v0=v-v0 Substituting the obtained values ​​of R, t1 and t2 into the equation: (u-u0)(r7X+r8Y+r9Z+t3)=fs(r1X+r2Y+r3Z+t1) (v-v0)(r7X+r8Y+r9Z+t3)=f(r4X+r5Y+r6Z+t2) Find f and t3; Taking f, t3 and s=0 as initial values, perform linear optimization on the following formula to obtain the true values ​​of f, t3 and s:

5. The camera posture automatic adjustment method according to claim 1, characterized in that: Based on the intrinsic parameters of the camera to be calibrated, the camera motion is restored by solving the epipolar geometry to obtain the initial pose change of the camera to be calibrated, and the two-dimensional coordinates and three-dimensional coordinates of the precise matching points are obtained by using the triangulation method, including: Determine the augmented matrix [R|t] based on the rotation matrix R and the three-dimensional translation vector t: The two-dimensional coordinates and three-dimensional coordinates of the exact matching point are obtained by conversion: Among them, (X, Y, Z) are three-dimensional coordinates and (u1, v1) are two-dimensional coordinates.

6. The camera posture automatic adjustment method according to claim 5, characterized in that: The DLT algorithm is used to calculate the two-dimensional coordinates and the three-dimensional coordinates to obtain the camera pose, including: From the two-dimensional coordinates and the three-dimensional coordinates, we get: Define the row vector of T as follows: <h2 style=";text-align:left;direction:ltr">t1 = (t1,t2,t3,t4)<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> t2 = (t5,t6,t7,t8)<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> t3 = (t9,t)<h2 style=";text-align:left;direction:ltr"> 10 <h2 style=";text-align:left;direction:ltr"> ,t<h2 style=";text-align:left;direction:ltr"> 11 <h2 style=";text-align:left;direction:ltr"> ,t<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> )<h2 style=";text-align:left;direction:ltr"> T Substituting the row vector of T into the equation: Based on N feature points, we get the linear equation system: Solve the linear equations to obtain the camera pose.

7. A camera posture automatic adjustment system, characterized in that: include: A matching module is used to obtain a reference image and a captured image of the camera to be calibrated, and to extract initial feature matching points of the reference image and the captured image using a deep learning algorithm and a feature matching algorithm; A removal module, used to remove abnormal matching points from the initial feature matching points by using an abnormal point removal algorithm to obtain accurate matching points; A calibration module is used to obtain the intrinsic parameters of the camera to be calibrated by using a first-order radial distortion RAC calibration method; A measurement module, for restoring the camera motion by solving the epipolar geometry based on the intrinsic parameters of the camera to be calibrated, obtaining the initial posture change of the camera to be calibrated, and obtaining the two-dimensional coordinates and three-dimensional coordinates of the precise matching points by using a triangulation method; A calculation module, used for calculating the two-dimensional coordinates and the three-dimensional coordinates by using a direct linear transformation DLT algorithm to obtain a camera pose; An adjustment module, used to adjust the camera to be calibrated using the camera posture, calculate the deviation value of the corresponding identifier in the acquired image and the reference image, and terminate the calibration adjustment process if the deviation value is within a preset deviation range, otherwise re-execute the calibration adjustment process; The adjustment module is specifically used for: Calculating the gradient strength and gradient direction of each pixel in the acquired image and the reference image, clustering the gradient strength and the gradient direction, and clustering pixels that may form a straight line; The weighted least square method is adopted, the gradient strength is used as the weight of each point in the straight line, and the straight line equation is fitted to obtain the detection line segment; The detected line segments are grouped, wherein the distance between the end point of any edge and the start point of any other edge is less than a preset distance threshold, and the connected line segments are in a counterclockwise direction, and the directed line segments and candidate line segments in the line segment tree formed after grouping are used to obtain the identifier quadrilateral structure feature; Detecting the corner points of the quadrilateral structural features of the identifier, extracting the brightness information in the squares around the corner points, encoding the brightness information, mapping it to a label to obtain a displacement identifier ID, and obtaining an identifier matching point from the corner points and the displacement identifier ID; Based on the identifier matching points, the identifiers in the collected image and the reference image are respectively calibrated, and deviation values ​​of the identifiers are calculated; If it is determined that the deviation value is less than the preset deviation threshold, the calibration adjustment process is terminated, otherwise the calibration adjustment process is executed again.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the camera posture automatic adjustment method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically adjusting the camera posture as described in any one of claims 1 to 6 is implemented.

Images