A telescope pose monitoring method based on computer vision

Abstract

This invention discloses a telescope pose monitoring method based on computer vision. The steps include: deploying cameras to acquire telescope images under different poses, obtaining raw image data; calibrating the deployed cameras to obtain their intrinsic parameters; establishing a deep learning model of the telescope pose using a dataset; calculating the telescope's rotation matrix using the Epnp algorithm based on the two-dimensional coordinates of the telescope's key points; converting the rotation matrix into rotation vectors and extracting the rotation angles along the two axes. This invention employs a YOLO deep learning-based approach for keypoint extraction, demonstrating good performance in complex environments. By deploying multiple cameras and simultaneously monitoring the telescope's pose using multiple monocular cameras, information is supplemented and blind spots are reduced. The Epnp algorithm is used for pose estimation, and minimum error optimization using reprojection improves the accuracy of the results.

Description

Technical Field

[0001] This invention relates to a method for monitoring the attitude of a telescope, and more particularly to a telescope attitude monitoring method based on computer vision. Background Technology

[0002] Under normal operating conditions, the telescope's attitude corresponds one-to-one with the rotation angle readings of its two axes. However, in some abnormal situations, the readings may not accurately reflect the telescope's attitude, such as when the rotation angle feedback device is not initialized, malfunctions in the device, or when the data acquisition device malfunctions. If these abnormalities are not detected in time due to human error or system failure, serious accidents may occur.

[0003] Typically, a straightforward method for determining whether the rotation angle readings of a telescope's two axes are abnormal during operation is for observers to roughly estimate the rotation angle readings of the two axes by monitoring the telescope's attitude on the screen and comparing them with the current readings to determine if there is an anomaly. However, continuous monitoring by humans is impossible. This invention uses computer technology to replace manual labor, employing computer vision to acquire images, identify the rotation angle reading range corresponding to the telescope's attitude in the images, and compare it with the current readings. If the two do not match, the telescope's rotation angle readings are likely abnormal, allowing for timely reporting and preventing potential problems at an early stage. This is of great significance for improving the reliability of telescope operation. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a telescope pose monitoring method based on computer vision.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A telescope pose monitoring method based on computer vision includes the following steps:

[0007] 1) Deploy cameras to collect telescope images in different postures and obtain raw image data;

[0008] 2) Calibrate the deployed cameras to obtain their intrinsic parameters;

[0009] By establishing a camera imaging model and the relationship between the image pixel coordinate system and the world coordinate system, the expression for the intrinsic parameters is obtained; at the same time, considering the tangential and radial distortion of the camera, a distortion model of the camera is established; and the expression for the intrinsic parameters is calculated based on the image.

[0010] 3) Build a deep learning model for telescope pose using the dataset; the model uses the YOLO algorithm, taking the coordinates of key points and camera numbers in manually labeled images as input, and the predicted coordinates of key points and camera numbers as output;

[0011] 4) Use the Epnp algorithm to calculate the telescope's rotation matrix based on the two-dimensional coordinates of the telescope's key points;

[0012] 5) Attitude optimization;

[0013] After obtaining the initial results, the LM algorithm based on minimum projection error is used to optimize them;

[0014] 6) Convert the rotation matrix into a rotation vector and extract the rotation angles of the two axes.

[0015] Furthermore, in step 1), multiple cameras are used to monitor the telescope from different angles. When a camera captures certain postures of the telescope and key points are obscured, images from cameras at other angles are used to supplement the information.

[0016] Furthermore, step 2) specifically includes:

[0017] 2.1) Establish a camera imaging model;

[0018] Based on the pinhole imaging model, four reference coordinate systems are proposed: world coordinate system, camera coordinate system, image physical coordinate system, and image pixel coordinate system.

[0019] 2.2) Solve for the camera parameters;

[0020] The homography matrix is ​​solved using Zhang's calibration method; the equations are solved using three calibration plots; when there are more than three calibration plots, the least squares method can also be used to solve the corresponding overdetermined equations.

[0021] 2.3) Solve for camera distortion parameters.

[0022] Furthermore, step 2.3) specifically includes:

[0023] 2.3.1) For the coordinates (x, y) of each pixel, convert them to coordinates in a normalized coordinate system. Right now and Among them (c x ,c y ) are the coordinates of the center point of the intrinsic parameter matrix;

[0024] 2.3.2) Calculate the radial distortion correction parameters k1, k2 and the tangential distortion parameters p1, p2; according to the camera model, the radial distortion term is expressed as:

[0025] Δxk =x(1+k1r) 2 +k2r 4 )

[0026] Δx p =x + 2p1xy + p2(r 2 +2x 2 )

[0027] Where Δx k Δx p It is the x-direction offset in the normalized coordinate system after radial and tangential distortion correction, and r is the radial distance in the normalized coordinate system;

[0028] 2.3.3) For each pixel, based on the calculated distortion parameters k1, k2, p1, p2, combined with the normalized coordinates... Distortion correction calculations are performed using the radial distance r:

[0029]

[0030] 2.3.4) Transform the corrected normalized coordinates back to pixel coordinates (x′, y′):

[0031] x'=xf x +c x

[0032] y'=yf y +c y .

[0033] Furthermore, step 3) specifically includes:

[0034] 3.1) The main steps in creating the dataset are image acquisition and image labeling. The open-source software LabelMe is used for labeling, and the labeling content includes the bounding box of the telescope and the pixel coordinates of the target key points.

[0035] 3.2) For model building and training, in the network configuration file, change the number of YOLOv8 keypoints to the number of labeled keypoints, and change the number of image types to the number of surveillance cameras, representing images taken from different cameras.

[0036] 3.3) For network training, set the number of network iterations, specify the location of the network dataset, and map images to their labels; set a reasonable batch size to train on multiple images and speed up the training process; and divide the dataset into training, validation, and test sets.

[0037] 3.4) For evaluation metrics, AP is introduced to measure the accuracy of training results.

[0038] Furthermore, in step 3.2), the model mainly consists of four parts: input end, backbone network, Neck module, and output end;

[0039] The input end of the network employs data augmentation methods to increase the richness of the dataset and the robustness of the network. The steps are as follows:

[0040] (1) Take a batch from the dataset for network training; then take img0, img1, img2, and img3 from the batch for cropping and splicing, with each image having a size of h*w.

[0041] (2) Randomly generate a center coordinate (cut_x, cut_y), which is the width and height of the top left corner of the enhanced image;

[0042] (3) Based on this random position, the four images are cropped and spliced ​​into a new image;

[0043] (4) Gaussian noise is added to the newly stitched image to simulate the complex changes in light inside the dome;

[0044] (5) Repeat the process more than batchsize times to obtain the enhanced dataset;

[0045] The backbone network includes a CBS module, a C2f module, and an SPPF module;

[0046] The CBS module extracts spatial information from the image through convolution, performs batch normalization through BN layers, and finally performs non-linear activation through an activation function, thereby realizing feature transformation and extraction of the input image. The C2f module is the residual feature learning module of YOLOv8. By branching and connecting the feature maps output by the CBS module, that is, cutting and fusing the feature maps at different layers, the gradient flow of the model is enriched, forming a neural network module with stronger feature representation capabilities. The SPPF module is the pooling module of the backbone network, which applies receptive fields of different sizes to the same image.

[0047] The Neck module adopts a Pan feature fusion structure, which first downsamples and then upsamples. There are also two cross-layer fusion connections between the two branches of upsampling and downsampling to enhance the network's ability to fuse features of objects at different scaling scales, making the network more sensitive to telescope rotation monitoring.

[0048] The output terminal is used to decouple classification and detection from key point location prediction, including loss calculation and target detection box filtering.

[0049] Furthermore, in step 3), an SE attention mechanism is added; the SE attention mechanism is added to the ninth layer of the backbone network, and the steps for establishing the attention mechanism are as follows:

[0050] (1) The eighth convolution layer produces a feature map with dimensions C'*W'*H', and the spatial dimensions are compressed by F. sq (.), outputting a feature map of size 1*1*C;

[0051] (2) The compressed feature map is learned through a fully connected network F. ex (.) Predict the correlation between C channels, and represent the results using weights W;

[0052] (3) Finally, the output is regarded as the importance of each channel after feature selection. It is multiplied onto the previous features by multiplication weighting to enhance important features and suppress unimportant features.

[0053] Furthermore, the method includes keypoint location loss and bounding box loss, uses the OKS method to evaluate the fitting results of the keypoints, and uses the CIOU method to evaluate the fitting results of the bounding boxes.

[0054] Furthermore, step 5) specifically involves the following steps:

[0055] 5.1) Calculate the initial telescope attitude, use the EPnP algorithm to estimate the camera rotation matrix and translation vector, and obtain the initial telescope attitude estimate;

[0056] 5.2) Define the reprojection error, calculate the reprojection points of all three-dimensional points projected onto the two-dimensional plane; then calculate the difference between the projected points and the actual observed points.

[0057] 5.3) Define the optimization objective function, using the reprojection error as the optimization objective function;

[0058] 5.4) Calculate the Jacobian matrix. In each iteration, calculate the partial derivatives of the objective function with respect to the gradient of the rotation matrix and translation vector; arrange these partial derivatives in order to form the Jacobian matrix; adjust the direction of the projection error optimization according to the Jacobian matrix.

[0059] 5.5) Optimize the objective function using the Levenberg-Marquardt algorithm;

[0060] 5.6) Convergence Criterion: Define the convergence criterion;

[0061] 5.7) Update the camera pose by using the new rotation matrix and translation vector obtained from the Levenberg-Marquardt algorithm to update the telescope pose;

[0062] 5.8) Repeat the optimization. If the convergence condition is not met, repeat steps 5.4) to 5.7) until the optimization converges.

[0063] Furthermore, in step 6), the cv2.Rodrigues function of OpenCV is used, with the optimized rotation matrix as input and the rotation angle of the two axes as output.

[0064] Compared with the prior art, the beneficial effects of the present invention are:

[0065] This invention aims to achieve telescope attitude monitoring through computer vision. For feature extraction, a YOLO deep learning-based approach is used to extract key points, demonstrating good performance in complex environments. By deploying multiple cameras, multiple monocular cameras simultaneously monitor the telescope's attitude, supplementing information and reducing blind spots. The Epnp algorithm is used for attitude estimation, and minimum error optimization using reprojection improves the accuracy of the results. Attached Figure Description

[0066] Figure 1 This is a flowchart of the method of the present invention;

[0067] Figure 2 It is a black and white checkered pattern;

[0068] Figure 3 It is an SPPF flowchart;

[0069] Figure 4 This is a structural diagram of the SE attention mechanism. Detailed Implementation

[0070] The present invention will now be described in further detail with reference to the accompanying drawings.

[0071] This invention presents a telescope pose monitoring method based on computer vision, aiming to achieve telescope pose monitoring through computer vision. For feature extraction, a YOLO deep learning-based approach is used to extract key points, demonstrating good performance in complex environments. By deploying multiple cameras, multiple monocular cameras simultaneously monitor the telescope's pose, supplementing information and reducing blind spots. The Epnp algorithm is used for pose estimation, and minimum error optimization using reprojection improves the accuracy of the results. The method flow is as follows: Figure 1 As shown, it includes the following steps:

[0072] 1) Deploy cameras to collect images of the telescope in different postures and obtain raw image data. Multiple cameras are used to monitor the telescope from different angles; if key points are obstructed when a camera captures images of the telescope in certain postures, images from cameras at other angles are used to supplement the information.

[0073] 2) Calibrate the deployed cameras to obtain their intrinsic parameters.

[0074] 3) A deep learning model for telescope pose was built using the dataset. This model uses the YOLO (You Only Look Once) algorithm, taking the coordinates of keypoints and camera numbers in manually labeled images as input layers, and the predicted keypoint coordinates and camera numbers as output layers. Specifically, the model input consists of a combination of object detection boxes and keypoint pixel coordinates, with at least 10 pairs of keypoint coordinates selected. The model is based on the YOLOv8 keypoint model, and an SE attention mechanism is added to the ninth layer of the network.

[0075] 4) The Epnp algorithm is used to calculate the telescope's rotation matrix based on the two-dimensional coordinates of the telescope's key points. The LM (Levenberg-Marquardt) algorithm, based on minimum projection error, is incorporated to optimize the results.

[0076] 5) Convert the rotation matrix into a rotation vector and extract the rotation angles of the two axes.

[0077] The specific steps of the telescope pose monitoring method based on computer vision in this embodiment are as follows:

[0078] Step 1: Deploy multiple cameras inside the dome of the telescope, positioned at the edge of the dome and approximately x meters above the ground, ensuring the telescope is completely within the image. The cameras are spaced approximately 30° apart, facilitating multi-angle monitoring.

[0079] Step 2: Calibrate the camera. This design uses Zhang's calibration method to obtain the camera's intrinsic parameters. By establishing a camera imaging model and the relationship between the image pixel coordinate system and the world coordinate system, the expression for the intrinsic parameters is obtained. Next, a camera distortion model is established, considering both tangential and radial distortion to increase the accuracy of the results. Finally, the expression for the intrinsic parameters is calculated based on the image. The specific process is as follows:

[0080] 2.1 Establishing a camera imaging model

[0081] Based on the pinhole camera model, four reference coordinate systems are proposed: world coordinate system, camera coordinate system, image physical coordinate system, and image pixel coordinate system. The relationships between the specific parameters can be expressed by the following formula:

[0082]

[0083] Where u and v represent the positions in the primitive coordinate system, u0 and v0 represent the offsets from the origin of the pixel in the primitive coordinate system, and f x ,f yThe ratio of the scale factor in the u and v directions to the camera focal length f is related to the camera's structure. R and t represent the rotation and translation matrices, respectively. X W Y W Z W Represents world coordinate system coordinates.

[0084] 2.2 Solving for camera parameters

[0085] After establishing the model, the camera's intrinsic parameters need to be solved to complete the camera calibration. The steps for solving the camera's intrinsic parameters are as follows:

[0086] (1) Solve the homography matrix using Zhang's method;

[0087] (2) In Zhang's planar template method, each calibration plot can provide a constraint relationship (corresponding to a homography matrix), and each constructed equation will provide two constraint relationships. The calibration plot style is as follows: Figure 2 The graph is shown in black and white. The equation itself contains six unknowns, so theoretically, it can be solved using three calibration graphs. When there are more than three calibration graphs, the least squares method can be used to solve the corresponding overdetermined equation. The solved intrinsic parameters are shown below:

[0088]

[0089] Among them B ij The symmetry matrix can be obtained from the homography matrix, where u0 and v0 represent the offsets from the origin of the pixel in the prime coordinate system, and f x f y γ represents the ratio of the scale factor in the u and v directions to the camera focal length f, γ represents the distortion parameter, and λ is the homography matrix scale factor.

[0090] 2.3 Solving for camera distortion parameters

[0091] In practical use, distortion occurs due to some optical lenses and manufacturing errors. Therefore, it is necessary to establish a model to calculate the error. The distortion mainly includes tangential distortion and radial distortion. Currently, most calibration methods only consider second-order radial distortion, but in reality, ignoring the tangential distortion coefficient will inevitably greatly reduce the calibration accuracy. Therefore, this design includes both second-order radial and tangential distortion. The specific steps are as follows:

[0092] (1) For the coordinates (x, y) of each pixel, convert them to coordinates in a normalized coordinate system. Right now and Among them (c x ,c y ) are the coordinates of the center point of the intrinsic parameter matrix.

[0093] (2) Calculate the radial distortion correction parameters k1, k2 and the tangential distortion parameters p1, p2. Based on the camera model, the radial distortion term can be expressed as...

[0094] Δx k =x(1+k1r) 2 +k2r 4 )

[0095] Δx p =x + 2p1xy + p2(r 2 +2x 2 )

[0096] Where Δx k Δx p It is the x-direction offset in the normalized coordinate system after radial and tangential distortion correction, and r is the radial distance in the normalized coordinate system.

[0097] (3) For each pixel, based on the calculated distortion parameters k1, k2, p1, p2, combined with the normalized coordinates Distortion correction calculations are performed using the radial distance r:

[0098]

[0099] (4) Transform the corrected normalized coordinates back to the pixel coordinate system (x′, y′):

[0100] x'=xf x +c x

[0101] y'=yf y +c y

[0102] Step 3: Creating the dataset mainly involves image acquisition and image labeling. The open-source software LabelMe is used for labeling. The labeling includes the bounding box of the telescope and the pixel coordinates of the target key points. This invention selects N key points with the following specific format:

[0103] Class Image type w Width of the bounding box h Height of bounding box x x-coordinate of the center point of the bounding box y The ordinate of the center point of the bounding box <![CDATA[(x1,y1)]]> Key point 1 coordinates <![CDATA[(x2,y2)]]> Key point 2 coordinates <![CDATA[(x3,y3)]]> Key point 3 coordinates <![CDATA[(x4,y4)]]> Key point 4 coordinates <![CDATA[(x5,y5)]]> Key point 5 coordinates <![CDATA[(x6,y6)]]> Key point 6 coordinates …… <![CDATA[(x N ,and N )]]> Key point N coordinates

[0104] Class represents the source of the image. h, w, x, and y represent the width and height of the telescope bounding box in the image, and the position of the center point, respectively. The data has been normalized to facilitate data convergence. Keypoints 1 to N represent the pixel coordinates of the keypoints in the image.

[0105] Step 4: For model building and training, the number of YOLOv8 keypoints in the network configuration file needs to be modified to the number of labeled keypoints. Simultaneously, the number of image types needs to be changed to the number of surveillance cameras, representing images captured from different cameras. The model mainly consists of four parts: input, backbone, neck module, and output.

[0106] 4.1 For the input end, data augmentation methods are used to increase the richness of the dataset and the robustness of the network. This design employs a combination of mosaic enhancement and Gaussian noise to both increase data richness and simulate camera shake caused by complex lighting conditions and strong winds at the telescope site, thereby enhancing the network's robustness. The steps are as follows:

[0107] (1) Take a batch from the dataset, such as 16, 32, etc., for network training. Then take img0, img1, img2, img3 from the batch for cropping and splicing, with each image having a size of h*w;

[0108] (2) Randomly generate a center coordinate (cut_x, cut_y), which is the width and height of the top left corner of the enhanced image;

[0109] (3) Based on this random position, the four images are cropped and spliced ​​into a new image;

[0110] (4) Gaussian noise is added to the newly stitched image to simulate the complex changes in light inside the dome;

[0111] (5) Repeat the process more than batchsize times to obtain the enhanced dataset.

[0112] 4.2 The backbone network mainly consists of the CBS module and the C2f module. The CBS module primarily extracts spatial information from the image through convolution, performs batch normalization using BN layers, and finally performs non-linear activation using an activation function, thereby achieving feature transformation and extraction from the input image. The C2f module is the residual feature learning module of YOLOv8. By branching and connecting the feature maps output by the CBS module across layers—that is, by cutting and fusing the feature maps at different layers—it enriches the gradient flow of the model, forming a neural network module with stronger feature representation capabilities.

[0113] SPPF is a pooling module in the backbone network that applies receptive fields of different sizes to the same image. For example... Figure 3 As shown, the feature map after CBS convolution is fused with maxpool (maximum pooling) performed 1, 2, and 3 times respectively, which not only filters information but also preserves information to the greatest extent.

[0114] 4.3 The Neck module adopts a Pan feature fusion structure. By first downsampling and then upsampling, and there are two cross-layer fusion connections between the upsampling and downsampling branches, the network's ability to fuse features of objects at different scaling scales can be enhanced, making the network more sensitive to telescope rotation monitoring.

[0115] 4.4 Output. The classification and detection, and keypoint location prediction are decoupled, mainly including loss calculation and target detection box selection.

[0116] 4.5. Add SE attention mechanism

[0117] An SE attention mechanism is added to the ninth layer of the backbone network to improve the accuracy of the output. The attention mechanism structure diagram is shown below. Figure 4 As shown, the steps to establish this are as follows:

[0118] (1) The eighth convolution layer produces a feature map with dimensions C'*W'*H', and the spatial dimensions are compressed by F. sq (.), outputting a feature map of size 1*1*C;

[0119] (2) The compressed feature map is learned through a fully connected network F. ex (.) Predict the correlation between C channels, and represent the results using weights W;

[0120] (3) Finally, the output is regarded as the importance of each channel after feature selection. It is multiplied onto the previous features by multiplication weighting to enhance important features and suppress unimportant features.

[0121] 4.6 Regarding the loss function, the losses in this design mainly include confidence loss, classification loss, keypoint location loss, and bounding box loss. The keypoint location loss and bounding box loss use OKS (object keypoint similarity) and CIOU respectively to evaluate the fitting results of the keypoints and bounding boxes. It combines the Euclidean distance between the actual and predicted keypoints and the area of ​​the bounding box as evaluation criteria, achieving better fitting results. The calculation method for OKS is as follows:

[0122]

[0123] Where, d n This represents the Euclidean distance between the currently detected set of keypoints and the actual keypoints. S represents the scale factor, which is the square root of the area of ​​the telescope's bounding box. σ i This represents the key point normalization factor.

[0124] The loss function for the bounding box is CIoU loss, used for bounding box supervision. For a given ground truth bounding box... It is related to the kth anchor. If the scale is s, then the loss between them can be defined as:

[0125]

[0126] Therefore, the total loss function is given by , where the constant before the loss function represents the weights of the loss function:

[0127]

[0128] Step 5: For network training, set the number of iterations, specify the location of the dataset, and map each image to its label. Set a reasonable batch size to train on multiple images, accelerating the training process. Simultaneously, divide the dataset into training, validation, and test sets.

[0129] Step 6: For evaluation metrics, AP (average precision) is introduced to measure the accuracy of the training results.

[0130] The specific calculation method is shown in the following formula, where T is the set threshold, which is set to 0.90 in this design.

[0131]

[0132] Step 7: For attitude estimation, the SOLVEPNP function in OpenCV is used to calculate the rotation matrix of the telescope in the image. The input includes the positions of N keypoints of the telescope in the pixel coordinate system, the intrinsic parameters of the calibrated camera, and the 3D coordinate positions of the keypoints. Four control points are selected using PCA (Principal Component Analysis). The coordinates of the telescope's 3D model and the camera coordinate system are represented by these control points. The rotation matrix between the two coordinate systems is calculated using SVD (Non-Singular Decomposition).

[0133] Step 8: For pose optimization, after obtaining the initial result, further optimization is needed to obtain a more accurate result. This invention uses the LM (Levenberg-Marquardt) algorithm. The specific steps are as follows:

[0134] 8.1 Calculate the initial telescope attitude. Use the EPnP algorithm to estimate the camera rotation matrix R and translation vector t to obtain the initial telescope attitude estimate.

[0135] 8.2 Define reprojection error and calculate X for all 3D points. i The reprojection point p(X) projected onto the two-dimensional plane iThen calculate the projection point p(X). i ) and the actual observation point x i The difference between them.

[0136] e i =p(X i )-x i

[0137] 8.3 Define the optimization objective function and use the reprojection error e i The objective function is to minimize the sum of squared reprojection errors.

[0138]

[0139] 8.4 Calculate the Jacobian matrix. In each iteration, calculate the partial derivatives of the objective function's gradient with respect to the rotation matrix R and the translation vector t. Arrange these partial derivatives in order to form the Jacobian matrix. Adjust the direction of the projection error optimization based on the Jacobian matrix.

[0140] 8.5. Use the Levenberg-Marquardt algorithm to optimize the objective function. This algorithm is a nonlinear least squares optimization algorithm that updates the camera attitude parameters by adjusting the step size in each iteration.

[0141] [R t] new =[R t] old -(J T J+λI) -1 J T e

[0142] Among them, [R t] old It is the old parameter vector, [Rt]. new This is the new parameter vector, where λ is a parameter of the LM algorithm used to control the magnitude of the step size adjustment.

[0143] 8.6 Convergence Criterion: Define convergence criteria, such as the change in the objective function being less than a certain threshold or reaching the maximum number of iterations.

[0144] 8.7 Update the camera attitude. Based on the new rotation matrix R and translation vector t obtained by the Levenberg-Marquardt algorithm, update the telescope attitude.

[0145] 8.8 Repeat the optimization. If the convergence condition is not met, repeat steps 8.4 to 8.7 until the optimization converges.

[0146] Step 9: Calculate the rotation angle using the cv2.Rodrigues function of OpenCV. The input is the optimized 3*3 rotation matrix, and the output is the rotation angle of the two axes.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A telescope pose monitoring method based on computer vision, characterized in that, Includes the following steps: 1) Deploy cameras to collect telescope images in different postures and obtain raw image data; 2) Calibrate the deployed cameras to obtain their intrinsic parameters; By establishing a camera imaging model and the relationship between the image pixel coordinate system and the world coordinate system, the expression for the intrinsic parameters is obtained; at the same time, considering the tangential and radial distortion of the camera, a distortion model of the camera is established; and the expression for the intrinsic parameters is calculated based on the image. 3) Establish a deep learning model for the telescope's pose using the dataset; this model uses the YOLO algorithm, taking the coordinates of keypoints and camera numbers in manually labeled images as input, and the predicted coordinates of keypoints and camera numbers as output; step 3) specifically includes: 3.1) The main steps in creating the dataset are image acquisition and image labeling. The open-source software LabelMe is used for labeling, and the labeling content includes the bounding box of the telescope and the pixel coordinates of the target key points. 3.2) For model building and training, in the network configuration file, change the number of YOLOv8 keypoints to the number of labeled keypoints, and change the number of image types to the number of surveillance cameras, representing images taken from different cameras. 3.3) For network training, set the number of network iterations, specify the location of the network dataset, and map images to their labels; set a reasonable batch size to train on multiple images and speed up the training process; and divide the dataset into training, validation, and test sets. 3.4) For evaluation metrics, AP is introduced to measure the accuracy of training results; In step 3.2), the model mainly consists of four parts: input, backbone network, Neck module, and output. The input end of the network employs data augmentation methods to increase the richness of the dataset and the robustness of the network. The steps are as follows: (1) Take a batch from the dataset for network training; then take images img0, img1, img2, and img3 from the batch for cropping and splicing, with each image having a size of h*w. (2) Randomly generate a center coordinate (cut_x, cut_y), which is the width and height of the top left corner of the enhanced image; (3) Based on this random position, the four images img0, img1, img2, and img3 are cropped and spliced ​​together to form a new image; (4) Gaussian noise is added to the newly stitched image to simulate the complex changes in light inside the dome; (5) Repeat steps (1) to (4) a batchsize times to obtain the enhanced dataset; The backbone network includes a CBS module, a C2f module, and an SPPF module; The CBS module extracts spatial information from the image through convolution, performs batch normalization through BN layers, and finally performs non-linear activation through an activation function, thereby realizing feature transformation and extraction of the input image. The C2f module is the residual feature learning module of YOLOv8. By branching and connecting the feature maps output by the CBS module, that is, cutting and fusing the feature maps at different layers, the gradient flow of the model is enriched, forming a neural network module with stronger feature representation capabilities. The SPPF module is the pooling module of the backbone network, which applies receptive fields of different sizes to the same image. The Neck module adopts a Pan feature fusion structure, which first downsamples and then upsamples. There are also two cross-layer fusion connections between the two branches of upsampling and downsampling to enhance the network's ability to fuse features of objects at different scaling scales, making the network more sensitive to telescope rotation monitoring. The output terminal is used to decouple classification and detection from key point location prediction, including loss calculation and target detection box filtering; 4) Use the Epnp algorithm to calculate the telescope's rotation matrix based on the two-dimensional coordinates of the telescope's key points; 5) Attitude optimization; After obtaining the initial results, the LM algorithm based on minimum projection error is used to optimize them; 6) Convert the rotation matrix into a rotation vector and extract the rotation angles of the two axes.

2. The telescope pose monitoring method based on computer vision according to claim 1, characterized in that, In step 1), multiple cameras are used to monitor the telescope from different angles. When a camera captures certain postures of the telescope, if key points are obstructed, images from cameras at other angles are used to supplement the information.

3. The telescope pose monitoring method based on computer vision according to claim 1, characterized in that, Step 2) specifically includes: 2.1) Establish a camera imaging model; Based on the pinhole imaging model, four reference coordinate systems are proposed: world coordinate system, camera coordinate system, image physical coordinate system, and image pixel coordinate system. 2.2) Solve for the camera parameters; The homography matrix is ​​solved using Zhang's calibration method; the equations are solved using three calibration plots; when there are more than three calibration plots, the least squares method can also be used to solve the corresponding overdetermined equations. 2.3) Solve for camera distortion parameters.

4. The telescope pose monitoring method based on computer vision according to claim 3, characterized in that, Step 2.3) specifically includes: 2.3.1) For the coordinates (x, y) of each pixel, convert them to coordinates in a normalized coordinate system. Right now and Among them (c x ,c y ) are the coordinates of the center point of the intrinsic parameter matrix; 2.3.2) Calculate the radial distortion correction parameters k1, k2 and the tangential distortion parameters p1, p2; according to the camera model, the radial distortion term is expressed as: Δx k =x(1+k1r 2 +k2r 4 ) Δx p =x+2p1xy+p2(r 2 +2x 2 ) Where Δx k Δx p It is the x-direction offset in the normalized coordinate system after radial and tangential distortion correction, and r is the radial distance in the normalized coordinate system; 2.3.3) For each pixel, based on the calculated distortion parameters k1, k2, p1, p2, combined with the normalized coordinates... Distortion correction calculations are performed using the radial distance r: 2.3.4) Transform the corrected normalized coordinates back to pixel coordinates (x′, y′): x'=xf x +c x y'=yf y +c y 。 5. The telescope pose monitoring method based on computer vision according to claim 1, characterized in that, In step 3), an SE attention mechanism is added; the SE attention mechanism is added to the ninth layer of the backbone network, and the steps for establishing the attention mechanism are as follows: (1) The eighth convolution layer produces a feature map with dimensions C'*W'*H', and the spatial dimensions are compressed by F. sq (.), outputting a feature map of size 1*1*C; (2) The compressed feature map is learned through a fully connected network F. ex (.) Predict the correlation between C channels, and represent the results using weights W; (3) Finally, the output is regarded as the importance of each channel after feature selection. It is multiplied onto the previous features by multiplication weighting to enhance important features and suppress unimportant features.

6. The telescope pose monitoring method based on computer vision according to claim 1, characterized in that, The method includes keypoint location loss and bounding box loss. The OKS method is used to evaluate the fitting results of the keypoints, and the CIOU method is used to evaluate the fitting results of the bounding boxes.

7. The telescope pose monitoring method based on computer vision according to claim 1, characterized in that, The specific steps of step 5) are as follows: 5.1) Calculate the initial telescope attitude, use the EPnP algorithm to estimate the camera rotation matrix and translation vector, and obtain the initial telescope attitude estimate; 5.2) Define the reprojection error, calculate the reprojection points of all three-dimensional points projected onto the two-dimensional plane; then calculate the difference between the projected points and the actual observed points. 5.3) Define the optimization objective function, using the reprojection error as the optimization objective function; 5.4) Calculate the Jacobian matrix. In each iteration, calculate the partial derivatives of the gradient of the objective function with respect to the rotation matrix and translation vector. Arrange these partial derivatives in order to form the Jacobian matrix; adjust the direction of projection error optimization based on the Jacobian matrix; 5.5) Optimize the objective function using the Levenberg-Marquardt algorithm; 5.6) Convergence Criterion: Define the convergence criterion; 5.7) Update the camera pose by using the new rotation matrix and translation vector obtained from the Levenberg-Marquardt algorithm to update the telescope pose; 5.8) Repeat the optimization. If the convergence condition is not met, repeat steps 5.4) to 5.7) until the optimization converges.

8. The telescope pose monitoring method based on computer vision according to claim 1, characterized in that, In step 6), the cv2.Rodrigues function of OpenCV is used, with the optimized rotation matrix as input and the rotation angles of the two axes as output.

Images