Coordinate reconstruction method, device and storage medium for coupling fiber faceplate with sensor

Abstract

The application discloses a kind of optical fiber panel and sensor coupling coordinate reconstruction method, device and storage medium, the present application is due to using monocular camera, not only greatly reduce hardware cost, avoid the need of using expensive laser ranging equipment or binocular camera, simplify the installation and maintenance of system, suitable for the integration of large-scale production line;The regional information of target sensor is detected by a deep learning target detection model, to improve the accuracy and robustness of detection.The present application can improve the real-time performance and work efficiency of the system, meet the requirements of industrial production for efficient and automated operation;At the same time, the optical fiber panel and sensor coupling coordinate reconstruction method in the embodiment has strong environmental adaptability, can maintain high reliability and stability in complex and harsh working environment, realize simple process, flexible deployment, can effectively improve the production efficiency and quality consistency of optical fiber panel and sensor coupling.The present application is widely used in machine vision technology field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a coordinate reconstruction method, device and storage medium for coupling of a fiber panel and a sensor. BACKGROUND

[0002] In the rapid development of contemporary technology, photoelectric device manufacturing plays a crucial role in the electronic manufacturing industry. Precise coupling of photoelectric sensors and fiber panels is essential for achieving high-performance optical devices. In the future, how to further improve the intelligent and automated level of photoelectric device manufacturing has become the focus of the industry. For example, in the process of coupling the fiber panel and the sensor, it is necessary to ensure that the fiber panel can be accurately moved above the sensor to achieve precise alignment. Any error in the coupling process may not only cause surface scratches, but also cause loss of light transmission. In this process, the movement of the fiber panel relies on a precise machine vision system to obtain real-time three-dimensional coordinate information of the sensor, including XY plane coordinates and the height (Z-axis coordinates) of the sensor, and then use a mechanical arm or other device to perform coupling operations such as clamping and assembling the sensor according to the three-dimensional coordinate information. Traditional coupling methods usually rely on manual operation or simple mechanical devices, resulting in low efficiency, insufficient precision, and poor coupling quality and consistency. Therefore, there is an urgent need for a real-time, high-precision, and low-cost sensor three-dimensional coordinate acquisition method to improve the coupling precision, efficiency, and consistency of the fiber panel and the sensor.

[0003] Some related technologies use laser range finders or binocular vision devices to solve the precision problem, however, these technologies usually require complex hardware configurations and high-cost investment. For example, although the laser ranging system has high precision, it is expensive and susceptible to interference in complex working environments; the binocular vision system can estimate depth information, but requires high hardware synchronization and algorithm calculation capability, increasing the complexity and cost of the system. SUMMARY

[0004] In view of the technical problems of high cost and complexity of the coupling technology of the fiber panel and the sensor, the purpose of the present application is to provide a coordinate reconstruction method, device and storage medium for coupling of a fiber panel and a sensor.

[0005] In one aspect, the present application embodiment includes a coordinate reconstruction method for coupling of a fiber panel and a sensor, which includes the following steps:

[0006] A monocular camera is used to take a picture of a target sensor to obtain a to-be-identified image; the target sensor is a sensor to be coupled with the fiber panel;

[0007] inputting the image to be identified into a trained deep learning target detection model for identification, and obtaining region information of the target sensor output by the deep learning target detection model;

[0008] determining, according to the region information, planar position coordinate information of the target sensor in an image coordinate system;

[0009] obtaining internal parameters and external parameters of the monocular camera;

[0010] performing coordinate reconstruction according to the internal parameters, the external parameters and the planar position coordinate information, to obtain three-dimensional world coordinate information of the target sensor in a world coordinate system.

[0011] Further, the determining, according to the region information, of the planar position coordinate information of the target sensor comprises:

[0012] performing edge detection according to the region information to obtain original edge information;

[0013] performing edge fitting on the original edge information to obtain fitted edge information;

[0014] determining, according to the fitted edge information, a center point coordinate of the target sensor as the planar position coordinate information.

[0015] Further, the obtaining of the internal parameters and the external parameters of the monocular camera comprises:

[0016] setting a calibration pattern;

[0017] setting an intrinsic matrix and an extrinsic matrix of the monocular camera;

[0018] establishing a world coordinate system on the calibration pattern;

[0019] capturing the calibration pattern using the monocular camera to obtain multiple calibration images; different capturing angles and capturing distances corresponding to different calibration images are different;

[0020] identifying multiple feature points in each of the calibration images respectively;

[0021] obtaining actual two-dimensional point coordinate information of each of the feature points in an image coordinate system respectively;

[0022] jointly optimizing the intrinsic matrix and the extrinsic matrix using the actual two-dimensional point coordinate information;

[0023] using the jointly optimized intrinsic matrix as the internal parameters.

[0024] Further, the joint optimization of the intrinsic parameter matrix and the extrinsic parameter matrix using the actual two-dimensional point coordinate information comprises:

[0025] According to the formula

[0026]

[0027] optimization; wherein K is the intrinsic parameter matrix, R i is a rotation matrix in the extrinsic parameter matrix corresponding to the i-th calibration image, t i is a translation vector in the extrinsic parameter matrix corresponding to the i-th calibration image, P j is actual three-dimensional point coordinate information of the j-th feature point in any one of the calibration images in the world coordinate system, p ij is the actual two-dimensional point coordinate information of the j-th feature point in the i-th calibration image, is the projection point coordinate information of the j-th feature point in the i-th calibration image according to the intrinsic parameter matrix K, the rotation matrix R i , the translation vector t i and the actual three-dimensional point coordinate information P j , N is the number of the calibration images, and M is the number of feature points in any one of the calibration images.

[0028] Further, the obtaining of the internal parameter and the external parameter of the monocular camera further comprises:

[0029] using the actual two-dimensional point coordinate information and the joint-optimized intrinsic parameter matrix, the extrinsic parameter matrix is separately optimized;

[0030] the separately-optimized extrinsic parameter matrix is taken as the external parameter.

[0031] Further, the separate optimization of the extrinsic parameter matrix using the actual two-dimensional point coordinate information and the joint-optimized intrinsic parameter matrix comprises:

[0032] According to the formula

[0033]

[0034] optimization; wherein K is the joint-optimized intrinsic parameter matrix, R is a rotation matrix in the extrinsic parameter matrix corresponding to the calibration image used for separate optimization, t is a translation vector in the extrinsic parameter matrix corresponding to the calibration image used for separate optimization, P i is actual three-dimensional point coordinate information of the i-th feature point in any one of the calibration images in the world coordinate system, p iThe actual two-dimensional point coordinate information of the i-th feature point in the calibration image used for separate optimization, The i-th feature point in the calibration image used for separate optimization is calculated based on the intrinsic parameter matrix K, the rotation matrix R, the translation vector t and the actual three-dimensional point coordinate information P i The projection point coordinate information obtained by projection, N is the number of feature points in any of the calibration images.

[0035] Furthermore, the performing coordinate reconstruction according to the internal parameters, the external parameters and the plane position coordinate information to obtain three-dimensional world coordinate information of the target sensor in the world coordinate system includes:

[0036] Obtaining the actual size of the target sensor;

[0037] Determining the pixel size of the target sensor in the image coordinate system according to the plane position coordinate information;

[0038] According to the formula

[0039]

[0040] Z W =Zc-Z c '

[0041] Calculate and determine the third dimension coordinate Z of the target sensor in the world coordinate system W Wherein, X and Y are the actual sizes of the target sensor, x and y are the pixel sizes of the target sensor, and f x and f y is the focal length included in the internal parameters;

[0042] The first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system are determined according to the third dimension coordinates.

[0043] Furthermore, determining the first-dimensional coordinates and the second-dimensional coordinates of the target sensor in the world coordinate system according to the third-dimensional coordinates includes:

[0044] According to the formula

[0045]

[0046] Calculate and determine the first dimension coordinate X W and the second dimension coordinate Y W ; where r 11 -r 33wherein R is an element of a rotation matrix in the extrinsic parameters, u0 and v0 are principal points in the intrinsic parameters, u and v are the planar position coordinate information, L is a coefficient for converting from a normalized coordinate system to an absolute coordinate system, and T1-T3 are elements of a translation vector in the extrinsic parameters.

[0047] In another aspect, the embodiments of the present application also include a computer device comprising a memory and a processor, the memory being configured to store at least one program, and the processor being configured to load the at least one program to execute the coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments.

[0048] In another aspect, the embodiments of the present application also include a computer readable storage medium having stored therein a program executable by a processor, the program executable by the processor when executed by the processor being configured to execute the coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments.

[0049] The coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments has the following beneficial effects: since a monocular camera is used, the hardware cost is greatly reduced, the need for using expensive laser ranging equipment or a binocular camera is avoided, the installation and maintenance of the system are simplified, and the system is suitable for integration in a large-scale production line; the area information of the target sensor is detected by using a deep learning target detection model, the detection accuracy and robustness are improved, and high-precision three-dimensional coordinate reconstruction is achieved by using a monocular depth estimation algorithm combined with calibration parameters, so that sub-millimeter-level coupling accuracy is ensured; compared with methods based on binocular vision or laser, the coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments can improve the real-time performance and working efficiency of the system, and meet the requirements of industrial production for high efficiency and automatic operation; at the same time, the coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments has strong environmental adaptability, and can still maintain high reliability and stability in complex and harsh working environments. The system implementation process is simple, the deployment is flexible, and the production efficiency and quality consistency of the coupling between the optical fiber panel and the sensor can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A schematic diagram of the steps of the coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments;

[0051] Figure 2 A schematic diagram of the structure and principle of the coordinate reconstruction system for coupling the optical fiber panel and the sensor in the embodiments;

[0052] Figure 3 And Figure 4 A schematic diagram of the actual recognition effect of the coordinate reconstruction method for coupling the optical fiber panel and the sensor in the embodiments on the real target sensor. DETAILED DESCRIPTION

[0053] For the problems of cost and complexity in identifying the three-dimensional coordinate information of the sensor by laser range finder or binocular vision device, a monocular camera can be considered to replace these devices. When using a monocular camera to identify the three-dimensional coordinate information of the sensor, one of the difficulties is to obtain the depth information of the sensor, which generally refers to the distance between the sensor and the monocular camera. Although some monocular depth estimation techniques have made progress in monocular depth estimation using deep learning networks, they can infer depth information from a single image and perform well in non-industrial scenarios. However, monocular depth models usually require a large amount of computing resources and have a long inference time, making it difficult to achieve real-time performance in industrial scenarios, especially in high-speed operating environments on production lines. The computing bottleneck becomes a major problem that restricts system performance. In addition, the deep learning network models used by these monocular depth estimation techniques usually require a large amount of labeled data, and the process of obtaining and labeling these data in actual industrial environments is complex and time-consuming.

[0054] To address the above problems of monocular cameras, monocular vision technology can be combined with depth estimation algorithms to achieve a low-cost, efficient, and real-time technical solution. The idea of this technical solution is to use a monocular camera and calibration parameters combined with an edge detection algorithm to not only accurately identify the sensor but also estimate its three-dimensional world coordinates, achieving sub-millimeter accuracy. This method significantly improves production efficiency and ensures coupling consistency. Compared with laser ranging and binocular vision solutions, monocular cameras have low hardware cost, simple system, good real-time performance, and better robustness in harsh environments, making them suitable for precision coupling operations in industrial scenarios.

[0055] Based on the idea of the above technical solution, in this embodiment, a coordinate reconstruction method for coupling a fiber panel and a sensor is provided. Referring to Figure 1 , the coordinate reconstruction method for coupling a fiber panel and a sensor includes the following steps:

[0056] S1. Use a monocular camera to take a picture of the target sensor to obtain a to-be-identified image;

[0057] S2. Input the to-be-identified image into a trained deep learning target detection model for identification to obtain the region information of the target sensor output by the deep learning target detection model;

[0058] S3. Determine the planar position coordinate information of the target sensor in the image coordinate system according to the region information;

[0059] S4. Obtain the internal and external parameters of the monocular camera;

[0060] S5. Coordinate reconstruction is performed according to the internal parameters, the external parameters, and the planar position coordinate information, to obtain three-dimensional world coordinate information of the target sensor in the world coordinate system.

[0061] Figure 1 The coordinate reconstruction method of the fiber panel coupled with the sensor shown can be applied to Figure 2 The coordinate reconstruction system of the fiber panel coupled with the sensor shown. Referring to Figure 2 , the system includes a perception device conversion module, a calibration reconstruction module, a detection calculation module, and a mechanical arm. The system processes the sensor to be coupled with the fiber panel, that is, the target sensor, and performs steps S1-S5, thereby identifying the three-dimensional world coordinate information of the target sensor in the world coordinate system, which represents the accurate position of the target sensor, so that the mechanical arm can be controlled according to the three-dimensional world coordinate information to accurately pick up, move, and couple with the target sensor and other operations.

[0062] In this embodiment, Figure 1 The working principle of the system shown is:

[0063] The perception device conversion module obtains the image of the region where the sensor target is located through devices such as monocular cameras, and converts it into digital signals to be processed for subsequent position detection and calculation of the sensor. That is, in this embodiment, the perception device conversion module can perform step S1;

[0064] The detection calculation module quickly and accurately locates the approximate position coordinates of the sensor in the image using a deep learning model, and then selects part of it as the region of interest. Further processing of the preprocessed image information using an edge detection algorithm successfully detects the edge of the sensor coupling region. That is, in this embodiment, the detection calculation module can perform steps S2-S3;

[0065] In the calibration reconstruction module, Zhang Zhengyou calibration method is used to establish the mapping relationship between the real three-dimensional space and the image pixel coordinate system, and the pixel coordinates in the two-dimensional image are converted into three-dimensional coordinates in the real world by combining the imaging principle. Ultimately, sub-millimeter level accuracy can be achieved. That is, in this embodiment, the calibration reconstruction module can perform steps S4-S5.

[0066] Before steps S1-S5 are performed, a monocular camera can be arranged on a sensor production line, an installation angle of the monocular camera needs to be ensured to clearly observe the target sensor in a captured image, and a suitable shooting distance is selected. A calibration pattern (specifically, a calibration chessboard) can also be placed in a visual field range of the monocular camera, and a world coordinate system is determined, wherein the world coordinate system can be established on the calibration chessboard by default, so that coordinate axis positions of the world coordinate system are fixed in a subsequent process. The number of monocular cameras installed and specific parameters such as device configuration can be determined according to an actual production scene.

[0067] In step S1, the monocular camera can be used as the perception device conversion module or called by the perception device conversion module to perform. The target sensor is captured by using the monocular camera to obtain the to-be-recognized image. Specifically, the to-be-recognized image can be in the form of a single static image or in the form of a video stream.

[0068] After the perception device conversion module obtains the to-be-recognized image in step S1, the to-be-recognized image is sent to the calibration reconstruction module.

[0069] The calibration reconstruction module performs step S2, inputs the to-be-recognized image into the trained deep learning target detection model for recognition, and obtains region information of the target sensor output by the deep learning target detection model.

[0070] Specifically, the deep learning target detection model used in step S2 is a network model capable of identifying target objects and giving the position of the object in the image and the object category. The pre-trained deep learning target detection model can obtain the sensor classification detection result. In the embodiment, the YOLOv7 model can be used as the deep learning target detection model. The basic principle of the YOLOv7 model is as follows: the core idea of the YOLO target detection algorithm is to convert the target detection task into a regression problem, by dividing the image into a grid and simultaneously predicting the bounding box and category information of the target in each grid; first, the YOLO target detection algorithm needs to extract the sensor video image information captured by the fixed camera, which is used for target position prediction. Then the pictures are subjected to a single forward propagation convolution layer network, down-sampling and feature extraction to form a plurality of feature maps of different scales; further, a multi-input stacking structure is introduced for feature extraction to achieve data enhancement. At the beginning, the feature maps of different scales are placed in the SPPCSPC module for processing, then the feature maps are subjected to sampling rate change and convolution fusion and stacking fusion with feature maps of different scales to realize feature fusion and strengthening, which is convenient for subsequent prediction; further, the feature layer needs to be transmitted to the Yolo Head to obtain the prediction result; at this time, the front-end processing of the RepConv structure is introduced, which improves the prediction performance of the network by introducing a special residual structure to assist training, then the feature maps processed by the RepConv network are connected to the Yolo Head convolution network with different output channel numbers to obtain target detection results of different scales; finally, the prediction result can be decoded. First, the outputs of the three scales are decoded to obtain the position, confidence and category probability of the prediction box at each scale, then the predicted center coordinates and width-height adjustment parameters are converted into actual relative values by the sigmoid function, and the predicted bounding box coordinates are calculated according to the position of the current grid unit and the size of the anchor box.

[0071] In step S2, the deep learning target detection model used can be trained through the following training process:

[0072] P1. Obtain the image data of the sensor to be coupled by targeted shooting of the sensor, wherein the shooting range should include all sensor categories to be detected to ensure the comprehensiveness of the image data;

[0073] P2. After pre-processing and pre-classification of sensors, the sensor positions and categories in the image dataset are labeled to obtain a training set. The pre-processing includes data cleaning and deletion. If the data is insufficient, data augmentation can be used to expand the dataset, such as image rotation, translation, scaling, or color adjustment, to ensure that the weights obtained in subsequent training are correct.

[0074] P3. The labeled dataset is used to train the YOLOv7 network to obtain training weights and import them into the YOLOv7 network.

[0075] In steps P1-P3, for the Yolo Head convolutional network, an adaptive multi-positive sample matching model can be used to determine the positive samples and the category information of the matching box by calculating the IoU of the real box and the prior box and the cost of the category. Then, according to the matching of the positive samples, the target for training is constructed. Next, by comparing the matching of the predicted box and the real box, the classification loss function, the review loss function, and the confidence loss function are constructed. The losses of these three parts are jointly optimized to gradually improve the accuracy of the model in classifying, positioning, and confidence prediction during training, thereby improving the performance of the YOLOv7 model in object detection.

[0076] In the actual use of the YOLOv7 network for prediction, since the YOLOv7 network needs to ensure that the input image size is the same, the pixel length and width of the image to be recognized can be constrained to a multiple of 32. If the image to be recognized is in the form of a video stream, the image size of each video frame in the video stream can be set, and a gray bar can be used to fill the difference in image size. To ensure the robustness of the YOLOv7 network algorithm, only prediction boxes with a confidence of >0.6 can be accepted. Specifically, non-maximum suppression is introduced, and for multiple possible prediction boxes identified from the image to be recognized, only the prediction box with the highest confidence is retained to ensure the uniqueness of each sensor recognition box. Different categories are subjected to different degrees of cropping and scaling, i.e., region of interest (ROI) division, so that the edge detection result can directly recognize smaller images, significantly optimizing the recognition speed and edge detection effect. It is worth mentioning that the classification of categories can be customized according to business needs, such as for sensor edge detection, which can be divided into two categories according to the presence or absence of sensors. That is, the YOLOv7 network can provide a status detection function for whether the sensor is within the field of view, thereby improving the robustness and practicality of the YOLOv7 network.

[0077] According to the principle of the YOLOv7 model, using the trained YOLOv7 model as a deep learning target detection model can recognize the to-be-recognized image, thereby outputting region information in the form of a bounding box coordinate. That is, the region information output by the YOLOv7 model can be represented as a bounding box in the to-be-recognized image, and the region framed by the bounding box in the to-be-recognized image is the region of the target sensor in the to-be-recognized image.

[0078] In summary, when quickly positioning the position coordinates of the target sensor by using the deep learning target detection model, a large amount of image data obtained by the perception device conversion module can be collected by using the perception device conversion module to capture the sensor at different angles and distances, and the target training data set can be constructed by using artificial labeling. The collection and production process of the data set will affect the target recognition effect of the final deep learning model. At the same time, after obtaining the approximate position coordinates of the sensor in the image, the image is divided into a region of interest, and the edge of the target sensor is detected by using the edge detection algorithm, so that sub-millimeter level accuracy can be achieved in the case of monocular vision.

[0079] In this embodiment, when the detection calculation module executes step S3, that is, determines the planar position coordinate information of the target sensor according to the region information, the following steps can be executed:

[0080] S301. Perform edge detection according to the region information to obtain original edge information;

[0081] S302. Perform edge fitting on the original edge information to obtain fitted edge information;

[0082] S303. According to the fitted edge information, determine the center point coordinates of the target sensor as the planar position coordinate information.

[0083] In step S301, the detection calculation module performs edge detection on the region information obtained in step S2, thereby detecting the edge of the region information, that is, the original edge information.

[0084] Since the edge detection algorithm used by the detection calculation module in step S301 can produce a large amount of noise and unstable edge contours, affecting the subsequent information extraction, the edge represented by the original edge information obtained in step S301 can be not smooth, and therefore step S302 can be performed to perform edge fitting on the original edge information to obtain fitted edge information.

[0085] Specifically, in step S302, an edge fitting technique is adopted to smooth and optimize the original edge information. Specifically, in step S302, the detection calculation module can fit the edge lines in the original edge information to generate corresponding straight lines, and calculate the distance between each edge line in the original edge information and the corresponding straight line, if the distance between the edge line and the straight line is less than a preset threshold, the straight line is used to replace the original edge line, so as to simplify the edge contour to a straight line, and convert the original edge information to smooth fitting edge information. This method significantly improves the contour stability and provides a reliable foundation for subsequent depth calculation.

[0086] Since the shape of the target sensor is generally rectangular, the fitting edge information obtained by performing step S302 is generally represented as a rectangle. In step S303, four corner points of the fitting edge information can be extracted and connected, thereby obtaining two diagonal lines of the fitting edge information, and determining the planar position coordinate information of the target sensor with the intersection of the diagonal lines as the center of the coordinates (u, v) in the to-be-identified image.

[0087] By performing steps S301-S303, the planar position coordinate information (u, v) obtained represents the coordinates of the target sensor in the image coordinate system, wherein the image coordinate system is established on the to-be-identified image, that is, the image coordinate system is in the same plane as the to-be-identified image.

[0088] In step S4, the internal and external parameters of the monocular camera are calibrated by the calibration reconstruction module, and the internal and external parameters of the monocular camera are solved, which represent the displacement and rotation vector of the position and posture of the monocular camera relative to the world coordinate axis.

[0089] In the embodiment, when the calibration reconstruction module performs step S4, that is, when the internal and external parameters of the monocular camera are obtained, the following steps can be performed:

[0090] S401. Set a calibration pattern;

[0091] S402. Set the internal parameter matrix and the external parameter matrix of the monocular camera;

[0092] S403. Establish a world coordinate system on the calibration pattern;

[0093] S404. Capture the calibration pattern using the monocular camera to obtain multiple calibration images;

[0094] S405. Identify multiple feature points in each calibration image, respectively;

[0095] S406. Obtain the actual two-dimensional point coordinate information of each feature point in the image coordinate system, respectively;

[0096] S407. Jointly optimize the intrinsic matrix and the extrinsic matrix using the actual two-dimensional point coordinate information;

[0097] S408. Take the jointly optimized intrinsic matrix as the internal parameter.

[0098] S409. Independently optimize the extrinsic matrix using the actual two-dimensional point coordinate information and the jointly optimized intrinsic matrix;

[0099] S410. Take the independently optimized extrinsic matrix as the external parameter.

[0100] In step S401, a calibration chessboard can be placed in the field of view of the monocular camera as a calibration pattern, so that the monocular camera can capture the calibration pattern.

[0101] In step S402, the calibration reconstruction module sets the initial value of the intrinsic matrix and the initial value of the extrinsic matrix of the monocular camera. In this embodiment,

[0102] The form of the intrinsic matrix of the monocular camera is

[0103]

[0104] where f x and f y represent the focal length of the monocular camera, and u0 and v0 represent the principal point of the monocular camera.

[0105] The form of the extrinsic matrix of the monocular camera is

[0106]

[0107] where R is a rotation matrix and T is a translation vector.

[0108] In step S403, a world coordinate system is established on the calibration pattern, so that the three-dimensional world coordinate information (X W , Y W , Z W ) of the target sensor can be represented in the world coordinate system. Where (X W , Y W ) is the coordinate of the target sensor in the plane parallel to the image to be recognized, Z W is the coordinate of the target sensor in the coordinate axis perpendicular to the image to be recognized, and Z W can be understood as the actual height of the target sensor.

[0109] Through the intrinsic matrix and the extrinsic matrix, the planar position coordinate information (u, v) of the target sensor in the image coordinate system and the three-dimensional world coordinate information (X W , Y W , Z Wrelationship between the two:

[0110]

[0111] In formula (1), Z c is the depth of the monocular camera from the target sensor in the image coordinate system.

[0112] After performing steps S1-S3, only the plane position coordinate information (u, v) in formula (1) is known, and the final goal is to obtain three-dimensional world coordinate information (X W , Y W , Z W ), so steps S404-S410 can be performed first to obtain the internal and external parameters.

[0113] In step S404, the monocular camera is used to capture the calibration pattern to obtain multiple calibration images. Each calibration image corresponds to a unique shooting angle and shooting distance. In this embodiment, a total of N calibration images are captured. Since the feature points in a single image may be distributed relatively concentratedly in the implementation process, resulting in inaccurate estimation of some parameters, multiple images can ensure that the feature points are distributed more uniformly at different angles and distances, thereby improving the calibration accuracy. At the same time, using multiple images can reduce the influence of inaccurate corner detection or noise in some images, so it is preferred to use more than 10 calibration images, i.e. N>10.

[0114] In step S405, multiple feature points are identified in each calibration image, respectively. In this embodiment, the feature points can be determined at the same position in different calibration images, and M feature points are determined in each calibration image.

[0115] In step S406, since the feature points are located in the calibration image, and the calibration image is captured by the monocular camera as the image to be identified, the respective coordinate points of each feature point can be determined in the image coordinate system, i.e. the actual two-dimensional point coordinate information of each feature point. Since the position of the calibration pattern can be specially designed, the coordinate points of the feature points in the world coordinate system can be determined, i.e. the actual three-dimensional point coordinate information of the feature points in the world coordinate system. In this embodiment, the actual two-dimensional point coordinate information of the jth feature point in the ith calibration image is denoted as p ij , and the actual three-dimensional point coordinate information of the jth feature point in any calibration image in the world coordinate system is denoted as P j .

[0116] In step S407, the calibration reconstruction module can perform a nonlinear least squares method to jointly optimize the intrinsic matrix and the extrinsic matrix using the actual two-dimensional point coordinate information.

[0117] Specifically, the calibration reconstruction module can be according to the formula

[0118]

[0119] Optimization. In formula (2), K is an intrinsic matrix, R i is a rotation matrix in the extrinsic matrix corresponding to the i-th calibration image, t i is a translation vector in the extrinsic matrix corresponding to the i-th calibration image, is the j-th feature point in the i-th calibration image according to the intrinsic matrix K, the rotation matrix R i , the translation vector t i and the actual three-dimensional point coordinate information P j projected to obtain the projection point coordinate information.

[0120] In formula (2), the Levenberg-Marquardt optimization algorithm can be used to solve the above target, which is similar to the Gauss-Newton method when the update step is small, and is similar to gradient descent when the step is large. This combination can accelerate convergence when approaching the solution, while avoiding falling into local minimum when far from the solution. Wherein, when the re-projection error reaches the minimum, the intrinsic matrix K at this time is taken out, and the joint optimization of the intrinsic matrix K and the extrinsic matrix (R i , t i ) is completed.

[0121] In formula (2), by jointly optimizing the intrinsic matrix K and the extrinsic matrix (R i , t i ), the intrinsic matrix K obtained after joint optimization in step S408 can be used as the internal parameter in step S5.

[0122] The intrinsic matrix obtained by step S407 is globally optimized, which is the global optimal solution on all calibration images. Since the calibration images obtained by the monocular camera shooting the calibration pattern come from different angles and distances, the extrinsic matrix obtained in the calibration process is not a global extrinsic matrix, and can only be used to help obtain the optimal intrinsic matrix K. Therefore, in step S408, the extrinsic matrix after joint optimization can not be output, but step S409 is executed to optimize the extrinsic matrix of each calibration image separately to ensure that the projection error of each calibration image is minimized.

[0123] To ensure the robustness of the algorithm, a stable and unique extrinsic parameter matrix needs to be obtained, so the PnP algorithm can be used to solve the homography matrix and calculate the extrinsic parameter matrix. Since the intrinsic parameter matrix used as the internal parameter in step S5 has been obtained by calibration in step S407, a unique extrinsic parameter can be obtained for a single calibration image in the case of a fixed monocular camera, and a coordinate system can be established based on the extrinsic parameter. Based on this principle, step S409 can be performed to separately optimize the extrinsic parameter matrix using the actual two-dimensional point coordinate information and the jointly optimized intrinsic parameter matrix.

[0124] Specifically, in step S409, the optimization is performed according to the formula

[0125]

[0126] where K is the jointly optimized intrinsic parameter matrix, R is the rotation matrix in the extrinsic parameter matrix corresponding to the calibration image used for separate optimization, t is the translation vector in the extrinsic parameter matrix corresponding to the calibration image used for separate optimization, P i is the actual three-dimensional point coordinate information of the i-th feature point in any calibration image in the world coordinate system, p i is the actual two-dimensional point coordinate information of the i-th feature point in the calibration image used for separate optimization, is the projection point coordinate information obtained by projecting the i-th feature point in the calibration image used for separate optimization according to the intrinsic parameter matrix K, the rotation matrix R, the translation vector t, and the actual three-dimensional point coordinate information P i .

[0127] In step S409, R directly solved by formula (3) is in vector form, in order to facilitate subsequent calculation, R can be converted into the form of a rotation matrix. Specifically, R can be obtained by the following formula:

[0128]

[0129] where θ = ||R|| is the rotation angle of R in vector form, [R] x is the skew-symmetric matrix corresponding to R in vector form. By formula (4), R in vector form can be converted into R mat in the form of a rotation matrix. In this embodiment, R in vector form and R mat in the form of a rotation matrix can not be distinguished, for example, R can be directly considered as a rotation matrix.

[0130] In step S410, the extrinsic parameter matrix (R, t) obtained after separate optimization can be used as the external parameter used in step S5.

[0131] In this embodiment, when performing the step S5, i.e., the step of performing coordinate reconstruction according to the internal parameters, the external parameters and the planar position coordinate information to obtain the three-dimensional world coordinate information of the target sensor in the world coordinate system, the calibration reconstruction module can perform the following steps:

[0132] S501. Obtain the actual size of the target sensor;

[0133] S502. Determine the pixel size of the target sensor in the image coordinate system according to the planar position coordinate information;

[0134] S503. Calculate the third-dimensional coordinate Z of the target sensor in the world coordinate system according to the formula

[0135]

[0136] Z W =Zc-Z c ′(5)

[0137] to determine the third-dimensional coordinate Z of the target sensor in the world coordinate system. W ; wherein X and Y are the actual size of the target sensor, x and y are the pixel size of the target sensor, f x and f y are the focal length contained in the internal parameters (wherein f x is the same dimension as x, and f y is the same dimension as y);

[0138] S504. Determine the first-dimensional coordinate and the second-dimensional coordinate of the target sensor in the world coordinate system according to the third-dimensional coordinate.

[0139] In step S501, the actual size of the target sensor is generally fixed on a specific production line, so the fixed length X and width Y of the target sensor can be obtained.

[0140] In step S502, the calibration reconstruction module can determine the pixel size x and y of the region of the target sensor in the image coordinate system in the to-be-recognized image according to the planar position coordinate information (u, v) or the corresponding fitted edge information, wherein x and X are the same dimension, and y and Y are the same dimension.

[0141] In step S503, the current depth of the target sensor can be estimated. First, the similar triangle in the imaging principle is used to estimate the depth Z cThe transformation based on similar triangles depends on the length of the pixels of the edges, and the length of the pixels of the four edges of the sensor will be different due to different observation angles, for example, the rectangular target area can become a rhombus or a trapezoid due to perspective. Therefore, before extracting the pixel coordinates, an affine transformation needs to be performed to convert the four corners of the sensor into a relatively flat 90 degrees. If not, only the central part close to the monocular camera can get relatively accurate results.

[0142] The principle of step S503 is that the depth of the monocular camera (the height of the monocular camera relative to the origin of the world coordinate system) is Z c , the depth of the target sensor (the height of the target sensor relative to the monocular camera) is Z c ′, according to the pinhole imaging principle, two similar triangles are obtained:

[0143]

[0144] Since the length of the sensor in the world coordinate system is the same as the length of the camera coordinate system, and the calibration process obtains the translation vector T, which contains [t x t y t z ] T , where the distance from the camera to the Z-axis direction of the world coordinate is given as t z , and the distance from the world coordinate Z w = 0 is Z c = t z *L (L is the coefficient of converting the normalized coordinate system to the absolute coordinate), so Z W can be calculated. In the case where the monocular camera is above the target sensor, the actual height Z W of the target sensor is:

[0145] Z W = Zc-Z c ′

[0146] According to the above principle, formula (5) used in step S503 can be obtained.

[0147] For Z c ′ in formula (5) used in step S503, in addition to being calculated directly according to the imaging principle by formula

[0148]

[0149] , it can also be calculated by the following formula:

[0150]

[0151]

[0152] wherein P is the number of image frames contained in the video stream if the image to be processed is in the form of a video stream; x ij is the pixel size of the jth edge of the region where the target sensor is located in the ith image in the image to be processed in the image coordinate system (same dimension as X), y ik is the pixel size of the kth edge of the region where the target sensor is located in the ith image in the image to be processed in the image coordinate system (same dimension as Y).

[0153] The principle of formula (6) is that the target sensor is small in size, the pixels of the region of interest are low, and a slight change in the edge can cause a large disturbance in the Z axis (the direction perpendicular to the image to be recognized), so the four edges of the target sensor are calculated to obtain the Z i , wherein Z i represents the distance between the sensor recognized in the ith image in the image to be processed and the monocular camera in the Z axis. The mean value of the respective Z i of P images is taken as the final Z c ', which can smooth the value of the Z axis to ensure the stability of the Z axis coordinate.

[0154] Z c ' is calculated by using formula (6), formula (5) can be optimized as:

[0155]

[0156] That is, Z W can be calculated by formula (7), which represents the height of the target sensor relative to the origin of the world coordinate system.

[0157] In the case of only knowing the pixel coordinates u, v, it is impossible to directly obtain a unique solution, because each image coordinate point can correspond to an infinite number of three-dimensional space points in the direction of the camera optical axis. That is, without specifying the distance of the object from the camera, there will be infinitely many solutions. In monocular vision, this phenomenon can be geometrically understood as a lack of depth information: although the pixel coordinates (u, v) give the position of the target on the image plane, due to the lack of the third dimension (i.e. depth Zw), the actual position of the object in three-dimensional space is uncertain. Each solution corresponds to a point in space along the direction of the camera light. However, after steps S1-S4 and S501-S503 in step S5 are performed, the third dimension coordinate Z W of the target sensor in the world coordinate system is obtained, and the internal parameters (internal matrix after joint optimization)

[0158]

[0159] and external parameters (external matrix optimized separately)

[0160]

[0161] and the depth Z of the monocular camera c is known, thus formula (1) can be solved to obtain the first dimension coordinate X W and the second dimension coordinate Y W of the target sensor in the world coordinate system. Specifically, formula (1) is solved to obtain

[0162]

[0163] In formula (7), r 11 , r 12 , r 13 , r 21 , r 22 , r 23 , r 31 , r 32 and r 33 are each element in the 3x3 rotation matrix in the external parameters, T1, T2 and T3 are each element in the 3x1 translation vector in the external parameters. L is the coefficient for converting the normalized coordinate system to the absolute coordinate, the value of which depends on the absolute length of one square black and white grid in the calibration pattern (calibration chessboard) in the Zhang Zhengyou calibration method used, in the embodiment, the value of L is 10mm, which can be changed according to the specific situation.

[0164] In formula (7), in addition to X w and Y W , the values of other parameters have been obtained in each step before step S503, thus in step S503, the first dimension coordinate X w and the second dimension coordinate Y w of the target sensor in the world coordinate system can be obtained by solving formula (7). Finally, the three-dimensional world coordinate information (X w , Y w , Z w ) of the target sensor in the world coordinate system is obtained.

[0165] In the embodiment, by performing steps S1-S5 on the real target sensor, the three-dimensional world coordinate information (X w , Y w , Z W ) of the target sensor in the world coordinate system is obtained, which has the effects as Figure 3 and Figure 4as shown. Figure 3 and Figure 4 In the three-dimensional world coordinate information is displayed in the upper right corner of each figure, by Figure 4 It can be seen that the accuracy of the three-dimensional world coordinate information reaches sub-millimeter level.

[0166] Since the three-dimensional world coordinate information (X W ,Y W ,Z W ) describes the position of the target sensor more accurately, the three-dimensional world coordinate information (X W ,Y W ,Z W ) can be input to the robot arm for the next operation, and the robot arm can accurately move to the position of the target sensor described by the three-dimensional world coordinate information (X W ,Y W ,Z W ), thereby accurately coupling the target sensor and the fiber panel.

[0167] The coordinate reconstruction method for coupling the fiber panel and the sensor in the embodiment can be implemented by writing a computer program for executing the coordinate reconstruction method for coupling the fiber panel and the sensor in the embodiment, writing the computer program into a computer device or a storage medium, and executing the coordinate reconstruction method for coupling the fiber panel and the sensor in the embodiment when the computer program is read and run, thereby achieving the same technical effects as the coordinate reconstruction method for coupling the fiber panel and the sensor in the embodiment.

[0168] It should be noted that, unless otherwise specified, when a certain feature is referred to as being "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. In addition, the up, down, left, right and other descriptions used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the drawings. In the present disclosure, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used in the present embodiment are the same as those commonly understood by those skilled in the art. The terms used in the present embodiment are only used to describe the specific embodiments, and are not intended to limit the present application. The term "and / or" used in the present embodiment includes any combination of one or more related listed items.

[0169] It should be understood that, although the terms first, second, third, etc. can be used herein to describe various elements, components, regions, layers and / or sections, these elements should not be limited to these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. The use of any and all examples, or exemplary language (e.g., "such as", "for instance", etc.) provided herein, is intended merely to better illuminate the present embodiments and does not pose a limitation on the scope of the present disclosure unless otherwise claimed.

[0170] It will be appreciated that embodiments of the present application can be realized by computer hardware, a combination of hardware and software, or by computer instructions stored on a non-transitory computer-readable storage medium. The methods can be implemented in a computer program, using standard programming techniques, including the configuration of non-transitory computer-readable storage media with a computer program, wherein the storage medium so configured with the computer program instructs the computer to operate in a specific and predefined manner according to the method described in the specific embodiments and the accompanying drawings. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, the program can be implemented in assembly or machine language, if desired. In any case, the language can be a compiled or interpreted language. Furthermore, the program can be able to run on a specially programmed integrated circuit for this purpose.

[0171] Furthermore, the operations of the processes described in the present embodiments can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described in the present embodiments (or variations and / or combinations thereof) can be implemented under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications), and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. A computer program includes a plurality of instructions executable by one or more processors.

[0172] Further, the methods can be implemented in any type of computing platform operably connected to the appropriate, including but not limited to a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. In addition, the machine readable code, or portions thereof, can be transmitted over wired or wireless networks. The present embodiments of the application include these and other different types of non-transitory computer readable storage media when the instructions or programs incorporating the above steps are implemented in conjunction with a microprocessor or other data processor. The present application also includes the computer itself when programmed in accordance with the methods and techniques of the present application.

[0173] The computer program can be applied to input data to perform the functions of the present embodiments, thereby transforming the input data to generate output data that is stored to non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In the preferred embodiments of the present application, the transformed data represents a physical and tangible object, including a particular visual depiction of the physical and tangible object produced on a display.

[0174] The above merely preferred embodiments of the present application and are not intended to limit the present application thereto. The present application is not limited to the above-described embodiments, but any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application. The technical solutions and / or embodiments of the present application can have various modifications and changes within the scope of the present application.

Claims

1. A coordinate reconstruction method for coupling an optical fiber panel with a sensor, characterized in that: The coordinate reconstruction method of the optical fiber panel coupled with the sensor includes: Use a monocular camera to shoot the target sensor to obtain an image to be identified; the target sensor is a sensor to be coupled to the optical fiber panel; Inputting the image to be identified into a trained deep learning target detection model for identification, and obtaining the area information of the target sensor output by the deep learning target detection model; Determining, based on the region information, plane position coordinate information of the target sensor in the image coordinate system; Obtaining internal and external parameters of the monocular camera; Coordinate reconstruction is performed according to the internal parameters, the external parameters and the plane position coordinate information to obtain three-dimensional world coordinate information of the target sensor in a world coordinate system.

2. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 1, characterized in that: Determining the plane position coordinate information of the target sensor according to the area information includes: Perform edge detection based on the region information to obtain original edge information; Performing edge fitting on the original edge information to obtain fitted edge information; According to the fitted edge information, the center point coordinates of the target sensor are determined as the plane position coordinate information.

3. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 1, characterized in that: The obtaining of internal parameters and external parameters of the monocular camera includes: Set the calibration pattern; Setting the intrinsic parameter matrix and extrinsic parameter matrix of the monocular camera; Establishing a world coordinate system on the calibration pattern; Using the monocular camera to shoot the calibration pattern to obtain multiple calibration images; different calibration images correspond to different shooting angles and shooting distances; Identifying a plurality of feature points in each of the calibration images; Obtaining actual two-dimensional point coordinate information of each of the feature points in the image coordinate system; Using the actual two-dimensional point coordinate information to jointly optimize the intrinsic parameter matrix and the extrinsic parameter matrix; The internal parameter matrix after joint optimization is used as the internal parameter.

4. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 3, characterized in that: The jointly optimizing the intrinsic parameter matrix and the extrinsic parameter matrix using the actual two-dimensional point coordinate information includes: According to the formula Optimize; where K is the internal parameter matrix, R i is the rotation matrix corresponding to the i-th calibration image in the external parameter matrix, t i is the translation vector corresponding to the i-th calibration image in the extrinsic parameter matrix, P j is the actual three-dimensional point coordinate information of the j-th feature point in any of the calibration images in the world coordinate system, p ij is the actual two-dimensional point coordinate information of the j-th feature point in the i-th calibration image, For the j-th feature point in the ith calibration image, the internal parameter matrix K and the rotation matrix R are used. i , translation vector t i and the actual 3D point coordinate information P j The projection point coordinate information obtained by projection, N is the number of the calibration images, and M is the number of feature points in any of the calibration images.

5. The coordinate reconstruction method of the optical fiber panel coupled with the sensor according to claim 3, characterized in that: The obtaining of the internal parameters and external parameters of the monocular camera further includes: Using the actual two-dimensional point coordinate information and the jointly optimized intrinsic parameter matrix, the extrinsic parameter matrix is ​​optimized separately; The independently optimized external parameter matrix is ​​used as the external parameter.

6. The coordinate reconstruction method of the optical fiber panel coupled with the sensor according to claim 5, characterized in that: The step of separately optimizing the extrinsic parameter matrix using the actual two-dimensional point coordinate information and the jointly optimized intrinsic parameter matrix includes: According to the formula Optimize; wherein K is the internal parameter matrix after joint optimization, R is the rotation matrix in the external parameter matrix corresponding to the calibration image used for separate optimization, t is the translation vector in the external parameter matrix corresponding to the calibration image used for separate optimization, P i is the actual three-dimensional point coordinate information of the i-th feature point in any of the calibration images in the world coordinate system, p i The actual two-dimensional point coordinate information of the i-th feature point in the calibration image used for separate optimization, The i-th feature point in the calibration image used for separate optimization is calculated based on the intrinsic parameter matrix K, the rotation matrix R, the translation vector t and the actual three-dimensional point coordinate information P i The projection point coordinate information obtained by projection, N is the number of feature points in any of the calibration images.

7. The coordinate reconstruction method of the optical fiber panel coupled with the sensor according to any one of claims 1 to 6, characterized in that: The performing coordinate reconstruction according to the internal parameters, the external parameters, and the plane position coordinate information to obtain three-dimensional world coordinate information of the target sensor in the world coordinate system includes: Obtaining the actual size of the target sensor; Determining the pixel size of the target sensor in the image coordinate system according to the plane position coordinate information; According to the formula WITH W =Z c -WITH c ′ Calculate and determine the third dimension coordinate Z of the target sensor in the world coordinate system W Wherein, X and Y are the actual sizes of the target sensor, x and y are the pixel sizes of the target sensor, and f x and f y is the focal length included in the internal parameters; The first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system are determined according to the third dimension coordinates.

8. The coordinate reconstruction method of the optical fiber panel coupled with the sensor according to claim 7, characterized in that: Determining the first-dimensional coordinates and the second-dimensional coordinates of the target sensor in the world coordinate system according to the third-dimensional coordinates includes: According to the formula Calculate and determine the first dimension coordinate X W and the second dimension coordinate Y W ; where r 11 -r 33 are the elements of the rotation matrix in the external parameters, u0 and v0 are the principal points in the internal parameters, u and v are the plane position coordinate information, L is the coefficient for converting from the normalized coordinate system to the absolute coordinate system, and T1-T3 are the elements of the translation vector in the external parameters.

9. A computer device, characterized in that: It includes a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the coordinate reconstruction method of the optical fiber panel and sensor coupling according to any one of claims 1-8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the coordinate reconstruction method of the optical fiber panel and sensor coupling as described in any one of claims 1 to 8 when executed by the processor.

Images