Coordinate reconstruction method and device for coupling of optical fiber panel and sensor, and storage medium

Abstract

The invention discloses a coordinate reconstruction method and device for coupling an optical fiber panel and a sensor and a storage medium, and the method employs a monocular camera, thereby greatly reducing the hardware cost, avoiding the use of expensive laser ranging equipment or a binocular camera, simplifying the installation and maintenance of a system, and being suitable for the integration of a large-scale production line. The regional information of the target sensor is detected through the deep learning target detection model, and the detection precision and robustness are improved. The real-time performance and the working efficiency of the system can be improved, and the requirements of industrial production for efficient and automatic operation are met; meanwhile, the coordinate reconstruction method for coupling of the optical fiber panel and the sensor in the embodiment has high environment adaptability, high reliability and stability can be kept in a complex and severe working environment, the implementation process is simple, deployment is flexible, and the production efficiency and quality consistency of coupling of the optical fiber panel and the sensor can be effectively improved. The method is widely applied to the technical field of machine vision.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a coordinate reconstruction method, device and storage medium for coupling an optical fiber panel with a sensor. Background Art

[0002] In the rapid development of contemporary science and technology, optoelectronic equipment manufacturing plays a vital role in the electronics manufacturing industry. The precise coupling of photoelectric sensors and fiber panels is essential to the realization of high-performance optical devices. In the future, how to further improve the intelligence and automation level of optoelectronic equipment manufacturing has become a key focus of the industry. For example, in the process of bonding the fiber panel and the sensor, it is necessary to ensure that the fiber panel can be accurately moved above the sensor to achieve precise alignment. Any error in the coupling process may not only cause surface scratches, but also cause loss of light transmission. In this process, the movement of the fiber panel depends on a precise machine vision system to obtain the three-dimensional coordinate information of the sensor in real time, including the XY plane coordinates and the height of the sensor (Z-axis coordinate), and then use a robotic arm and other devices to perform coupling operations such as clamping and assembly of the sensor according to the three-dimensional coordinate information. Traditional coupling methods usually rely on manual operation or simple mechanical devices, resulting in low efficiency, insufficient precision, and poor coupling quality and consistency. Therefore, there is an urgent need for a real-time, high-precision, low-cost sensor three-dimensional coordinate acquisition method to improve the coupling accuracy, efficiency and consistency of the fiber panel and the sensor.

[0003] Some related technologies use laser rangefinders or binocular vision devices to solve the accuracy problem. However, these technologies usually require complex hardware configuration and high cost investment. For example, although the laser rangefinder system has high accuracy, the hardware cost is expensive and it is easily disturbed in complex working environments; although the binocular vision system can estimate depth information, it has high requirements for hardware synchronization and algorithm computing power, which increases the complexity and cost of the system. Summary of the invention

[0004] In view of the technical problems of high cost and complexity of the current coupling technology of optical fiber panel and sensor, the object of the present invention is to provide a coordinate reconstruction method, device and storage medium for coupling optical fiber panel and sensor.

[0005] On the one hand, an embodiment of the present invention includes a coordinate reconstruction method for coupling an optical fiber panel with a sensor, and the coordinate reconstruction method for coupling an optical fiber panel with a sensor includes the following steps:

[0006] Use a monocular camera to shoot the target sensor to obtain an image to be identified; the target sensor is a sensor to be coupled to the optical fiber panel;

[0007] Inputting the image to be identified into a trained deep learning target detection model for identification, and obtaining the area information of the target sensor output by the deep learning target detection model;

[0008] Determining the plane position coordinate information of the target sensor in the image coordinate system according to the area information;

[0009] Obtaining internal and external parameters of the monocular camera;

[0010] Coordinate reconstruction is performed according to the internal parameters, the external parameters and the plane position coordinate information to obtain three-dimensional world coordinate information of the target sensor in the world coordinate system.

[0011] Further, determining the plane position coordinate information of the target sensor according to the area information includes:

[0012] Perform edge detection according to the region information to obtain original edge information;

[0013] Performing edge fitting on the original edge information to obtain fitted edge information;

[0014] According to the fitting edge information, the center point coordinates of the target sensor are determined as the plane position coordinate information.

[0015] Furthermore, the obtaining of the internal parameters and external parameters of the monocular camera includes:

[0016] Set the calibration pattern;

[0017] Setting the intrinsic parameter matrix and extrinsic parameter matrix of the monocular camera;

[0018] Establishing a world coordinate system on the calibration pattern;

[0019] Using the monocular camera to shoot the calibration pattern to obtain a plurality of calibration images; different calibration images correspond to different shooting angles and shooting distances;

[0020] Respectively identifying a plurality of feature points in each of the calibration images;

[0021] Obtaining actual two-dimensional point coordinate information of each of the feature points in the image coordinate system;

[0022] Using the actual two-dimensional point coordinate information to jointly optimize the intrinsic parameter matrix and the extrinsic parameter matrix;

[0023] The internal parameter matrix after joint optimization is used as the internal parameter.

[0024] Furthermore, the using the actual two-dimensional point coordinate information to jointly optimize the intrinsic parameter matrix and the extrinsic parameter matrix includes:

[0025] According to the formula

[0026]

[0027] Optimize; where K is the internal parameter matrix, R i is the rotation matrix corresponding to the i-th calibration image in the external parameter matrix, t i is the translation vector corresponding to the i-th calibration image in the extrinsic parameter matrix, P j is the actual 3D point coordinate information of the jth feature point in any of the calibration images in the world coordinate system, p ij is the actual two-dimensional point coordinate information of the j-th feature point in the i-th calibration image, For the jth feature point in the ith calibration image, according to the internal parameter matrix K and the rotation matrix R i , translation vector t i and the actual 3D point coordinate information P j The projection point coordinate information obtained by projection, N is the number of the calibration images, and M is the number of feature points in any of the calibration images.

[0028] Furthermore, the obtaining of the internal parameters and external parameters of the monocular camera further includes:

[0029] Using the actual two-dimensional point coordinate information and the jointly optimized internal parameter matrix, the external parameter matrix is ​​optimized separately;

[0030] The independently optimized external parameter matrix is ​​used as the external parameter.

[0031] Furthermore, the using the actual two-dimensional point coordinate information and the jointly optimized intrinsic parameter matrix to separately optimize the extrinsic parameter matrix includes:

[0032] According to the formula

[0033]

[0034] Optimize; wherein K is the internal parameter matrix after joint optimization, R is the rotation matrix in the external parameter matrix corresponding to the calibration image used for separate optimization, t is the translation vector in the external parameter matrix corresponding to the calibration image used for separate optimization, P i is the actual 3D point coordinate information of the i-th feature point in any of the calibration images in the world coordinate system, p iThe actual two-dimensional point coordinate information of the i-th feature point in the calibration image used for separate optimization, The i-th feature point in the calibration image used for separate optimization is calculated based on the intrinsic parameter matrix K, the rotation matrix R, the translation vector t and the actual three-dimensional point coordinate information P i The projection point coordinate information obtained by projection, N is the number of feature points in any of the calibration images.

[0035] Further, the performing coordinate reconstruction according to the internal parameters, the external parameters and the plane position coordinate information to obtain the three-dimensional world coordinate information of the target sensor in the world coordinate system includes:

[0036] Obtaining the actual size of the target sensor;

[0037] Determining the pixel size of the target sensor in the image coordinate system according to the plane position coordinate information;

[0038] According to the formula

[0039]

[0040] Z W =Zc-Z c '

[0041] Calculate and determine the third dimension coordinate Z of the target sensor in the world coordinate system W ; Wherein, X and Y are the actual sizes of the target sensor, x and y are the pixel sizes of the target sensor, and f x and f y is the focal length included in the internal parameters;

[0042] The first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system are determined according to the third dimension coordinates.

[0043] Further, determining the first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system according to the third dimension coordinates includes:

[0044] According to the formula

[0045]

[0046] Calculate and determine the first dimension coordinate X W and the second dimension coordinate Y W ; where r 11 -r 33 is the element of the rotation matrix in the external parameters, u 0 and v 0is the principal point in the internal parameters, u and v are the plane position coordinate information, L is the coefficient for converting from the normalized coordinate system to the absolute coordinate system, T 1 -T 3 is the element of the translation vector in the external parameters.

[0047] On the other hand, an embodiment of the present invention also includes a computer device including a memory and a processor, the memory being used to store at least one program, and the processor being used to load at least one program to execute the coordinate reconstruction method of the fiber optic panel and sensor coupling in the embodiment.

[0048] On the other hand, an embodiment of the present invention further includes a computer-readable storage medium storing a program executable by a processor, and the program executable by the processor is used to execute the coordinate reconstruction method of the optical fiber panel and the sensor coupling in the embodiment when executed by the processor.

[0049] The beneficial effects of the present invention are as follows: the coordinate reconstruction method of the optical fiber panel coupled with the sensor in the embodiment, due to the use of a monocular camera, not only greatly reduces the hardware cost, avoids the need to use expensive laser ranging equipment or binocular cameras, but also simplifies the installation and maintenance of the system, and is suitable for the integration of large-scale production lines; the regional information of the target sensor is detected by a deep learning target detection model, the detection accuracy and robustness are improved, and the high-precision three-dimensional coordinate reconstruction is achieved by combining the monocular depth estimation algorithm with the calibration parameters, ensuring the coupling accuracy of the sub-millimeter level; compared with methods based on binocular vision or laser, the coordinate reconstruction method of the optical fiber panel coupled with the sensor in the embodiment can improve the real-time performance and work efficiency of the system, and meet the requirements of industrial production for efficient and automated operation; at the same time, the coordinate reconstruction method of the optical fiber panel coupled with the sensor in the embodiment has a strong environmental adaptability, and can still maintain high reliability and stability in complex and harsh working environments. The system implementation process is simple and the deployment is flexible, which can effectively improve the production efficiency and quality consistency of the optical fiber panel coupled with the sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the steps of a coordinate reconstruction method for coupling an optical fiber panel with a sensor in an embodiment;

[0051] Figure 2 A schematic diagram of the structure and principle of a coordinate reconstruction system of a fiber optic panel coupled with a sensor in an embodiment;

[0052] Figure 3 and Figure 4 It is a schematic diagram of the actual recognition effect of the physical target sensor on the coordinate reconstruction method of the optical fiber panel and sensor coupling in the embodiment. DETAILED DESCRIPTION

[0053] Monocular cameras can be used to replace the high cost and complexity of laser rangefinders or binocular vision devices to identify the three-dimensional coordinate information of sensors. When using a monocular camera to identify the three-dimensional coordinate information of a sensor, one of the difficulties is to obtain the depth information of the sensor. The depth information generally refers to the distance between the sensor and the monocular camera. Although some monocular depth estimation technologies have made progress in monocular depth estimation using deep learning networks, they can infer depth information from a single image and perform well in non-industrial scenarios. However, monocular depth models usually require a lot of computing resources and have a long inference time. It is difficult to achieve real-time performance in industrial scenarios, especially in the high-speed operation environment of the production line. The computing bottleneck becomes a major problem that restricts system performance. In addition, the deep learning network models used by these monocular depth estimation technologies usually require a large amount of labeled data. The process of obtaining and labeling this data in an actual industrial environment is complex and time-consuming.

[0054] In view of the above problems of monocular cameras, it is possible to combine monocular vision technology with depth estimation algorithms to achieve a low-cost, efficient and real-time technical solution. The idea of ​​this technical solution is: through the monocular camera and calibration parameters, combined with the edge detection algorithm, it is possible not only to accurately identify the sensor, but also to estimate its three-dimensional world coordinates to achieve sub-millimeter accuracy. This method significantly improves production efficiency and ensures the consistency of coupling. Compared with laser ranging and binocular vision solutions, monocular cameras have low hardware cost, simple system, good real-time performance, and better robustness in harsh environments, which are suitable for precision coupling operations in industrial scenarios.

[0055] Based on the idea of ​​the above technical solution, in this embodiment, a coordinate reconstruction method for coupling an optical fiber panel with a sensor is provided. Figure 1 , the coordinate reconstruction method of the optical fiber panel coupled with the sensor includes the following steps:

[0056] S1. Use a monocular camera to shoot the target sensor to obtain an image to be recognized;

[0057] S2. Input the image to be identified into the trained deep learning target detection model for identification, and obtain the area information of the target sensor output by the deep learning target detection model;

[0058] S3. Determine the plane position coordinate information of the target sensor in the image coordinate system according to the area information;

[0059] S4. Obtaining internal and external parameters of the monocular camera;

[0060] S5. Reconstruct coordinates according to the internal parameters, external parameters and plane position coordinate information to obtain the three-dimensional world coordinate information of the target sensor in the world coordinate system.

[0061] Figure 1 The coordinate reconstruction method of the fiber optic panel coupled with the sensor shown can be applied to Figure 2 The fiber optic panel is shown in the coordinate reconstruction system coupled with the sensor. Figure 2 The system includes a sensing device conversion module, a calibration and reconstruction module, a detection and calculation module, and a mechanical arm. The system processes the sensor to be coupled to the optical fiber panel, that is, the target sensor, and executes steps S1-S5 to identify the three-dimensional world coordinate information of the target sensor in the world coordinate system. The three-dimensional world coordinate information represents the precise position of the target sensor, so that the control mechanical arm can accurately pick up, move, and couple the target sensor with the optical fiber panel according to the three-dimensional world coordinate information.

[0062] In this embodiment, Figure 1 The system shown works as follows:

[0063] The sensing device conversion module obtains the image of the area where the sensor target is located through a monocular camera or other equipment, and converts it into a digital signal to be processed so as to facilitate the subsequent position detection and calculation of the sensor. That is, in this embodiment, the sensing device conversion module can execute step S1;

[0064] The detection and calculation module uses the deep learning model to quickly and accurately locate the approximate position coordinates of the sensor in the image, and selects part of it as the area of ​​interest. The pre-processed image information is further processed using the edge detection algorithm, and the edge of the sensor coupling area is successfully detected. That is, in this embodiment, the detection and calculation module can execute steps S2-S3;

[0065] In the calibration and reconstruction module, Zhang Zhengyou's calibration method is used to establish a mapping relationship between the real three-dimensional space and the image pixel coordinate system. Combined with the imaging principle, the pixel coordinates in the two-dimensional image are converted into three-dimensional coordinates in the real world, and finally sub-millimeter accuracy can be successfully achieved. That is, in this embodiment, the calibration and reconstruction module can execute steps S4-S5.

[0066] Before executing steps S1-S5, a monocular camera can be arranged on the sensor production line. The installation angle of the monocular camera must ensure that the target sensor can be clearly observed in the shooting picture, and a suitable shooting distance is selected. A calibration pattern (specifically, a calibration chessboard) can also be placed within the field of view of the monocular camera to determine the world coordinate system, wherein the world coordinate system can be established on the calibration chessboard by default, so that the coordinate axis position of the world coordinate system is fixed in the subsequent process. Specific parameters such as the number of monocular cameras installed and the equipment configuration can be determined according to the actual production scenario.

[0067] In step S1, a monocular camera can be used as a perception device conversion module, or the perception device conversion module can call a monocular camera to execute. The monocular camera is used to shoot the target sensor to obtain an image to be recognized. Specifically, the image to be recognized can be in the form of a single static image or in the form of a video stream.

[0068] After executing step S1 to obtain the image to be recognized, the perception device conversion module sends the image to be recognized to the calibration and reconstruction module.

[0069] The calibration and reconstruction module executes step S2, inputs the image to be identified into the trained deep learning target detection model for identification, and obtains the area information of the target sensor output by the deep learning target detection model.

[0070] Specifically, the deep learning target detection model used in step S2 is a network model that can identify the target object and give the position and category of the object in the image. The input of the pre-trained deep learning target detection model can obtain the sensor classification detection result. In this embodiment, the YOLOv7 model can be used as a deep learning target detection model. The basic principle of the YOLOv7 model is as follows: The core idea of ​​the YOLO target detection algorithm is to convert the target detection task into a regression problem by dividing the image into grids and predicting the bounding box and category information of the target in each grid at the same time; first, the YOLO target detection algorithm needs to extract the sensor video image information taken by the fixed camera for the position prediction of the target. After that, these images need to pass through a single forward propagation convolutional layer network for downsampling and feature extraction to form feature maps of multiple scales. Furthermore, a multi-input stacking structure is introduced for feature extraction to achieve data enhancement. At the beginning, feature maps of different scales are placed in the SPPCSPC module for processing. Then, the feature maps are subjected to sampling rate changes and convolution fusion and stacking fusion with feature maps of different scales to achieve feature fusion and enhancement, which is convenient for subsequent prediction. Furthermore, the feature layer needs to be passed to Yolo Head to obtain the prediction result. Here, the front-end processing of the RepConv structure is introduced. By introducing a special residual structure to assist training, the prediction performance of the network is improved. Then, the feature maps processed by the RepConv network are connected to the Yolo Head convolutional network with different output channels to obtain target detection results of different scales. Finally, the prediction results can be decoded. First, the outputs of the three scales are decoded to obtain the position, confidence and category probability of the prediction box at each scale. Then, the predicted center coordinates and width and height adjustment parameters are converted into actual relative values ​​through the sigmoid function, and the predicted bounding box coordinates are calculated according to the position of the current grid unit and the size of the anchor box.

[0071] In step S2, the deep learning target detection model used can be trained through the following training process:

[0072] P1. Obtain image data of the sensor that needs to be coupled by taking targeted photos of the sensor. The shooting range should include all sensor types that need to be tested to ensure the comprehensiveness of the image data.

[0073] P2. After pre-processing and pre-division of sensor types, the sensor locations and categories in the image data set are annotated to obtain a training set. The annotation content includes the location and category of the sensor. Pre-processing includes data cleaning and deletion. If the data is insufficient, data augmentation can be used, such as image rotation, translation, scaling or color adjustment, to expand the data set to ensure that the weights obtained in subsequent training are correct.

[0074] P3. Use the labeled dataset to train the YOLOv7 network, obtain the training weights and import them into the YOLOv7 network.

[0075] In steps P1-P3, for the Yolo Head convolutional network, which requires data set training, the adaptive multi-positive sample matching model can be used to determine the category information of the positive sample and the matching box by calculating the IoU and category cost of the real box and the prior box. Then, according to the matching of the positive sample, the target for training is constructed; then, by comparing the matching of the predicted box and the real box, the classification loss function, the review loss function and the confidence loss function are constructed. The loss of these three parts is jointly optimized, so that the model gradually improves the accuracy of classification, positioning and confidence prediction of the target during the training process, thereby improving the object detection performance of the YOLOv7 model.

[0076] In the actual prediction process using the YOLOv7 network, since the YOLOv7 network needs to ensure that the size of the input image is the same, the pixel length and width of the image to be identified can be constrained to a multiple of 32; if the image to be identified is in the form of a video stream, the image size can be set once for each video frame in the video stream, and gray bars can be used to fill the difference in image size; in order to ensure the robustness of the YOLOv7 network execution algorithm, only prediction boxes with confidence > 0.6 can be accepted; specifically, non-maximum suppression is introduced, and for multiple possible prediction boxes identified from the image to be identified, only the prediction box with the highest confidence is retained to ensure the uniqueness of the identification box of each sensor. Different categories are intercepted and scaled to different degrees, that is, the region of interest (ROI) is divided, so the edge detection result can directly identify smaller images, significantly optimizing the recognition speed and edge detection effect. It is worth mentioning that the classification of categories can be customized according to business needs. For example, for sensor edge detection, it can be divided into two categories according to whether there is a sensor or not. In other words, the YOLOv7 network can provide a state detection function to determine whether the sensor is within the field of view, thereby improving the robustness and practicality of the YOLOv7 network.

[0077] According to the principle of the YOLOv7 model, using the trained YOLOv7 model as a deep learning target detection model can identify the image to be identified, thereby outputting the region information in the form of bounding box coordinates. In other words, the region information output by the YOLOv7 model can be represented as a bounding box in the image to be identified, and the region in the image to be identified framed by the bounding box is the region of the target sensor in the image to be identified.

[0078] In summary, when using the deep learning target detection model to quickly locate the position coordinates of the target sensor, the perception device conversion module can be used to collect a large number of image data obtained by cameras shooting sensors at different angles and distances, and manually annotate to build a training data set that meets the target. The data set collection and production process will affect the final target recognition effect of the deep learning model. At the same time, after obtaining the approximate position coordinates of the sensor in the image, the image is divided into regions of interest, and then the edge detection algorithm is used to detect the edge of the target sensor, which can achieve sub-millimeter accuracy in the case of monocular vision.

[0079] In this embodiment, when the detection and calculation module executes step S3, that is, the step of determining the plane position coordinate information of the target sensor according to the area information, the following steps may be specifically performed:

[0080] S301. Perform edge detection based on the region information to obtain original edge information;

[0081] S302. Perform edge fitting on the original edge information to obtain fitted edge information;

[0082] S303. Determine the center point coordinates of the target sensor as the plane position coordinate information according to the fitted edge information.

[0083] In step S301, the detection and calculation module performs edge detection on the region information obtained by executing step S2, thereby detecting the edge of the region information, that is, original edge information.

[0084] Since the edge detection algorithm used by the detection calculation module to execute step S301 may generate a large amount of noise and unstable edge contours, affecting subsequent information extraction, the edge represented by the original edge information obtained by executing step S301 may not be smooth. Therefore, step S302 can be executed to perform edge fitting on the original edge information to obtain fitted edge information.

[0085] Specifically, in step S302, edge fitting technology is used to smooth and optimize the original edge information. Specifically, in step S302, the detection calculation module can fit the edge lines in the original edge information to generate corresponding straight lines, and calculate the distance between each edge line in the original edge information and the corresponding straight line. If the distance between the edge line and the straight line is less than a preset threshold, the straight line is used to replace the original edge line, thereby simplifying the edge contour into a flat straight line and converting the original edge information into smooth fitting edge information. This method significantly improves the contour stability and provides a reliable basis for subsequent depth calculation.

[0086] Since the shape of the target sensor is generally a rectangle, the fitting edge information obtained by executing step S302 is generally represented as a rectangle. In step S303, the four corner points of the fitting edge information can be extracted and connected to obtain two diagonal lines of the fitting edge information, and the coordinates (u, v) of the intersection of the diagonal lines in the image to be identified are used as the center to determine the plane position coordinate information of the target sensor.

[0087] By executing steps S301-S303, the obtained plane position coordinate information (u, v) represents the coordinates of the target sensor in the image coordinate system, wherein the image coordinate system is established on the image to be identified, that is, the image coordinate system and the image to be identified are in the same plane.

[0088] In step S4, the calibration and reconstruction module calibrates the internal and external parameters of the monocular camera to solve the internal and external parameters of the monocular camera. These parameters represent the displacement and rotation vector of the position and posture of the front monocular camera relative to the world coordinate axis.

[0089] In this embodiment, when the calibration and reconstruction module executes step S4, that is, the step of obtaining the internal parameters and external parameters of the monocular camera, the following steps may be specifically performed:

[0090] S401. Set the calibration pattern;

[0091] S402. Setting the internal parameter matrix and external parameter matrix of the monocular camera;

[0092] S403. Establishing a world coordinate system on the calibration pattern;

[0093] S404. Use a monocular camera to shoot the calibration pattern to obtain multiple calibration images;

[0094] S405. Identify multiple feature points in each calibration image respectively;

[0095] S406. Obtaining the actual two-dimensional point coordinate information of each feature point in the image coordinate system;

[0096] S407. Use the actual two-dimensional point coordinate information to jointly optimize the internal parameter matrix and the external parameter matrix;

[0097] S408. Use the internal parameter matrix after joint optimization as the internal parameter.

[0098] S409. Use the actual two-dimensional point coordinate information and the internal parameter matrix after joint optimization to optimize the external parameter matrix separately;

[0099] S410. Use the independently optimized external parameter matrix as the external parameter.

[0100] In step S401, a calibration chessboard may be placed within the field of view of the monocular camera as a calibration pattern, so that the monocular camera can capture the calibration pattern.

[0101] In step S402, the calibration and reconstruction module sets the initial value of the internal parameter matrix and the initial value of the external parameter matrix of the monocular camera.

[0102] The intrinsic parameter matrix of a monocular camera is in the form of

[0103]

[0104] where f x and f y Represents the focal length of the monocular camera, u 0 and v 0 Represents the principal point of the monocular camera.

[0105] The form of the extrinsic parameter matrix of a monocular camera is

[0106]

[0107] Where R is the rotation matrix and T is the translation vector.

[0108] In step S403, a world coordinate system is established on the calibration pattern, so that the three-dimensional world coordinate information (X W ,Y W ,Z W ). Among them, (X W ,Y W ) is the coordinate of the target sensor on the plane parallel to the image to be recognized, Z W is the coordinate of the target sensor on the coordinate axis perpendicular to the image to be identified, Z W It can be understood as the actual height of the target sensor.

[0109] Through the internal parameter matrix and the external parameter matrix, the plane position coordinate information (u, v) of the target sensor in the image coordinate system and the three-dimensional world coordinate information (X W ,YW ,Z W ):

[0110]

[0111] In formula (1), Z c is the depth of the monocular camera from the target sensor in the image coordinate system.

[0112] After executing steps S1-S3, only the plane position coordinate information (u, v) in formula (1) is known, and the ultimate goal is to obtain the three-dimensional world coordinate information (X W ,Y W ,Z W ), therefore, steps S404-S410 may be performed first to obtain internal parameters and external parameters.

[0113] In step S404, a monocular camera is used to shoot the calibration pattern to obtain multiple calibration images. Each calibration image corresponds to a unique shooting angle and shooting distance. In this embodiment, a total of N calibration images are taken. Since the feature points in a single image may be distributed more concentratedly during the implementation process, resulting in inaccurate estimation of some parameters, multiple images can ensure that the feature points are more evenly distributed at different angles and distances, thereby improving the calibration accuracy. At the same time, the use of multiple images can reduce the impact of inaccurate corner point detection or noise in some images. Therefore, it is preferred to use more than 10 calibration images, that is, N>10.

[0114] In step S405, a plurality of feature points are identified in each calibration image. In this embodiment, feature points can be determined at the same position in different calibration images, and M feature points are determined in each calibration image.

[0115] In step S406, since the feature points are located in the calibration image, and the calibration image is obtained by taking a monocular camera like the image to be identified, the coordinate points of each feature point can be determined in the image coordinate system, that is, the actual two-dimensional point coordinate information of each feature point. Since the position of the calibration pattern can be specially designed, the coordinate points of the feature points in the world coordinate system can be determined, that is, the actual three-dimensional point coordinate information of the feature points in the world coordinate system. In this embodiment, the actual two-dimensional point coordinate information of the jth feature point in the i-th calibration image is recorded as p ij , the actual 3D point coordinate information of the jth feature point in any calibration image in the world coordinate system is recorded as P j .

[0116] In step S407, the calibration and reconstruction module may perform a nonlinear least square method to jointly optimize the intrinsic parameter matrix and the extrinsic parameter matrix using the actual two-dimensional point coordinate information.

[0117] Specifically, the calibration and reconstruction module can be based on the formula

[0118]

[0119] In formula (2), K is the internal parameter matrix, R i is the rotation matrix corresponding to the i-th calibration image in the external parameter matrix, t i is the translation vector corresponding to the i-th calibration image in the extrinsic matrix, For the jth feature point in the i-th calibration image, according to the internal parameter matrix K and the rotation matrix R i , translation vector t i and the actual 3D point coordinate information P j The projection point coordinate information obtained by projection.

[0120] In formula (2), the Levenberg-Marquardt optimization algorithm can be used to solve the above objective. When the update step size is small, it is similar to the Gauss-Newton method, and when the step size is large, it is similar to gradient descent. This combination can accelerate convergence when approaching the solution, and avoid falling into the local minimum when far away from the solution. Among them, when the reprojection error reaches the minimum, the internal parameter matrix K is taken out at this time, and the internal parameter matrix K and the external parameter matrix (R i ,t i ) joint optimization.

[0121] In formula (2), by changing the internal parameter matrix K and the external parameter matrix (R i ,t i ) are jointly optimized. In step S408, the internal parameter matrix K obtained after the joint optimization can be used as the internal parameter to be used in step S5.

[0122] The internal parameter matrix obtained by step S407 is globally optimized and is the global optimal solution for all calibration images. Since the calibration images obtained by the monocular camera shooting the calibration pattern come from different angles and distances, the external parameter matrix obtained during the calibration process is not a global external parameter and can only be used to help obtain the optimal internal parameter matrix K. Therefore, in step S408, the external parameter matrix after joint optimization may not be output, but step S409 may be executed to optimize the external parameter matrix of each calibration image separately to ensure that the projection error of each calibration image is minimized.

[0123] In order to ensure the robustness of the algorithm, a stable and unique external parameter matrix is ​​required. Therefore, the PnP algorithm can be used to solve the homography matrix and calculate the external parameter matrix. Since the internal parameter matrix used as the internal parameter for executing step S5 has been calibrated and obtained in step S407, a unique external parameter can be obtained for a single calibration image in the case of a fixed monocular camera, and the coordinate system can be established based on this external parameter. Based on this principle, step S409 can be executed to optimize the external parameter matrix separately using the actual two-dimensional point coordinate information and the internal parameter matrix after joint optimization.

[0124] Specifically, when executing step S409, according to the formula

[0125]

[0127] Among them, K is the internal parameter matrix after joint optimization, R is the rotation matrix in the external parameter matrix corresponding to the calibration image used for separate optimization, t is the translation vector in the external parameter matrix corresponding to the calibration image used for separate optimization, P i is the actual 3D point coordinate information of the i-th feature point in any calibration image in the world coordinate system, p i To optimize the actual two-dimensional point coordinate information of the i-th feature point in the calibration image used, The i-th feature point in the calibration image used for individual optimization is calculated based on the intrinsic parameter matrix K, the rotation matrix R, the translation vector t and the actual three-dimensional point coordinate information P i The projection point coordinate information obtained by projection.

[0128] In step S409, R directly solved by formula (3) is in vector form. In order to facilitate subsequent calculations, R can be converted into a rotation matrix. Specifically, it can be obtained by the following formula:

[0129]

[0130] Where: θ = ||R|| is the rotation angle of R in vector form, [R] x is the antisymmetric matrix corresponding to the vector form of R. By formula (4), the vector form of R can be converted into the rotation matrix form of R mat In this embodiment, it is not necessary to distinguish between R in vector form and R in rotation matrix form. mat , for example, we can directly think of R as the rotation matrix.

[0131] In step S410, the external parameter matrix (R, t) obtained after the individual optimization can be used as the external parameter to be used in step S5.

[0132] In this embodiment, when the calibration and reconstruction module executes step S5, that is, the step of performing coordinate reconstruction according to the internal parameters, the external parameters and the plane position coordinate information to obtain the three-dimensional world coordinate information of the target sensor in the world coordinate system, the following steps can be specifically performed:

[0133] S501. Obtain the actual size of the target sensor;

[0134] S502. Determine the pixel size of the target sensor in the image coordinate system according to the plane position coordinate information;

[0135] S503. According to the formula

[0136]

[0137] Z W =Zc-Z c ′ (5)

[0139] Calculate and determine the third dimension coordinate Z of the target sensor in the world coordinate system W ; where X and Y are the actual sizes of the target sensor, x and y are the pixel sizes of the target sensor, and f x and f y is the focal length contained in the internal parameters (where f x Same dimension as x, f y Same dimension as y);

[0140] S504. Determine the first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system according to the third dimension coordinates.

[0141] In step S501 , on a specific production line, the actual size of the target sensor is generally fixed, so the length X and width Y of the fixed target sensor can be obtained.

[0142] In step S502, the calibration and reconstruction module can determine the pixel sizes x and y of the area of ​​the target sensor in the image to be identified in the image coordinate system according to the plane position coordinate information (u, v) or the corresponding fitting edge information, where x and X have the same dimensions, and y and Y have the same dimensions.

[0143] In step S503, the current depth of the target sensor can be estimated. First, the depth Z is estimated by using similar triangles in the imaging principle. c′, the transformation based on similar triangles depends on the pixel length of the edge. Since the four edges of the sensor will have different lengths in pixels due to different observation angles, for example, a rectangular target area may become a rhombus or a trapezoid due to perspective. Therefore, before extracting the pixel coordinates, an affine transformation is required to convert the four angles of the sensor into a relatively flat 90 degrees. Otherwise, relatively accurate results can only be obtained near the center of the monocular camera.

[0144] The principle of step S503 is: let the depth of the monocular camera (the height of the monocular camera relative to the origin of the world coordinate system) be Z c , the depth of the target sensor (the height of the target sensor relative to the monocular camera) is Z c ′, according to the pinhole imaging principle, we get two similar triangles:

[0145]

[0146] Since the sensor length in the world coordinate system is the same as the length in the camera coordinate system, and the calibration process obtains the translation vector T, which contains [t x t y t z ] T , where the Z-axis distance t from the camera to the world coordinate is given z , and the distance from the world coordinate Z w =0 is: Z c =t z *L (L is the coefficient of converting the normalized coordinate system to the absolute coordinate system), so Z can be calculated W , when the monocular camera is located above the target sensor, the actual height Z of the target sensor W for:

[0147] Z W =Zc-Z c '

[0148] According to the above principle, formula (5) used in step S503 can be obtained.

[0149] For Z in the formula (5) used in step S503 c ′, in addition to being able to directly use the formula based on the imaging principle

[0150]

[0151] In addition to the calculation, it can also be calculated by the following formula:

[0152]

[0153]

[0154] Where P is the number of image frames contained in the case where the image to be processed is in the form of a video stream; x ij is the pixel size of the jth edge of the target sensor region in the i-th frame of the image to be processed in the image coordinate system (same dimension as X), y ik It is the pixel size of the kth edge of the target sensor area in the i-th frame of the image to be processed in the image coordinate system (same dimension as Y).

[0155] The principle of formula (6) is that the target sensor is small in size, the pixels in the area of ​​interest are low, and a slight change in the edge may cause a large disturbance in the Z axis (the direction perpendicular to the image to be recognized). Therefore, the four edges of the target sensor are calculated once to obtain the Z corresponding to the i-th frame image. i , where Z i The meaning of is the distance between the sensor and the monocular camera on the Z axis identified by the i-th frame image in the image to be processed. Find the Z corresponding to each of the P frame images i The final Z c ′, the Z-axis value can be smoothed to ensure the stability of the Z-axis coordinate.

[0156] By using formula (6) to calculate Z c ′, formula (5) can be optimized as follows:

[0157]

[0158] That is, Z can be calculated by formula (7): W , which represents the height of the target sensor relative to the origin of the world coordinate system.

[0159] When only the pixel coordinates u and v are known, it is impossible to directly obtain a unique solution, because each image coordinate point may correspond to countless three-dimensional space points in the direction of the camera optical axis. In other words, when the distance of the object from the camera is unclear, there will be infinite solutions. In monocular vision, this phenomenon can be geometrically understood as the lack of depth information: although the pixel coordinates (u, v) give the position of the target on the image plane, the actual position of the object in three-dimensional space is uncertain due to the lack of the third dimension (ie, depth Zw). Each solution corresponds to a point in space along the direction of the camera light. However, after executing steps S1-S4 and S501-S503 in step S5, the third-dimensional coordinate Z of the target sensor in the world coordinate system is obtained. W , the internal parameters are also determined by calibration (internal matrix after joint optimization)

[0160]

[0161] and external parameters (external matrices after separate optimization)

[0162]

[0163] Moreover, the plane position coordinate information (u, v) in formula (1) and the depth Z of the monocular camera c It is known that, therefore, formula (1) can be solved to obtain the first dimension coordinate X of the target sensor in the world coordinate system: W and the second dimension coordinate Y W Specifically, solve formula (1) to obtain

[0164]

[0165] In formula (7), r 11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 and r 33 are the elements of the 3×3 rotation matrix in the external parameters, T 1 , T 2 and T 3 are the elements of the 3×1 translation vector in the external parameters. L is the coefficient of converting the normalized coordinate system to the absolute coordinate system, and its value depends on the absolute length of one of the square black and white grids in the calibration pattern (calibration chessboard) in the Zhang Zhengyou calibration method used. In this embodiment, the value of L is 10 mm, and its value can be changed according to the specific situation.

[0166] In formula (7), except for X w and Y W In addition, the values ​​of other parameters have been obtained in the steps before step S503, so in step S503, the first dimension coordinate X of the target sensor in the world coordinate system can be obtained by solving formula (7): w and the second dimension coordinate Y w Finally, the three-dimensional world coordinate information (X w ,Y w ,Z w ).

[0167] In this embodiment, by executing steps S1-S5 on the physical target sensor, the three-dimensional world coordinate information (X w ,Y w ,ZW ), the effect is as follows Figure 3 and Figure 4 shown. Figure 3 and Figure 4 In the figure, the 3D world coordinate information is shown in the upper right corner of each figure. Figure 4 It can be seen that the accuracy of the three-dimensional world coordinate information reaches the sub-millimeter level.

[0168] Since the three-dimensional world coordinate information (X W ,Y W ,Z W ) describes the position of the target sensor more accurately, so the three-dimensional world coordinate information (X W ,Y W ,Z W ) is input to the robot arm for the next operation, and the robot arm can accurately move to the three-dimensional world coordinate information (X W ,Y W ,Z W ) to accurately couple the target sensor and the fiber optic panel.

[0169] A computer program for executing the coordinate reconstruction method of the fiber optic panel and the sensor coupled in this embodiment can be written and written into a computer device or a storage medium. When the computer program is read out and executed, the coordinate reconstruction method of the fiber optic panel and the sensor coupled in this embodiment is executed, thereby achieving the same technical effect as the coordinate reconstruction method of the fiber optic panel and the sensor coupled in the embodiment.

[0170] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0171] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0172] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0173] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0174] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0175] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0176] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A coordinate reconstruction method for coupling an optical fiber panel with a sensor, characterized in that: The coordinate reconstruction method of the optical fiber panel and the sensor coupling includes: Use a monocular camera to shoot the target sensor to obtain an image to be identified; the target sensor is a sensor to be coupled to the optical fiber panel; Inputting the image to be identified into a trained deep learning target detection model for identification, and obtaining the area information of the target sensor output by the deep learning target detection model; Determining the plane position coordinate information of the target sensor in the image coordinate system according to the area information; Obtaining internal and external parameters of the monocular camera; Coordinate reconstruction is performed according to the internal parameters, the external parameters and the plane position coordinate information to obtain three-dimensional world coordinate information of the target sensor in the world coordinate system.

2. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 1, characterized in that: Determining the plane position coordinate information of the target sensor according to the area information includes: Perform edge detection according to the region information to obtain original edge information; Performing edge fitting on the original edge information to obtain fitted edge information; According to the fitting edge information, the center point coordinates of the target sensor are determined as the plane position coordinate information.

3. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 1, characterized in that: The obtaining of internal parameters and external parameters of the monocular camera includes: Set the calibration pattern; Setting the intrinsic parameter matrix and extrinsic parameter matrix of the monocular camera; Establishing a world coordinate system on the calibration pattern; Using the monocular camera to shoot the calibration pattern to obtain a plurality of calibration images; different calibration images correspond to different shooting angles and shooting distances; Respectively identifying a plurality of feature points in each of the calibration images; Obtaining actual two-dimensional point coordinate information of each of the feature points in the image coordinate system; Using the actual two-dimensional point coordinate information to jointly optimize the intrinsic parameter matrix and the extrinsic parameter matrix; The internal parameter matrix after joint optimization is used as the internal parameter.

4. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 3, characterized in that: The using the actual two-dimensional point coordinate information to jointly optimize the intrinsic parameter matrix and the extrinsic parameter matrix includes: According to the formula Optimize; where K is the internal parameter matrix, R i is the rotation matrix corresponding to the i-th calibration image in the external parameter matrix, t i is the translation vector corresponding to the i-th calibration image in the extrinsic parameter matrix, P j is the actual 3D point coordinate information of the jth feature point in any of the calibration images in the world coordinate system, p ij is the actual two-dimensional point coordinate information of the j-th feature point in the i-th calibration image, For the jth feature point in the ith calibration image, according to the internal parameter matrix K and the rotation matrix R i , translation vector t i and the actual 3D point coordinate information P j The projection point coordinate information obtained by projection, N is the number of the calibration images, and M is the number of feature points in any of the calibration images.

5. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 3, characterized in that: The obtaining of the internal parameters and external parameters of the monocular camera also includes: Using the actual two-dimensional point coordinate information and the jointly optimized internal parameter matrix, the external parameter matrix is ​​optimized separately; The independently optimized external parameter matrix is ​​used as the external parameter.

6. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 5, characterized in that: The step of using the actual two-dimensional point coordinate information and the jointly optimized intrinsic parameter matrix to separately optimize the extrinsic parameter matrix includes: According to the formula Optimize; wherein K is the internal parameter matrix after joint optimization, R is the rotation matrix in the external parameter matrix corresponding to the calibration image used for separate optimization, t is the translation vector in the external parameter matrix corresponding to the calibration image used for separate optimization, P i is the actual 3D point coordinate information of the i-th feature point in any of the calibration images in the world coordinate system, p i The actual two-dimensional point coordinate information of the i-th feature point in the calibration image used for separate optimization, The i-th feature point in the calibration image used for separate optimization is calculated based on the intrinsic parameter matrix K, the rotation matrix R, the translation vector t and the actual three-dimensional point coordinate information P i The projection point coordinate information obtained by projection, N is the number of feature points in any of the calibration images.

7. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to any one of claims 1 to 6, characterized in that: The performing coordinate reconstruction according to the internal parameters, the external parameters and the plane position coordinate information to obtain the three-dimensional world coordinate information of the target sensor in the world coordinate system includes: Obtaining the actual size of the target sensor; Determining the pixel size of the target sensor in the image coordinate system according to the plane position coordinate information; According to the formula From W =Zc-Z c ′ Calculate and determine the third dimension coordinate Z of the target sensor in the world coordinate system W ; Wherein, X and Y are the actual sizes of the target sensor, x and y are the pixel sizes of the target sensor, and f x and f y is the focal length included in the internal parameters; The first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system are determined according to the third dimension coordinates.

8. The coordinate reconstruction method of the optical fiber panel and sensor coupling according to claim 7, characterized in that: Determining the first dimension coordinates and the second dimension coordinates of the target sensor in the world coordinate system according to the third dimension coordinates includes: According to the formula Calculate and determine the first dimension coordinate X W and the second dimension coordinate Y W ; where r 11 -r 33 are the elements of the rotation matrix in the external parameters, u0 and v0 are the principal points in the internal parameters, u and v are the plane position coordinate information, L is the coefficient for converting from the normalized coordinate system to the absolute coordinate system, and T1-T3 are the elements of the translation vector in the external parameters.

9. A computer device, characterized in that: It comprises a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the coordinate reconstruction method of the optical fiber panel coupled with the sensor as described in any one of claims 1-8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the coordinate reconstruction method of the optical fiber panel and sensor coupling as described in any one of claims 1-8 when executed by the processor.

Images