Camera calibration method, device and equipment of optical fiber positioning system

Abstract

The invention provides a camera calibration method of an optical fiber positioning system, which can be applied to the technical field of computer vision, and comprises the following steps: acquiring an initial image of a camera of a front lighting recognition system; preprocessing the initial image to obtain a preprocessed graph; the preprocessed images are input into the trained neural network model, a prediction result of camera parameters is obtained, camera calibration is completed, and the initial images comprise images shot by camera focusing planes of all the front lighting recognition systems and the whole focal plane area capable of being covered by the initial images; the neural network model is a convolutional neural network with deep residual learning capability, and the problem of gradient disappearance in a traditional deep network can be solved by introducing residual learning. The invention further provides a camera calibration device and equipment of the optical fiber positioning system.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a camera calibration method, device and equipment for an optical fiber positioning system. Background Art

[0002] Fiber optic spectroscopic telescopes are advanced observation equipment that integrates fiber optic technology and spectrometer analysis technology. The core of the telescope is to capture the light signals of celestial bodies through optical fibers, and then use spectrometers to deeply analyze these spectral information to reveal the physical characteristics, chemical composition and motion state of celestial bodies. The focal plane of this type of telescope is densely covered with a large number of optical fibers, each of which is precisely aimed at a celestial body, realizing efficient observation of many celestial bodies at the same time.

[0003] In order to further improve the accuracy and efficiency of observation, the fiber optic positioning system introduces a front-illuminated recognition system camera, which can monitor and measure the working status of the fiber optic positioning unit in real time. However, the camera must undergo a rigorous calibration process before use, including the determination of internal parameters (such as focal length) and external parameters (such as the position and attitude of the camera relative to the focal plane). This calibration process is crucial, as it establishes an accurate mapping between image pixels and the physical coordinates of the focal plane, thereby ensuring the accurate positioning of the fiber optic positioning unit and the accurate calculation of the distance between the optical fiber and the target star image, providing a basis for the precise positioning of the system.

[0004] At present, the traditional camera calibration method relies on the checkerboard calibration plate and Zhang Zhengyou calibration method. This process requires manual placement of the calibration plate and taking images at multiple angles, which is not only complicated but also time-consuming. For large-scale observation systems such as fiber optic spectroscopic telescopes, there is obviously a lack of a simple, fast and accurate intelligent automated camera calibration method.

[0005] At present, fiber optic spectroscopic telescopes that have been built or are under construction at home and abroad are equipped with a large number of optical fibers for efficient observation. Therefore, developing a fast and accurate intelligent automatic camera calibration method has extremely high practical value and wide applicability for improving the efficiency and accuracy of the fiber optic positioning system of these telescopes.

[0006] Although a single-view image camera calibration method based on deep learning has been proposed, this method is mainly applicable to daily life scenes and relies on geometric clues such as the horizon. It is not suitable for fiber positioning systems with low light conditions and where the objects are optical fibers or other mechanical structures. At the same time, although the collision detection method of the fiber unit of the multi-target fiber spectroscopy telescope also uses computer vision technology, it does not involve camera calibration methods.

[0007] In summary, for the specific industrial scenario of fiber spectroscopy telescope, it is particularly important to develop a fast, accurate, and intelligent automated camera calibration method, which will greatly promote the development of astronomical observation technology and improve observation efficiency and accuracy. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] In view of the above problems, the present invention provides a camera calibration method, device and equipment for a fiber optic positioning system, which uses massive image data rendered by a simulated three-dimensional model of the fiber optic positioning system to train a neural network, thereby achieving rapid and accurate calibration of the external parameters of the camera of the front-light recognition system.

[0010] (II) Technical solution

[0011] In view of the above technical problems, embodiments of the present invention provide a camera calibration method, device and equipment for a fiber optic positioning system.

[0012] According to a first aspect of the present invention, a camera calibration method for a fiber optic positioning system is provided, comprising: acquiring an initial image of a camera of a front-light recognition system; preprocessing the initial image to obtain a preprocessed graphic; inputting the preprocessed image into a trained neural network model to obtain a prediction result of camera parameters, and completing camera calibration, wherein the initial image includes an image taken of a focus surface of each front-light recognition system camera, and the initial image can cover the entire focus surface area; the neural network model is a convolutional neural network with deep residual learning capability, and by introducing residual learning, the gradient vanishing problem in traditional deep networks can be solved.

[0013] In some exemplary embodiments, preprocessing the initial image includes: using a target detection algorithm to detect the reference optical fiber unit in the initial image, removing the optical fiber positioning unit, retaining the reference optical fiber unit, and obtaining a reference optical fiber image; obtaining the center point of the reference optical fiber unit, and connecting the center points of the reference optical fiber units on the same straight line with a line; detecting the reference optical fiber unit having a specific arrangement shape in the reference optical fiber image, drawing the fixed shape as a fixed feature, and obtaining a preprocessed image, wherein the preprocessed image includes the connecting line of the center point of the optical fiber positioning unit and the fixed shape.

[0014] In some exemplary embodiments, the depth of the neural network model is 101 layers, including an initial layer, a maximum pooling layer, a network processing stage and a fully connected layer, wherein the initial layer is a 7×7 convolution layer with a step size of 2 and an output channel of 64; the maximum pooling layer is a 3×3 maximum pooling layer with a step size of 2; the network processing stage includes a first stage, a second stage, a third stage and a fourth stage, the first stage includes 3 residual blocks, the second stage includes 4 residual blocks, the third stage includes 23 residual blocks and the fourth stage includes 3 residual blocks, wherein each residual block includes a 1×1 convolution layer and a 3×3 convolution layer for dimensionality adjustment and feature extraction; the number of output features of the fully connected layer is 6; the activation function of the neural network model is a leaky linear rectification function.

[0015] In some exemplary embodiments, a method for obtaining a trained neural network model includes: constructing a training image data set; preprocessing the training image data set to obtain a preprocessed training image data set; training a neural network model using the preprocessed training image data set to obtain a trained neural network model, wherein the preprocessed training image data set has the same data features as the preprocessed image.

[0016] In some exemplary embodiments, constructing a training image data set includes: constructing a simulated three-dimensional model based on the size of an actual fiber optic positioning system; setting external parameters of a camera in the simulated three-dimensional model to obtain a simulated image of a corresponding front-light recognition system camera; rendering the simulated image of the front-light recognition system camera and annotating corresponding external parameters so that the simulated image corresponds one-to-one with the external parameters of the camera to form a training image data set; wherein, the focal plane, fiber optic positioning unit, single-head and multi-head reference fiber optic units in the simulated three-dimensional model are simulated and modeled according to actual sizes to ensure the accuracy of the model; the central axis and eccentric axis robotic arms in the simulated three-dimensional model can rotate according to actual functional angles to achieve a simulation effect.

[0017] In some exemplary embodiments, training a neural network model using a preprocessed training image dataset includes: inputting the preprocessed training image dataset into the neural network model to obtain predicted values ​​of camera external parameters; using a smooth L1 loss function to calculate the loss between the predicted value and the true value, and minimizing the loss function to gradually converge the model to obtain an optimal solution.

[0018] In some exemplary embodiments, training the neural network model using a preprocessed training image data set also includes: obtaining the pixel coordinates of a reference optical fiber point of a fixed shape in the preprocessed training image data set, recording the pixel coordinates as a two-dimensional array, and encoding them as a feature vector of a fixed length; downsampling the feature vector through average pooling, and then splicing it with the convolution feature map to obtain feature information; based on the feature information, training the neural network model to obtain a trained neural network model.

[0019] In some exemplary embodiments, training a neural network model using a preprocessed training image data set also includes: testing the trained neural network model using test set data to calculate the error mean and median of the camera's external parameters; comparing the error mean and median with an existing neural network camera calibration method to verify the superiority of the model.

[0020] The second aspect of the present invention provides a camera calibration device for a fiber optic positioning system, comprising: an acquisition module for acquiring an initial image of a front-light recognition system camera; a preprocessing module for preprocessing the initial image to obtain a preprocessed graphic; a prediction module for inputting the preprocessed image into a trained neural network model to obtain a prediction result of the camera parameters and complete the camera calibration, wherein the initial image includes an image taken of the focus surface of each front-light recognition system camera, and the initial image can cover the entire focus surface area; the neural network model is a convolutional neural network with deep residual learning capability, and by introducing residual learning, the gradient vanishing problem in traditional deep networks can be solved.

[0021] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0022] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0023] (III) Beneficial effects

[0024] It can be seen from the above technical solutions that the camera calibration method, device and equipment of the optical fiber positioning system provided by the embodiments of the present invention have at least the following beneficial effects:

[0025] (1) By constructing a simulated 3D model of the fiber optic positioning system, a large amount of training data can be generated, providing sufficient data support for the training of the neural network. The simulated 3D model can also simulate complex conditions such as low light and featureless reference objects in the real environment, so that the neural network can better adapt to the actual observation environment.

[0026] (2) Aiming at the unique industrial environment of the fiber optic positioning system, a series of targeted image preprocessing operations are proposed, such as identifying reference fiber units, detecting fiber point edges and fixed features, etc., to further improve the prediction accuracy of the neural network for camera parameters.

[0027] (3) The rotation parameters of the camera in the dataset are negative. In order to prevent the activation function ReLU in the network from excluding the negative parameter range, it is changed to the leaky linear rectifier function (LeakyReLU) function so that when the input is negative, a non-zero output is generated, which effectively handles the negative value characteristics of the rotation and retains more information. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0029] Figure 1 The structural diagram of the LAMOST optical fiber positioning system according to an embodiment of the present invention is schematically shown.

[0030] Figure 2 A flow chart of a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0031] Figure 3 The following is a schematic diagram of a front-light recognition system of LAMOST according to an embodiment of the present invention.

[0032] Figure 4 The figure schematically shows the positional relationship between the camera and the focal plane in the LAMOST according to an embodiment of the present invention.

[0033] Figure 5 The diagram schematically shows a real shot image obtained by a front camera according to an embodiment of the present invention.

[0034] Figure 6 A schematic diagram of a simulation image obtained according to an embodiment of the present invention is shown schematically.

[0035] Figure 7 A schematic diagram of a reference optical fiber image obtained according to an embodiment of the present invention is schematically shown.

[0036] Figure 8A schematic diagram of a horizontal connection line of a reference optical fiber unit center point according to an embodiment of the present invention is schematically shown.

[0037] Fig. 9 The schematic diagram of a fixed shape drawn based on a reference optical fiber unit having a specific arrangement shape in a reference optical fiber image according to an embodiment of the present invention is schematically shown.

[0038] Fig.10 A flowchart of a method for obtaining a trained neural network model according to an embodiment of the present invention is schematically shown.

[0039] Fig.11 A schematic diagram of a one-to-one simulation three-dimensional model of an optical fiber positioning system according to an embodiment of the present invention is shown schematically.

[0040] Fig.12 A schematic diagram of a fiber positioning unit model according to an embodiment of the present invention is shown schematically.

[0041] Fig.13 A structural block diagram of a camera calibration device for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0042] Fig.14 A block diagram of an electronic device for a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0043] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0044] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0045] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0046] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0047] To facilitate the understanding of the content of the invention, the fiber optic spectroscopic telescope is introduced by taking the Guo Shoujing Telescope (Large Sky Area Multi-Object Fiber Spectroscopic Telescope, referred to as LAMOST) as an example. LAMOST now occupies a very important position in the field of domestic optical survey telescopes. It has created many outstanding achievements relying on advanced equipment and technology. Among them, the parallel controllable fiber optic positioning unit technology is a key independent innovation technology of LAMOST. During the operation of LAMOST, the light of celestial bodies passes through the reflective Schmidt corrector MA and the spherical primary mirror MB in turn, and is finally reflected and imaged on the focal plane. Among them, the focal plane is covered with 4,000 fiber optic positioning units, which can quickly and accurately locate the star image target according to the position in the star catalog, thereby receiving celestial starlight. The LAMOST fiber optic positioning system is as follows Figure 1 As shown in the figure, there are 4,000 fiber positioning units in total, each unit has two rotatable mechanical arms, the rotation axis is divided into a central axis and an eccentric axis, and the length of the mechanical arm is 8.25 mm. By rotating and controlling these fiber positioning units, the optical fiber is positioned on the star image target. The system relies on 7 front-light detection cameras placed around the spherical primary mirror MB to perform photographic detection on the 4,000 optical fibers on the focus surface, obtain the unit position in real time, and determine whether the unit needs to continue positioning.

[0048] Figure 2 A flow chart of a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0049] like Figure 2 As shown, a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention includes steps S110 to S130.

[0050] In step S110, an initial image of the front-light recognition system camera is acquired, wherein the initial image includes an image captured by each front-light recognition system camera on the focus surface, and the initial image can cover the entire focus surface area.

[0051] For example, for LAMOST, its front-view recognition system cameras are located around the spherical primary mirror MB, such as Figure 3 As shown, there are 7 cameras in total (respectively Figure 3These cameras take photos and measure the focal plane at a distance of 20 meters. The positional relationship between the camera and the focal plane is as follows: Figure 4 As shown in FIG. 1 , the front camera is located at the same position as the spherical primary mirror MB. The real image obtained by the front camera is as follows: Figure 5 As shown, the fiber positioning units are all in a non-working zero position state, and the image size is 7920×6004 pixels. The area captured by each camera is different, and the range captured by the seven cameras covers the entire focal plane area. The ultimate purpose of the method of the embodiment of the present invention is to calibrate the seven cameras separately through a single camera image.

[0052] In step S120, the initial image is preprocessed to obtain a preprocessed image.

[0053] In order to improve the accuracy of camera calibration, the embodiments of the present invention propose a series of targeted image preprocessing operations in view of the unique working environment of the fiber optic positioning system with low light, relatively fixed objects, no real geometric references such as floors and horizons, and no obvious reference objects. The purpose is to make the neural network recognize and analyze camera images more accurately and predict camera parameters more precisely.

[0054] In some exemplary embodiments, step S120 includes steps S121 - S123 .

[0055] In step S121, a target detection algorithm is used to detect the reference optical fiber unit in the initial image, remove the optical fiber positioning unit, retain the reference optical fiber unit, and obtain a reference optical fiber image.

[0056] For example, an image contains at least 800 fiber units, such as Figure 6 As shown in the figure, it includes a fiber positioning unit and a reference fiber. The swing angles of the two arms of each fiber positioning unit are different, and the rotation angle of the fiber positioning unit changes every time LAMOST makes an observation, which makes the feature points in the image very numerous and complex, without any regularity. In order to make the neural network estimate the external parameters of the camera more accurately, the target detection algorithm is used in the preprocessing to detect the reference fiber in the image, and then other complex fiber positioning units are removed from the image, so that only about 50 reference fibers are left in the image, as shown in the figure. Figure 7 As shown, the reference optical fiber has a single shape, simple structure, unique arrangement pattern, and obvious feature points. The neural network can learn the features more easily, further improving the accuracy of camera parameter prediction.

[0057] In step S122, the center points of the reference optical fiber units are obtained, and the center points of the reference optical fiber units on the same straight line are connected with lines.

[0058] In conventional scenarios, calibration objects with known sizes, such as calibration plates, are important references for camera calibration. However, in the actual application scenarios of the LAMOST fiber optic positioning system, the lack of such traditional calibration objects poses a challenge to camera calibration. In order to overcome this problem and enrich the geometric information in the image so that the neural network can more accurately determine the position and posture of the camera, this embodiment adopts a novel strategy. Specifically, the fiber points of the reference fiber units distributed on the vertical and horizontal lines in the system are used as key features. By accurately detecting these fiber points and connecting them in the direction of the straight line, such as Figure 8 The horizontal connection line diagram shown in the figure constructs a series of simulated vanishing lines. These connection lines form artificial geometric constraints in the image, which simulate the characteristics of vanishing lines in perspective projection and provide valuable information for the calibration of the camera's external parameters. Through these geometric constraints, the neural network can better understand the translation and rotation state of the camera, thereby more accurately estimating the camera's pose. In addition, this method of using artificial geometric constraints formed by fiber points not only improves the accuracy of the neural network in predicting the camera's pose, but also enhances the system's robustness to changes in camera parameters.

[0059] In step S123, a reference optical fiber unit having a specific arrangement shape is detected in the reference optical fiber image, and the fixed shape is drawn as a fixed feature to obtain a preprocessed image, wherein the preprocessed image includes a connecting line of the center point of the optical fiber positioning unit and the fixed shape.

[0060] There are several holes on the focal plane of LAMOST for mounting other instruments, and cleverly, the optical fiber units around these holes are arranged in a fixed shape. This fixed shape provides a natural auxiliary calibration object for camera calibration, which can help neural networks recognize geometric spatial structures more accurately.

[0061] By using this fixed shape, the neural network can infer the translation and rotation parameters of the camera relative to the scene. Fig. 9 As shown in the figure, a schematic diagram of this fixed shape is drawn to intuitively demonstrate its role in camera calibration. In addition, in order to further improve the prediction accuracy of the neural network, the pixel coordinates of the reference fiber points contained in this fixed shape can also be recorded. These pixel coordinates, as prior information, will play an important role in the subsequent camera calibration process and provide strong support for the accurate prediction of the neural network.

[0062] In step S130, the preprocessed image is input into the trained neural network model to obtain the prediction results of the camera parameters and complete the camera calibration. The neural network model is a convolutional neural network with deep residual learning capability. By introducing residual learning, the gradient vanishing problem in traditional deep networks can be solved.

[0063] In some exemplary embodiments, the neural network model is modified, optimized and improved with Resnet101 as the framework. The depth of the neural network model is 101 layers, including an initial layer, a maximum pooling layer, a network processing stage and a fully connected layer, wherein the initial layer is a 7×7 convolution layer with a step size of 2 and an output channel of 64; the maximum pooling layer is a 3×3 maximum pooling layer with a step size of 2; the network processing stage includes the first stage, the second stage, the third stage and the fourth stage, the first stage includes 3 residual blocks, the second stage includes The first stage includes 4 residual blocks, the third stage includes 23 residual blocks, and the fourth stage includes 3 residual blocks, wherein each residual block includes a 1×1 convolution layer and a 3×3 convolution layer for dimensionality adjustment and feature extraction; the number of output features of the fully connected layer is 6. The purpose of the neural network in the embodiment of the present invention is to predict the external parameters of the camera. For the regression task of predicting continuous values, the fully connected layer parameters of the last layer of the network are set to 6, that is, the translation and rotation of the camera on the X, Y, and Z axes are output; the activation function of the neural network model is a leaky linear rectification function.

[0064] The rotation parameters of the camera in the dataset are negative. In order to avoid the activation function ReLU in the network from eliminating the negative parameter range, it is changed to the LeakyReLU function so that when the input is negative, a non-zero output is generated, which effectively handles the negative value characteristics of the rotation and retains more information. The specific formula is as follows:

[0065] (1)

[0066] in, is a small positive number, which is set to 0.1 in this embodiment.

[0067] In some exemplary embodiments, the method for obtaining a trained neural network model includes steps S210-S230, see Fig.10 .

[0068] In step S210, a training image dataset is constructed.

[0069] In some exemplary embodiments, step S210 includes steps S211 to S213.

[0070] In step S211, a simulation three-dimensional model is constructed based on the size of the actual fiber optic positioning system, wherein the focal panel, fiber optic positioning unit, single-head and multi-head reference fiber optic units in the simulation three-dimensional model are simulated and modeled according to the actual size to ensure the accuracy of the model; the central axis and eccentric axis robotic arms in the simulation three-dimensional model can rotate according to the actual function angle to achieve the simulation effect.

[0071] In order to provide a large amount of training and testing image data for the neural network model, the embodiment of the present invention uses three-dimensional modeling software to build a simulated three-dimensional model according to the actual fiber positioning system size. Fig.11 As shown in the figure, the front view and partial magnification of the simulated 3D model of the LAMOST fiber positioning system are shown. The focal plane, fiber positioning unit, single-head and multi-head reference fiber units are also simulated and modeled according to the actual size, as shown in the figure. Fig.12 As shown, this is a schematic diagram of the fiber optic positioning unit model, in which the central axis and eccentric axis mechanical arms can also rotate according to the actual functions to achieve the simulation effect.

[0072] In step S212, external parameters of the camera in the simulated three-dimensional model are set to obtain a corresponding simulated image of the camera of the front-light recognition system.

[0073] In step S213, the simulated image of the camera of the front-light recognition system is rendered and the corresponding external parameters are annotated, so that the simulated image corresponds to the camera external parameters one by one, forming a training image data set.

[0074] Relying on modeling software, the external parameters of tens of thousands of cameras with different external parameters can be set, and the images of these cameras can be rendered with rendering software and the corresponding external parameters, that is, the translation and rotation of the camera on the X, Y, and Z axes, can be marked, so that a single image has a one-to-one correspondence with the camera's external parameters, forming an image data set for training and testing neural networks.

[0075] For example, using rendering software to render a simulated image, such as Figure 6 As shown, Figure 6 For simulation Figure 5 The simulated rendering image of the real shot image, in which the optical fiber positioning unit has its central axis and eccentric axis angle adjusted by random parameters to test the simulation effect.

[0076] In the actual operation of the astronomical telescope, in order to prevent the instrument surface from reflecting light and thus interfering with the received starlight, the telescope focal plane and most of the instrument surfaces of the fiber optic positioning system are made of materials with low reflectivity as much as possible. In order to simulate such low-light or even dark conditions, the simulation 3D model also changes the instrument material and color according to the corresponding requirements, and the rendering light is also adjusted to a darker color to simulate the actual situation, simulating the real environment as much as possible to achieve the purpose of accurate simulation.

[0077] In step S220, the training image data set is preprocessed to obtain a preprocessed training image data set, wherein the preprocessed training image data set has the same data features as the preprocessed image.

[0078] The method for preprocessing the training image data set may adopt the method for preprocessing the initial image in step S120.

[0079] In step S230, the neural network model is trained using the preprocessed training image data set to obtain a trained neural network model.

[0080] In some exemplary embodiments, step S230 includes step S231 - step S232.

[0081] In step S231, the preprocessed training image data set is input into the neural network model to obtain the predicted values ​​of the camera external parameters.

[0082] In step S232, the smooth L1 loss function is used to calculate the loss between the predicted value and the true value, and the model is gradually converged to the optimal solution by minimizing the loss function. This method uses square penalties for small errors and linear penalties for large errors, so that the gradient of the loss function is more stable. The specific formula is as follows:

[0083] (2)

[0084] Where x represents the difference between the predicted value and the true value. is a preset threshold, optionally, Set to 1.

[0085] In order to further improve the prediction accuracy of the neural network and help the neural network better learn the mapping relationship between the image and the camera parameters, step S230 may also include: obtaining the pixel coordinates of the reference optical fiber points of a fixed shape in the preprocessed training image data set; recording the pixel coordinates as a two-dimensional array and encoding them as a feature vector of a fixed length; downsampling the feature vector through average pooling, and then splicing it with the convolution feature map to obtain feature information; based on the feature information, training the neural network model to obtain a trained neural network model.

[0086] In order to verify the effect of the trained neural network model of the embodiment of the present invention, the trained neural network model is tested using test set data, and the error mean and median of the camera external parameters are calculated; the error mean and median are compared with the existing neural network camera calibration method to verify the superiority of the model.

[0087] The average and median values ​​of the errors (the difference between the network predicted value and the true value) of the six camera parameters predicted by the neural network model trained according to the embodiment of the present invention are shown in Table 1.

[0088] Table 1 Prediction error of the neural network model trained in the embodiment of the present invention

[0089]

[0090] The prediction results of the rotation parameters of this embodiment are compared with the existing neural network camera calibration method as shown in Table 2, where Upright is the "Automatic Upright Adjustment of Photographs with Robust Camera Calibration" method by Hyunjoon Lee et al. in 2014; Perceptual is the "A Perceptual Measure for Deep Single Image Camera Calibration" method by Yannick Hold-Geoffroy et al. in 2018; UprightNet is the "UprightNet: Geometry-Aware Camera Orientation Estimation From Single Images" method by Wenqi Xian et al. in 2019; GPNet is the "Neural Geometric Parser for Single Image Camera Calibration" by Jinwoo Lee et al. in 2020; CTRL-C is the "CTRL-C: Camera calibration transformer with line-classification" method by J. Lee et al. in 2021), and the last row Ours is the result of the method of this embodiment.

[0091] Table 2 Comparison of prediction results of different calibration methods

[0092]

[0093] It should be noted that the other methods in Table 2 are mainly aimed at the diverse daily outdoor or indoor scenes, and the datasets used are public street view datasets. The method of the present invention is aimed at the fiber optic positioning system with completely different situations. As can be seen from the above table, the experimental results of the method of the present invention not only meet the accuracy requirements of the existing neural network camera calibration method, but also outperform other camera calibration methods.

[0094] Fig.13 A structural block diagram of a camera calibration device for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0095] like Fig.13As shown, the camera calibration device 800 of the optical fiber positioning system of this embodiment includes an acquisition module 810 , a preprocessing module 820 and a prediction module 830 .

[0096] The acquisition module 810 is used to acquire an initial image of a front-light recognition system camera, wherein the initial image includes an image taken of a focus surface of each front-light recognition system camera, and the initial image can cover the entire focus surface area.

[0097] The preprocessing module 820 is used to preprocess the initial image to obtain a preprocessed image.

[0098] The prediction module 830 is used to input the preprocessed image into the trained neural network model to obtain the prediction results of the camera parameters and complete the camera calibration. The neural network model is a convolutional neural network with deep residual learning capabilities. By introducing residual learning, the gradient vanishing problem in traditional deep networks can be solved.

[0099] According to an embodiment of the present invention, any multiple modules in the acquisition module 810, the preprocessing module 820 and the prediction module 830 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 810, the preprocessing module 820 and the prediction module 830 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 810, the preprocessing module 820 and the prediction module 830 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0100] Fig.14 A block diagram of an electronic device for a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0101] like Fig.14As shown, the electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0102] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0103] According to an embodiment of the present invention, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0104] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0105] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0106] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

Claims

1. A camera calibration method for a fiber optic positioning system, characterized in that: The method comprises: Acquire an initial image of a front-view recognition system camera; Preprocessing the initial image to obtain a preprocessed graphic; The preprocessed image is input into the trained neural network model to obtain the prediction result of the camera parameters and complete the camera calibration. The initial image includes an image captured by each front-light recognition system camera on the focus surface, and the initial image can cover the entire focus surface area; The neural network model is a convolutional neural network with deep residual learning capability. By introducing residual learning, the gradient vanishing problem in traditional deep networks can be solved.

2. The method according to claim 1, characterized in that The preprocessing of the initial image comprises: Using a target detection algorithm to detect the reference optical fiber unit in the initial image, removing the optical fiber positioning unit, retaining the reference optical fiber unit, and obtaining a reference optical fiber image; Obtaining the center points of the reference optical fiber units, and connecting the center points of the reference optical fiber units on the same straight line with lines; Detecting a reference optical fiber unit having a specific arrangement shape in the reference optical fiber image, drawing the fixed shape as a fixed feature, and obtaining a preprocessed image, Wherein, the preprocessed image includes a connecting line of the center point of the optical fiber positioning unit and the fixed shape.

3. The method according to claim 1, characterized in that The neural network model has a depth of 101 layers, including the initial layer, the maximum pooling layer, the network processing stage and the fully connected layer. The initial layer is a 7×7 convolutional layer with a step size of 2 and an output channel of 64; The maximum pooling layer is a 3×3 maximum pooling layer with a step size of 2; The network processing stage includes a first stage, a second stage, a third stage and a fourth stage, the first stage includes 3 residual blocks, the second stage includes 4 residual blocks, the third stage includes 23 residual blocks and the fourth stage includes 3 residual blocks, wherein each of the residual blocks includes a 1×1 convolution layer and a 3×3 convolution layer for dimensionality adjustment and feature extraction; The number of output features of the fully connected layer is 6; The activation function of the neural network model is a leaky linear rectification function.

4. The method according to claim 2, characterized in that: Methods for obtaining a trained neural network model include: Build a training image dataset; Preprocessing the training image data set to obtain a preprocessed training image data set; The neural network model is trained using the preprocessed training image data set to obtain a trained neural network model. The preprocessed training image data set and the preprocessed image data set have the same data features.

5. The method according to claim 4, characterized in that The constructing of the training image data set comprises: Based on the dimensions of the actual fiber positioning system, a simulation 3D model is constructed; Setting external parameters of the camera in the simulated three-dimensional model to obtain a corresponding simulated image of the camera of the front-light recognition system; Rendering the simulated image of the camera of the front light recognition system and marking the corresponding external parameters so that the simulated image corresponds to the camera external parameters one by one to form a training image data set; The focal plane, optical fiber positioning unit, single-head and multi-head reference optical fiber units in the simulated three-dimensional model are simulated and modeled according to actual sizes to ensure the accuracy of the model; The central axis and eccentric axis mechanical arms in the simulation three-dimensional model can rotate according to actual functions to achieve simulation effects.

6. The method according to claim 4, characterized in that The training of the neural network model using the preprocessed training image data set includes: Inputting the preprocessed training image data set into a neural network model to obtain predicted values ​​of camera extrinsic parameters; A smooth L1 loss function is used to calculate the loss between the predicted value and the true value, and the model is gradually converged to obtain the optimal solution by minimizing the loss function.

7. The method according to claim 6, characterized in that The training of the neural network model using the preprocessed training image data set also includes: Obtaining pixel coordinates of reference optical fiber points of a fixed shape in the preprocessed training image data set; Record the pixel coordinates as a two-dimensional array and encode them into a fixed-length feature vector; Downsampling the feature vector by average pooling, and then concatenating it with the convolution feature map to obtain feature information; Based on the feature information, the neural network model is trained to obtain a trained neural network model.

8. The method according to claim 7, characterized in that The training of the neural network model using the preprocessed training image data set also includes: The trained neural network model is tested using the test set data to calculate the error mean and median of the camera's external parameters; The error mean and median are compared with the existing neural network camera calibration method to verify the superiority of the model.

9. A camera calibration device for a fiber optic positioning system, characterized in that: The device comprises: An acquisition module, used for acquiring an initial image of a front-view recognition system camera; A preprocessing module, used for preprocessing the initial image to obtain a preprocessed image; A prediction module is used to input the preprocessed image into the trained neural network model to obtain the prediction results of the camera parameters and complete the camera calibration. The initial image includes an image captured by each front-light recognition system camera on the focus surface, and the initial image can cover the entire focus surface area; The neural network model is a convolutional neural network with deep residual learning capability. By introducing residual learning, the gradient vanishing problem in traditional deep networks can be solved.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images