Camera Calibration Method, Device and Equipment for Fiber Optic Positioning System

Abstract

The present invention provides a camera calibration method for an optical fiber positioning system, which can be applied to the field of computer vision technology. The method includes: obtaining an initial image of a headlamp recognition system camera; preprocessing the initial image to obtain a preprocessed graph; inputting the preprocessed image into a trained neural network model to obtain a prediction result of camera parameters, thereby completing camera calibration. Among them, the initial image includes images taken by the focus planes of each headlamp recognition system camera, and the initial image can cover the entire focal plane area; the neural network model is a convolutional neural network with deep residual learning ability. By introducing residual learning, it can solve the problem of gradient disappearance in traditional deep networks. The present invention also provides a camera calibration device and equipment for an optical fiber positioning system.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a camera calibration method, device, and equipment for an optical fiber positioning system. Background Art

[0002] The fiber optic spectroscopic astronomical telescope, as an advanced observation device integrating fiber optic technology and spectrometer analysis technology, its core lies in capturing the optical signals of celestial bodies through optical fibers, and then deeply analyzing these spectral information by the spectrometer to reveal the physical properties, chemical compositions, and motion states of celestial bodies. A large number of optical fibers are densely arranged on the focal plane of such a telescope, and each optical fiber is accurately aligned with a celestial body, achieving efficient observation of multiple celestial bodies simultaneously.

[0003] In order to further improve the accuracy and efficiency of observation, the fiber optic positioning system introduces a front illumination recognition system camera, which can monitor and measure the working state of the fiber optic positioning unit in real time. However, the camera needs to go through a strict calibration process before use, including the determination of internal parameters (such as focal length) and external parameters (such as the position and attitude of the camera relative to the focal plane). This calibration process is crucial as it establishes an accurate mapping between image pixels and physical coordinates on the focal plane, thus ensuring the accurate position positioning of the fiber optic positioning unit and the precise calculation of the distance between the optical fiber and the target star image, providing a basis for the precise positioning of the system.

[0004] Currently, traditional camera calibration methods rely on checkerboard calibration plates and Zhang Zhengyou calibration method. This process requires manual placement and shooting of the calibration plate multiple times and from multiple angles, which is not only complex in operation but also time-consuming. For a large-scale observation system such as a fiber optic spectroscopic astronomical telescope, there is clearly a lack of a simple, fast, and accurate intelligent automated camera calibration method.

[0005] Currently, fiber optic spectroscopic astronomical telescopes that have been built or are being built at home and abroad are all equipped with a large number of optical fibers for efficient observation. Therefore, developing a fast and accurate intelligent automated camera calibration method has extremely high practical value and wide applicability for improving the efficiency and accuracy of the fiber optic positioning systems of these telescopes.

[0006] Although a single-view image camera calibration method based on deep learning has been proposed, this method is mainly applicable to daily life scenarios and relies on geometric clues such as the horizon, and is not applicable to fiber optic positioning systems with low light and the photographed objects being optical fibers and other mechanical structures. At the same time, although the collision detection method for fiber optic units of multi-object fiber optic spectroscopic astronomical telescopes also uses computer vision technology, it does not involve camera calibration methods.

[0007] In summary, for the specific industrial scenario of fiber optic spectroscopic astronomical telescopes, it is particularly important to develop a fast, accurate and intelligent automated camera calibration method, which will greatly promote the development of astronomical observation technology and improve the observation efficiency and accuracy. Summary of the Invention

[0008] (I) Technical Problems to be Solved

[0009] In view of the above problems, the present invention provides a camera calibration method, device and equipment for a fiber optic positioning system. The neural network is trained using the massive image data rendered by the simulation three-dimensional model of the fiber optic positioning system, so as to quickly and accurately calibrate the external parameters of the camera of the headlight recognition system.

[0010] (II) Technical Solutions

[0011] In view of the above technical problems, embodiments of the present invention propose a camera calibration method, device and equipment for a fiber optic positioning system.

[0012] According to a first aspect of the present invention, there is provided a camera calibration method for a fiber optic positioning system, including: obtaining an initial image of a camera of a headlight recognition system; preprocessing the initial image to obtain a preprocessed graph; inputting the preprocessed image into a trained neural network model to obtain a predicted result of camera parameters, thereby completing camera calibration. The initial image includes images taken by the focus planes of each camera of the headlight recognition system, and the initial image can cover the entire focal plane area. The neural network model is a convolutional neural network with deep residual learning ability. By introducing residual learning, the problem of gradient disappearance in traditional deep networks can be solved.

[0013] In some exemplary embodiments, preprocessing the initial image includes: using a target detection algorithm to detect reference fiber units in the initial image, removing fiber positioning units, and retaining reference fiber units to obtain a reference fiber image; obtaining the center points of the reference fiber units, and connecting the center points of the reference fiber units on the same straight line with a line; detecting reference fiber units with a specific arrangement shape in the reference fiber image, and drawing the fixed shape as a fixed feature to obtain a preprocessed image, where the preprocessed image includes the connecting lines of the center points of the fiber positioning units and the fixed shape.

[0014] In some exemplary embodiments, the neural network model has a depth of 101 layers, including an initial layer, a max pooling layer, a network processing stage, and a fully connected layer. Among them, the initial layer is a 7×7 convolutional layer with a stride of 2 and an output channel of 64; the max pooling layer is a 3×3 max pooling layer with a stride of 2; the network processing stage includes a first stage, a second stage, a third stage, and a fourth stage. The first stage includes 3 residual blocks, the second stage includes 4 residual blocks, the third stage includes 23 residual blocks, and the fourth stage includes 3 residual blocks. Each residual block includes a 1×1 convolutional layer and a 3×3 convolutional layer for dimension adjustment and feature extraction; the number of output features of the fully connected layer is 6; the activation function of the neural network model is the leaky rectified linear unit function.

[0015] In some exemplary embodiments, the method for obtaining a trained neural network model includes: constructing a training image dataset; preprocessing the training image dataset to obtain a preprocessed training image dataset; using the preprocessed training image dataset to train the neural network model to obtain a trained neural network model. Among them, the preprocessed training image dataset has the same data characteristics as the preprocessed images.

[0016] In some exemplary embodiments, constructing a training image dataset includes: based on the size of the actual fiber positioning system, constructing a simulation 3D model; setting the external parameters of the camera in the simulation 3D model to obtain the simulated images of the front camera recognition system; rendering the simulated images of the front camera recognition system and annotating the corresponding external parameters so that the simulated images and the camera external parameters are in one-to-one correspondence to form a training image dataset; among them, the focal plane, fiber positioning unit, single-head and multi-head reference fiber units in the simulation 3D model are simulated and modeled according to the actual size to ensure the accuracy of the model; the central axis and eccentric axis robotic arms in the simulation 3D model can rotate at the actual functional angles to achieve the effect of simulation.

[0017] In some exemplary embodiments, using the preprocessed training image dataset to train the neural network model includes: inputting the preprocessed training image dataset into the neural network model to obtain the predicted values of the camera external parameters; using the smooth L1 loss function to calculate the loss between the predicted values and the true values, and gradually converging the model by minimizing the loss function to obtain the optimal solution.

[0018] In some exemplary embodiments, training the neural network model using the preprocessed training image dataset further includes: obtaining the pixel coordinates of the reference optical fiber points with a fixed shape in the preprocessed training image dataset, recording the pixel coordinates as a two-dimensional array, and encoding them into a feature vector with a fixed length; performing downsampling on the feature vector through average pooling, and then splicing it with the convolutional feature map to obtain feature information; training the neural network model based on the feature information to obtain a trained neural network model.

[0019] In some exemplary embodiments, training the neural network model using the preprocessed training image dataset further includes: using the test set data to test the trained neural network model, and calculating the average value and median of the errors of the external camera parameters; comparing the average value and median of the errors with the existing neural network camera calibration methods to verify the superiority of the model.

[0020] The second aspect of the present invention provides a camera calibration device for an optical fiber positioning system, including: an acquisition module, configured to acquire the initial images of the front recognition system cameras; a preprocessing module, configured to preprocess the initial images to obtain preprocessed graphics; a prediction module, configured to input the preprocessed images into the trained neural network model to obtain the prediction results of the camera parameters, and complete camera calibration, where the initial images include the images captured by each front recognition system camera on the focus plane, and the initial images can cover the entire focal plane area; the neural network model is a convolutional neural network with deep residual learning ability, and by introducing residual learning, it can solve the problem of gradient disappearance in traditional deep networks.

[0021] The third aspect of the present invention provides an electronic device, including: one or more processors; a memory, configured to store one or more computer programs, where the above-mentioned one or more processors execute the above-mentioned one or more computer programs to implement the steps of the above method.

[0022] The fourth aspect of the present invention further provides a computer-readable storage medium, on which computer programs or instructions are stored, and when the above-mentioned computer programs or instructions are executed by a processor, the steps of the above method are implemented.

[0023] (III) Beneficial effects

[0024] As can be seen from the above technical solutions, a camera calibration method, device, and equipment for an optical fiber positioning system provided by the embodiments of the present invention have at least the following beneficial effects:

[0025] (1) By constructing a simulation 3D model of the fiber optic positioning system, a large amount of training data can be generated, providing sufficient data support for the training of the neural network. The simulation 3D model can also simulate complex conditions such as low light and no feature reference objects in the real environment, enabling the neural network to better adapt to the actual observation environment.

[0026] (2) A series of targeted image preprocessing operations are proposed for the unique industrial environment of the fiber optic positioning system, such as identifying reference fiber optic units, detecting the edges of fiber optic points, and fixed features, etc., further improving the prediction accuracy of the neural network for camera parameters.

[0027] (3) There are negative values in the rotation amount parameter of the camera in the dataset. In order to avoid the rectified linear unit (ReLU) in the network from removing the negative parameter range, it is changed to the leaky rectified linear unit (LeakyReLU) function, so that when the input is negative, a non-zero output is generated, effectively processing the negative value feature of the rotation amount and retaining more information. Brief Description of the Drawings

[0028] Through the following description of the embodiments of the present invention with reference to the drawings, the above content and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0029] Figure 1 Schematically shows the structural diagram of the LAMOST fiber optic positioning system according to an embodiment of the present invention.

[0030] Figure 2 Schematically shows the flowchart of a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention.

[0031] Figure 3 Schematically shows the schematic diagram of the front illumination recognition system of LAMOST according to an embodiment of the present invention.

[0032] Figure 4 Schematically shows the schematic diagram of the positional relationship between the camera and the focal plane in LAMOST according to an embodiment of the present invention.

[0033] Figure 5 Schematically shows the schematic diagram of the actual captured image obtained by the front camera according to an embodiment of the present invention.

[0034] Figure 6 Schematically shows the schematic diagram of the simulated simulation image obtained according to an embodiment of the present invention.

[0035] Figure 7 Schematically shows the schematic diagram of the reference fiber optic image obtained according to an embodiment of the present invention.

[0036] Figure 8Schematically shows a schematic diagram of a horizontal connection line of the center point of a reference optical fiber unit according to an embodiment of the present invention.

[0037] Figure 9 Schematically shows a schematic diagram of a fixed shape drawn based on a reference optical fiber unit with a specific arrangement shape in a reference optical fiber image according to an embodiment of the present invention.

[0038] Figure 10 Schematically shows a flowchart of a method for obtaining a trained neural network model according to an embodiment of the present invention.

[0039] Figure 11 Schematically shows a schematic diagram of a simulation three-dimensional model constructed in a one-to-one ratio of the size of an optical fiber positioning system according to an embodiment of the present invention.

[0040] Figure 12 Schematically shows a schematic diagram of an optical fiber positioning unit model according to an embodiment of the present invention.

[0041] Figure 13 Schematically shows a structural block diagram of a camera calibration device of an optical fiber positioning system according to an embodiment of the present invention.

[0042] Figure 14 Schematically shows a block diagram of an electronic device of a camera calibration method of an optical fiber positioning system according to an embodiment of the present invention. Detailed implementation manners

[0043] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0044] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising", etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0045] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0046] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0047] In order to facilitate the understanding of the content of the invention, the fiber optic spectroscopic telescope is introduced by taking the Guo Shoujing Telescope (Large Sky Area Multi-Object Fiber Spectroscopic Telescope, referred to as LAMOST) as an example. LAMOST now occupies a very important position in the field of domestic optical survey telescopes. It has created many outstanding achievements relying on advanced equipment and technology. Among them, the parallel controllable fiber optic positioning unit technology is a key independent innovation technology of LAMOST. During the operation of LAMOST, the light of the celestial body passes through the reflective Schmidt corrector MA and the spherical primary mirror MB in turn, and is finally reflected and imaged on the focal plane. Among them, there are 4,000 fiber optic positioning units distributed throughout the focal plane, which can quickly and accurately locate the star image target according to the position in the star catalog, thereby receiving celestial starlight. The LAMOST fiber optic positioning system is as follows Figure 1 As shown, there are 4,000 fiber positioning units, each with two rotatable robotic arms, each with a central and eccentric axis. The arms are 8.25 mm long. These units are controlled by rotation to position the optical fibers on the star image target. The system relies on seven front-view detection cameras placed around the spherical primary mirror (MB) to photographically inspect the 4,000 optical fibers on the focal plane, providing real-time information on the unit's position and determining whether further positioning is required.

[0048] Figure 2 The flowchart of a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0049] like Figure 2 As shown, a camera calibration method for a fiber optic positioning system according to an embodiment of the present invention includes steps S110 to S130.

[0050] In step S110 , an initial image of a front-view recognition system camera is acquired, wherein the initial image includes images captured by each front-view recognition system camera on a focus plane, and the initial image can cover the entire focus plane area.

[0051] For example, for LAMOST, its front-view recognition system cameras are located around the spherical primary mirror MB, such as Figure 3 As shown, there are 7 cameras in total (respectively Figure 3A, B, C, D, E, F, and G in). These cameras take pictures and measurements of the focal plane at a distance of 20 meters from the focal plane. The positional relationship between the cameras and the focal plane is as Figure 4 shown, where the front camera is in the same position as the spherical primary mirror MB. The actual captured image obtained by the front camera is as Figure 5 shown, where the fiber optic positioning units are all in the non-operating zero position state, and the image size is 7920×6004 pixels. The areas captured by each camera are different, and the ranges captured by the 7 cameras cover the entire focal plane area. The ultimate goal of the method in the embodiment of the present invention is to calibrate these 7 cameras respectively through a single camera image.

[0052] In step S120, the initial image is preprocessed to obtain a preprocessed graph.

[0053] In order to improve the accuracy of camera calibration, in the unique working environment of the fiber optic positioning system with low light, relatively fixed objects to be photographed, no real geometric references such as floors and horizons, and no obvious reference objects with features, the embodiment of the present invention proposes a series of targeted image preprocessing operations, aiming to make the neural network more accurate in recognizing and analyzing camera images and more precise in predicting camera parameters.

[0054] In some exemplary embodiments, step S120 includes steps S121 - S123.

[0055] In step S121, a target detection algorithm is used to detect the reference fiber optic units in the initial image, remove the fiber optic positioning units, and retain the reference fiber optic units to obtain a reference fiber optic image.

[0056] For example, an image contains at least 800 fiber optic units, as Figure 6 shown, including fiber optic positioning units and reference fibers, and the swinging angles of the two arms of each fiber optic positioning unit are different, and every time LAMOST makes an observation, the rotation angle of the fiber optic positioning unit will change, which makes the feature points in the image very numerous and complex without any rules. In order to make the neural network estimate the external parameters of the camera with higher accuracy, a target detection algorithm is used in the preprocessing to detect the reference fibers in the image, and then other complex fiber optic positioning units are removed from the image. In this way, only about 50 reference fibers remain in the image, as Figure 7 shown, and the reference fibers have a single shape, simple structure, and unique arrangement rules, with obvious feature points, and the neural network can more easily learn the features, further improving the accuracy of predicting camera parameters.

[0057] In step S122, the center points of the reference fiber optic units are obtained, and the center points of the reference fiber optic units belonging to the same straight line are connected by lines.

[0058] In a conventional scenario, calibration objects with known dimensions such as calibration plates are important references for camera calibration. However, in the actual application scenario of the LAMOST fiber positioning system, the lack of such traditional calibration objects poses a challenge to camera calibration. To overcome this problem and enrich the geometric information in the image so that the neural network can more accurately judge the pose of the camera, this embodiment adopts a novel strategy. Specifically, the fiber points of the reference fiber units distributed on the vertical and horizontal lines in the system are used as key features. By precisely detecting these fiber points and connecting them in the straight line direction, as Figure 8 shown in the schematic diagram of the horizontal connecting line, a series of simulated vanishing lines are constructed. These connecting lines form artificial geometric constraints in the image. They simulate the characteristics of vanishing lines in perspective projection and provide valuable information for the calibration of the external parameters of the camera. Through these geometric constraints, the neural network can better understand the translation and rotation states of the camera, and thus more accurately estimate the pose of the camera. In addition, this method of using the artificial geometric constraints formed by fiber points not only improves the accuracy of the neural network in predicting the camera pose, but also enhances the robustness of the system to changes in camera parameters.

[0059] In step S123, the reference fiber units with a specific arrangement shape in the reference fiber image are detected, and this fixed shape is drawn as a fixed feature to obtain a preprocessed image, where the preprocessed image includes the connecting lines of the central points of the fiber positioning units and the fixed shape.

[0060] On the LAMOST focal plane, there are several holes for loading other instruments. Interestingly, the fiber units around these holes are exactly arranged in a fixed shape. This fixed shape provides a natural auxiliary calibration object for camera calibration, which can help the neural network more accurately identify the geometric space structure.

[0061] By using this fixed shape, the neural network can deduce the translation and rotation parameters of the camera relative to the scene. As Figure 9 shown, a schematic diagram of this fixed shape is drawn to intuitively show its role in camera calibration. In addition, to further improve the prediction accuracy of the neural network, the pixel coordinates of the reference fiber points included in this fixed shape can also be recorded. These pixel coordinates, as prior information, will play an important role in the subsequent camera calibration process and provide strong support for the accurate prediction of the neural network.

[0062] In step S130, the preprocessed image is input into the trained neural network model to obtain the prediction result of the camera parameters, and the camera calibration is completed. The neural network model is a convolutional neural network with deep residual learning ability. By introducing residual learning, the problem of gradient disappearance in traditional deep networks can be solved.

[0063] In some exemplary embodiments, the neural network model is based on Resnet101 and undergoes a series of targeted modifications, optimizations, and improvements. The depth of this neural network model is 101 layers, including an initial layer, a max pooling layer, a network processing stage, and a fully connected layer. Among them, the initial layer is a 7×7 convolutional layer with a stride of 2 and an output channel of 64; the max pooling layer is a 3×3 max pooling layer with a stride of 2; the network processing stage includes a first stage, a second stage, a third stage, and a fourth stage. The first stage includes 3 residual blocks, the second stage includes 4 residual blocks, the third stage includes 23 residual blocks, and the fourth stage includes 3 residual blocks. Each residual block includes a 1×1 convolutional layer and a 3×3 convolutional layer for dimension adjustment and feature extraction; the number of output features of the fully connected layer is 6. The purpose of the neural network in the embodiments of the present invention is to predict the external parameters of the camera, which is a regression task for predicting continuous values. Therefore, the parameters of the fully connected layer in the last layer of the network are set to 6, that is, the translation and rotation amounts of the camera on the X, Y, and Z axes are output; the activation function of the neural network model is the leaky rectified linear unit function.

[0064] There are negative values in the rotation amount parameters of the cameras in the dataset. To avoid the activation function ReLU in the network from removing the negative parameter range, it is changed to the LeakyReLU function, so that when the input is negative, a non-zero output is generated, effectively handling the negative value features of the rotation amount and retaining more information. The specific formula is as follows:

[0065] (1)

[0066] Where is a small positive number, which is set to 0.1 in this embodiment.

[0067] In some exemplary embodiments, the method for obtaining the trained neural network model includes steps S210 - S230, see Figure 10 .

[0068] In step S210, a training image dataset is constructed.

[0069] In some exemplary embodiments, step S210 includes steps S211 - S213.

[0070] In step S211, based on the size of the actual fiber optic positioning system, a simulation three-dimensional model is constructed. Among them, the focal panel, fiber optic positioning unit, single-head, and multi-head reference fiber optic unit in the simulation three-dimensional model are simulated and modeled according to the actual size to ensure the accuracy of the model; the central axis and eccentric axis robotic arms in the simulation three-dimensional model can rotate at the actual functional angles to achieve the effect of simulation.

[0071] To provide a large amount of training and test image data for the neural network model, in the embodiments of the present invention, a three-dimensional modeling software is used to construct a simulation three-dimensional model one-to-one according to the actual size of the optical fiber positioning system, as Figure 11 shown. The front view and partial enlarged view of the simulation three-dimensional model example of the LAMOST optical fiber positioning system are shown. Among them, the focal panel, the optical fiber positioning unit, the single-head and multi-head reference optical fiber units are also simulated and modeled according to the actual size, as Figure 12 shown. The schematic diagram of the optical fiber positioning unit model is shown. The central axis and eccentric axis robotic arms therein can also rotate angles according to the actual functions to achieve the simulation effect.

[0072] In step S212, the external parameters of the camera in the simulation three-dimensional model are set to obtain the simulated image of the front view recognition system camera.

[0073] In step S213, the simulated image of the front view recognition system camera is rendered and the corresponding external parameters are marked, so that the simulated image corresponds one-to-one with the external parameters of the camera, forming a training image data set.

[0074] Relying on the modeling software, the external parameters of tens of thousands of cameras with different external parameters can be set. The images of these cameras are rendered with the rendering software and the corresponding external parameters are marked, that is, the translation and rotation amounts of the camera on the X, Y, and Z axes, so that the one-to-one correspondence between a single image and the external parameters of the camera is formed, forming an image data set for the training and testing of the neural network.

[0075] For example, the simulation image is rendered using the rendering software, as Figure 6 shown. Among them Figure 6 is the simulated rendering image of the simulated Figure 5 real-shot image. The central axis and eccentric axis angles of the optical fiber positioning unit are randomly adjusted for testing the simulation effect.

[0076] During the actual operation of the astronomical telescope, in order to prevent the instrument surface from reflecting light and interfering with the received starlight, most of the instrument surfaces of the telescope focal plane and the optical fiber positioning system are made of materials with a relatively low reflectivity as much as possible. In order to simulate this low-light or even dark condition, the simulation three-dimensional model also changes the instrument material and color according to the corresponding requirements, and the rendering light is also adjusted to be darker to simulate the actual situation as much as possible, achieving the purpose of accurate simulation.

[0077] In step S220, the training image data set is preprocessed to obtain the preprocessed training image data set. Among them, the preprocessed training image data set has the same data characteristics as the preprocessed image.

[0078] The method for preprocessing the training image dataset can adopt the method for preprocessing the initial image in step S120.

[0079] In step S230, the neural network model is trained using the preprocessed training image dataset to obtain a trained neural network model.

[0080] In some exemplary embodiments, step S230 includes steps S231 - S232.

[0081] In step S231, the preprocessed training image dataset is input into the neural network model to obtain predicted values of the external camera parameters.

[0082] In step S232, the smooth L1 loss function is used to calculate the loss between the predicted values and the true values, and the model is gradually converged to the optimal solution by minimizing the loss function. This method uses square penalty for small errors and linear penalty for large errors, thus making the gradient of the loss function more stable. The specific formula is as follows:

[0083] (2)

[0084] where x represents the difference between the predicted value and the true value. is a preset threshold. Optionally, is set to 1.

[0085] To further improve the prediction accuracy of the neural network and help the neural network better learn the mapping relationship between the image and the camera parameters, step S230 may further include: obtaining the pixel coordinates of the reference fiber points with a fixed shape in the preprocessed training image dataset; recording the pixel coordinates as a two-dimensional array and encoding them into a feature vector with a fixed length; performing downsampling on the feature vector through average pooling, and then splicing it with the convolutional feature map to obtain feature information; training the neural network model based on the feature information to obtain a trained neural network model.

[0086] To verify the effect of the trained neural network model of the embodiment of the present invention, the trained neural network model is tested using the test set data, and the average value and median of the errors of the external camera parameters are calculated; the average value and median of the errors are compared with the existing neural network camera calibration methods to verify the superiority of the model.

[0087] The average value and median of the errors (the difference between the network predicted value and the true value) of the six camera parameters predicted by using the trained neural network model of the embodiment of the present invention are shown in Table 1.

[0088] Table 1 Prediction errors of the trained neural network model of the embodiment of the present invention

[0089]

[0090] The prediction results of this embodiment in terms of rotation parameters are compared with existing neural network camera calibration methods as shown in Table 2. Among them, Upright is the method of "Automatic Upright Adjustment of Photographs with Robust Camera Calibration" by Hyunjoon Lee et al. in 2014; Perceptual is the method of "A Perceptual Measure for Deep Single Image Camera Calibration" by Yannick Hold-Geoffroy et al. in 2018; UprightNet is the method of "UprightNet: Geometry-Aware Camera Orientation Estimation From Single Images" by Wenqi Xian et al. in 2019; GPNet is the method of "Neural Geometric Parser for Single Image Camera Calibration" by Jinwoo Lee et al. in 2020; CTRL-C is the method of "CTRL-C: Camera calibration transformer with line-classification" by J. Lee et al. in 2021). The result of the method of this present embodiment is shown in the last line "Ours".

[0091] Table 2 Comparison of prediction results of different calibration methods

[0092]

[0093] It should be noted that other methods in Table 2 mainly target diverse daily outdoor or indoor scenes, and the dataset used is the publicly available street view dataset. The method of the present invention targets a completely different fiber optic positioning system. As can be seen from the above table, the experimental results of the method of the present invention not only meet the accuracy requirements of existing neural network camera calibration methods, but also are superior to other camera calibration methods.

[0094] Figure 13 The structural block diagram of a camera calibration device for a fiber optic positioning system according to an embodiment of the present invention is schematically shown.

[0095] As Figure 13As shown, the camera calibration device 800 of the optical fiber positioning system according to this embodiment includes an acquisition module 810, a preprocessing module 820, and a prediction module 830.

[0096] The acquisition module 810 is configured to acquire an initial image of the headlamp recognition system camera. Among them, the initial image includes images taken by the focus planes of each headlamp recognition system camera, and the initial image can cover the entire focal plane area.

[0097] The preprocessing module 820 is configured to preprocess the initial image to obtain a preprocessed graph.

[0098] The prediction module 830 is configured to input the preprocessed image into the trained neural network model to obtain a prediction result of the camera parameters, thereby completing camera calibration. The neural network model is a convolutional neural network with deep residual learning ability. By introducing residual learning, the problem of gradient disappearance in traditional deep networks can be solved.

[0099] According to an embodiment of the present invention, any multiple of the acquisition module 810, the preprocessing module 820, and the prediction module 830 can be combined and implemented in one module, or any one of them can be split into multiple modules. Or, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 810, the preprocessing module 820, and the prediction module 830 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable means such as integrating or packaging circuits, etc., in hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in any appropriate combination of several of them. Or, at least one of the acquisition module 810, the preprocessing module 820, and the prediction module 830 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions can be executed.

[0100] Figure 14 Schematically shows a block diagram of an electronic device for the camera calibration method of the optical fiber positioning system according to an embodiment of the present invention.

[0101] As Figure 14As shown, the electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include on-board memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0102] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method flow according to an embodiment of the present invention by executing a program in the ROM 902 and / or the RAM 903. It should be noted that the program may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to an embodiment of the present invention by executing a program stored in one or more memories.

[0103] According to an embodiment of the present invention, the electronic device 900 may further include an input / output (I / O) interface 905, and the input / output (I / O) interface 905 is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read from it can be installed into the storage section 908 as needed.

[0104] The present invention also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist alone without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present invention is implemented.

[0105] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the above-described ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903.

[0106] Those skilled in the art can understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A camera calibration method for an optical fiber positioning system, characterized in that The method includes: Obtaining an initial image of the headlamp recognition system camera; Preprocessing the initial image to obtain a preprocessed graph; Inputting the preprocessed image into a trained neural network model to obtain a prediction result of the camera parameters and complete camera calibration. Wherein, the initial image includes images taken by the focus planes of each headlamp recognition system camera, and the initial image can cover the entire focal plane area; The neural network model is a convolutional neural network with deep residual learning ability. By introducing residual learning, the problem of gradient disappearance in traditional deep networks can be solved; The preprocessing of the initial image includes: Using an object detection algorithm to detect the reference optical fiber units in the initial image, removing the optical fiber positioning units, and retaining the reference optical fiber units to obtain a reference optical fiber image; Obtaining the center points of the reference optical fiber units and connecting the center points of the reference optical fiber units on the same straight line with a line; Detecting the reference optical fiber units with a specific arrangement shape in the reference optical fiber image, and drawing the specific arrangement shape as a fixed feature to obtain a preprocessed image; Wherein, the preprocessed image includes the connection lines of the center points of the optical fiber positioning units and the specific arrangement shape; The method for obtaining a trained neural network model includes: Constructing a training image data set; Preprocessing the training image data set to obtain a preprocessed training image data set; Training a neural network model using the preprocessed training image data set to obtain a trained neural network model; Wherein, the preprocessed training image data set has the same data characteristics as the preprocessed image; The constructing of the training image data set includes: Based on the size of the actual optical fiber positioning system, constructing a simulation three-dimensional model; Setting the external parameters of the camera in the simulation three-dimensional model to obtain a simulated image of the corresponding headlamp recognition system camera; Rendering the simulated image of the headlamp recognition system camera and annotating the corresponding external parameters so that the simulated image corresponds one-to-one with the camera external parameters to form a training image data set; Wherein, the focal plane panel, optical fiber positioning unit, single-head and multi-head reference optical fiber units in the simulation three-dimensional model are simulated and modeled according to the actual size to ensure the accuracy of the model; The central axis and eccentric axis robotic arms in the simulation three-dimensional model can rotate at the actual functional angles to achieve the effect of simulation.

2. The method according to claim 1, wherein The depth of the neural network model is 101 layers, including an initial layer, a max pooling layer, a network processing stage, and a fully connected layer. Wherein, the initial layer is a 7×7 convolutional layer with a stride of 2 and an output channel of 64; The max pooling layer is a 3×3 max pooling layer with a stride of 2; The network processing stage includes a first stage, a second stage, a third stage, and a fourth stage. The first stage includes 3 residual blocks, the second stage includes 4 residual blocks, the third stage includes 23 residual blocks, and the fourth stage includes 3 residual blocks. Each of the residual blocks includes a 1×1 convolutional layer and a 3×3 convolutional layer for dimension adjustment and feature extraction. The output feature number of the fully connected layer is 6; The activation function of the neural network model is the leaky rectified linear unit function.

3. The method according to claim 1, wherein The training of the neural network model using the preprocessed training image dataset includes: Inputting the preprocessed training image dataset into the neural network model to obtain the predicted values of the camera external parameters; Using the smooth L1 loss function to calculate the loss between the predicted values and the true values, and gradually converging the model by minimizing the loss function to obtain the optimal solution.

4. The method according to claim 3, wherein The training of the neural network model using the preprocessed training image dataset further includes: Obtaining the pixel coordinates of the reference optical fiber points with a specific arrangement shape in the preprocessed training image dataset; Recording the pixel coordinates as a two-dimensional array and encoding them into a feature vector with a fixed length; Performing downsampling on the feature vector through average pooling, and then splicing it with the convolutional feature map to obtain feature information; Based on the feature information, training the neural network model to obtain the trained neural network model.

5. The method according to claim 4, characterized in that The training of the neural network model using the preprocessed training image dataset further includes: Using the test set data to test the trained neural network model, and calculating the average value and median of the errors of the camera external parameters; Comparing the average value and median of the errors with the existing neural network camera calibration methods to verify the superiority of the model.

6. A camera calibration device for an optical fiber positioning system, characterized in that The device includes: An acquisition module for acquiring the initial images of the headlamp recognition system camera; A preprocessing module for preprocessing the initial images to obtain preprocessed graphics; A prediction module for inputting the preprocessed images into the trained neural network model to obtain the predicted results of the camera parameters and complete camera calibration, wherein the initial images include the images captured by the focus planes of each headlamp recognition system camera, and the initial images can cover the entire focal plane area; The neural network model is a convolutional neural network with deep residual learning ability. By introducing residual learning, it can solve the problem of gradient disappearance in traditional deep networks; The preprocessing of the initial images includes: Using a target detection algorithm to detect the reference optical fiber units in the initial images, removing the optical fiber positioning units, and retaining the reference optical fiber units to obtain reference optical fiber images; Obtaining the center points of the reference optical fiber units, and connecting the center points of the reference optical fiber units on the same straight line with lines; Detecting the reference optical fiber units with a specific arrangement shape in the reference optical fiber images, and drawing the specific arrangement shape as a fixed feature to obtain preprocessed images, wherein the preprocessed images include the connecting lines of the center points of the optical fiber positioning units and the specific arrangement shape; The method for obtaining the trained neural network model includes: Constructing a training image dataset; Preprocessing the training image dataset to obtain a preprocessed training image dataset; Using the preprocessed training image dataset to train the neural network model to obtain the trained neural network model, wherein the preprocessed training image dataset has the same data characteristics as the preprocessed images; The construction of the training image dataset includes: Construct a simulation 3D model based on the size of the actual fiber optic positioning system; Set the external parameters of the camera in the simulation 3D model to obtain the simulated images of the camera of the headlamp recognition system; Render the simulated images of the camera of the headlamp recognition system and label the corresponding external parameters, so that the simulated images correspond one-to-one with the external parameters of the camera, forming a training image dataset; Among them, the focal plane panel, fiber optic positioning unit, single-head and multi-head reference fiber optic units in the simulation 3D model are simulated and modeled according to the actual size to ensure the accuracy of the model; The central axis and eccentric axis robotic arms in the simulation 3D model can rotate at the actual functional angle to achieve the effect of simulation; 7. An electronic device, comprising: One or more processors; A memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

Images