Single-view image camera calibration method and system

By constructing encoder and decoder networks and combining self-attention mechanisms and loss functions, the problem of insufficient accuracy of traditional single-view image camera calibration methods in unknown scenarios is solved, achieving higher reliability and accuracy, and making it suitable for a wider range of application scenarios.

CN118552626BActive Publication Date: 2024-11-26CENT SOUTH UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411002916.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

Traditional single-view camera calibration methods are not accurate enough in unknown scenes or when there is noise interference. Deep learning-based methods rely on large-scale data and ignore camera imaging geometry modeling, resulting in poor practicality and accuracy.

Method used

An encoder network and a decoder network are constructed. Feature vectors are extracted through image geometric features. The network is trained using a self-attention mechanism and frequency domain analysis, combined with a log-space loss function and a similarity loss function, to achieve accurate calibration of camera parameters for single-view images.

Benefits of technology

It improves the reliability and accuracy of calibration without relying on markers, is applicable to a wider range of scenarios, and has better practicality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118552626B_ABST
    Figure CN118552626B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for calibrating a single-view image camera, including obtaining an existing image data set; extracting a feature vector corresponding to the image based on the geometric features of the image and constructing a training data set; constructing a preliminary model for calibrating a single-view image camera including an encoder network and a decoder network and training to obtain a single-view image camera calibration model; and using the single-view image camera calibration model to complete parameter calibration of a target single-view image camera. The present invention also discloses a system for implementing the single-view image camera calibration method. The present invention can obtain more accurate calibration results in scenes without preset markers, and has higher reliability, better accuracy, and better practicality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a single-view image camera calibration method and system. Background Art

[0002] Camera calibration is a key pre-step in the field of vision and imaging, and plays a vital role in many applications such as 3D reconstruction, target tracking, and robot navigation. Only with accurate camera calibration can an accurate mapping of a 2D image to a 3D world be established. In recent years, with the continuous development of computer vision technology, the application scope of camera calibration has been continuously expanding, from industrial inspection and drone driving to emerging fields such as virtual reality and unmanned driving. At present, the application scenarios of single-view image cameras are gradually increasing, such as Internet images; therefore, higher requirements are placed on the accuracy of single-view image camera calibration.

[0003] Traditional single-view image camera calibration methods require the presence of specified landmarks in the calibration image. Although this calibration method can achieve high accuracy under ideal conditions, its calibration performance depends largely on the design and quality of the landmarks. When faced with unknown scenes, due to the lack of preset landmarks or the distortion of the landmarks, or when there is a lot of noise interference in the image, the feature information extracted by this type of traditional single-view image camera calibration method often has deviations; this causes this type of scheme to often perform unsatisfactorily under non-ideal conditions. In response to the limitations of this type of traditional method, researchers proposed a single-view image camera calibration scheme based on deep learning technology.

[0004] However, most of the current single-view image camera calibration solutions based on deep learning are based on a data-driven approach, that is, using a large-scale dataset to directly regress the camera focal length or related geometric constraints. Although such methods have reduced the dependence on landmarks and scenes to a certain extent, the effectiveness of such solutions depends on the quantity and quality of data, and large-scale and diverse training data are required to achieve good results. In addition, such methods treat the calibration problem as a black-box regression problem, thereby ignoring the explicit modeling and reasoning of camera imaging geometry, and lack the full utilization of the geometric structure features associated with different prediction targets. Therefore, the practical practicality and accuracy of such solutions are relatively poor. Summary of the invention

[0005] One of the purposes of the present invention is to provide a single-view image camera calibration method with high reliability, good accuracy and good practicality.

[0006] A second objective of the present invention is to provide a system for implementing the single-view image camera calibration method.

[0007] The single-view image camera calibration method provided by the present invention comprises the following steps:

[0008] S1. Obtain an existing image dataset;

[0009] S2. For the image data set obtained in step S1, extracting feature vectors corresponding to the image based on the geometric features of the image, thereby constructing a training data set;

[0010] S3. Build a preliminary model for single-view image camera calibration including encoder network and decoder network;

[0011] The encoder network is used to extract geometric features and contextual information in the input image and adaptively focus on key areas in the input image;

[0012] The decoder network is used to map the output features of the encoder network and the features of the input image to the single-view image camera parameter space, thereby predicting the internal and external parameters of the single-view image camera;

[0013] S4. Using the training data set constructed in step S2, the single-view image camera calibration preliminary model constructed in step S3 is trained to obtain a single-view image camera calibration model;

[0014] S5. Using the single-view image camera calibration model constructed in step S4, complete the parameter calibration of the target single-view image camera.

[0015] The step S2 specifically includes the following steps:

[0016] For the image dataset obtained in step S1, the LSD (Line Segment Detector) algorithm is used to extract the lines L in the image, which can be expressed as ,in is the i-th image, is the number of detected line segments;

[0017] Each line segment is divided into two parts according to its line equation in the image plane. Indicates that is the image coordinate, a is the normal vector component of the line in the x-axis direction, b is the normal vector component of the line in the y-axis direction, and c is the normal vector component from the line to the origin distance;

[0018] Transform to get upper triangular matrix , and encoded as a 6-dimensional vector , as the feature vector corresponding to the image.

[0019] The step S3 comprises the following steps:

[0020] Based on the spatial domain geometric features and the frequency domain geometric features, an encoder network is constructed;

[0021] Based on the self-attention mechanism, a decoder network is constructed.

[0022] The construction of the encoder network specifically includes the following steps:

[0023] The constructed encoder network includes a serially connected encoding module, a frequency domain module, an attention module, and an inference module; the input of the encoder network is the image in the training data set;

[0024] The encoding module is used to perform preliminary feature extraction and representation learning on the input image; the encoding module includes a first encoder layer, a second encoder layer, a third encoder layer and a fourth encoder layer; the input of the first encoder layer and the output of the first encoder layer are added as the input of the second encoder layer; the input of the second encoder layer and the output of the second encoder layer are added as the input of the third encoder layer; the input of the third encoder layer and the output of the third encoder layer are added as the input of the fourth encoder layer; the input of the fourth encoder layer and the output of the fourth encoder layer are added as the output of the encoding module; the structures of the first encoder layer, the second encoder layer, the third encoder layer and the fourth encoder layer are the same, and all include a first convolutional layer, a second convolutional layer, a third convolutional layer, a batch normalization layer and a ReLU activation function layer; the first convolutional layer is The second convolutional layer is The third convolutional layer is The convolutional layer;

[0025] The frequency domain module is used to extract the geometric feature distribution of the image at different frequencies; the frequency domain module includes a first normalization layer, a fast Fourier transform layer, a complex parameter layer, a fast inverse Fourier transform layer and a second normalization layer connected in series; the output of the encoding module is normalized by the first normalization layer and then fast Fourier transformed by the fast Fourier transform layer; the output of the fast Fourier transform layer is processed by the complex parameter layer to control the importance of different frequency components; the output of the complex parameter layer is fast inverse Fourier transformed by the fast inverse Fourier transform layer, and finally normalized by the second normalization layer to obtain the output of the frequency domain module; wherein the processing process of the complex parameter processing layer is expressed as , where w represents the weight parameter, represents the real part tensor after Fourier transformation, represents the imaginary part tensor after Fourier transformation, i represents the imaginary unit;

[0026] The attention module is used to extract and enhance the geometric features related to the single-view image camera calibration in the image; the attention module includes a linear transformation layer, a first scaling dot product layer and a geometric feature perception layer connected in series; the linear transformation layer is composed of a fully connected layer and is used to perform a linear transformation on the input feature data; the first scaling dot product layer is used to capture the relative importance and correlation between features. The processing process of the first scaling dot product layer is expressed as ,in and represents the matrix obtained after the linear transformation layer, express The dimension size of s represents the softmax function; the geometric feature perception layer is used to extract the geometric information feature representation of the input feature. The geometric feature perception layer includes a first branch, a second branch and a GeLU activation function sublayer. After the input features pass through the first branch and the second branch respectively, the output of the first branch is added to the output of the second branch, and then processed by the GeLU activation function sublayer, and multiplied by the output of the scaled dot product layer to obtain the output of the geometric feature perception layer. The first branch includes the sequentially connected Convolutional layer, batch normalization layer and sigmoid activation function layer. The second branch includes Convolutional layer, batch normalization layer and sigmoid activation function layer;

[0027] The inference module is used to make nonlinear changes to the output features of the attention module to enhance the feature extraction capability of the model; the inference module includes a first fully connected layer, a ReLu activation function layer, and a second fully connected layer connected in series.

[0028] The construction of the decoder network specifically includes the following steps:

[0029] The decoder network constructed includes a convolutional layer, a first attention layer, a second attention layer, and a feedforward neural network layer connected in series. The input of the decoder network is the output of the encoder network and the feature vector corresponding to the image in the training data set.

[0030] The convolutional layer is The convolutional layer;

[0031] The first attention layer and the second attention layer have the same structure, both of which include a linear projection sublayer, a second scaled dot product sublayer, a softmax sublayer, and a third scaled dot product sublayer; the linear projection sublayer is used to transform the input features through linear projection to obtain the Q value matrix Q, the K value matrix K, and the V value matrix V; the second scaled dot product sublayer is used to calculate the correlation between the Q value matrix Q and the K value matrix K. The processing process of the second scaled dot product sublayer is expressed as ,in and represents the matrix obtained after the linear projection sublayer, express The softmax sublayer is used to normalize the input data; the third scaling dot product sublayer is used to extract global context information. The processing process of the third scaling dot product sublayer is expressed as ,in Represents the matrix obtained after the linear projection sublayer;

[0032] The feedforward neural network layer uses a feedforward neural network to transform the obtained global context information into the intrinsic parameter space and extrinsic parameter space of the single-view image camera, thereby realizing the parameter calibration of the single-view image camera.

[0033] The training described in step S4 specifically includes the following steps:

[0034] The following formula is used as the logarithmic space loss function : Where w is the width of the image; in is the true value of the camera intrinsic parameter of the single-view image; is the predicted value of the camera intrinsic parameter of the single-view image; is the set threshold;

[0035] When predicting the vanishing point constraints of the camera extrinsics, the following formula is used as the first similarity loss function: : Where vp is the true value of the vanishing point coordinates; is the predicted value of the vanishing point coordinates; is the norm of the vector;

[0036] When predicting the horizon loss of the camera extrinsic parameters, the following formula is used as the second similarity loss function : Where n is the number of selected endpoints; hor is the true value of the horizon; is the predicted value of the horizon; Calculate the coordinates of the left and right endpoints of the horizon; is the Manhattan distance of the vector;

[0037] Finally, the total loss function LL is constructed as , is the first weight, is the second weight, The third weight.

[0038] The present invention also provides a system for implementing the single-view image camera calibration method, comprising a data acquisition module, a training set construction module, a model construction module, a model training module and a camera calibration module; the data acquisition module, the training set construction module, the model construction module, the model training module and the camera calibration module are connected in series in sequence; the data acquisition module is used to acquire an existing image data set, and upload the data information to the training set construction module; the training set construction module is used to extract the feature vector corresponding to the image of the acquired image data set according to the received data information, thereby constructing the training data set, and uploading the data information to the model construction module; the model construction module is used to construct a single-view image camera calibration preliminary model including an encoder network and a decoder network according to the received data information, and upload the data information to the model training module; the model training module is used to train the constructed single-view image camera calibration preliminary model according to the received data information, using the constructed training data set, to obtain the single-view image camera calibration model, and upload the data information to the camera calibration module; the camera calibration module is used to complete the parameter calibration of the target single-view image camera according to the received data information, using the constructed single-view image camera calibration model.

[0039] The single-view image camera calibration method and system provided by the present invention extract feature vectors based on the geometric features of the image, effectively extract geometric features related to the target in the image, provide richer and more targeted feature information for the model training process, and guide the constructed calibration network to perform corresponding training, so that the scheme of the present invention can obtain more accurate calibration results in scenes without preset markers, and the present invention has higher reliability, better accuracy and better practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a schematic diagram of the method flow of the calibration method of the present invention.

[0041] Figure 2 is a visual schematic diagram of the calibration method of the present invention, Figure 2 (a) is a visualization diagram of the vanishing point. Figure 2 (b) is a visualization diagram of the horizon. Figure 2 (c) is a visualization diagram of the internal reference.

[0042] Figure 3 is a schematic diagram of the calibration results of the calibration method of the present invention, Figure 3 (a1)-(e1) are schematic diagrams showing the comparison between the calibrated horizon results and the true values. Figure 3 (a2)-(e2) are schematic diagrams of the calibrated horizon and vanishing point results.

[0043] Figure 4 Schematic diagram of the functional modules of the system of the present invention. DETAILED DESCRIPTION

[0044] like Figure 1 The figure shows a schematic diagram of the method flow of the calibration method of the present invention: the single-view image camera calibration method disclosed in the present invention comprises the following steps:

[0045] S1. Obtain an existing image dataset;

[0046] In the specific implementation, the Google Street View dataset and the HLW dataset can be obtained, and the dataset includes images and the parameters of the corresponding single-view image camera;

[0047] S2. For the image data set obtained in step S1, extract the feature vector corresponding to the image based on the geometric features of the image, thereby constructing a training data set; specifically comprising the following steps:

[0048] For the image dataset obtained in step S1, the LSD (Line Segment Detector) algorithm is used to extract the lines L in the image, which can be expressed as ,in is the i-th image, is the number of detected line segments;

[0049] Each line segment is divided into two parts according to its line equation in the image plane. Indicates that is the image coordinate, a is the normal vector component of the line in the x-axis direction, b is the normal vector component of the line in the y-axis direction, and c is the normal vector component from the line to the origin distance;

[0050] Transform to get upper triangular matrix , and encoded as a 6-dimensional vector , as the feature vector corresponding to the image;

[0051] S3. Build a preliminary model for single-view image camera calibration including encoder network and decoder network;

[0052] The encoder network is used to extract geometric features and context information in the input image, and adaptively focus on key areas in the input image; the key areas are key areas containing significant geometric clues;

[0053] The decoder network is used to map the output features of the encoder network and the features of the input image to the single-view image camera parameter space, thereby predicting the internal and external parameters of the single-view image camera;

[0054] The specific implementation includes the following steps:

[0055] Based on the spatial domain geometric features and the frequency domain geometric features, an encoder network is constructed; specifically, the following steps are included:

[0056] The constructed encoder network includes a serially connected encoding module, a frequency domain module, an attention module, and an inference module; the input of the encoder network is the image in the training data set;

[0057] The encoding module is used to perform preliminary feature extraction and representation learning on the input image; the encoding module includes a first encoder layer, a second encoder layer, a third encoder layer and a fourth encoder layer; the input of the first encoder layer and the output of the first encoder layer are added as the input of the second encoder layer; the input of the second encoder layer and the output of the second encoder layer are added as the input of the third encoder layer; the input of the third encoder layer and the output of the third encoder layer are added as the input of the fourth encoder layer; the input of the fourth encoder layer and the output of the fourth encoder layer are added as the output of the encoding module; the structures of the first encoder layer, the second encoder layer, the third encoder layer and the fourth encoder layer are the same, and all include a first convolutional layer, a second convolutional layer, a third convolutional layer, a batch normalization layer and a ReLU activation function layer; the first convolutional layer is The second convolutional layer is The third convolutional layer is The convolutional layer;

[0058] The frequency domain module is used to extract the geometric feature distribution of the image at different frequencies. The geometric features will show a special distribution in the frequency domain. The frequency domain module includes a first normalization layer, a fast Fourier transform layer, a complex parameter layer, a fast inverse Fourier transform layer and a second normalization layer connected in series. The output of the encoding module is normalized by the first normalization layer and then fast Fourier transformed by the fast Fourier transform layer. The output of the fast Fourier transform layer is processed by the complex parameter layer to control the importance of different frequency components. The output of the complex parameter layer is fast inverse Fourier transformed by the fast inverse Fourier transform layer and finally normalized by the second normalization layer to obtain the output of the frequency domain module. The processing process of the complex parameter processing layer is expressed as , where w represents the weight parameter, represents the real part tensor after Fourier transformation, represents the imaginary part tensor after Fourier transformation, i represents the imaginary unit;

[0059] The attention module is used to extract and enhance the geometric features related to the single-view image camera calibration in the image; the attention module includes a linear transformation layer, a first scaling dot product layer and a geometric feature perception layer connected in series; the linear transformation layer is composed of a fully connected layer and is used to perform a linear transformation on the input feature data; the first scaling dot product layer is composed of matrix multiplication and scaling operations to obtain a weight matrix to capture the relative importance and correlation between features. The processing process of the first scaling dot product layer is expressed as ,in and represents the matrix obtained after the linear transformation layer, express The dimension size of s represents the softmax function; the geometric feature perception layer is used to extract the geometric information feature representation of the input feature. The geometric feature perception layer includes a first branch, a second branch and a GeLU activation function sublayer. After the input features pass through the first branch and the second branch respectively, the output of the first branch is added to the output of the second branch, and then processed by the GeLU activation function sublayer, and multiplied by the output of the scaled dot product layer to obtain the output of the geometric feature perception layer. The first branch includes the sequentially connected Convolutional layer, batch normalization layer and sigmoid activation function layer. The second branch includes Convolutional layer, batch normalization layer and sigmoid activation function layer;

[0060] The inference module is used to perform nonlinear changes on the output features of the attention module to enhance the feature extraction capability of the model; the inference module includes a first fully connected layer, a ReLu activation function layer, and a second fully connected layer connected in series;

[0061] Based on the self-attention mechanism, the decoder network is constructed; the specific steps include:

[0062] The decoder network constructed includes a convolutional layer, a first attention layer, a second attention layer, and a feedforward neural network layer connected in series. The input of the decoder network is the output of the encoder network and the feature vector corresponding to the image in the training data set.

[0063] The convolutional layer is The convolutional layer;

[0064] The first attention layer and the second attention layer have the same structure, both of which include a linear projection sublayer, a second scaled dot product sublayer, a softmax sublayer, and a third scaled dot product sublayer; the linear projection sublayer is used to transform the input features through linear projection to obtain the Q value matrix Q, the K value matrix K, and the V value matrix V; the second scaled dot product sublayer is used to calculate the correlation between the Q value matrix Q and the K value matrix K. The processing process of the second scaled dot product sublayer is expressed as ,in and represents the matrix obtained after the linear projection sublayer, express The softmax sublayer is used to normalize the input data; the third scaling dot product sublayer is used to extract global context information. The processing process of the third scaling dot product sublayer is expressed as ,in Represents the matrix obtained after the linear projection sublayer;

[0065] The feedforward neural network layer uses a feedforward neural network to transform the obtained global context information into the intrinsic parameter space and extrinsic parameter space of the single-view image camera, thereby realizing the parameter calibration of the single-view image camera;

[0066] S4. Using the training data set constructed in step S2, the preliminary model of the single-view image camera calibration constructed in step S3 is trained to obtain a single-view image camera calibration model; the training specifically comprises the following steps:

[0067] The following formula is used as the logarithmic space loss function : Where w is the width of the image; in is the true value of the camera intrinsic parameter of the single-view image; is the predicted value of the camera intrinsic parameter of the single-view image; is the set threshold;

[0068] When predicting the vanishing point constraints of the camera extrinsics, the following formula is used as the first similarity loss function: : Where vp is the true value of the vanishing point coordinates; is the predicted value of the vanishing point coordinates; is the norm of the vector;

[0069] When predicting the horizon loss of the camera extrinsic parameters, the following formula is used as the second similarity loss function : Where n is the number of selected endpoints; hor is the true value of the horizon; is the predicted value of the horizon; Calculate the coordinates of the left and right endpoints of the horizon; is the Manhattan distance of the vector;

[0070] Finally, the total loss function L is constructed as , is the first weight, is the second weight, is the third weight;

[0071] S5. Using the single-view image camera calibration model constructed in step S4, complete the parameter calibration of the target single-view image camera.

[0072] The solution of the present invention can predict the intrinsic and extrinsic parameters of the camera without the need for markers by perceiving geometric clues in the image, so that the present invention can be used in a wider range of scenarios with better practicality and higher accuracy.

[0073] like Figure 2 Shown is a visualization schematic diagram of the calibration method of the present invention:

[0074] Figure 2 (a) Corresponding to the feature representation of learning vanishing points, it can be found that the network focuses on the parallel lines on both sides of the road surface. The network captures these parallel line segments and their extension directions to extract geometric information closely related to the vanishing point. The network uses the convergence trend of these parallel lines to effectively learn the feature representation related to the vanishing point;

[0075] Figure 2 (b) Corresponding to the feature representation of the learned horizon, the heat map highlights the outline of the bottom of the building and the boundary between the sky and the ground; by focusing on capturing the difference between the bottom of the building and the ground objects, the network is able to use the feature representation of the geometric boundary to locate the position of the horizon in the image;

[0076] Figure 2 (c) Corresponding to the feature representation of the learned intrinsic parameters, the heat map focuses on the vicinity of the image center and presents a concentric circle form; this feature distribution represents the geometric properties related to perspective projection. By capturing the features around the center, the network provides important constraints for calculating the intrinsic parameters of the image.

[0077] pass Figure 2 It can be seen that the network of the present invention has the ability to perceive the geometric clues corresponding to different prediction targets, and adaptively focuses on the key geometric clues in the image for specific prediction targets, and can obtain more accurate calibration results on the basis of certain interpretability.

[0078] Figure 3 is a schematic diagram of the calibration results of the calibration method of the present invention, Figure 3 The comparison results between the experimental results and the true values, as well as the predicted results of the horizon and the vanishing point are shown respectively. The green line in the figure represents the predicted result of the horizon, the first row of red lines represents the true value of the horizon, and the second row of red lines represents the predicted result of the vanishing point. Figure 3 It can be seen that the calibration method of the present invention is generally close to the true value and shows high accuracy and consistency, which once again illustrates the effectiveness and superiority of the calibration method of the present invention.

[0079] The effect of the calibration method of the present invention is further described below in conjunction with the embodiments:

[0080] The method proposed in the present invention is compared with the existing calibration methods (traditional method, 2018 method, 2020 method, and 2021 method); among them, the traditional method is the method proposed by Hyunjoon Lee in "Automatic Upright Adjustment of Photographs With Robust Camera Calibration" in 2014; the 2018 method is the method proposed by Yannick Hold-Geoffroy in "A Perceptual Measure for Deep SingleImage Camera Calibration" in 2018; the 2020 method is the method proposed by Jinwoo Lee in "Neural Geometric Parser for Single Image Camera Calibration" in 2020; the 2021 method is the method proposed by Jinwoo Lee (author) in "CTRL-C: Camera calibration TRansformerwith Line-Classification" in 2021 (time);

[0081] The experiment was conducted on the public Google Street View dataset, which includes 13,214 training sets and 1,333 test sets. The average error of the internal reference calibration, the average error of the vanishing point prediction, and the AUC value of the horizon prediction were used as evaluation criteria. All experimental results were obtained on the test set.

[0082] The specific comparison data is shown in Table 1:

[0083] Table 1 Schematic diagram of comparative experimental data

[0084]

[0085] It can be seen from the experimental results in Table 1 that the calibration method proposed in the present invention has achieved the best results in various indicators, which is better than traditional methods and other deep learning methods, proving the effectiveness and superiority of the method proposed in the present invention.

[0086] like Figure 4The functional module schematic diagram of the system of the present invention is shown as follows: the system for implementing the single-view image camera calibration method disclosed in the present invention comprises a data acquisition module, a training set construction module, a model construction module, a model training module and a camera calibration module; the data acquisition module, the training set construction module, the model construction module, the model training module and the camera calibration module are connected in series in sequence; the data acquisition module is used to acquire an existing image data set and upload the data information to the training set construction module; the training set construction module is used to extract the feature vector corresponding to the image from the acquired image data set according to the received data information, thereby constructing the training data set, and uploading the data information to the model construction module; the model construction module is used to construct a single-view image camera calibration preliminary model including an encoder network and a decoder network according to the received data information, and upload the data information to the model training module; the model training module is used to train the constructed single-view image camera calibration preliminary model according to the received data information using the constructed training data set, obtain the single-view image camera calibration model, and upload the data information to the camera calibration module; the camera calibration module is used to complete the parameter calibration of the target single-view image camera according to the received data information using the constructed single-view image camera calibration model.

[0087] In addition, the single-view image camera calibration method and system of the present invention can also be used for a single-view image camera, wherein the single-view image camera includes the single-view image camera calibration method, and the single-view image camera calibration method is used to calibrate the single-view image camera. In specific applications, the single-view image camera calibration model obtained after training in the calibration method of the present invention is integrated into the single-view image camera; when in use, a single-view image camera with the integrated single-view image camera calibration model is used to take a photo at random, and the acquired photo is input into the single-view image camera calibration model. The single-view image camera calibration model can realize parameter calibration of the single-view image camera through the input photo, and the calibrated parameters are input into the single-view image camera to complete the calibration process.

Claims

1. A single-view image camera calibration method, Features The steps include: S1. Obtain an existing image dataset; S2. For the image data set obtained in step S1, extracting feature vectors corresponding to the image based on the geometric features of the image, thereby constructing a training data set; S3. Build a preliminary model for single-view image camera calibration including encoder network and decoder network; The encoder network is used to extract geometric features and contextual information in the input image and adaptively focus on key areas in the input image; The decoder network is used to map the output features of the encoder network and the features of the input image to the single-view image camera parameter space, thereby predicting the internal and external parameters of the single-view image camera; The specific implementation includes the following steps: Based on the spatial domain geometric features and the frequency domain geometric features, an encoder network is constructed; specifically, the following steps are included: The constructed encoder network includes a serially connected encoding module, a frequency domain module, an attention module, and an inference module; the input of the encoder network is the image in the training data set; The encoding module is used to perform preliminary feature extraction and representation learning on the input image; the encoding module includes a first encoder layer, a second encoder layer, a third encoder layer and a fourth encoder layer; the input of the first encoder layer and the output of the first encoder layer are added as the input of the second encoder layer; the input of the second encoder layer and the output of the second encoder layer are added as the input of the third encoder layer; the input of the third encoder layer and the output of the third encoder layer are added as the input of the fourth encoder layer; the input of the fourth encoder layer and the output of the fourth encoder layer are added as the output of the encoding module; the structures of the first encoder layer, the second encoder layer, the third encoder layer and the fourth encoder layer are the same, and all include a first convolutional layer, a second convolutional layer, a third convolutional layer, a batch normalization layer and a ReLU activation function layer; the first convolutional layer is The second convolutional layer is The third convolutional layer is The convolutional layer; The frequency domain module is used to extract the geometric feature distribution of the image at different frequencies; the frequency domain module includes a first normalization layer, a fast Fourier transform layer, a complex parameter layer, a fast inverse Fourier transform layer and a second normalization layer connected in series; the output of the encoding module is normalized by the first normalization layer and then fast Fourier transformed by the fast Fourier transform layer; the output of the fast Fourier transform layer is processed by the complex parameter layer to control the importance of different frequency components; the output of the complex parameter layer is fast inverse Fourier transformed by the fast inverse Fourier transform layer, and finally normalized by the second normalization layer to obtain the output of the frequency domain module; wherein the processing process of the complex parameter processing layer is expressed as , where w represents the weight parameter, represents the real part tensor after Fourier transformation, represents the imaginary part tensor after Fourier transformation, i represents the imaginary unit; The attention module is used to extract and enhance the geometric features related to the single-view image camera calibration in the image; the attention module includes a linear transformation layer, a first scaling dot product layer and a geometric feature perception layer connected in series; the linear transformation layer is composed of a fully connected layer and is used to perform a linear transformation on the input feature data; the first scaling dot product layer is used to capture the relative importance and correlation between features. The processing process of the first scaling dot product layer is expressed as ,in and represents the matrix obtained after the linear transformation layer, express The dimension size of s represents the softmax function; the geometric feature perception layer is used to extract the geometric information feature representation of the input feature. The geometric feature perception layer includes a first branch, a second branch and a GeLU activation function sublayer. After the input features pass through the first branch and the second branch respectively, the output of the first branch is added to the output of the second branch, and then processed by the GeLU activation function sublayer, and multiplied by the output of the scaled dot product layer to obtain the output of the geometric feature perception layer. The first branch includes the sequentially connected Convolutional layer, batch normalization layer and sigmoid activation function layer. The second branch includes Convolutional layer, batch normalization layer and sigmoid activation function layer; The inference module is used to perform nonlinear changes on the output features of the attention module to enhance the feature extraction capability of the model; the inference module includes a first fully connected layer, a ReLu activation function layer, and a second fully connected layer connected in series; Construct a decoder network based on the self-attention mechanism; S4. Using the training data set constructed in step S2, the single-view image camera calibration preliminary model constructed in step S3 is trained to obtain a single-view image camera calibration model; S5. Using the single-view image camera calibration model constructed in step S4, complete the parameter calibration of the target single-view image camera.

2. The method for calibrating a single-view image camera according to claim 1, Features The step S2 specifically includes the following steps: For the image data set obtained in step S1, the LSD algorithm is used to extract the lines L in the image, which can be expressed as ,in is the i-th image, is the number of detected line segments; Each line segment is divided into two parts according to its line equation in the image plane. Indicates that is the image coordinate, a is the normal vector component of the line in the x-axis direction, b is the normal vector component of the line in the y-axis direction, and c is the normal vector component from the line to the origin distance; Transform to get upper triangular matrix , and encoded as a 6-dimensional vector , as the feature vector corresponding to the image.

3. The method for calibrating a single-view image camera according to claim 2, Features The construction of the decoder network specifically includes the following steps: The decoder network constructed includes a convolutional layer, a first attention layer, a second attention layer, and a feedforward neural network layer connected in series. The input of the decoder network is the output of the encoder network and the feature vector corresponding to the image in the training data set. The convolutional layer is The convolutional layer; The first attention layer and the second attention layer have the same structure, both of which include a linear projection sublayer, a second scaled dot product sublayer, a softmax sublayer, and a third scaled dot product sublayer; The linear projection sublayer is used to transform the input features through linear projection to obtain the Q value matrix Q, K value matrix K and V value matrix V; The second scaled dot product sublayer is used to calculate the correlation between the Q value matrix Q and the K value matrix K. The processing process of the second scaled dot product sublayer is expressed as ,in and represents the matrix obtained after the linear projection sublayer, express The softmax sublayer is used to normalize the input data; the third scaling dot product sublayer is used to extract global context information. The processing process of the third scaling dot product sublayer is expressed as ,in Represents the matrix obtained after the linear projection sublayer; The feedforward neural network layer uses a feedforward neural network to transform the obtained global context information into the intrinsic parameter space and extrinsic parameter space of the single-view image camera, thereby realizing the parameter calibration of the single-view image camera.

4. The method for calibrating a single-view image camera according to claim 3, Features The training described in step S4 specifically includes the following steps: The following formula is used as the logarithmic space loss function : Where w is the width of the image; in is the true value of the camera intrinsic parameter of the single-view image; is the predicted value of the camera intrinsic parameter of the single-view image; is the set threshold; When predicting the vanishing point constraints of the camera extrinsics, the following formula is used as the first similarity loss function: : Where vp is the true value of the vanishing point coordinates; is the predicted value of the vanishing point coordinates; is the norm of the vector; When predicting the horizon loss of the camera extrinsic parameters, the following formula is used as the second similarity loss function : Where n is the number of selected endpoints; hor is the true value of the horizon; is the predicted value of the horizon; Calculate the coordinates of the left and right endpoints of the horizon; is the Manhattan distance of the vector; Finally, the total loss function LL is constructed as , is the first weight, is the second weight, The third weight.

5. A system for implementing the single-view image camera calibration method according to any one of claims 1 to 4, Features It includes a data acquisition module, a training set construction module, a model construction module, a model training module and a camera calibration module; the data acquisition module, the training set construction module, the model construction module, the model training module and the camera calibration module are connected in series in sequence; the data acquisition module is used to acquire the existing image data set and upload the data information to the training set construction module; the training set construction module is used to extract the feature vector corresponding to the image of the acquired image data set according to the received data information, so as to construct the training data set, and upload the data information to the model construction module; The model building module is used to build a preliminary model for single-view image camera calibration including an encoder network and a decoder network according to the received data information, and upload the data information to the model training module; The model training module is used to train the constructed single-view image camera calibration preliminary model based on the received data information and the constructed training data set to obtain the single-view image camera calibration model, and upload the data information to the camera calibration module; The camera calibration module is used to complete the parameter calibration of the target single-view image camera based on the received data information and the constructed single-view image camera calibration model.

Citation Information

Patent Citations

  • Camera calibration system and calibration method thereof

    CN107808400A

  • Unmanned aerial vehicle crop state visual identification method for air-ground cooperation

    CN117409339A

  • Image defocusing fuzzy model training method and camera parameter calibration method

    CN117671024A

  • RGB-D camera depth module calibration method based on adversarial neural network

    CN118379358A

  • Neural network-based camera calibration

    US20190164312A1