Camera intrinsic parameter calibration method, device, unmanned equipment and storage medium

Through the deep learning network model, the depth space features are extracted from the camera internal references at the previous moment and the image data at the current moment, and the camera internal references are quickly and accurately predicted, which solves the problems of low accuracy and low efficiency of camera internal reference calibration results in the prior art, and improves the The three-dimensional reconstruction accuracy and speed of unmanned equipment.

CN114494461BActive Publication Date: 2024-12-03GUANGZHOU XAIRCRAFT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210089793.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-25
Publication Date
2024-12-03
Estimated Expiration
2042-01-25

AI Technical Summary

Technical Problem

The existing camera internal parameter calibration method is low in accuracy and efficiency when real-time three-dimensional reconstruction of unmanned equipment. Especially when image data cannot be constructed, the accuracy of internal parameter calibration results cannot be guaranteed.

Method used

Using a pre-trained deep learning network model, the depth spatial characteristics of image data of past moments are parsed from the camera internal reference predicted at the previous moment, and combined with the depth spatial characteristics of image data of current moments, through the first and second networks The model performs feature fusion to quickly and accurately predict camera internal parameters.

Benefits of technology

Without waiting for multiple image data to be collected, the accuracy of camera internal parameters is ensured, calibration efficiency is improved, and the calculation speed and accuracy of unmanned equipment for three-dimensional reconstruction are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114494461B_ABST
    Figure CN114494461B_ABST
Patent Text Reader

Abstract

The present application discloses a camera intrinsic parameter calibration method, device, unmanned equipment and storage medium. The technical solution provided by the present application includes: obtaining first image data and first camera intrinsic parameters, the first image data is the image data collected at the current moment, and the first camera intrinsic parameters are the camera intrinsic parameters predicted at the previous moment; inputting the first image data into a pre-trained first network model to obtain the depth space features output by the first network model; inputting the depth space features and the first camera intrinsic parameters into a pre-trained second network model to obtain the second camera intrinsic parameters output by the second network model. Through the above technical means, the problem of low accuracy of camera intrinsic parameter calibration results in the prior art is solved, the accuracy of the intrinsic parameter calibration results is improved, and the efficiency of the intrinsic parameter calibration is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a camera intrinsic parameter calibration method, device, unmanned equipment and storage medium. Background Art

[0002] After the unmanned equipment collects image data of the surveying area through the camera, it can perform real-time 3D reconstruction of the image data based on the camera's internal parameters, external parameters, and posture parameters. However, the internal parameters calibrated by the camera at the factory may deviate due to the impact of the unmanned equipment during transportation or operation, resulting in a decrease in the accuracy of the 3D reconstruction model. Therefore, the camera's internal parameters need to be calibrated when performing real-time 3D reconstruction.

[0003] The current camera intrinsic calibration method is to reconstruct the 3D point cloud from the image data based on the discrete 3D reconstruction algorithm, and jointly estimate the camera intrinsic parameters with the camera intrinsic parameters, camera posture and 3D point cloud. However, if the image data involved in the 3D reconstruction are all collected by the camera when the unmanned equipment is sailing on the same straight channel, then these image data cannot construct effective triangular constraints and cannot guarantee the accuracy of the intrinsic calibration results. In this case, the unmanned equipment needs to collect image data from multiple channels to accurately calibrate the camera intrinsic parameters, and the camera calibration efficiency is low. Summary of the invention

[0004] The present application provides a camera intrinsic parameter calibration method, device, unmanned equipment and storage medium, which solves the problem of low accuracy of camera intrinsic parameter calibration results in the prior art, improves the accuracy of the intrinsic parameter calibration results, and improves the efficiency of the intrinsic parameter calibration.

[0005] In a first aspect, the present application provides a camera intrinsic parameter calibration method, comprising:

[0006] Acquire first image data and first camera intrinsic parameters, where the first image data is image data collected at a current moment, and the first camera intrinsic parameters is camera intrinsic parameters predicted at a previous moment;

[0007] Inputting the first image data into a pre-trained first network model to obtain a deep spatial feature output by the first network model;

[0008] The depth space feature and the first camera intrinsic parameter are input into a pre-trained second network model to obtain a second camera intrinsic parameter output by the second network model.

[0009] In a second aspect, the present application provides a camera intrinsic parameter calibration device, comprising:

[0010] A data acquisition module is configured to acquire first image data and a first camera intrinsic parameter, wherein the first image data is image data collected at a current moment, and the first camera intrinsic parameter is a camera intrinsic parameter predicted at a previous moment;

[0011] A feature extraction module is configured to input the first image data into a pre-trained first network model to obtain a deep spatial feature output by the first network model;

[0012] The internal parameter prediction module is configured to input the depth space feature and the first camera internal parameter into a pre-trained second network model to obtain a second camera internal parameter output by the second network model.

[0013] In a third aspect, the present application provides an unmanned device, including:

[0014] One or more processors; a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the camera intrinsic parameter calibration method as described in the first aspect.

[0015] In a fourth aspect, the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the camera intrinsic parameter calibration method as described in the first aspect.

[0016] This application uses a pre-trained deep learning network model to parse the depth spatial features of image data collected at past moments from the camera intrinsic parameters predicted at the previous moment, and extract the depth spatial features of image data collected by the camera at the current moment. The camera intrinsic parameters are accurately and quickly predicted through the depth spatial features of image data at past moments and the depth spatial features of image data at the current moment, and the accuracy of the camera intrinsic parameters can be guaranteed even when the image data collected by the camera cannot construct effective triangular constraints. The camera intrinsic parameters at the current moment can be predicted through the image data collected by the camera at the current moment, without waiting for the camera to collect multiple image data and then calibrating the camera intrinsic parameters. While ensuring the accuracy of the calibration results, the calibration efficiency is also improved, and the calculation speed and accuracy of three-dimensional reconstruction of unmanned equipment are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a camera intrinsic parameter calibration method provided in an embodiment of the present application;

[0018] Figure 2 is a flowchart of training a deep learning network provided in an embodiment of the present application;

[0019] Figure 3 is a flow chart of extracting deep spatial features through a first network model provided by an embodiment of the present application;

[0020] Figure 4 is a flow chart of obtaining deep spatial features through a first self-attention model provided in an embodiment of the present application;

[0021] Figure 5 is a flowchart of predicting camera intrinsic parameters through a second network model provided by an embodiment of the present application;

[0022] Figure 6 is a flow chart of determining the second camera internal parameter through a decoder provided in an embodiment of the present application;

[0023] Figure 7 is a flow chart of determining the second camera intrinsic parameter by the third self-attention model provided in an embodiment of the present application;

[0024] Figure 8 It is a structural diagram of the first network model and the second network model provided in the embodiment of the present application;

[0025] Fig. 9 is a structural schematic diagram of a camera intrinsic parameter calibration device provided in an embodiment of the present application;

[0026] Fig.10 It is a structural schematic diagram of an unmanned device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The camera intrinsic calibration method provided in the embodiment of the present application can be executed by a camera intrinsic calibration device, which can be implemented by software and / or hardware, and can be composed of two or more physical entities, or one physical entity. For example, the camera intrinsic calibration device can be the unmanned device itself, or it can be the processor of the unmanned device.

[0030] The camera intrinsic calibration device is installed with at least one type of operating system, wherein the operating system includes but is not limited to Android system, Linux system and Windows system. The camera intrinsic calibration device can install at least one application based on the operating system, and the application can be an application that comes with the operating system, or an application downloaded from a third-party device or server. In the embodiment, the camera intrinsic calibration device is at least based on an application that can execute the camera intrinsic calibration method, so the camera intrinsic calibration device can also be the application itself.

[0031] For ease of understanding, this embodiment is described by taking an unmanned device as the subject of executing the camera intrinsic parameter calibration method as an example.

[0032] In one embodiment, when the unmanned equipment performs a surveying task, real-time three-dimensional reconstruction can be performed based on the real-time collected image data to construct a three-dimensional model of the surveying area. When the unmanned equipment performs three-dimensional reconstruction, the intrinsic parameters of the current camera need to be applied, and the accuracy of the camera intrinsic parameters will affect the accuracy of the three-dimensional reconstruction. Therefore, before the unmanned equipment performs a three-dimensional reconstruction task, the camera intrinsic parameters can be calibrated to prevent the intrinsic parameters of the current camera from having errors with the camera intrinsic parameters calibrated at the factory or the camera intrinsic parameters calibrated at the last time, thereby reducing the accuracy of the three-dimensional model. The existing camera intrinsic parameter calibration method aims to reconstruct a three-dimensional point cloud from image data based on a discrete three-dimensional reconstruction algorithm, and jointly estimate the camera intrinsic parameters based on the factory-calibrated camera intrinsic parameters, camera posture, and three-dimensional point cloud. However, if the existing camera intrinsic parameter calibration method is used in a real-time three-dimensional reconstruction algorithm, the unmanned equipment needs to collect multiple image data to construct effective triangular constraints, thereby ensuring the accuracy of the intrinsic parameter calibration results. Therefore, the calibration efficiency of the existing camera intrinsic parameter calibration method is low. Moreover, if the image data involved in 3D reconstruction are all collected by the camera when the unmanned equipment is sailing on the same straight course, then these image data cannot construct effective triangular constraints and cannot guarantee the accuracy of the internal parameter calibration results. This requires the unmanned equipment to spend more time collecting more image data to calibrate the camera internal parameters, resulting in poor real-time 3D reconstruction.

[0033] To solve the above problems, the present application provides a camera intrinsic parameter calibration method, which improves the calibration efficiency while ensuring the accuracy of the calibration results.

[0034] Figure 1A flow chart of a camera intrinsic parameter calibration method provided in an embodiment of the present application is given. Figure 1 , the camera intrinsic parameter calibration method specifically includes:

[0035] S110, obtaining first image data and first camera intrinsic parameters, where the first image data is image data collected at a current moment, and the first camera intrinsic parameters is camera intrinsic parameters predicted at a previous moment.

[0036] Exemplarily, after the unmanned device starts to perform the surveying and mapping task, the unmanned device navigates to the first shooting point and controls the camera to collect the first image data, and based on the first image data and the camera intrinsic parameters predicted last time, the camera intrinsic parameters are predicted for the first time. Since the image data currently collected by the camera is the first image data, the unmanned device did not predict the camera intrinsic parameters at the last moment, so the camera intrinsic parameters calibrated in the last surveying and mapping task or the camera intrinsic parameters calibrated at the factory are used to predict the current camera intrinsic parameters. Further, when the unmanned device navigates to the second shooting point, the camera is controlled to collect the second image data, and the camera intrinsic parameters are predicted for the second time based on the second image data and the camera intrinsic parameters predicted for the first time. As the camera collects more image data, the predicted camera intrinsic parameters are more accurate. When the number of predictions reaches a certain number or when the predicted camera intrinsic parameters no longer change, the prediction of the camera intrinsic parameters can be stopped, and in the subsequent real-time three-dimensional reconstruction process, the camera intrinsic parameters predicted last time are used to build a three-dimensional model.

[0037] Since the image data contains rich spatial information, the offline 3D reconstruction method also determines the camera intrinsic parameters based on the spatial information of multiple image data. In this regard, the present embodiment proposes to extract the deep spatial features in the image data and predict the camera intrinsic parameters through a deep learning network with a coding and decoding structure. Among them, the deep learning network includes a first network model and a second network model, the first network model and the second network model are set in series, and the output of the first network model is used as the input of the second network model. In the encoding stage, the deep spatial features in the image data collected at the current moment are extracted by the first network model; in the decoding stage, the camera intrinsic parameters predicted at the previous moment are decoded by the second network model to parse out the deep spatial features of the image data collected at the past moment, and the deep spatial features of the image data collected at the past moment are feature fused and analyzed with the deep spatial features of the image data collected at the current moment to accurately predict the camera intrinsic parameters.

[0038] In one embodiment, Figure 2 It is a flowchart of training a deep learning network provided in an embodiment of the present application.

[0039] like Figure 2 As shown, the steps of training the deep learning network specifically include S1101-S1103:

[0040] S1101. Extracting deep spatial features of each sample image in training data through a first network model, where the training data includes multiple sample images and internal reference labels of the same flight.

[0041] The image data of the same flight refers to the image data collected by the same unmanned equipment when performing the same surveying and mapping task. Exemplarily, the image data collected by multiple unmanned equipment when performing different surveying and mapping tasks are obtained, and the image data is divided into multiple sample images of different flights. The sample images of the same flight are processed by an offline 3D reconstruction method to obtain the camera intrinsic parameters corresponding to the flight.

[0042] In one embodiment, assuming that the projection point of the spatial point P = (X, Y, Z) in the sample image is p, and the original coordinates of the projection point p in the sample image are (u, v), the original coordinates of the projection point p are dedistorted by the camera distortion parameters in the camera intrinsic parameters, and the dedistorted coordinates of the projection point are (u′, v′). The calculation formula of the dedistorted coordinates is as follows:

[0043]

[0044] Among them, k1, k2, k3, p1, p2 are camera distortion parameters,

[0045] Furthermore, the camera intrinsic parameter matrix The homogeneous transformation matrix from the world coordinate system to the camera coordinate system is The 3D reconstruction constraint function F is:

[0046]

[0047] Among them, the variable m of the constraint function F is (R, t, P, f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2), p=(u′,v′). By constructing a 3D reconstruction constraint function F for all sample images of a flight and performing joint optimization, the camera intrinsic parameters of the flight can be determined. After determining the camera intrinsic parameters of the flight, the camera intrinsic parameters are used as the intrinsic parameter labels of the sample images of the flight. In this embodiment, since the structure of the photosensitive element in the camera is relatively good, f x =f y , f x and f x The camera internal parameters include 8 parameters, namely, the camera internal parameters focal length F = (f, c x ,c y ), pixel parameter ρ = (k1, k2, k3, p1, p2).

[0048] Furthermore, the sample images of multiple flights and the corresponding internal reference labels are packaged to obtain multiple sets of training data. For example, assume that a set of training data includes n sample images, and the n sample images are respectively from the time t1 to the time t n Image data is collected at all times. n sample images are sequentially input into the first network model to obtain the depth spatial features corresponding to each sample image output by the first network model.

[0049] S1102: Input the depth spatial features of the sample image at the first moment and the camera intrinsic parameters at the second moment into the second network model to obtain the camera intrinsic parameters at the first moment output by the second network model, where the first moment is the next moment of the second moment.

[0050] Exemplarily, the depth spatial features of the sample image at time t1 and the camera intrinsic parameters calibrated at the factory are input into the second network model, and the camera intrinsic focal length F predicted at time t1 by the second network model is obtained. 1 and pixel parameter ρ 1 The depth spatial features of the sample image at time t2 and the camera internal reference focal length F at time t1 are predicted 1 and pixel parameter ρ 1 Input the second network model and obtain the camera intrinsic focal length F predicted at time t1 output by the second network model 2 and pixel parameter ρ 2 . And so on, we get n The camera internal parameter F predicted at each moment 1 …F n and ρ 1 …ρ n .

[0051] In this embodiment, when the t i The sample image at time t and the i-1 The camera’s intrinsic focal length F predicted at any moment i-1 and pixel parameter ρ i-1 When the second network model is input, the second network model is responsible for i The deep spatial features of all sample images at time t1 are fused in time series, and the deep spatial features of all sample images at time t2 are fused in time series, and the deep spatial features of all sample images at time t3 are fused in time series, and the ... i Analyze the fusion features at time t and get i The camera internal parameters predicted at the moment. i When the moment is larger, the second network model integrates more deep spatial features of the sample image, so the predicted camera intrinsic parameters are more accurate.

[0052] S1103. Train the first network model and the second network model according to a preset difference constraint and a difference between the camera's internal reference labels at each moment in the same flight; the difference constraint is that the difference between the camera's internal reference labels at the first moment is less than the difference between the camera's internal reference labels at the corresponding second moment.

[0053] For example, due to the t i The larger the moment is, the more accurate the camera internal parameters predicted by the second network model are. i The deviation of the camera internal reference label predicted at time t is less than i-1 The deviation of the camera intrinsic parameter predicted at the time is the internal parameter label. If the camera intrinsic parameter output by the second network model does not meet the difference constraint condition between the previous and next moments, it indicates that the first network model and the second network model do not have the ability to predict the camera intrinsic parameter. The model parameters of the first network model and the second network model can be optimized according to the internal parameter result output by the second network model and the corresponding internal parameter label. In this embodiment, the camera intrinsic parameter from time t1 to time t i The camera’s intrinsic focal length F predicted at any moment 1 …F n Respectively with the camera intrinsic focal length F in the intrinsic label l Subtract the time from t1 to t i The pixel parameter ρ predicted at the moment 1 …ρ n Respectively with the pixel parameters ρ in the internal reference label l Make a difference and construct the difference constraint condition as 0≤||F l -F 1 ||≤||F l -F 2 ||≤…≤||F l -F n || and 0≤||ρ l -ρ 1 ||≤||ρ l -ρ 2 ||≤…≤||ρ l -ρ n ||.

[0054] S120. Input the first image data into a pre-trained first network model to obtain the deep spatial features output by the first network model.

[0055] In one embodiment, the first network model includes a convolutional neural network and a self-attention network, and the deep features of different spaces in the first image data are extracted by the convolutional neural network, and the deep features of different spaces are complexly fused by the self-attention network to obtain the deep spatial features in the first image data. In this embodiment, Figure 3is a flow chart of extracting deep spatial features through a first network model provided by an embodiment of the present application. Figure 3 As shown, the step of extracting deep spatial features through the first network model specifically includes S1201-S1203:

[0056] S1201. Input the first image data into the residual network model to obtain a depth feature map output by the residual network model.

[0057] Exemplarily, the first image data is convolved by a residual network model to obtain a depth feature map output by the residual network. The residual network is a type of convolutional neural network. When the depth features of the image data are extracted by the residual network, the original features of the image data can be retained.

[0058] Furthermore, since the dimensions of the depth feature map and the camera intrinsic feature are different, the depth feature map can be projected into a space of the same dimension as the camera intrinsic feature through a one-dimensional convolutional layer.

[0059] S1202. Convert the depth feature map into a feature map sequence according to the position encoding, where the feature map sequence includes multiple feature maps.

[0060] Exemplarily, the position information of each pixel in the depth feature map also belongs to the spatial feature information. In order to ensure the position information of each pixel in the depth feature map, the depth feature map is processed by position coding to split the depth feature map into multiple small-sized feature maps, and the multiple small-sized feature maps are sorted according to the position information to obtain a feature map sequence. Since some pixels in the continuous image data collected by the camera overlap when the unmanned equipment performs surveying and mapping tasks, the feature map sequence obtained by splitting the depth feature map based on position coding can be regarded as a feature map generated based on the continuous image data collected by the camera. This embodiment converts the spatial features in an image data into the time series features of multiple image data through position coding, so as to predict the camera intrinsic parameters according to the time series features.

[0061] S1203: Input the feature map sequence into the encoder to obtain the deep spatial features output by the encoder.

[0062] In one embodiment, the encoder includes a first self-attention model, and the deep spatial features are extracted from the feature map sequence by the first self-attention model. Exemplarily, the self-attention model is good at capturing the internal correlation of data or features, so the correlation between each temporal feature map in the feature map sequence can be captured by the self-attention model to obtain more complex high-order semantic features.

[0063] In this embodiment, Figure 4 is a flow chart of obtaining deep spatial features through the first self-attention model provided by an embodiment of the present application. Figure 4As shown, the step of obtaining the deep spatial features through the encoder specifically includes S12031-S12032:

[0064] S12031. Input the feature map sequence into the first multi-head self-attention model, add the output of the first multi-head self-attention model to the input and perform normalization to obtain first feature data. The first multi-head self-attention model includes multiple first self-attention models set in parallel.

[0065] S12032. Input the first feature data into the first fully connected layer, add the output of the first fully connected layer to the input and perform normalization to obtain a deep spatial feature.

[0066] Exemplarily, the multi-head self-attention network has a stronger global time series feature analysis capability than the convolutional neural network. Therefore, this embodiment uses a multi-head self-attention model to extract features from the feature map sequence to obtain high-order semantic features. This embodiment is described by taking the first multi-head self-attention model as an example in which six first self-attention models are set in parallel. The feature map sequence is input into the six first self-attention models together, and the six first self-attention models fuse the feature maps of different time series respectively to obtain the features output by each self-attention model. The features output by all the first self-attention models are spliced, and the spliced ​​features are linearly transformed to obtain the features output by the first multi-head self-attention model.

[0067] Furthermore, the feature map sequence is added to the features output by the first multi-head self-attention model and normalized to prevent the first multi-head self-attention model from degrading due to increased depth. However, since the dimensions of the feature map sequence and the features output by the first multi-head self-attention model are different, before the addition and normalization, the feature map sequence needs to be projected to a space with the same dimension as the features output by the first multi-head self-attention model through a one-dimensional convolution layer. Furthermore, the first feature data is input to the first fully connected layer, and the first fully connected layer performs feature analysis on the first feature data. The feature vector output by the first fully connected layer is added to the first feature data and normalized to obtain a deep spatial feature rich in high-order semantic information.

[0068] S130: Input the depth space feature and the first camera intrinsic parameter into a pre-trained second network model to obtain a second camera intrinsic parameter output by the second network model.

[0069] In one embodiment, the second network model includes a self-attention network, which parses the camera intrinsic parameters predicted at the previous moment through the self-attention network to parse the depth space features of the image data at the past moment, and performs feature fusion and analysis based on the depth space features of the image data at the past moment and the depth space features of the image data at the current moment to predict a more accurate camera intrinsic parameter. The second camera intrinsic parameter is the camera intrinsic parameter predicted at the current moment. In this embodiment, Figure 5 : is a flowchart of predicting camera intrinsic parameters through a second network model provided by an embodiment of the present application. Figure 5 As shown, the step of predicting the camera intrinsic parameter by using the second network model specifically includes S1301-S1302:

[0070] S1301. Input a first camera intrinsic parameter into a one-dimensional convolutional layer to obtain second feature data output by the one-dimensional convolutional layer.

[0071] Exemplarily, since the deep spatial features of the first camera participating in the output of the first network model are not in the same dimensional space, the first camera internal parameters are projected into the space of the same dimensionality as the deep spatial features through a one-dimensional convolutional layer.

[0072] S1302: Input the second feature data and the depth space feature into a decoder to obtain a second camera intrinsic parameter output by the decoder.

[0073] In one embodiment, the decoder includes a second self-attention model and a third self-attention model, and the first camera intrinsic parameter is decoded by the second self-attention model to obtain the depth spatial features of the image data collected at the past moment. For example, if the current moment is moment t, then the image data collected at the past moment includes the image data from moment 1 to moment t-1. It should be noted that the depth spatial features of the image data at the past moment are not based on the high-order semantic features extracted from the image data collected at the past moment, but are extracted based on the camera intrinsic parameters predicted last time. Since the currently predicted camera intrinsic parameters are predicted based on the depth spatial features of the camera intrinsic parameters predicted last time and the image data at the current moment, the camera intrinsic parameters predicted last time include the depth spatial features of the image data at the past moment and the previous moment, so the depth spatial features obtained by parsing the camera intrinsic parameters predicted last time can be regarded as the depth spatial features of the image data collected at the past moment.

[0074] Furthermore, the third self-attention model is used to perform a fusion analysis on the depth spatial features of the current image data and the depth spatial features of the past image data, and the camera intrinsic parameters related to the past and current image data are predicted. Figure 6 1 is a flow chart of determining the second camera internal parameter by a decoder according to an embodiment of the present application. Figure 6As shown, the step of determining the second camera internal parameter by the decoder specifically includes S13021-S13022:

[0075] S13021. Extracting camera intrinsic features from the second feature data through a second self-attention model.

[0076] Among them, the camera intrinsic feature can be regarded as a high-order semantic feature of the image data at the past moment. In this embodiment, the second feature data is input into the second multi-head self-attention model, and the output and input of the second multi-head self-attention model are added and normalized to obtain the camera intrinsic feature; the second multi-head self-attention model includes multiple second self-attention models set in parallel. This embodiment is described by taking the second multi-head self-attention model as an example to set six second self-attention models in parallel. The second feature data is input into the six second self-attention models together, and the six second self-attention models respectively analyze the high-order semantic features of the image data at different moments in the past to obtain the features output by each second self-attention model. The features output by all the second self-attention models are spliced, and the spliced ​​features are linearly transformed to obtain the features output by the second multi-head self-attention model. Further, the second feature data is added to the features output by the second self-attention model and normalized to obtain the camera intrinsic feature containing the deep spatial features of the image data at the past moment.

[0077] S13022. Fuse the camera intrinsic parameter features and the depth space features through a third self-attention model, and determine the second camera intrinsic parameter based on the fused features.

[0078] Exemplarily, the camera intrinsic parameter features and deep spatial features are fused through the third self-attention model, that is, the high-order semantic features of the image data at the past moment and the current moment are fused in feature time series, and the fused features of the past moment and the current moment are analyzed to obtain the second camera intrinsic parameter.

[0079] In one embodiment, Figure 7 is a flow chart of determining the second camera internal parameter by the third self-attention model provided in an embodiment of the present application. Figure 7 As shown, the step of determining the second camera internal parameter by the third self-attention model specifically includes S130221-S130222:

[0080] S130221. Input the camera intrinsic parameter features and the deep space features into the third multi-head self-attention model, add the output of the third multi-head attention model to the camera intrinsic parameter features and perform normalization to obtain third feature data; the third multi-head self-attention model includes multiple third self-attention models set in parallel.

[0081] S130222. Input the third feature data into the second fully connected layer, add the output of the second fully connected layer to the input and perform normalization processing to obtain the second camera intrinsic parameter.

[0082] This embodiment is described by taking the third multi-head self-attention model as an example in which six third self-attention models are set in parallel. The third feature data and the deep spatial features are input into the six third self-attention models together, and the six third self-attention models fuse the high-order semantic features of the image data at different moments in the past and the current moment to obtain the features output by each third self-attention model. The features output by all the third self-attention models are spliced, and the spliced ​​features are linearly transformed to obtain the features output by the third multi-head self-attention model. The features output by the third multi-head self-attention model are added to the camera intrinsic reference features and normalized to obtain the third feature data. Among them, the third feature data can be regarded as a temporal fusion feature obtained by fusing the high-order semantic features of the image data at the past moment and the current moment.

[0083] Furthermore, the third feature data is input into the second fully connected layer, the second fully connected layer performs feature analysis on the fourth feature data, and the feature output by the second fully connected layer is added to the fourth feature data and normalized to obtain a second added internal reference.

[0084] In one embodiment, Figure 8 Schematic diagram of the structure of the first network model and the second network model provided in the embodiment of the present application. Figure 8 As shown, the first camera intrinsic parameter includes the first camera intrinsic parameter focal length and the first pixel parameter, and the second camera intrinsic parameter includes the second camera intrinsic parameter focal length and the second pixel parameter. The first image data is input into the first network model to obtain the depth space feature output by the first network model. The depth space feature and the first camera intrinsic parameter are input into the second network model to obtain the second camera intrinsic parameter output by the second network model.

[0085] In summary, the camera intrinsic parameter calibration method provided in the embodiment of the present application parses the depth space features of the image data collected at the past moment from the camera intrinsic parameters predicted at the previous moment through a pre-trained deep learning network model, and extracts the depth space features of the image data collected by the camera at the current moment. The camera intrinsic parameters are accurately and quickly predicted through the depth space features of the image data at the past moment and the depth space features of the image data at the current moment, and the accuracy of the camera intrinsic parameters can be guaranteed even when the image data collected by the camera cannot construct effective triangular constraints. The camera intrinsic parameters at the current moment can be predicted through the image data collected by the camera at the current moment, without waiting for the camera to collect multiple image data and then calibrating the camera intrinsic parameters. While ensuring the accuracy of the calibration results, the calibration efficiency is also improved, and the calculation speed and accuracy of the three-dimensional reconstruction of unmanned equipment are improved.

[0086] Based on the above embodiments, Fig. 9 A schematic diagram of the structure of a camera intrinsic parameter calibration device provided in an embodiment of the present application. Fig. 9 The camera intrinsic parameter calibration device provided in this embodiment specifically includes: a data acquisition module 21, a feature extraction module 22 and an intrinsic parameter prediction module 23.

[0087] The data acquisition module is configured to acquire first image data and first camera intrinsic parameters, wherein the first image data is image data collected at the current moment, and the first camera intrinsic parameters are camera intrinsic parameters predicted at the previous moment;

[0088] A feature extraction module is configured to input the first image data into a pre-trained first network model to obtain a deep spatial feature output by the first network model;

[0089] The internal parameter prediction module is configured to input the depth space feature and the first camera internal parameter into a pre-trained second network model to obtain the second camera internal parameter output by the second network model.

[0090] Based on the above embodiment, the feature extraction module includes: a residual network sub-module, configured to input the first image data into the residual network model to obtain a depth feature map output by the residual network model; a position encoding sub-module, configured to convert the depth feature map into a feature map sequence according to the position encoding, and the feature map sequence includes multiple feature maps; an encoding sub-module, configured to input the feature map sequence into the encoder to obtain the deep spatial features output by the encoder.

[0091] Based on the above embodiment, the encoder includes a first self-attention model; accordingly, the encoding submodule includes: a first self-attention unit, configured to extract deep spatial features from the feature map sequence through the first self-attention model.

[0092] Based on the above embodiments, the first self-attention unit includes: a first multi-head self-attention sub-unit, configured to input the feature map sequence into the first multi-head self-attention model, and add the output of the first multi-head self-attention model to the input and perform normalization to obtain first feature data, the first multi-head self-attention model includes multiple first self-attention models set in parallel; a first fully connected layer sub-unit, configured to input the first feature data into the first fully connected layer, and add the output of the first fully connected layer to the input and perform normalization to obtain deep spatial features.

[0093] Based on the above embodiment, the intrinsic parameter prediction module includes: a one-dimensional convolution sub-module, configured to input the first camera intrinsic parameter into the one-dimensional convolution layer to obtain the second feature data output by the one-dimensional convolution layer; a decoding sub-module, configured to input the second feature data and the deep space feature into the decoder to obtain the second camera intrinsic parameter output by the decoder.

[0094] Based on the above embodiment, the decoding submodule includes: a decoding unit, configured to extract camera intrinsic parameter features from the second feature data through a second self-attention model; a feature fusion unit, configured to fuse the camera intrinsic parameter features and the deep space features through a third self-attention model, and determine the second camera intrinsic parameter based on the fused features.

[0095] Based on the above embodiment, the decoding unit includes: a second multi-head self-attention sub-unit, configured to input the second feature data into a second multi-head self-attention model, and add the output and input of the second multi-head self-attention model and normalize them to obtain the camera intrinsic parameter feature; the second multi-head self-attention model includes multiple second self-attention models set in parallel.

[0096] Based on the above embodiments, the feature fusion unit includes: a third multi-head self-attention sub-unit, configured to input the camera intrinsic parameter feature and the deep space feature into a third multi-head self-attention model, add the output of the third multi-head attention model to the camera intrinsic parameter feature and perform normalization to obtain third feature data; the third multi-head self-attention model includes multiple third self-attention models set in parallel; a second fully connected layer sub-unit, configured to input the third feature data into the second fully connected layer, add the output of the second fully connected layer to the input and perform normalization to obtain a second camera intrinsic parameter.

[0097] On the basis of the above embodiment, the camera intrinsic parameter calibration device also includes: a first training module, configured to extract the depth spatial features of each sample image in the training data through the first network model, and the training data includes multiple sample images and intrinsic reference labels of the same flight; a second training module, configured to input the depth spatial features of the sample image at the first moment and the camera intrinsic reference at the second moment into the second network model, and obtain the camera intrinsic reference at the first moment output by the second network model, and the first moment is the next moment of the second moment; a third training module, configured to train the first network model and the second network model according to a preset difference constraint condition and the difference of the camera intrinsic reference labels at each moment in the same flight; the difference constraint condition is that the difference of the camera intrinsic reference labels at the first moment is less than the difference of the camera intrinsic reference labels at the corresponding second moment.

[0098] As mentioned above, the camera intrinsic parameter calibration device provided in the embodiment of the present application parses the depth space features of the image data collected at the past moment from the camera intrinsic parameters predicted at the previous moment through a pre-trained deep learning network model, and extracts the depth space features of the image data collected by the camera at the current moment. The camera intrinsic parameters are accurately and quickly predicted through the depth space features of the image data at the past moment and the depth space features of the image data at the current moment, and the accuracy of the camera intrinsic parameters can be guaranteed even when the image data collected by the camera cannot construct effective triangular constraints. The camera intrinsic parameters at the current moment can be predicted through the image data collected by the camera at the current moment, without waiting for the camera to collect multiple image data and then calibrating the camera intrinsic parameters. While ensuring the accuracy of the calibration results, the calibration efficiency is also improved, and the calculation speed and accuracy of the three-dimensional reconstruction of unmanned equipment are improved.

[0099] The camera intrinsic parameter calibration device provided in the embodiment of the present application can be used to execute the camera intrinsic parameter calibration method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0100] Fig.10 This is a schematic diagram of the structure of an unmanned device provided in an embodiment of the present application, with reference to Fig.10 The unmanned device includes: a processor 31, a memory 32, a communication device 33, an input device 34 and an output device 35. The number of processors 31 in the unmanned device can be one or more, and the number of memories 32 in the unmanned device can be one or more. The processor 31, memory 32, communication device 33, input device 34 and output device 35 of the unmanned device can be connected through a bus or other methods.

[0101] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the camera intrinsic calibration method of any embodiment of the present application (for example, the data acquisition module 21, the feature extraction module 22 and the intrinsic parameter prediction module 23 in the camera intrinsic calibration device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0102] The communication device 33 is used for data transmission.

[0103] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned camera intrinsic parameter calibration method.

[0104] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0105] The unmanned equipment provided above can be used to execute the camera intrinsic parameter calibration method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0106] An embodiment of the present application also provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute a camera intrinsic parameter calibration method. The camera intrinsic parameter calibration method includes: obtaining first image data and first camera intrinsic parameters, the first image data is the image data collected at the current moment, and the first camera intrinsic parameters are the camera intrinsic parameters predicted at the previous moment; inputting the first image data into a pre-trained first network model to obtain depth space features output by the first network model; inputting the depth space features and the first camera intrinsic parameters into a pre-trained second network model to obtain second camera intrinsic parameters output by the second network model.

[0107] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0108] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present application is not limited to the above camera intrinsic parameter calibration method, and the computer executable instructions can also execute related operations in the camera intrinsic parameter calibration method provided in any embodiment of the present application.

[0109] The camera intrinsic parameter calibration device, storage medium and unmanned equipment provided in the above embodiments can execute the camera intrinsic parameter calibration method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the camera intrinsic parameter calibration method provided in any embodiment of the present application.

[0110] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the protection scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A camera intrinsic parameter calibration method, characterized in that: include: Acquire first image data and first camera intrinsic parameters, where the first image data is image data collected at a current moment, and the first camera intrinsic parameters is camera intrinsic parameters predicted at a previous moment; Inputting the first image data into a pre-trained first network model to obtain a deep spatial feature output by the first network model; The method comprises: inputting the first image data into a residual network model to obtain a depth feature map output by the residual network model; converting the depth feature map into a feature map sequence according to position coding, wherein the feature map sequence includes a plurality of feature maps; inputting the feature map sequence into an encoder to obtain a depth spatial feature output by the encoder; The depth space feature and the first camera intrinsic parameter are input into a pre-trained second network model to obtain a second camera intrinsic parameter output by the second network model.

2. The camera intrinsic parameter calibration method according to claim 1, characterized in that: The encoder includes a first self-attention model; accordingly, inputting the feature map sequence into the encoder to obtain the deep spatial features output by the encoder includes: The deep spatial features are extracted from the feature map sequence through the first self-attention model.

3. The camera intrinsic parameter calibration method according to claim 2, characterized in that: The extracting the deep spatial feature from the feature map sequence by using the first self-attention model comprises: Inputting the feature graph sequence into a first multi-head self-attention model, adding the output of the first multi-head self-attention model to the input and performing normalization processing to obtain first feature data, wherein the first multi-head self-attention model includes a plurality of first self-attention models arranged in parallel; The first feature data is input into a first fully connected layer, and the output of the first fully connected layer is added to the input and normalized to obtain the deep spatial feature.

4. The camera intrinsic parameter calibration method according to any one of claims 1 to 3, characterized in that: The step of inputting the depth space feature and the first camera intrinsic parameter into a pre-trained second network model to obtain a second camera intrinsic parameter output by the second network model includes: Inputting the first camera intrinsic parameter into a one-dimensional convolution layer to obtain second feature data output by the one-dimensional convolution layer; The second feature data and the depth space feature are input into a decoder to obtain a second camera intrinsic parameter output by the decoder.

5. The camera intrinsic parameter calibration method according to claim 4, characterized in that: The decoder includes a second self-attention model and a third self-attention model; accordingly, the second feature data and the deep space feature are input into the decoder to obtain a second camera intrinsic parameter output by the decoder, including: Extracting camera intrinsic reference features from the second feature data by using the second self-attention model; The camera intrinsic parameter features and the depth space features are fused through the third self-attention model, and the second camera intrinsic parameter is determined based on the fused features.

6. The camera intrinsic parameter calibration method according to claim 5, characterized in that: The extracting the camera intrinsic feature from the second feature data by using the second self-attention model includes: The second feature data is input into a second multi-head self-attention model, and the output and input of the second multi-head self-attention model are added and normalized to obtain the camera intrinsic feature; the second multi-head self-attention model includes multiple second self-attention models set in parallel.

7. The camera intrinsic parameter calibration method according to claim 6, characterized in that: The fusing the camera intrinsic parameter feature and the depth space feature through the third self-attention model, and determining the second camera intrinsic parameter based on the fused feature, includes: Inputting the camera intrinsic parameter feature and the deep space feature into a third multi-head self-attention model, adding the output of the third multi-head attention model to the camera intrinsic parameter feature and performing normalization processing to obtain third feature data; the third multi-head self-attention model includes multiple third self-attention models set in parallel; The third feature data is input into a second fully connected layer, and the output of the second fully connected layer is added to the input and normalized to obtain the second camera intrinsic parameter.

8. The camera intrinsic parameter calibration method according to claim 1, characterized in that: Before acquiring the first image data and the first camera intrinsic parameter, the method further includes: Extracting deep spatial features of each sample image in training data through the first network model, wherein the training data includes multiple sample images and internal reference labels of the same flight; Inputting the depth spatial features of the sample image at the first moment and the camera intrinsic parameters at the second moment into the second network model, obtaining the camera intrinsic parameters at the first moment output by the second network model, wherein the first moment is the next moment of the second moment; The first network model and the second network model are trained according to a preset difference constraint and the difference of the intrinsic reference label participated in by the camera at each moment in the same flight; the difference constraint is that the difference of the intrinsic reference label participated in by the camera at the first moment is less than the difference of the intrinsic reference label participated in by the camera at the corresponding second moment.

9. A camera intrinsic parameter calibration device, characterized in that: include: A data acquisition module is configured to acquire first image data and a first camera intrinsic parameter, wherein the first image data is image data collected at a current moment, and the first camera intrinsic parameter is a camera intrinsic parameter predicted at a previous moment; A feature extraction module is configured to input the first image data into a pre-trained first network model to obtain a deep spatial feature output by the first network model; The feature extraction module includes: a residual network submodule, configured to input the first image data into a residual network model to obtain a depth feature map output by the residual network model; a position encoding submodule, configured to convert the depth feature map into a feature map sequence according to the position encoding, wherein the feature map sequence includes a plurality of feature maps; an encoding submodule, configured to input the feature map sequence into an encoder to obtain a depth spatial feature output by the encoder; The internal parameter prediction module is configured to input the depth space feature and the first camera internal parameter into a pre-trained second network model to obtain a second camera internal parameter output by the second network model.

10. An unmanned device, characterized in that: include: one or more processors; A storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the camera intrinsic parameter calibration method as described in any one of claims 1-8.

11. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the camera intrinsic parameter calibration method as described in any one of claims 1-8 when executed by a computer processor.