Image de-warping method for spatial pose measurement

Abstract

The application discloses an image de-distortion method for spatial pose measurement. The method generates a non-distorted true value data set based on a digital twin principle, and then adds random distortion to the true value image by using a Radtan distortion model or a fisheye distortion model, so that the distortion conditions of ordinary perspective lenses or fisheye lenses can be simulated, and a large amount of data set for network training can be realized. The data set can be directly used as the input of a U-Net network, and the true value data set is used as the label output of the U-Net network. Further, the network is trained to realize distortion correction of collected images. The method has the advantages of wide application range, high accuracy, flexible application and the like.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to an image distortion correction method for spatial pose measurement and its network training dataset, which can be used to create datasets for deep learning network training and can solve the distortion correction problem of images acquired by cameras. Background Technology

[0002] Accurately measuring the relative position and attitude (collectively referred to as pose) of spacecraft, especially non-cooperative targets, at close range within 50 meters in space is a crucial prerequisite for completing operations such as space rendezvous and docking, offensive and defensive confrontation, and on-orbit capture and maintenance. It is also one of the key technologies for accomplishing related major cutting-edge defense and aerospace missions. Due to its advantages such as high accuracy, strong autonomy, low cost, simple equipment, low power consumption, and large amount of optical image information, vision-based measurement methods have become one of the main methods for accomplishing the aforementioned space missions.

[0003] The principle of machine vision-based pose measurement is to observe feature points on the object being measured using a calibrated camera and record the image. Then, the pose of the object relative to the camera is determined using the known pose transformation relationship between the world coordinate system and the camera coordinate system. Capturing distortion-free images is crucial. However, images acquired by machine vision systems are often distorted due to factors such as lens manufacturing precision and the difficulty of aligning the camera's optical axis perpendicular to the plane of the object being measured. This distortion ultimately affects the accuracy of pose measurement.

[0004] Traditional distortion correction methods obtain distortion coefficients based on camera calibration and then use a polynomial distortion model to correct the distortion in the image. This method has two major drawbacks: First, it is cumbersome, requiring separate calibration for cameras of the same model. Furthermore, during the long-term, harsh space environment, intrinsic parameters and distortion coefficients change, and this method cannot update the distortion coefficients in real time, thus failing to obtain accurate pose measurement results. Second, image correction is a non-linear process, and the polynomial distortion model-based correction method cannot replace the actual distortion model. Using high-order polynomials for fitting results in excessive computation, while using low-order polynomials introduces significant systematic errors during distortion removal. Therefore, existing traditional methods suffer from cumbersome operation and low accuracy when achieving high-precision pose measurement for various complex space missions.

[0005] In recent years, with the continuous optimization of deep learning network structures and the improvement of computational efficiency, deep learning methods have demonstrated powerful performance. Numerous studies have proven that deep learning exhibits greater advantages over traditional algorithms in terms of speed and reliability, thus leading to its widespread application in various fields. Unlike traditional model-based analytical methods that assume a clear analytical formula for solving the problem and then establish a model through certain linear approximations, deep learning methods bypass the strict requirement of establishing analytical models. By training a given deep network with input and output data, it learns reliable prior knowledge in a non-parametric manner, defining the function of the given network and thus achieving regularization of nonlinear problems. Therefore, deep learning methods provide a new approach to nonlinear problems such as image distortion correction. However, deep learning networks require reliable, diverse, and large-scale sample data for training to ensure the accuracy and generalization ability of the trained network model.

[0006] While datasets created experimentally in real-world or simulated environments are highly realistic, the process is time-consuming and labor-intensive, making it difficult to meet the scale requirements of such datasets. Using mathematical models to generate training samples through computer simulation is convenient and feasible, but this method suffers from insufficient realism in the generated data, thus affecting the accuracy of the trained network model.

[0007] To address this issue, this patent proposes a method for creating a dataset based on digital twins. By constructing a virtual measurement system using the structural parameters of a real system, it approximates the actual measurement environment and the characteristics of the measured object to the greatest extent possible, thereby creating a high-fidelity dataset for training network models. Simultaneously, a distortion-reducing U-Net network is established. Combined with the created high-fidelity dataset, a deep learning network is used to enhance, denoise, and correct distortion of the acquired images, thus solving the problem of difficult correction of spacecraft pose measurement images. Summary of the Invention

[0008] This invention discloses an image distortion correction method for spatial pose measurement and its network training dataset. The method generates a distortion-free ground truth dataset based on the principle of digital twins. Then, it adds random distortion to the ground truth images using a Radtan distortion model or a fisheye distortion model, thereby simulating the distortion of ordinary perspective lenses or fisheye lenses and enabling the creation of a large dataset for network training. This dataset can be directly used as input to the U-Net network, with the ground truth dataset serving as the U-Net network's label output. The network is then trained to correct distortion in acquired images. This method has advantages such as wide applicability, high accuracy, and flexible application.

[0009] An image distortion correction method for spatial pose measurement includes the following steps:

[0010] Step one: In the digital twin software, the required spacecraft model, fluorescent lamp model, measurement background, etc., are digitally established according to the actual measurement process to generate the measurement environment.

[0011] Step two involves constructing a digital model of the measuring camera based on its design parameters (focal length, sensor size, output image size, etc.) and controlling its rotation and translation under set lighting conditions. This allows for the comprehensive acquisition of image data of the object under test, resulting in a distortion-free ground truth label dataset.

[0012] Step 3: Process the above ground value dataset according to the distortion model, random lighting conditions, and noise to form a distorted data subset, a data subset under random lighting, and a data subset contaminated by noise, and finally form the training dataset.

[0013] Step four: Construct a distortion correction network and train it using the aforementioned dataset to enable it to perform distortion correction, image enhancement, and noise suppression, thereby achieving distortion correction of pose images.

[0014] This invention provides an image distortion correction method for spatial pose measurement. Compared with the prior art, the advantages of this invention are:

[0015] 1. By using digital twin technology to build a virtual measurement scenario and collect images of the object being measured, a real, distortion-free image can be obtained. This eliminates the need for time-consuming and laborious experimental dataset creation and overcomes the problem of insufficient accuracy in dataset creation using mathematical models. It also effectively solves the problem of obtaining ground truth labels for deep learning networks.

[0016] 2. This method utilizes digital twin technology to generate a labeled dataset, which can then be used to create a high-fidelity network training input dataset that reflects the characteristics of extreme darkness, overexposure, noise pollution, and lens distortion in spatial applications, providing a new approach to improve the accuracy and generalization ability of network models.

[0017] 3. A deep learning-based image distortion correction method was established. Unlike traditional distortion correction methods, this method eliminates the need for time-consuming and laborious camera calibration. It can perform distortion correction on images captured by specific cameras (ordinary perspective cameras, fisheye cameras) using only a deep learning network. Applying this method to the task of distortion correction for measurement targets in space applications can solve the problem of real-time distortion correction of images of measured targets in this field. Attached Figure Description

[0018] Figure 1 It is a virtual measurement environment

[0019] Figure 2 It is the construction of a simulated camera

[0020] Figure 3The definition of α and β

[0021] Figure 4 It is a distortion correction network structure

[0022] Figure 5 This is a data sample. Detailed Implementation

[0023] This invention discloses an image distortion correction method for spatial pose measurement. The technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. These embodiments are merely one implementation of the invention and are not intended to limit the invention in any way. Therefore, any simple modifications, equivalent changes, or modifications made to the above embodiments based on the technical essence of this invention shall still fall within the scope of this invention.

[0024] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] 1. Setting up the initial scene

[0026] Step 1: Constructing a Virtual Measurement Environment

[0027] To obtain a realistic, distortion-free spacecraft image dataset, a virtual measurement environment was first constructed in digital twin software, establishing a world coordinate system, a spacecraft twin, scene lighting, background, etc.

[0028] The center of the spacecraft twin was aligned with the origin of the world coordinate system. To achieve uniform illumination, six fluorescent lamps with constant intensity (1W / m²) were placed along the six axes (X+, X-, Y+, Y-, Z+, Z-) of the spacecraft twin. 2 Set the fluorescent lamp parameters as required, and arrange the fluorescent lamp positions as follows: Figure 1 As shown.

[0029] Step 2: Construct a camera twin

[0030] A camera twin is constructed and its internal parameters are set. To capture images of the spacecraft in different poses, the camera is controlled to move along a spherical trajectory. The camera's position coordinate control equation is: Formula (1). Simultaneously, to ensure the spacecraft remains within the camera's field of view during movement, a constraint is added to the camera that the optical axis points towards the world origin. The camera's motion trajectory is illustrated below. Figure 2 As shown in (a).

[0031]

[0032] Where (X) O ,Y O Z O (X) is the world origin coordinate, (X) C ,YC Z C ) represents the camera's position coordinates; r C It is the distance from the camera's optical center to the world origin. α and β take uniform values ​​within their range as needed. The definitions of α and β are as follows: Figure 3 As shown.

[0033] 2. Dataset Creation

[0034] Step 1: Creating the Labeled Dataset

[0035] Based on the size M of the dataset, determine the appropriate value of n according to formula (2), and the values ​​of α and β are given in formula (3). Turn on 6 fluorescent lamps to obtain constant illumination (e.g., 1W / m²). 2 We take pictures of the spacecraft according to the required resolution. This yields a distortion-free ground truth dataset (denoted as Data_True).

[0036] M = (n-1) × 2n (n > 1) (2)

[0037]

[0038] Step 2: Creating a subset of distorted data

[0039] The distorted data subset is generated by adding distortion to the above ground value dataset.

[0040] Based on the camera's design parameters, for a standard perspective camera, the absolute values ​​of the radial distortion coefficient ki (i = 1, 2, 3) and the tangential distortion coefficient pi (i = 1, 2) are set to random values ​​within the range of 0 to 1. Perspective distortion is then added to each frame of the ground truth dataset Data_True according to the Radtan distortion model to generate standard perspective camera distortion data, as shown in formula (5). To improve the model's generalization ability, the distortion variable can be appropriately enlarged or reduced by a certain factor, as shown in Table (1).

[0041] Table 1. Range of distortion coefficients for perspective cameras (10 is the magnification).

[0042]

[0043] For fisheye cameras, referencing the mapping coefficient Q of mainstream fisheye cameras on the market, the mapping coefficient is set as: q i =rand(0.9~1.1)×Q,(i=1,2,3,4). Fisheye camera distortion data can be generated by adding random fisheye distortion to each frame of the data in the Data_True dataset according to the fisheye distortion model, as shown in formula (6). In order to improve the generalization ability of the model, the distortion variable can be appropriately enlarged or reduced by a certain factor, such as the value method in Table (2).

[0044] Table 2. Range of values ​​for fisheye camera mapping coefficients

[0045]

[0046] The resulting distorted data subset is denoted as Data_Distortion. The specific process of adding distortions is as follows:

[0047] Let the transformation relationship between image pixel coordinates in the pixel coordinate system and the camera coordinate system be as follows:

[0048]

[0049] Where (u,v) are the coordinates of the image pixels in the pixel coordinate system, (x... c ,y c () represents the coordinates of the image pixels in the camera coordinate system.

[0050] According to the Radtan distortion model, the pixel coordinates of an image containing distortion can be represented as:

[0051]

[0052] Where (x) d ,y d ) represents the pixel coordinates of the image after perspective distortion has been added, k1, k2, and k3 are the radial distortion coefficients, and p1 and p2 are the tangential distortion coefficients.

[0053] According to the fisheye distortion model, the pixel coordinates of an image containing distortion can be represented as:

[0054]

[0055] Where (x) d ,y d ) represents the pixel coordinates of the image after fisheye distortion has been added, and k1, k2, k3, and k4 are mapping coefficients.

[0056] Then, the pixel coordinates of the distorted image are transformed to the pixel coordinate system:

[0057]

[0058] Where (u d ,v d ) is the coordinate of the distorted pixel in the pixel coordinate system.

[0059] Step 3: Noise Data Subset Creation

[0060] Based on the noise parameters of the real scene, Gaussian noise is added to each frame of the Data_True dataset. This results in a noisy subset of data (denoted as Data_Noise).

[0061] Step 4: Production of Extreme Lighting Data Subset

[0062] Define the light intensity range for extremely dim lighting, such as: (0.0015, 0.015) W / m². 2 Define the light intensity range of an overexposed image, such as: (15, 20) W / m 2 This creates an extremely dark, overexposed lighting scene. The camera is controlled to take pictures of the spacecraft according to formula (1) and the required resolution. This yields a subset of data under extreme lighting conditions (denoted as Data_Radom_Light).

[0063] The datasets for Step 2 and Step 3 were created in a Python environment, while the datasets for Step 1 and Step 4 were created in a Blender environment. The resulting images were saved in PNG format.

[0064] 3. Distortion-free network structure

[0065] The distortion correction network uses the U-Net network, the structure of which is shown in the appendix. Figure 4 It comprises three sub-networks: an image enhancement network, an image denoising network, and a distortion correction network. The input to the image enhancement network is the dataset `Data_Radom_Light`, and the label is the dataset `Data_True`. The input to the image denoising network is the dataset `Data_Noise`, and the label is the dataset `Data_True`. The input to the distortion correction network is the dataset `Data_Distortion`, and the label is the dataset `Data_True`. To avoid overfitting, techniques such as skip connections between layers, batch regularization, and learning rate decay are employed. The ReLU activation function is used in the intermediate layers, the Sigmoid function is used in the last layer, the root mean square error loss function is used, the learning rate is 0.001, and the Adam optimizer is used.

[0066] 4. Online training

[0067] During network training, the dataset was divided into training, validation, and test sets in a 6:2:2 ratio. For large images, the `fit_generator` strategy was used to read the dataset in batches, followed by normalization preprocessing. The Adam optimizer was used to update the weights. To eliminate edge artifacts, a 3×3 window median filter was used to smooth the distortion-free images.

[0068] 5. After training the network, use the real distorted images captured by the camera as input to the network. The trained network can then be used to complete the distortion removal task in complex scenes.

Claims

1. An image distortion correction method for spatial pose measurement, characterized in that: The method generates a distortion-free ground truth dataset based on the principle of digital twins. Then, it adds random distortion to the ground truth images using a Radtan distortion model or a fisheye distortion model to simulate the distortion of ordinary perspective lenses or fisheye lenses, thereby enabling the creation of a large dataset for network training. This dataset can be directly used as the input to the U-Net network, and the ground truth dataset can be used as the label output of the U-Net network. In this way, the network can be trained to achieve distortion correction of the acquired images. Follow these steps: Step 1: In the digital twin software, digitally establish the required spacecraft model, fluorescent lamp model, and measurement background according to the actual measurement process, and generate the measurement environment; Step two: Based on the design parameters of the camera used, under the set lighting conditions, construct a digital model of the measuring camera and control the camera's rotation and translation; then, collect image data of the object under test from all directions to form a distortion-free ground truth label dataset. Step 3: Process the above ground value dataset according to the distortion model, random lighting conditions, and noise to form a distorted data subset, a data subset under random lighting, and a data subset contaminated by noise, and finally form the training dataset. Step 4: Construct a distortion correction network and train it using the dataset mentioned above to enable it to perform distortion correction, image enhancement, and noise suppression, thereby achieving distortion correction of pose images. The distortion correction network structure is constructed according to the following steps: The distortion correction network uses the U-Net network, which comprises three sub-networks: an image augmentation network, an image denoising network, and a distortion correction network. The input to the image augmentation network is the dataset Data_Radom_Light, and the label is the dataset Data_True. The input to the image denoising network is the dataset Data_Noise, and the label is the dataset Data_True. The input to the distortion correction network is the dataset Data_Distortion, and the label is the dataset Data_True. To avoid overfitting, skip connections between layers, batch regularization, and learning rate decay are employed. The ReLU function is used for activation in the intermediate layers, and the Sigmoid function is used for activation in the last layer. The root mean square error function is used for loss, the learning rate is 0.001, and the Adam optimizer is used.

2. The image distortion correction method for spatial pose measurement as described in claim 1, characterized in that, The initial scene setup follows these steps: Step 1: Constructing a Virtual Measurement Environment To obtain a realistic dataset of distortion-free spacecraft images, a virtual measurement environment was first constructed in digital twin software, establishing a world coordinate system, a spacecraft twin, scene lighting, and background. The center of the spacecraft twin is aligned with the origin of the world coordinate system. To obtain uniform illumination, six fluorescent lamps with constant light intensity are arranged along the six axes of the spacecraft twin, and the parameters of the fluorescent lamps are set as required. Step 2: Construct a camera twin Construct a camera twin and set internal parameters; in order to capture images of the spacecraft in different poses, control the camera to move along a spherical trajectory. The position coordinate control equation of the camera is: Formula (1). At the same time, in order to ensure that the spacecraft is always in the camera's field of view during the movement, add a constraint that the optical axis of the camera points to the world origin. （1）； in It is the world origin coordinate. These are the camera's position coordinates; It is the distance from the optical center of the camera to the origin of the world; The value is taken evenly within its range as needed.

3. The image distortion correction method for spatial pose measurement as described in claim 1, characterized in that, The dataset was created according to the following steps: Step 1: Creating the Labeled Dataset Based on the size of the dataset M Determine the appropriate formula according to formula (2). n value, The value of is given in formula (3); turn on 6 fluorescent lamps to obtain constant illumination, and take pictures of the spacecraft according to the required resolution; This yields a distortion-free true dataset, denoted as Data_True; （2）； （3）； Step 2: Creating a subset of distorted data The distorted data subset is generated by adding distortion to the above ground value dataset; Based on the camera's design parameters, for a standard perspective camera, set the radial distortion coefficient. k i , i = 1, 2, 3, and tangential distortion coefficient p i , i = 1 and 2 are both random values ​​in the range of 0-1. According to the Radtan distortion model, perspective random distortion is added to each frame of the true dataset Data_True to generate ordinary perspective camera distortion data, see formula (5); in order to improve the generalization ability of the model, the distortion variable is appropriately enlarged or reduced by a certain factor. For fisheye cameras, refer to the mapping coefficients of mainstream fisheye cameras on the market. Q The mapping coefficient is set as follows: ; Add random fisheye distortion to each frame of the data in the Data_True dataset according to the fisheye distortion model to generate fisheye camera distortion data, see formula (6) for details; In order to improve the generalization ability of the model, the distortion variable is appropriately enlarged or reduced by a certain factor; The resulting distorted data subset is denoted as Data_Distortion; the specific process of adding distortions is as follows: Let the transformation relationship between image pixel coordinates in the pixel coordinate system and the camera coordinate system be as follows: （4）； in These are the coordinates of image pixels in the pixel coordinate system. These are the coordinates of the image pixels in the camera coordinate system; According to the Radtan distortion model, the pixel coordinates of an image containing distortion can be represented as: （5）； in These are the pixel coordinates of the image after perspective distortion has been added. It is the radial distortion coefficient. It is the tangential distortion coefficient; According to the fisheye distortion model, the pixel coordinates of an image containing distortion can be represented as: （6）； in These are the pixel coordinates of the image after adding fisheye distortion. These are mapping coefficients; Then, the pixel coordinates of the distorted image are transformed to the pixel coordinate system: （7）； in These are the coordinates of the distorted pixel in the pixel coordinate system; Step 3: Noise Data Subset Creation Based on the noise parameters in the real scene, Gaussian noise is added to each frame of the Data_True dataset; thus, a noisy subset of data is obtained, denoted as Data_Noise. Step 4: Production of Extreme Lighting Data Subset Define the light intensity range of extremely dark illumination, such as (0.0015, 0.015) W / m2, and define the light intensity range of overexposed illumination, such as (15, 20) W / m2, to form extremely dark and overexposed illumination scenes; control the camera to take pictures of the spacecraft according to the formula (1) and the required resolution; thereby obtain the data subset under extreme illumination, denoted as Data_Radom_Light; The datasets for Step 2 and Step 3 were created in a Python environment, while the datasets for Step 1 and Step 4 were created in a Blender environment. The resulting images were saved in PNG format.

4. The image distortion correction method for spatial pose measurement as described in claim 2, characterized in that, The network training is performed according to the following steps: When training the network, the established dataset is divided into training set, validation set, and test set, with an allocation ratio of 6:2:

2. For large images, the fit_generator strategy is used to read the dataset in batches, and then normalization preprocessing is performed. The Adam optimizer is used to update the weights. In order to eliminate edge effects, a 3×3 window median filter is used to smooth the distortion-free image. After training the network, the real distorted images captured by the camera are used as input to the network, and the trained network can complete the distortion removal task in complex scenes.

Images