Image dedistortion method and device

Abstract

The present invention discloses a method and device for image dedistortion. The method comprises: extracting the center coordinates of multiple calibration plate images acquired by an image acquisition device; establishing a mapping relationship between the center coordinates and the standard grid points after dedistortion; training a first mapping model for each calibration plate image according to the above mapping relationship; establishing a unified standard grid coordinate system, updating the mapping relationship between the center coordinates of each calibration plate and the unified standard grid; training a second mapping model according to the updated mapping relationship; inferring the mapping relationship between the image acquired by the image acquisition device and the corresponding standard grid points according to the trained second mapping model, and remapping the acquired image to obtain the dedistorted image. The present invention establishes a dedistortion method for large-size and high-resolution images based on a machine learning model, which solves the problem that the subtle distortion of large-size and high-resolution images affects image post-processing operations such as image stitching.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a method and device for image dedistortion. Background Art

[0002] Image distortion generally refers to radial distortion or tangential distortion inside the lens, or distortion caused by the instability of the scanning process of scanning imaging equipment. High-end imaging equipment such as electron microscopes can obtain large-size and high-resolution images. Such equipment often has built-in de-distortion algorithms or devices to de-distort the collected images.

[0003] However, even if camera equipment such as electron microscopes have undergone distortion correction before leaving the factory, their accuracy level still does not meet the technical requirements of image stitching in certain industrial fields, and will leave obvious stitching marks, affecting the final stitching effect. Summary of the invention

[0004] In order to solve the problems in the prior art, the present invention provides a method and device for image dedistortion to solve the problem that subtle distortion of large-size and high-resolution images affects image post-processing operations such as image stitching.

[0005] In a first aspect, an embodiment of the present invention provides a method for image dedistortion, comprising:

[0006] S1, extracting the center coordinates of multiple calibration plate images acquired by the image acquisition device;

[0007] S2. For the center coordinates extracted from each calibration plate image, establish a mapping relationship between them and the standard grid points after dedistortion; each calibration plate image corresponds to a different standard grid;

[0008] S3, training a first mapping model for each calibration plate image according to the mapping relationship between the center coordinates of each calibration plate image and its corresponding standard grid;

[0009] S4, establishing a unified standard grid coordinate system, and updating the mapping relationship between the center coordinates of each calibration plate and the unified standard grid according to the trained first mapping model;

[0010] S5. Train a second mapping model based on the updated mapping relationship between the center coordinates of each calibration plate and the unified standard grid, infer the mapping relationship between the image captured by the image acquisition device and the corresponding standard grid points based on the trained second mapping model, and remap the image captured by the image acquisition device according to the mapping relationship to obtain a dedistorted image.

[0011] Optionally, the S1 specifically includes:

[0012] S11, according to the brightness of the calibration plate image, obtaining a binarization empirical threshold of the calibration plate image and performing binarization processing on the calibration plate image;

[0013] S12, performing a filtering operation on the binarized calibration plate image to remove the noise caused by the binarization empirical threshold;

[0014] S13, using the centroid coordinates of the connected domain of the filtered binary calibration plate image as the center coordinates of the calibration plate image.

[0015] Optionally, the S2 specifically includes:

[0016] S21, selecting a number of circle centers in the middle area of ​​the calibration plate image, mapping the center coordinates of the number of circle centers to the area of ​​the standard grid, and using the mapping relationship of the coordinates as an initialized mapping relationship data set;

[0017] S22, training a linear regression model according to the initialized mapping relationship data set;

[0018] S23, bringing the coordinates of the centers of circles in the calibration plate image that are not in the initialized mapping relationship data set into the linear regression model to obtain the predicted values ​​of the standard grid points, and expanding the initialized mapping relationship data set according to the predicted values ​​of the standard grid points.

[0019] Optionally, the S23 specifically includes:

[0020] The center coordinates of each circle in the calibration plate image that are not in the initialized mapping relationship data set are sorted according to the distance;

[0021] In order from near to far, the coordinates of the circle center are input into the linear regression equation to obtain the predicted values ​​of the standard grid points;

[0022] If the difference between the rounded predicted value and the predicted value is less than a certain threshold, the predicted value is considered to be credible, and the rounded predicted value and the corresponding circle center coordinates are added to the initialized mapping relationship data set to obtain the final mapping relationship data set.

[0023] Optionally, the S3 specifically includes:

[0024] The center coordinates of the calibration plate image in the mapping relationship are used as the input of the first mapping model, and the corresponding standard grid point coordinates in the mapping relationship are used as the output of the first mapping model, and the first mapping model is established for each calibration plate image.

[0025] Optionally, the training of the second mapping model in S5 specifically includes:

[0026] The center coordinates of each calibration plate image are used as the input of the second mapping model, and the unified standard grid point coordinates are used as the output of the second mapping model, and a unified second mapping model is obtained through training.

[0027] In a second aspect, an embodiment of the present invention further provides an image dedistortion device, comprising:

[0028] A circle center coordinate extraction module is used to extract the circle center coordinates of multiple calibration plate images acquired by the image acquisition device;

[0029] A mapping relationship establishment module is used to establish a mapping relationship between the center coordinates extracted from each calibration plate image and the standard grid points after dedistortion; each calibration plate image corresponds to a different standard grid;

[0030] A first mapping model training module is used to train a first mapping model for each calibration plate image according to a mapping relationship between the center coordinates of each calibration plate image and its corresponding standard grid;

[0031] A standard grid coordinate system module is used to establish a unified standard grid coordinate system and update the mapping relationship between the center coordinates of each calibration plate and the unified standard grid according to the trained first mapping model;

[0032] The dedistortion module is used to train a second mapping model according to the mapping relationship between the updated center coordinates of each calibration plate and the unified standard grid, infer the mapping relationship between the image captured by the image acquisition device and the corresponding standard grid points according to the trained second mapping model, and remap the image captured by the image acquisition device according to the mapping relationship to obtain a dedistorted image.

[0033] The present invention is based on machine learning and uses data from multiple calibration plates to establish a mapping relationship between an image and a standard grid. The machine learning model is trained according to the mapping relationship and a method for dedistorting an image based on a full-pixel dense mapping relationship is established. This method solves the problem that slight distortion in high-precision large-size images affects subsequent tasks such as image stitching. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for image dedistortion provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a calibration plate image provided by an embodiment of the present invention;

[0036] Figure 3 A flowchart of establishing a mapping relationship between a calibration plate center and a standard grid is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings. Example

[0038] Figure 1 A flowchart of a method for image dedistortion provided by an embodiment of the present invention specifically includes the following steps:

[0039] S1. Extracting the center coordinates of a plurality of calibration plate images acquired by an image acquisition device.

[0040] In particular, the image acquisition device in this embodiment can be a high-end imaging device such as an electron microscope, which is used to obtain large-size and high-resolution images. Figure 2 A schematic diagram of a calibration plate image provided in this embodiment.

[0041] Specifically, the above steps include:

[0042] S11. According to the brightness of the calibration plate image, an empirical threshold value for binarization of the calibration plate image is obtained to perform binarization processing on the calibration plate image.

[0043] S12, performing a filtering operation on the binarized calibration plate image to remove noise caused by the binarization empirical threshold.

[0044] In this embodiment, the geometric features of the image are mainly selected to implement denoising filtering.

[0045] As an optional implementation, the filtering operation of this embodiment includes: reading all connected domain information of the calibration plate image after binarization, calculating and counting the connected domain area mode M, and adjusting M by adding a reasonable fluctuation value to generate the connected domain area mode maximum value M max and the minimum value M min To filter the area. min The noise points are usually caused by unreasonable binarization, while those greater than M max The intersection of points is considered to be caused by dirty points, and the above two situations are eliminated.

[0046] Furthermore, under the premise of selecting a circular calibration plate, a secondary filtering is performed on the connected domains whose area values ​​meet the conditions, and the roundness is selected as the feature descriptor to remove outliers to prevent the influence of noise whose area meets the conditions.

[0047] S13, using the centroid coordinates of the connected domain of the filtered binary calibration plate image as the center coordinates of the calibration plate image.

[0048] S2. For the center coordinates extracted from each calibration plate image, a mapping relationship between them and the standard grid points after dedistortion is established; each calibration plate image corresponds to a different standard grid.

[0049] In this embodiment, the standard grid point is the corresponding area on the standard grid. The standard grid is automatically generated according to the calibration plate image. Each calibration plate image corresponds to a standard grid, and the standard grid coordinate system of each calibration plate image is established independently and differently. By establishing a mapping relationship, the corresponding position information of the center position of the circle on the distorted calibration plate image on the undistorted standard grid can be obtained.

[0050] Specifically, S2 includes:

[0051] S21. Since the distortion degree of the middle area of ​​the calibration plate image is relatively weak, several circle centers are selected in the middle area of ​​the calibration plate image, and the center coordinates of these circle centers are mapped to the area of ​​the standard grid, and the mapping relationship of the coordinates is used as the initialized mapping relationship data set.

[0052] S22. A linear regression model is trained according to the initialized mapping relationship data set, wherein the input of the linear regression model is the coordinates of the center of the circle in the initialized mapping relationship, and the output is the coordinates of the standard grid points in the initialized mapping relationship data set.

[0053] The above-trained linear regression model can be used to automatically find the mapping relationship between each circle center and the standard grid point in the calibration plate image, see S23 for details.

[0054] S23, bringing the coordinates of the centers of circles in the calibration plate image that are not in the initialized mapping relationship data set into the linear regression model to obtain the predicted values ​​of the standard grid points, and expanding the initialized mapping relationship data set according to the predicted values ​​of the standard grid points.

[0055] See Figure 3 , sort the circle center coordinates that are not in the initialized mapping relationship data set, find the circle center closest to the data set, input it into the linear regression model, and get the predicted value of the standard grid point. Round the predicted value. If the difference between it and the predicted value is less than a certain threshold, the predicted value is considered to be credible, and the rounded predicted value and the circle center coordinates of the input model are added to the mapping relationship data set. Repeat this process and continuously expand the mapping relationship data set until all the circle center coordinates on the calibration plate are traversed.

[0056] S3. Train a first mapping model for each calibration plate image according to the mapping relationship between the center coordinates of each calibration plate image and its corresponding standard grid.

[0057] The first mapping model in this embodiment can be a regression-type machine learning model. When performing model training, the expanded mapping relationship data set in S23 is used for training. The center coordinates of the circle in the mapping relationship are used as the model input, and the standard grid point coordinates are used as the model output. Each calibration plate image is independently modeled, and the number of models is the same as the number of calibration plate images.

[0058] By adopting a regression-based machine learning model for training, the sparse correspondence between image feature points and standard grid point coordinates can be fitted into a dense mapping relationship of all pixel points in the image through interpolation operations during the training process.

[0059] S4. Establish a unified standard grid coordinate system, and update the mapping relationship between the center coordinates of each calibration plate and the unified standard grid according to the trained first mapping model.

[0060] Specifically, for i A calibration plate image, f i Represents the trained first mapping model. The standard grid point corresponding to the circle center coordinate (0, 0) is obtained by inferring the trained model. f i (0, 0). For other circle center coordinates on the calibration plate image ( x , y ), let the new standard grid points be F i ( x , y )= f i ( x , y )- f i (0, 0). Map all calibration plate images by f Updated to F , which is the mapping relationship from the center of the circle to the unified standard grid coordinate system.

[0061] By transforming the mapping relationship established by multiple calibration plate images into a unified standard grid coordinate system, the training data set is expanded and the model fitting effect is better.

[0062] S5. Train a second mapping model based on the updated mapping relationship between the center coordinates of each calibration plate and the unified standard grid, infer the mapping relationship between the image captured by the image acquisition device and the corresponding standard grid points based on the trained second mapping model, and remap the image captured by the image acquisition device according to the mapping relationship to obtain a dedistorted image.

[0063] Similar to S3, the second mapping model in this embodiment also adopts a regression-type machine learning model, takes the center coordinates obtained from each calibration plate image as the model input, and the unified standard grid point coordinates as the model output, and uses the mapping relationship of all calibration plate images as the data set to train a unified machine learning model. By inputting the image obtained by the image acquisition device into the machine learning model, the mapping relationship between the image and the corresponding standard grid point can be obtained, and the image can be remapped according to the mapping relationship. For example, the mapping relationship is input into the remap function of OpenCV to obtain a dedistorted image.

[0064] Furthermore, an embodiment of the present invention also provides an image dedistortion device, comprising:

[0065] A circle center coordinate extraction module is used to extract the circle center coordinates of multiple calibration plate images acquired by the image acquisition device;

[0066] A mapping relationship establishment module is used to establish a mapping relationship between the center coordinates extracted from each calibration plate image and the standard grid points after dedistortion; each calibration plate image corresponds to a different standard grid;

[0067] A first mapping model training module is used to train a first mapping model for each calibration plate image according to a mapping relationship between the center coordinates of each calibration plate image and its corresponding standard grid;

[0068] A standard grid coordinate system module is used to establish a unified standard grid coordinate system and update the mapping relationship between the center coordinates of each calibration plate and the unified standard grid according to the trained first mapping model;

[0069] The dedistortion module is used to train a second mapping model according to the mapping relationship between the updated center coordinates of each calibration plate and the unified standard grid, infer the mapping relationship between the image captured by the image acquisition device and the corresponding standard grid points according to the trained second mapping model, and remap the image captured by the image acquisition device according to the mapping relationship to obtain a dedistorted image.

[0070] The circle center coordinate extraction module is specifically used to perform:

[0071] S11, according to the brightness of the calibration plate image, obtaining a binarization empirical threshold of the calibration plate image and performing binarization processing on the calibration plate image;

[0072] S12, performing a filtering operation on the binarized calibration plate image to remove the noise caused by the binarization empirical threshold;

[0073] S13, using the centroid coordinates of the connected domain of the filtered binary calibration plate image as the center coordinates of the calibration plate image.

[0074] The mapping relationship establishment module is specifically used to execute:

[0075] S21, selecting a number of circle centers in the middle area of ​​the calibration plate image, mapping the center coordinates of the number of circle centers to the area of ​​the standard grid, and using the mapping relationship of the coordinates as an initialized mapping relationship data set;

[0076] S22, training a linear regression model according to the initialized mapping relationship data set;

[0077] S23, bringing the coordinates of the centers of circles in the calibration plate image that are not in the initialized mapping relationship data set into the linear regression model to obtain the predicted values ​​of the standard grid points, and expanding the initialized mapping relationship data set according to the predicted values ​​of the standard grid points.

[0078] Furthermore, the S23 specifically includes:

[0079] The center coordinates of each circle in the calibration plate image that are not in the initialized mapping relationship data set are sorted according to the distance;

[0080] In order from near to far, the coordinates of the circle center are input into the linear regression equation to obtain the predicted values ​​of the standard grid points;

[0081] If the difference between the rounded predicted value and the predicted value is less than a certain threshold, the predicted value is considered to be credible, and the rounded predicted value and the corresponding circle center coordinates are added to the initialized mapping relationship data set to obtain the final mapping relationship data set.

[0082] The first mapping model training module is specifically used to perform:

[0083] The center coordinates of the calibration plate image in the mapping relationship are used as the input of the first mapping model, and the corresponding standard grid point coordinates in the mapping relationship are used as the output of the first mapping model, and the first mapping model is established for each calibration plate image.

[0084] The training of the second mapping model specifically includes:

[0085] The center coordinates of each calibration plate image are used as the input of the second mapping model, and the unified standard grid point coordinates are used as the output of the second mapping model to train a unified second mapping model.

[0086] The center coordinates of each calibration plate image are used as the input of the second mapping model, and the unified standard grid point coordinates are used as the output of the second mapping model, and a unified second mapping model is obtained through training.

[0087] An image dedistortion device provided in an embodiment of the present invention can execute an image dedistortion method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0088] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for image dedistortion, characterized in that: include: S1, extracting the center coordinates of multiple calibration plate images acquired by the image acquisition device; S2. For the center coordinates extracted from each calibration plate image, establish a mapping relationship between them and the distortion-free standard grid points; each calibration plate image corresponds to a different standard grid; S3, training a first mapping model for each calibration plate image according to the mapping relationship between the center coordinates of each calibration plate image and its corresponding standard grid; S4, establishing a unified standard grid coordinate system, and updating the mapping relationship between the center coordinates of each calibration plate and the unified standard grid according to the trained first mapping model; S5, training a second mapping model according to the updated mapping relationship between the center coordinates of each calibration plate and the unified standard grid, inferring the mapping relationship between the image collected by the image acquisition device and the corresponding standard grid points according to the trained second mapping model, and remapping the image collected by the image acquisition device according to the mapping relationship to obtain a dedistorted image; The S2 specifically includes: Select at least two circle centers in the middle area of ​​the calibration plate image, correspond the center coordinates of the at least two circle centers to the area of ​​the standard grid, and use the mapping relationship of the coordinates as an initialized mapping relationship data set; A linear regression model is trained according to the initialized mapping relationship data set, wherein the input of the linear regression model is the coordinates of the center of the circle in the initialized mapping relationship, and the output is the coordinates of the standard grid points in the initialized mapping relationship data set; The coordinates of the centers of circles in the calibration plate image that are not in the initialized mapping relationship data set are brought into the linear regression model to obtain the predicted values ​​of the standard grid points, and the initialized mapping relationship data set is expanded according to the predicted values ​​of the standard grid points.

2. The method according to claim 1, characterized in that The S1 specifically includes: S11, according to the brightness of the calibration plate image, obtaining a binarization empirical threshold of the calibration plate image and performing binarization processing on the calibration plate image; S12, performing a filtering operation on the binarized calibration plate image to remove the noise caused by the binarization empirical threshold; S13, using the centroid coordinates of the connected domain of the filtered binary calibration plate image as the center coordinates of the calibration plate image.

3. The method according to claim 1, characterized in that The S23 specifically includes: The center coordinates of each circle in the calibration plate image that are not in the initialized mapping relationship data set are sorted according to the distance; In order from near to far, the coordinates of the circle center are input into the linear regression equation to obtain the predicted values ​​of the standard grid points; If the difference between the rounded predicted value and the predicted value is less than a certain threshold, the predicted value is considered to be credible, and the rounded predicted value and the corresponding circle center coordinates are added to the initialized mapping relationship data set to obtain the final mapping relationship data set.

4. The method according to claim 1, characterized in that: The S3 specifically includes: The center coordinates of the calibration plate image in the mapping relationship are used as the input of the first mapping model, and the corresponding standard grid point coordinates in the mapping relationship are used as the output of the first mapping model, and the first mapping model is established for each calibration plate image.

5. The method according to claim 1, characterized in that The training of the second mapping model in S5 specifically includes: The center coordinates of each calibration plate image are used as the input of the second mapping model, and the unified standard grid point coordinates are used as the output of the second mapping model, and a unified second mapping model is obtained through training.

6. An image dedistortion device, characterized in that: include: A circle center coordinate extraction module is used to extract the circle center coordinates of multiple calibration plate images acquired by the image acquisition device; A mapping relationship establishment module is used to establish a mapping relationship between the center coordinates extracted from each calibration plate image and the distortion-free standard grid points; each calibration plate image corresponds to a different standard grid; A first mapping model training module is used to train a first mapping model for each calibration plate image according to a mapping relationship between the center coordinates of each calibration plate image and its corresponding standard grid; A standard grid coordinate system module is used to establish a unified standard grid coordinate system and update the mapping relationship between the center coordinates of each calibration plate and the unified standard grid according to the trained first mapping model; A dedistortion module is used to train a second mapping model according to the updated mapping relationship between the center coordinates of each calibration plate and the unified standard grid, infer the mapping relationship between the image collected by the image acquisition device and the corresponding standard grid points according to the trained second mapping model, and remap the image collected by the image acquisition device according to the mapping relationship to obtain a dedistorted image; The mapping relationship establishment module is specifically used to execute: Select at least two circle centers in the middle area of ​​the calibration plate image, correspond the center coordinates of the at least two circle centers to the area of ​​the standard grid, and use the mapping relationship of the coordinates as an initialized mapping relationship data set; A linear regression model is trained according to the initialized mapping relationship data set, wherein the input of the linear regression model is the coordinates of the center of the circle in the initialized mapping relationship, and the output is the coordinates of the standard grid points in the initialized mapping relationship data set; The coordinates of the centers of circles in the calibration plate image that are not in the initialized mapping relationship data set are brought into the linear regression model to obtain the predicted values ​​of the standard grid points, and the initialized mapping relationship data set is expanded according to the predicted values ​​of the standard grid points.

Images