Distortion mapping data generation method and distortion correction method for camera module

Abstract

Embodiments of the present application disclose a method for generating distortion mapping data of a camera module and a method for correcting distortion. An embodiment of the method comprises: calibrating a camera module based on a calibration image captured by the camera module on a calibration board to obtain parameters of the camera module; finding actual mapping points of each corner point in the calibration image, and determining ideal mapping points of each corner point in the calibration image based on the parameters; determining distortion offsets corresponding to each corner point based on coordinates of the actual mapping points and the ideal mapping points in the calibration image; and generating distortion mapping data of the camera module based on the determined distortion offsets. The embodiment improves the accuracy of the generated data, reduces the operation difficulty, corrects a distorted image based on the data, and improves the correction effect of the distorted image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to a distortion mapping data generation method and a distortion correction method of a camera module. BACKGROUND

[0002] With the popularity of terminal devices and the increasing demand of users for taking photos, the types of camera modules are becoming more and more diverse. When using various camera modules to take images, the images taken usually have different degrees of distortion. It is usually necessary to express the distortion of the images taken by the camera module in the form of data, and then correct the distortion of the images taken based on the data.

[0003] In the prior art, a camera calibration method or a grid-based method can be used to determine the data used to express the distortion. However, the data obtained by using the camera calibration method has a large error when the image has non-uniform distortion, resulting in poor distortion image correction effect. The grid-based method has very high requirements for the collection operation of the grid image (such as the lens needs to be vertically photographed to the grid image, the center of the lens needs to be directly opposite the corner point in the grid image, etc.), and the operation is difficult. SUMMARY

[0004] Embodiments of the present application provide a distortion mapping data generation method and a distortion correction method of a camera module to solve the technical problems of poor distortion image correction effect and large data acquisition difficulty in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a distortion mapping data generation method of a camera module, which comprises: calibrating the camera module based on a calibration image taken by the camera module on a calibration board to obtain parameters of the camera module; finding actual mapping points of each corner point in the calibration board in the calibration image, and determining ideal mapping points of each corner point in the calibration image based on the parameters; determining distortion offsets corresponding to each corner point based on coordinates of the actual mapping points and the ideal mapping points in the calibration image; and generating distortion mapping data of the camera module based on the determined distortion offsets.

[0006] In a second aspect, the embodiments of the present application provide a distortion correction method, which comprises: obtaining a distorted image; correcting the distorted image based on pre-generated distortion mapping data to obtain a corrected image, wherein the distortion mapping data is generated based on the method described in the first aspect.

[0007] In a third aspect, an electronic device is provided, including: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors to implement the method described in the first aspect or the second aspect.

[0008] In a fourth aspect, a computer readable medium is provided, having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or the second aspect.

[0009] In a fifth aspect, a computer program product is provided, including a computer program which, when executed by a processor, implements the method described in the first aspect or the second aspect.

[0010] The camera module distortion mapping data generation method and the distortion correction method provided by the embodiments of the present application can calibrate the camera module based on the calibration image captured by the camera module on the calibration board, obtain the parameters of the camera module, then find the actual mapping points of each corner point in the calibration image in the calibration board, determine the ideal mapping points of each corner point in the calibration image based on the parameters, then determine the distortion offset corresponding to each corner point based on the coordinates of the actual mapping points and the ideal mapping points in the calibration image, and generate the distortion mapping data of the camera module based on the determined distortion offset. On the one hand, since the generated distortion mapping data is obtained based on the calibration result and the distortion offset, the image distortion can be more accurately expressed, and the distortion image correction based on the data can improve the correction effect of the distortion image. On the other hand, the calibration image captured by the camera module in the conventional manner is directly used to calculate the distortion offset, and high requirement grid images do not need to be captured, thereby reducing the complexity of the operation. BRIEF DESCRIPTION OF DRAWINGS

[0011] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments thereof as read in conjunction with the accompanying drawings:

[0012] Figure 1 is a flowchart of one embodiment of the camera module distortion mapping data generation method according to the present application;

[0013] Figure 2 is a schematic diagram of the calibration image in the camera module distortion mapping data generation method according to the present application;

[0014] Figure 3 is a flowchart of one embodiment of the distortion correction method according to the present application;

[0015] Figure 4is a structural schematic diagram of an embodiment of a distortion mapping data generation apparatus of a camera module according to the present application;

[0016] Figure 5 is a structural schematic diagram of an embodiment of a distortion correction apparatus according to the present application;

[0017] Figure 6 is a structural schematic diagram of a computer system of an electronic device for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0018] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and are not a limitation on the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0019] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0020] It should be noted that all actions of obtaining signals, information or data in the present application are carried out in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0021] In recent years, important progress has been made in the research of computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence. Artificial intelligence (AI) is a new science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline that involves chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks and many other technology categories. Computer vision, as an important branch of artificial intelligence, is specifically to enable machines to recognize the world. Computer vision technology generally includes face recognition, liveness detection, fingerprint recognition and anti-forgery verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, character recognition, video processing, video content recognition, behavior recognition, three-dimensional reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning, and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as security, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile imaging, cloud services, smart home, wearable devices, driverless vehicles, autonomous driving, intelligent medical care, face payment, face unlocking, fingerprint unlocking, face and certificate verification, smart screens, smart televisions, cameras, mobile Internet, network live streaming, beauty, makeup, medical cosmetology, intelligent temperature measurement, and other fields.

[0022] In the above-mentioned fields of application of artificial intelligence, image acquisition is usually required. However, when using various camera modules for image capture, the captured images usually have different degrees of distortion. It is usually necessary to express the distortion of the images captured by the camera module in the form of data, and then correct the distortion of the images captured based on the data. The present application provides a distortion mapping data generation method of a camera module capable of improving the correction effect of distorted images.

[0023] Reference is made to Figure 1 which shows a flow 100 of one embodiment of a distortion mapping data generation method of a camera module according to the present application. The distortion mapping data generation method of the camera module can be applied to various electronic devices, for example, which can include but are not limited to: servers, smartphones, tablet computers, laptop computers, palm computers, desktop computers, etc. The distortion mapping data generation method of the camera module includes the following steps:

[0024] Step 101, calibrating the camera module based on a calibration image captured by the camera module on a calibration board to obtain parameters of the camera module.

[0025] In the embodiment, the execution subject of the distortion mapping data generation method of the camera module (such as the electronic device described above) can be communicatively connected with the camera module through wired connection or wireless connection to obtain a calibration image captured by the camera module on the calibration board. The camera module can be a module for image acquisition. The calibration board is a geometric model used to determine lens distortion and the conversion relationship between physical size and pixels in machine vision, image measurement, photogrammetry, three-dimensional reconstruction, etc. For example, a checkerboard calibration board can be used. The calibration image can be an image captured by the camera module described above with the calibration board as the shooting object.

[0026] In some scenarios, the calibration image can be a plurality of groups of images (e.g., at least three groups of images) captured for the same calibration board, each group of images showing a different position or posture of the calibration board to ensure that the calibration board is imaged in each region of the image. The number of images in each group can be determined by the number of camera lenses of the camera module. As an example, if the camera module is only configured with a monocular camera, the calibration image can be at least three monocular images captured by the camera module described above with the calibration board as the shooting object. As another example, if the camera module is only configured with a binocular camera, the calibration image can be at least three groups of binocular images captured by the camera module described above with the calibration board as the shooting object. In practice, the calibration image can be acquired in various ways. As an example, the position and posture of the calibration board can be fixed, and a plurality of different images can be acquired as calibration images by changing the position or posture of the camera module. As another example, the position and posture of the camera module can be fixed, and a plurality of different images can be acquired as calibration images by changing the position or posture of the calibration board.

[0027] In other scenarios, the calibration image can also be a group of images captured for a plurality of calibration boards of the same specification and different postures, and a plurality of calibration boards of the same specification and different postures can be displayed in the group of images at the same time. As an example, referring to FIG. 6, the calibration image displays four calibration boards of the same specification and different postures. Here, the number of images in each group can also be determined by the number of camera lenses of the camera module, which will not be described again here. Figure 2

[0028] ​After obtaining the calibration image collected by the camera module, the above execution subject can calibrate the camera module by using a camera calibration method such as Zhang's calibration method, to obtain the parameters of the camera module. In practice, in the image measurement process and machine vision application, in order to determine the mutual relationship between the three-dimensional geometric position of a point on the surface of a space object and the corresponding point in the image, the camera parameters must be determined, and the process of solving the parameters is called camera calibration. Through camera calibration, at least one of the following can be obtained: camera intrinsic parameters (which can be denoted as K, K is a 3x3 parameter matrix), extrinsic parameters (which can include a rotation matrix R and a translation matrix T, R is a 3x3 parameter matrix, and T is a 3x1 parameter matrix), distortion coefficients (which can be denoted as D, D is a 1x8 parameter matrix). The extrinsic parameters can be used to convert data from the world coordinate system to the camera coordinate system. The intrinsic parameters can be used to convert data from the camera coordinate system to the image coordinate system. Among them, the world coordinate system refers to the coordinate system of the three-dimensional world defined by the user, which is introduced to describe the position of the target object (such as the corner points in the calibration board) in the real world. The camera coordinate system is a coordinate system established on the camera, which is used to describe the position of the object from the perspective of the camera. The image coordinate system is the pixel coordinate system, which is introduced to describe the projection and transmission relationship of the object from the camera coordinate system to the image coordinate system in the imaging process.

[0029] It should be noted that the camera module can be configured with any number of cameras, such as a monocular camera, a binocular camera, a multi-camera, etc., which is not limited here. When the camera module includes at least two cameras, each camera can be calibrated separately to obtain the corresponding parameters of each camera, and the following process can be performed for each camera to obtain the distortion mapping coefficients corresponding to each camera.

[0030] Step 102, find the actual mapping points of each corner point in the calibration board in the calibration image, and determine the ideal mapping points of each corner point in the calibration image based on the parameters.

[0031] In this embodiment, the above execution subject can first find the actual mapping points of each corner point in the calibration board in the calibration image. Specifically, the coordinates of the actual mapping points in the calibration image can be obtained based on the functions (such as the findChessboardCorners() function and the cvFindCornerSubpix function) in the visual software library (such as the opencv library).

[0032] Then, the above execution subject can determine the ideal mapping points of each corner point in the calibration image based on the above parameters. The coordinates of the ideal mapping points in the calibration image can be calculated based on the above parameters and the coordinates of the corner points in the world coordinate system through the projection formula. For example, the coordinates of the corner points in the world coordinate system are denoted as X, and the coordinates of the ideal mapping points of the corner points in the calibration image are denoted as P, and the projection formula is P = K x [R T] x X.

[0033] Step 103, based on the coordinates of the actual mapping points and the ideal mapping points in the calibration image, determine the distortion offset corresponding to each corner point.

[0034] In the embodiment, for each corner point, the execution body can determine the difference between the coordinates of the actual mapping point and the coordinates of the ideal mapping point in the calibration image as the distortion offset corresponding to each corner point, which can be denoted as delta(dx, dy). Wherein, dx can represent the difference between the horizontal coordinates of the actual mapping point and the horizontal coordinates of the ideal mapping point, and dy can represent the difference between the vertical coordinates of the actual mapping point and the vertical coordinates of the ideal mapping point.

[0035] In some optional implementations, after determining the distortion offset corresponding to each corner point, the execution body can further perform smoothing processing on the obtained distortion offset to improve the distortion correction effect. Specifically, the following steps can be used for smoothing processing:

[0036] First, determine the amplitude of the distortion offset corresponding to each corner point. Here, the amplitude of the distortion offset can be denoted as d, d = (dx * dx + dy * dy) 1 / 2 .

[0037] Second, construct a three-dimensional space based on the size of the calibration image, and fit a three-dimensional surface S in the three-dimensional space based on the amplitude of the distortion offset corresponding to each corner point. As an example, the x-axis and y-axis of the three-dimensional space can correspond to the length and width of the calibration image respectively, and the z-axis of the three-dimensional space represents the amplitude of the distortion offset.

[0038] Third, update the distortion offset corresponding to each corner point based on the three-dimensional surface. Specifically, for each corner point, the new amplitude of the distortion offset of the corner point (i.e. the amplitude after smoothing) can be denoted as newd, newd = S(x, y). The new distortion offset can be denoted as new_delta(new_dx, new_dy). new_dx is the component of the smoothed distortion offset in the x-axis direction, new_dx = newd * cos(a). new_dy is the component of the smoothed distortion offset in the y-axis direction, newd * sin(a). a = atan(dy / dx).

[0039] Step 104, based on the determined distortion offset, generate the distortion mapping data of the camera module.

[0040] In the embodiment, the execution subject can generate the distortion mapping data of the camera module based on the determined distortion offset. The representation form of the distortion mapping data is not limited here, for example, the map table form can be used for representation. The distortion offset of each corner point can correspond to an element in the map table. The key of each pixel can be the pixel coordinate, and the value can be the distortion offset corresponding to the pixel.

[0041] In some optional implementations, the execution subject can first obtain the distortion offset of each non-corner point in the calibration image based on the distortion offset of each corner point by using an interpolation algorithm. Here, the cubic interpolation method can be used for interpolation of the distortion offset of the non-corner point. Then, the distortion mapping data of the camera module can be generated based on the distortion offset of each corner point and the distortion offset of each non-corner point. At this time, taking the map table form as an example, the elements in the map table can correspond to the pixel points of the image collected by the camera module one by one. Through interpolation, the distortion offset corresponding to all pixel points of the image collected by the camera module can be quickly obtained, and the mapping point of the non-corner point does not need to be looked up and responsible calculation, which improves the generation efficiency of the distortion mapping data.

[0042] The method provided by the above embodiment of the application can more accurately express the image distortion, because the generated distortion mapping data is obtained based on the calibration result and combined with the distortion offset. The distortion image correction based on the data can improve the correction effect of the distortion image. On the other hand, the distortion offset is calculated directly using the calibration image collected by the camera module in the conventional manner, and the high requirement grid image does not need to be collected, which reduces the operation complexity.

[0043] In some optional embodiments, in step 101, the camera module can be calibrated by using the following substeps S11 to S13 to obtain the parameters of the camera module.

[0044] In substep S11, the calibration image collected by the camera module on the calibration board is obtained.

[0045] In sub-step S12, the world coordinates of each corner point in the world coordinate system and the image coordinates of each corner point in the calibration image are determined. Taking the checkerboard calibration board as an example, the corner points are the vertices of each cell of the checkerboard calibration board. The origin of the world coordinate system can be set as the top-left corner point of the checkerboard calibration board, and the length direction and the width direction of the checkerboard calibration board can be respectively taken as the X-axis and the Y-axis of the world coordinate system. The Z-axis of the world coordinate system can be perpendicular to the checkerboard plane. The size of the checkerboard can be recorded as (col, row), and the length of the checkerboard can be recorded as grid_l. Thus, the world coordinate value of each corner point is X = (x = i * grid_l, y = j * grid_l, z = 0), where i is an integer in [0, col] and j is an integer in [0, row]. The image coordinates of each corner point (which can be recorded as P0) can be obtained based on functions (such as the findChessboardCorners() function and the cvFindCornerSubpix function) in a visual software library (such as the opencv library).

[0046] In sub-step S13, the camera module is calibrated based on the determined world coordinates and image coordinates to obtain the parameters of the camera module. Here, the Zhang calibration method can be used to complete the calibration to obtain the parameters of the camera module. Taking the camera module configured with a monocular camera as an example, the obtained parameters can be recorded as [K, D, R, T]. Among them, K is a 3x3 intrinsic matrix, R is a 3x3 rotation matrix, T is a 3x1 translation matrix, and D is a 1x8 distortion coefficient matrix. Taking the camera module configured with a binocular camera as an example, the obtained parameters can include the parameters [K1, D1, R1, T1] corresponding to the main camera in the binocular camera and the parameters [K2, D2, R2, T2] corresponding to the auxiliary camera. Among them, K1 and K2 are 3x3 intrinsic matrices, R1 and R2 are 3x3 rotation matrices, T1 and T2 are 3x1 translation matrices, and D1 and D2 are 1x8 distortion coefficient matrices.

[0047] In some optional embodiments, before step 102 is performed, if a non-calibration board region is displayed in the calibration image (i.e., the imaging of the calibration board does not occupy the camera field of view angle), the calibration board can also be virtually expanded to update the calibration board, so that as many mapping points as possible are obtained in each region of the calibration image. Here, the virtually expanded calibration board can be a virtual calibration board rather than a physical calibration board. For example, 1 / 4 of the checkerboard can be virtually expanded around the checkerboard, and more coordinates of corner points can be calculated and recorded according to the size of the unit cell.

[0048] In the step 102, after the extension of the calibration board, the corner points located in the original area of the updated calibration board can be taken as original corner points, the actual mapping points of the original corner points in the calibration image can be found, and the ideal mapping points of the original corner points in the calibration image can be determined based on the parameters. Specifically, the actual mapping point of the original corner point in the calibration image can be denoted as P0, and the coordinates of P0 in the calibration image can be obtained based on the functions in the above-mentioned vision software library, which will not be described herein. The ideal mapping point of the original corner point in the calibration image can be denoted as P1, and the coordinates of the original corner point in the world coordinate system can be denoted as X, and P1 = K x [R T] x X.

[0049] In addition, the corner points located in the extended area of the updated calibration board and capable of being mapped to the range of the calibration image can be taken as extended corner points, and the actual mapping points and the ideal mapping points of the extended corner points in the calibration image can be determined based on the parameters of the camera module. Specifically, the ideal mapping points of the extended corner points in the calibration image can be determined based on the intrinsic parameters and the extrinsic parameters of the camera module. Then, the actual mapping points of the extended corner points in the calibration image can be determined based on the intrinsic parameters, the distortion coefficients of the camera module, and the ideal mapping points of the extended corner points in the calibration image. The ideal mapping point of the extended corner point in the calibration image can be denoted as P2, the coordinates of the extended corner point in the world coordinate system can be denoted as X1, and P2 = K x [R T] x X1. The actual mapping point of the extended corner point in the calibration image can be denoted as D{P2}, where P2 represents the ideal mapping point of the extended corner point in the calibration image, and {} represents the addition of the distortion operation on P2 using the intrinsic parameters K and the distortion coefficients D.

[0050] Further, in the step 103, after the extension of the calibration board, the distortion offset (which can be denoted as delta_p1) of each original corner point can be determined based on the coordinates of the actual mapping point and the coordinates of the ideal mapping point of each original corner point in the calibration image, i.e., delta_p1 = P0 - P1. Similarly, the distortion offset (which can be denoted as delta_p2) of each extended corner point can be determined based on the coordinates of the actual mapping point and the coordinates of the ideal mapping point of each extended corner point in the calibration image, i.e., delta_p2 = D{P2} - P2. By virtually extending the calibration board and calculating the distortion offset of the extended corner point, the distortion of the edge area of the image can be expressed, and the distortion processing effect of the edge area of the image can be improved.

[0051] Further reference is made to Figure 3 which shows a flow 300 of yet another embodiment of a distortion correction method. The distortion correction method can be applied to various electronic devices, for example, which can include but are not limited to: servers, smartphones, tablet computers, laptop computers, palm computers, desktop computers, wearable devices, etc. The distortion correction method includes the following steps:

[0052] Step 301, obtaining a distorted image.

[0053] In this embodiment, the execution subject of the distortion correction method can be installed with a camera module, and can obtain an image captured by the camera module and take the image as the distorted image.

[0054] Step 302, correcting the distorted image based on the pre-generated distortion mapping data to obtain a corrected image.

[0055] In this embodiment, the execution subject can correct the distorted image based on the pre-generated distortion mapping data to obtain a corrected image. For example, for a certain pixel point (i, j) in the distorted image, the distortion mapping data can include a distortion offset corresponding to the pixel point, which can be denoted as (dx, dy). The execution subject can add the distortion offset to the pixel point coordinates (i, j) to obtain (i+dx, j+dy), so as to move the pixel value of the pixel point (i, j) to a new pixel point (i+dx, j+dy), thereby correcting the distortion of the pixel point.

[0056] It should be noted that the distortion mapping data is generated based on the distortion mapping data generation method of the camera module in any of the above embodiments, which will not be described here.

[0057] The method provided by the above embodiments of the present application can more accurately express the image distortion because the generated distortion mapping data is obtained based on the calibration result combined with the distortion offset. The distortion image correction with such data can improve the correction effect of the distorted image.

[0058] Further referring to Figure 4 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a distortion mapping data generation apparatus of a camera module, which corresponds to the method embodiment shown in Figure 1 , and the apparatus can be applied in various electronic devices.

[0059] As shown in Figure 4 , the data generation apparatus 400 of the present embodiment includes a calibration unit 401 configured to calibrate a camera module based on a calibration image captured by the camera module on a calibration board to obtain parameters of the camera module; a first determination unit 402 configured to find actual mapping points of each corner point in the calibration board in the calibration image, and determine ideal mapping points of each corner point in the calibration image based on the parameters; a second determination unit 403 configured to determine distortion offsets corresponding to each corner point based on coordinates of the actual mapping points and the ideal mapping points in the calibration image; and a generation unit 404 configured to generate distortion mapping data of the camera module based on the determined distortion offsets.

[0060] In some optional implementations of the present embodiment, the calibration unit 401 is further configured to acquire a calibration image of the calibration board captured by the camera module; determine world coordinates of each corner point in the world coordinate system and image coordinates of each corner point in the calibration image; and calibrate the camera module based on the determined world coordinates and image coordinates to obtain the parameters of the camera module.

[0061] In some optional implementations of the present embodiment, the device further includes an updating unit configured to determine the distortion offset amplitude of each corner point; construct a three-dimensional space based on the size of the calibration image, and fit a three-dimensional curved surface in the three-dimensional space based on the distortion offset amplitude of each corner point; and update the distortion offset of each corner point based on the three-dimensional curved surface.

[0062] In some optional implementations of the present embodiment, the generating unit 404 is further configured to obtain the distortion offset of each non-corner point in the calibration image by an interpolation algorithm based on the distortion offset of each corner point; and generate the distortion mapping data of the camera module based on the distortion offset of each corner point and the distortion offset of each non-corner point.

[0063] In some optional implementations of the present embodiment, the device further includes an expanding unit configured to virtually expand the calibration board to update the calibration board if a non-calibration board region is displayed in the calibration image; and the first determining unit 402 is further configured to take the corner points located in the original region of the updated calibration board as original corner points, find the actual mapping points of each original corner point in the calibration image, and determine the ideal mapping points of each original corner point in the calibration image based on the parameters; and take the corner points located in the expanded region of the updated calibration board and capable of being mapped to the range of the calibration image as expanded corner points, determine the actual mapping points and ideal mapping points of each expanded corner point in the calibration image based on the parameters.

[0064] In some optional implementations of the present embodiment, the parameters include intrinsic parameters, extrinsic parameters, and distortion coefficients; and the first determining unit 402 is further configured to determine the ideal mapping points of each expanded corner point in the calibration image based on the intrinsic parameters and the extrinsic parameters; and determine the actual mapping points of each expanded corner point in the calibration image based on the intrinsic parameters, the distortion coefficients, and the ideal mapping points of each expanded corner point in the calibration image.

[0065] The device provided by the above embodiments of the present application calibrates the camera module based on the calibration image of the calibration board captured by the camera module, obtains the parameters of the camera module, then finds the actual mapping points of each corner point in the calibration image in the calibration board, determines the ideal mapping points of each corner point in the calibration image based on the parameters, and then determines the distortion offset corresponding to each corner point based on the coordinates of the actual mapping points and the ideal mapping points in the calibration image, so as to generate the distortion mapping data of the camera module based on the determined distortion offset. On the one hand, since the generated distortion mapping data is obtained based on the calibration result in combination with the distortion offset, the image distortion can be more accurately expressed, and the distortion image correction can be performed based on the data, which can improve the correction effect of the distortion image. On the other hand, the calibration image captured by the camera module in the conventional manner is directly used to calculate the distortion offset, and high requirement grid images do not need to be captured, which reduces the complexity of the operation.

[0066] Further reference is made to Figure 5 As an implementation of the method shown in the above figures, the present application provides an embodiment of a distortion correction device, which corresponds to the method embodiment shown in Figure 1 The device can be applied in various electronic devices.

[0067] As shown in Figure 5 The distortion correction device 500 of the present embodiment includes an acquisition unit 501 configured to acquire a distortion image, and a distortion correction unit 502 configured to correct the distortion image based on the pre-generated distortion mapping data to obtain a corrected image. It should be noted that the distortion mapping data is generated based on the distortion mapping data generation method of the camera module in any of the above embodiments, which will not be described herein.

[0068] The device provided by the above embodiments of the present application can more accurately express the image distortion, and the distortion image correction can be performed based on the data, which can improve the correction effect of the distortion image.

[0069] The present embodiment also provides an electronic device including one or more processors, a storage device having one or more programs stored thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned distortion mapping data generation method of the camera module.

[0070] Reference is made to Figure 6 which shows a structural schematic diagram of an electronic device for implementing some embodiments of the present application. Figure 6 The electronic device shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0071] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0072] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, disks, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0073] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for generating distortion mapping data of a camera module.

[0074] In particular, according to some embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from ROM 602. When the computer program is executed by processing device 601, it performs the functions defined above in the methods of some embodiments of this application.

[0075] This application also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the above-described method for generating distortion mapping data of a camera module.

[0076] Note that the computer-readable medium in some embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus, or device to function or operate. In some embodiments of the present application, the computer-readable signal medium can include a computer-readable program code in a baseband or propagated as a carrier wave in a propagated signal, where the computer-readable program code can be used by or in connection with an instruction execution system, apparatus, or device. The propagated signal can take any of a variety of forms, including but not limited to, electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF (radio frequency), and the like, or any suitable combination of the foregoing.

[0077] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0078] The computer readable medium can be included in the electronic device; or can exist separately from the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to calibrate a camera module based on a calibration image captured by the camera module on a calibration board, to obtain parameters of the camera module; to find actual mapping points of each corner point in the calibration image in the calibration board, and to determine ideal mapping points of each corner point in the calibration image based on the parameters; to determine distortion offsets corresponding to each corner point based on coordinates of the actual mapping points and the ideal mapping points in the calibration image; and to generate distortion mapping data of the camera module based on the determined distortion offsets.

[0079] Computer program code for carrying out operations of some embodiments of the application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0080] The computer program code can also be loaded onto a computer or other programmable information processing system to cause a series of operational steps to be performed on the computer or other programmable information system to produce the operations described.

[0081] The units described in some embodiments of the present application can be implemented by means of software, and can also be implemented by hardware. The described units can also be arranged in a processor, for example, can be described as: a processor comprising a first determining unit, a second determining unit, a selecting unit and a third determining unit. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0082] The functions described above in the present document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, non-limiting examples of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0083] The above description is merely some of the preferred embodiments of the present application and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features can be replaced with technical features with similar functions disclosed in the embodiments of the present application (but not limited to) to form technical solutions.

Claims

1. A method for generating distortion mapping data for a camera module, characterized in that, The method includes: Based on the calibration images captured by the camera module on the calibration board, the camera module is calibrated to obtain the parameters of the camera module; wherein, the calibration images are multiple sets of images captured for the same calibration board, or, the calibration images are a set of images captured for multiple calibration boards of the same specifications but different orientations. If the calibration image shows a non-calibration board area, the calibration board is virtually expanded to update the calibration board; Find the actual mapping point of each corner point in the calibration plate in the calibration image, and determine the ideal mapping point of each corner point in the calibration image based on the parameters; Based on the coordinates of the actual mapping point and the ideal mapping point in the calibration image, the distortion offset corresponding to each corner point is determined. Based on the determined distortion offset, distortion mapping data of the camera module is generated; The step of finding the actual mapping point of each corner point in the calibration plate in the calibration image, and determining the ideal mapping point of each corner point in the calibration image based on the parameters, includes: The corner points located in the original area of ​​the updated calibration board are taken as the original corner points. The actual mapping points of each original corner point in the calibration image are found, and the ideal mapping points of each original corner point in the calibration image are determined based on the parameters. Corner points located in the extended region of the updated calibration board and capable of being mapped to the range of the calibration image are taken as extended corner points. Based on the parameters, the actual mapping point and ideal mapping point of each extended corner point in the calibration image are determined.

2. The method according to claim 1, characterized in that, The calibration of the camera module is performed based on the calibration image captured by the calibration board, to obtain the parameters of the camera module, including: Acquire calibration images captured by the camera module against the calibration board; Determine the world coordinates of each corner point in the calibration plate in the world coordinate system and the image coordinates in the calibration image; Based on the determined world coordinates and image coordinates, the camera module is calibrated to obtain the parameters of the camera module.

3. The method according to claim 1, characterized in that, After determining the distortion offset corresponding to each corner point, the method further includes: Determine the distortion offset magnitude corresponding to each corner point; A three-dimensional space is constructed based on the size of the calibrated image, and a three-dimensional surface is fitted in the three-dimensional space based on the distortion offset amplitude corresponding to each corner point. Based on the three-dimensional surface, update the distortion offset corresponding to each corner point.

4. The method according to claim 1, characterized in that, The step of generating distortion mapping data for the camera module based on the determined distortion offset includes: Based on the distortion offset of each corner point, the distortion offset of each non-corner point in the calibration image is obtained through an interpolation algorithm; The distortion mapping data of the camera module is generated based on the distortion offset of each corner point and the distortion offset of each non-corner point.

5. The method according to claim 1, characterized in that, The parameters include intrinsic parameters, extrinsic parameters, and distortion coefficients; The step of determining the actual and ideal mapping points of each extended corner point in the calibration image based on the parameters includes: Based on the intrinsic and extrinsic parameters, determine the ideal mapping point of each extended corner point in the calibration image; Based on the intrinsic parameters, the distortion coefficients, and the ideal mapping points of each extended corner point in the calibration image, the actual mapping points of each extended corner point in the calibration image are determined.

6. A distortion correction method, characterized in that, The method includes: Acquire distorted images; The distorted image is corrected based on the pre-generated distortion mapping data to obtain a corrected image, wherein the distortion mapping data is generated based on the method described in any one of claims 1-5.

7. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images