Camera distortion correction device and method therefor

Abstract

Proposed is a camera distortion correction method. The device may comprise a coordinate information collector for acquiring image coordinates of a plurality of points of a first group in an acquired image and actual measured GPS coordinates corresponding thereto, and a processor. The processor is configured to: acquire, from image coordinates of a plurality of points of a second group in the acquired image, estimated transformed GPS coordinates corresponding thereto by using a homography matrix acquired on the basis of the image coordinates of the plurality of points of the first group and actual measured transformed GPS coordinates based on the actual measured GPS coordinates; perform distortion correction on the image coordinates by using the image coordinates of the plurality of points of the first group and the plurality of points of the second group and the actual measured transformed GPS coordinates or the estimated transformed GPS coordinates corresponding thereto; and update the homography matrix on the basis of corrected image coordinates, which are acquired according to the distortion correction, of the plurality of points of the first group or the second group and the actual measured transformed GPS coordinates or the estimated transformed GPS coordinates.

Description

Camera distortion correction device and method therefor

[0001] The present invention relates to a device for camera distortion correction and a method therefor, and more specifically, to a technique for performing distortion correction using image coordinates of an acquired image and GPS coordinates thereof and estimating GPS coordinates of points within the image therefrom.

[0002] Figure 1 illustrates a traffic safety service system based on communication technology, one of the fields to which the present invention applies. As an example, it illustrates the use of V2X (vehicle to everything) and Soft V2X technologies.

[0003] The road side unit (RSU) (100) transmits road and traffic information provided by the Cooperative Intelligent Transport System (C-ITS) (200) to road users. Furthermore, the RSU (100) collects information on surrounding vehicles. Furthermore, the RSU (100) links C-ITS and Soft V2X services, which use different technologies, and utilizes information from various roadside sensors, including CCTV, to provide information that can predict road collisions.

[0004] In addition, the RSU (100) can detect various road users such as pedestrians, vehicles, two-wheeled vehicles, and kickboards using intelligent CCTV. The RSU (100) transmits road user detection information to surrounding users to enable collision prediction or avoidance. The RSU (100) can transmit road user detection information to a C-ITS terminal (311) and a Soft V2X terminal (321). The C-ITS terminal (311) is installed in a vehicle (310), and the Soft V2X terminal (321) is illustrated as being installed or located in a vehicle (320).

[0005] In this way, C-ITS and Soft V2X are technologies that help predict, avoid, or prevent traffic accidents, respectively, but C-ITS terminals (311) and Soft V2X terminals (321) cannot directly exchange information with each other. In this way, by linking C-ITS and Soft V2X through the RSU (100), information exchange between C-ITS terminals and Soft V2X terminals is enabled, thereby enabling collision prediction and avoidance between C-ITS terminal users and Soft V2X terminal users.

[0006] As described above, achieving collision prediction, avoidance, and other tasks requires, above all, the advancement of object location prediction, estimation, and tracking technologies. The present invention addresses technologies for detecting objects within an image and predicting, estimating, and tracking their locations.

[0007] The present invention proposes a method for more accurately estimating the location of an object when tracking an object in an acquired image.

[0008] More specifically, the present invention seeks to provide a method for estimating the actual location of an object, etc., within an image by using image coordinates within the image and corresponding measured GPS coordinate information.

[0009] In addition, the present invention seeks to correct image distortion by using image coordinates within the image and corresponding measured GPS coordinate information.

[0010] In addition, the present invention aims to more accurately estimate GPS coordinate information or a corresponding actual location using image coordinates within an image, that is, image information on which distortion correction has been performed, and to provide services or warning alarms, traffic information, etc. to objects or users in the vicinity based on the corrected image coordinates.

[0011] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0012] A device for camera distortion correction is proposed. The device may include a coordinate information collector that obtains image coordinates of a plurality of points of a first group in an acquired image and corresponding ground truth GPS coordinates; and a processor configured to obtain estimated transformed GPS coordinates corresponding to the image coordinates of a plurality of points of a second group in the acquired image by using a homography matrix obtained based on the image coordinates of the plurality of points of the first group and ground truth transformed GPS coordinates based on the ground truth GPS coordinates, perform distortion correction of the image coordinates by using the image coordinates of the plurality of points of the first group and the plurality of points of the second group and the ground truth transformed GPS coordinates or the estimated transformed GPS coordinates corresponding thereto, and update the homography matrix based on the corrected image coordinates of the plurality of points of the first group or the second group obtained according to the distortion correction and the ground truth transformed GPS coordinates or the estimated transformed GPS coordinates.

[0013] A method for camera distortion correction is proposed, the method comprising: obtaining image coordinates of a plurality of points of a first group in an acquired image and corresponding ground truth GPS coordinates; obtaining estimated transformed GPS coordinates corresponding to the image coordinates of a plurality of points of a second group in the acquired image using a homography matrix obtained based on the image coordinates of the plurality of points of the first group and ground truth transformed GPS coordinates based on the ground truth GPS coordinates; performing distortion correction of image coordinates using the image coordinates of the plurality of points of the first group and the plurality of points of the second group and the ground truth transformed GPS coordinates or the estimated transformed GPS coordinates corresponding thereto; and updating the homography matrix based on the corrected image coordinates of the plurality of points of the first group or the second group obtained according to the distortion correction and the ground truth transformed GPS coordinates or the estimated transformed GPS coordinates.

[0014] The above problem solving methods are only some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by a person having ordinary knowledge in the relevant technical field based on the detailed description of the present invention described below.

[0015] The present invention has the following effects.

[0016] The present invention can estimate the location of an object more accurately through an image.

[0017] The present invention can estimate the actual location of an object, etc., within an image by using image coordinates within the image and corresponding measured GPS coordinate information.

[0018] In addition, the present invention can correct image distortion by using image coordinates within the image and corresponding measured GPS coordinate information.

[0019] In addition, the present invention can more accurately estimate the actual location of an object, user, etc. by using image information on which distortion correction has been performed, and accordingly provide services, warning alarms, traffic information, etc. to the user, etc.

[0020] The present invention can correct or update observation values ​​in tracking objects to values ​​that better reflect the actual location of the object.

[0021] The present invention can improve the accuracy of object location estimation in object tracking.

[0022] The effects according to the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the detailed description of the invention below.

[0023] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0024] Figure 1 illustrates a system diagram of a field to which the present invention is applied.

[0025] Figure 2 illustrates a flowchart for camera-based object tracking and position estimation to which the present invention is applied.

[0026] Figure 3 illustrates a pinhole camera model for explaining camera calibration.

[0027] Figure 4 illustrates a flowchart for position estimation using camera-acquired images according to the present invention.

[0028] Figure 5 illustrates the conversion between image coordinates and GPS coordinates using a homography matrix according to the present invention.

[0029] Figure 6 illustrates a flowchart for camera distortion correction according to the present invention.

[0030] Figure 7 shows the camera distortion phenomenon and the results of distortion correction according to the present invention.

[0031] FIG. 8 describes the update of a homography matrix or a pair of image coordinates and ground truth transformed GPS coordinates according to a change in an external parameter of a camera according to the present invention.

[0032] Figure 9 illustrates a block diagram of a camera distortion correction device according to the present invention.

[0033] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0034] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0035] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0036] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0037] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0038]

[0039] Figure 2 illustrates a procedure in a system for camera-based object tracking and position estimation to which the present invention is applied.

[0040] First, an image is acquired using an image sensor such as a camera (S10).

[0041] Object detection is performed in the acquired image (S20).

[0042] Then, tracking of the detected object is performed (S30).

[0043] Finally, location estimation is performed to obtain location information such as GPS coordinates from the location of the object in the acquired image (S40).

[0044] Object tracking uses computer vision technology to detect and track objects in images. To achieve this, various algorithms are used to detect and track moving objects. Key algorithms include the Kalman filter, particle filter, and DeepSORT. Each tracked object is assigned an identifier (ID) and tracked individually. This allows the path and speed of each object to be estimated.

[0045]

[0046] Figure 3 illustrates a pinhole camera model for explaining camera calibration.

[0047] Camera calibration is the process of adjusting a camera to correctly capture and interpret images. This process improves image accuracy by correcting lens distortion, color distortion, and geometric distortion. The primary goal is to accurately reproduce real-world objects, which plays a crucial role in diverse fields such as computer vision, robotics, augmented and virtual reality, and autonomous vehicles.

[0048] Referring to Figure 3, a total of four coordinate systems are shown, with the camera coordinate system, normal coordinate system, pixel coordinate system, and world coordinate system shown from the lower right, respectively.

[0049] The world coordinate system and the camera coordinate system are three-dimensional coordinate systems, while the regular coordinate system and pixel coordinate system are two-dimensional coordinate systems. The world coordinate system is the coordinate system we use as a reference when expressing the position of an object (subject), and the camera coordinate system is a coordinate system based on the camera. The pixel coordinate system, also called the image coordinate system, is the coordinate system for the image we actually see with our eyes. It refers to the image plane formed in the right and downward directions with the upper left corner as the origin.

[0050] A point P = (X, Y, Z) in 3D space is projected to a point pimg = (x, y) on the image plane through the focus of the camera or lens.

[0051] The canonical coordinate system corresponds to the image coordinate system that removes the influence of the camera's intrinsic parameters. The canonical coordinate system is a unit-less (normalized) coordinate system that defines a virtual image plane whose distance from the camera focal point is 1. It is defined as a point P'=(u, v) in the canonical coordinate system. Accordingly, knowing the camera's intrinsic parameters allows conversion between image (pixel) coordinates and canonical coordinates.

[0052]

[0053] Here, fx is the focal length in the horizontal direction, fy is the focal length in the vertical direction, and cx and cy are principal points, which represent the image (pixel) coordinates where the optical axis and the image plane meet.

[0054] Additionally, the rotation and translation transformation matrix ([R│t]) for transforming the world coordinate system into the camera coordinate system is modeled, which can be expressed as follows.

[0055]

[0056] In mathematical terms, camera calibration is the process of finding the transformation relationship between 3D space coordinates and 2D image coordinates, or the parameters that describe this transformation relationship.

[0057] The matrix in mathematical expression 1 is referred to as an internal parameter, and [R│t] in mathematical expression 2 is referred to as an external parameter.

[0058] Using a wide-angle lens or ultra-wide-angle lens with a wide camera field of view allows you to see a wider range, but this has the problem of causing relatively more image distortion.

[0059] Beyond visual issues, this image distortion poses a particular problem when precise numerical calculations are required for image analysis. For example, converting image coordinates to physical coordinates to determine the actual location of an object detected in an image can lead to significant errors depending on the degree of image distortion.

[0060] Distortion caused by the camera, i.e. the lens, includes radial distortion and tangential distortion.

[0061] Assuming there is no lens distortion, a point (Xc, Yc, Zc) in three-dimensional space is projected to a point (x_n_u, y_n_u) on a regular image plane by deep projection.

[0062]

[0063] If the normal coordinates reflecting the lens distortion are expressed as (x_n_d, y_n_d), the lens distortion model can be expressed as follows.

[0064]

[0065] Here, am.

[0066] In Equation 4, the first term on the right side represents radial distortion, and the second term represents tangential distortion. K1, k2, and k3 are radial distortion coefficients, p1, and p2 are tangential distortion coefficients, and r_u is the distance (radius) to the center (principal point) when there is no distortion.

[0067] Again, using the internal parameters of Equation 1, the normal coordinates reflecting the distortion can be expressed as image pixel coordinates (x_p_d, y_p_d).

[0068]

[0069] Here, skew_cfx is an asymmetry coefficient, which is usually set to 0 (i.e., omitted).

[0070] That is, the coordinates on the regular image plane can be converted into coordinates that reflect the distortion on the image plane.

[0071] Through this relationship, image distortion correction or image coordinate distortion correction can be performed, and image coordinate distortion correction is described.

[0072] In actual image processing, obtaining the corrected image coordinates may be more important than obtaining the image with the distortion corrected.

[0073] However, obtaining the distorted image coordinates from the distorted image coordinates is a difficult problem without a closed-form solution, and the solution is usually obtained approximately.

[0074] One way to obtain the distorted coordinates p_u from the distorted image coordinates p_d is to assume p_u = p_d and distort p_u using Equation 4. The amount of error between the point obtained in this way and the actual p_d is calculated and the error is inversely reflected in p_u. For example, if the distorted point p_u is larger than p_d by δ, then p_u is reduced by δ. This process is repeated until the error falls below a threshold value.

[0075] In conclusion, camera calibration or camera distortion correction can correct the distortion of image coordinates based on the calculation of internal or external parameters using pairs of image (pixel) coordinates and their corresponding 3D coordinates (i.e., world coordinates).

[0076] Although camera calibration and camera distortion correction are briefly described in this specification, reference may be made to many materials published prior to the filing of this application.

[0077]

[0078] Figure 4 illustrates a flowchart for position estimation using camera-acquired images according to the present invention. Below, the process of the present invention is described as being performed by a camera distortion correction device. However, it should be noted that the name of the device does not limit the present invention.

[0079] The camera distortion correction device can perform camera calibration (S41). Camera calibration is described with reference to FIG. 3.

[0080] The difference between the existing camera calibration and the camera calibration of the present invention is that the present invention uses converted GPS coordinates based on GPS coordinates instead of 3D coordinates (world coordinates). One reason for this is that the cameras related to the present invention include, for example, CCTVs that are fixed to supports such as roads or streets, and the existing camera calibration uses images captured at close range by placing a chess board or a 3D jig structure, and the 3D coordinates of the intersection of the chess board and its image coordinates that are known in advance. However, in the case of cameras installed on supports such as CCTVs, the camera shooting range is wide, so testing itself using a specific structure such as a chess board is difficult, and the calibration effect is bound to be reduced.

[0081] Therefore, the present invention utilizes GPS coordinates measured from points within the camera's image range, i.e., the points used for camera calibration. Then, by matching these points with the image coordinates, multiple pairs of GPS coordinates and image coordinates are generated. An example is as follows.

[0082] PointIDXYLatitudeLongitude2101188492537.53367126.85392102133877737.533 68126.8539210368985737.53368126.8538210499458637.5337126.85392105756488 37.53372126.85382106108430437.53375126.8539210782023037.53378126.8538210891028337.53376126.8539210994622737.53378126.8539211098223437.53377126 .8539211181818937.5338126.8538211289419037.53379126.8539211396018437.5338126.8539211497316437.53381126.8539211584114737.53382126.853821168391 2437.53384126.8538211780813337.53384126.853821189988837.53387126.853921198637637.53389126.853821209856937.53389126.853921219721337.534126.8539

[0083] Then, assuming that the GPS coordinates are 3D coordinates on a plane, one of the measured points is set to p0 (i.e., set as the origin), and the x-axis and y-axis coordinates among the 3D coordinates are set to the longitude and latitude coordinates of the GPS coordinates, respectively, and the z-axis coordinate is set to 0. Then, the 3D coordinates of the remaining points are expressed as relative coordinates with respect to p0. For example, the coordinates of the remaining points pi can be set as follows.

[0084]

[0085] The 3D coordinates obtained in this way are referred to as “ground truth transformed GPS coordinates.” On the other hand, the 3D coordinates obtained from image coordinates using the homography matrix described later, without actual measurement, are referred to as “estimated transformed GPS coordinates.”

[0086] In addition, according to the above relationship, “actually measured GPS coordinates” can be converted into actual GPS coordinates, and “estimatedly measured GPS coordinates” described later can also be converted into estimated GPS coordinates.

[0087] Accordingly, the camera distortion correction device can perform camera calibration using a pair of image coordinates and ground truth transformed GPS coordinates.

[0088] The camera distortion correction device can correct the distortion of an image acquired through camera calibration to obtain an image with the distortion corrected, or can correct the image coordinates from a pair of image coordinates and actual measurement converted GPS coordinates (S42).

[0089] The camera distortion correction device can calculate a homography matrix using a pair of corrected image coordinates and actual transformed GPS coordinates (S43). The homography matrix is ​​a matrix for coordinate transformation, and expresses the transformation of image coordinates into GPS coordinates or vice versa. Therefore, if the homography matrix is ​​known, all image coordinates in the image can be transformed into GPS coordinates. Fig. 5 illustrates the transformation between image coordinates and GPS coordinates using the homography matrix according to the present invention.

[0090] Figure 5 (a) shows the conversion of image coordinates into GPS coordinates through a homography matrix, and Figure 5 (b) shows the conversion of GPS coordinates into image coordinates through a homography matrix.

[0091] Then, the camera distortion correction device can obtain GPS coordinates of the location of a point or object in the image based on the homography matrix, and ultimately estimate the GPS location (S44).

[0092] Meanwhile, S41 and S42 can be combined into one procedure.

[0093] Additionally, steps S41, S42, and S43 can be performed repeatedly to obtain more accurate location information. This will be described in more detail with reference to Fig. 6.

[0094]

[0095] Figure 6 illustrates a flowchart for camera distortion correction according to the present invention.

[0096] Hereinafter, the process of the present invention is described as being performed by a camera distortion correction device, but it is noted in advance that the name of the device does not limit the present invention.

[0097] The camera distortion correction device can calculate an initial homography matrix (S610). As mentioned above, the initial homography matrix is ​​calculated using pairs of ground truth transformed GPS coordinates and corresponding image coordinates based on GPS coordinates measured at the actual locations of multiple points within the acquired image.

[0098] The camera distortion correction device can select multiple points or their image coordinates within the acquired image (S620). The multiple points used to calculate the initial homography matrix may be referred to as a first group, and the multiple points selected in S620 may be referred to as a second group. For example, the second group may be randomly selected from among the multiple points within the acquired image.

[0099] In one embodiment, the points of the first group (or their corresponding image coordinates) and the points of the second group (or their corresponding image coordinates) may overlap at least partially.

[0100] In another embodiment, the points of the first group (or their corresponding image coordinates) and the points of the second group (or their corresponding image coordinates) may not overlap.

[0101] The camera distortion correction device can obtain estimated transformed GPS coordinates from the image coordinates of a plurality of points in the second group using the initial homography matrix (S630). Preferably, the estimated transformed GPS coordinates can be obtained only for the image coordinates of a plurality of points in the second group that do not overlap with the plurality of points in the first group.

[0102] Then, the camera distortion correction device can extend the "image coordinate and (ground truth) transformed GPS coordinate pair" into image coordinate and transformed GPS coordinate pairs for points of the first group and the second group (S630). In this case, the transformed GPS coordinate may include ground truth transformed GPS coordinates or estimated transformed GPS coordinates. This is because the coordinate pair includes ground truth transformed GPS coordinates based on measured GPS coordinates for points of the first group, and also includes points of the second group. That is, the image coordinate and transformed GPS coordinate pair is for points including points of the second group as well as points of the first group. However, as described above, at least some of the points of the first group and the points of the second group may be the same points.

[0103] The camera distortion correction device can perform camera distortion correction using image coordinate and transformed GPS coordinate pair information (S640). Camera distortion correction may include camera calibration, and reference should be made to the matters described above with reference to FIGS. 3 and 4.

[0104] Additionally, camera distortion correction includes distortion correction for image coordinates. That is, the camera distortion correction device can correct image coordinates for the union of points of the first group and the second group.

[0105] Then, the camera distortion correction device can update the homography matrix using the corrected image coordinate and transformed GPS coordinate pair information (S650).

[0106] Meanwhile, the paired information of corrected image coordinates and transformed GPS coordinates used to update the homography matrix may be limited to points in the first group. Since the transformed GPS coordinates for the points in the first group are based on actual GPS coordinates, more precise position estimation, camera calibration, and distortion correction are expected.

[0107] Alternatively, the pairs of corrected image coordinates and transformed GPS coordinates used to update the homography matrix may include not only points from the first group but also points from the second group. That is, updating the homography matrix may be performed using points belonging to the union of the first and second groups. As the number of points used to update the homography matrix increases, errors in position estimation, camera calibration, and distortion correction are expected to be reduced.

[0108] The procedure corresponding to S620 to S650 of FIG. 6 may be referred to as a “distortion correction process.” The distortion correction process may be repeated rather than being performed once.

[0109] Accordingly, the camera distortion correction device can calculate a distance error for a plurality of points of the first group (S660). The distance error refers to the difference between the actual GPS coordinates (or actual transformed GPS coordinates) and the estimated GPS coordinates (or estimated transformed GPS coordinates) of the plurality of points of the first group. The estimated GPS coordinates or the estimated transformed GPS coordinates can be obtained from the (corrected) image coordinates of the plurality of points of the first group using the updated homography matrix. It can be interpreted that the smaller the distance error, the higher the accuracy of the position estimation using the homography matrix.

[0110] Since the plurality of points in the first group have actual measured GPS coordinate information, the camera distortion correction device can determine how much the information based on the actual GPS coordinates differs from the information based on the estimated GPS coordinates obtained using the homography matrix. In addition, since the distance error is calculated for the plurality of points, the value of the distance error can be determined by adding up the differences between the actual GPS coordinates (or actual transformed GPS coordinates) and the estimated GPS coordinates (or estimated transformed GPS coordinates) of each point.

[0111] The camera distortion correction device can determine whether to repeat the distortion correction process based on the calculated distance error (S670).

[0112] There may be several criteria for determining whether to repeat. One such criterion is that the number of repetitions of the distortion correction process may be preset, and the count value may be increased by 1 after any one of S630 to S660, for example, updating the homography matrix (S650), is completed, and the count value may be determined based on whether the preset number of repetitions has been reached.

[0113] Meanwhile, the calculation of the distance error (S660) is necessary because, even if the number of repetitions of the distortion correction process is preset, the "distortion correction process" with the minimum distance error resulting from the update of each homography matrix must be identified. Since the repetitions must be performed as many times as the number of repetitions is preset, all calculated or updated homography matrices or camera calibration or distortion correction information must be stored. After the repetitions are completed, the camera distortion correction device can select the homography matrix or camera calibration or distortion correction information of the distortion correction process with the minimum distance error (S680).

[0114] Alternatively, the repetition can be performed until the distance error becomes small. That is, the most recently calculated distance error is compared with the distance error calculated this time, and if the distance error calculated this time is smaller, the distortion correction process is repeated. To this end, an initial comparison value of the distance error needs to be set. In addition, the value of the minimum distance error needs to be stored, and the homography matrix or camera calibration or distortion correction information from which the value of the minimum distance error is calculated needs to be stored. The camera distortion correction device can select the homography matrix or camera calibration or distortion correction information of the distortion correction process having the minimum distance error (S680).

[0115] A camera distortion correction device can acquire an image using selected camera calibration or distortion correction information, or can acquire estimated transformed GPS coordinates or estimated GPS coordinates of image coordinates within the acquired image using a selected homography matrix. Consequently, the camera distortion correction device can acquire image coordinates of an object within the acquired image, and thereby acquire estimated GPS coordinates, i.e., estimated position information, of the object. Since the correction of image coordinates becomes efficient and precise according to the camera calibration or distortion correction according to the present invention, the accuracy of position estimation of an object within the image can be improved.

[0116]

[0117] Figure 7 shows the camera distortion phenomenon and the results of distortion correction according to the present invention.

[0118] Figure 7 (a) is an image before distortion correction, and Figure 7 (b) is an image after distortion correction of image coordinates according to the present invention.

[0119] It can be confirmed that the road (R1) of Fig. 7 (a) is curved as it goes from the bottom to the top of the image, and it can be confirmed that the road (R1') of Fig. 7 (b) has greatly improved curve (i.e., distortion).

[0120]

[0121] FIG. 8 describes the update of a homography matrix or a pair of image coordinates and ground truth transformed GPS coordinates according to a change in an external parameter of a camera according to the present invention.

[0122] The camera's external parameters are primarily variables that describe its position and orientation. These external parameters define the camera's position and orientation in space and determine the point of view from which the camera observes. Key external parameters may include the camera's spatial position, its orientation, and its field of view.

[0123] As previously explained, the camera of the present invention is installed on a support like a CCTV, so once installed, it is used for a long period of time. However, depending on the usage environment, it may be reinstalled or its initial installation location or orientation may change. In such cases, reinstalling the camera to its initial installation location or performing camera calibration or distortion correction is a difficult problem. Therefore, the present invention proposes an efficient method for updating the homography matrix or updating pairs of image coordinates and transformed GPS coordinates.

[0124] Hereinafter, the process of the present invention is described as being performed by a camera distortion correction device, but it is noted in advance that the name of the device does not limit the present invention.

[0125] Figure 8 (a) shows an image captured by a camera at its initial installation location or settings, while Figure 8 (b) shows an image captured by a camera with a changed installation location or settings. Points p1 and p1' represent the same actual location points. However, the image coordinates represent different situations. Although only one point is illustrated, it is assumed that such image coordinate changes have occurred for multiple points.

[0126] The camera distortion correction device can extract feature values ​​for the first group of points (p1) where the aforementioned GPS coordinate measurements are performed. The feature values ​​can include information related to the shape, boundary, color texture, etc. of the object. In other words, the camera distortion correction device can extract feature values ​​for an area including at least the points of the first group.

[0127] Then, the camera distortion correction device can match and store the extracted feature values ​​with the image coordinates and ground truth transformed GPS coordinate pairs of the points of the first group.

[0128] That is, a data set can be constructed for pairs of image coordinates and actual transformed GPS coordinates of points in the first group and their corresponding extracted feature values.

[0129] The camera distortion correction device can detect changes in the image coordinates of the first group of points within the image based on the extracted feature values. That is, referring to Fig. 8(b), it can be confirmed that point (p1') is a point having the feature value of point (p1).

[0130] As changes in the image coordinates of points in the first group are detected, the camera distortion correction device can identify the changed image coordinates of the corresponding point (p1'). The camera distortion correction device can update the homography matrix based on the changed image coordinates and the ground truth transformed GPS coordinates (from the data set mentioned above).

[0131] According to the present invention, when the position or direction of the camera is changed from that at the time of initial installation, the image coordinates are corrected or the homography matrix is ​​updated through the above process, so that the accuracy of the estimated GPS information corresponding to the image coordinates can be continuously guaranteed.

[0132]

[0133] Figure 9 illustrates a block diagram of a camera distortion correction device according to the present invention.

[0134] The camera distortion correction device (10) may include a coordinate information collector (11) and a processor (12). Additionally, the camera distortion correction device (10) may include a transceiver (13).

[0135] The coordinate information collector (11) can obtain image coordinates of a plurality of points of the first group in the acquired image and the corresponding actual GPS coordinates.

[0136] The processor (12) may be configured to perform a distortion correction process using image coordinates of a plurality of points of the first group and corresponding actual GPS coordinates.

[0137] The distortion correction process is as follows:

[0138] - Obtaining the initial homography matrix based on the first group points

[0139] The processor (12) can obtain an initial homography matrix based on the image coordinates of a plurality of points of the first group and the actual transformed GPS coordinates based on the actual GPS coordinates.

[0140] - Obtain estimated transformed GPS coordinates for the second group of points

[0141] The processor (12) may select a plurality of points of a second group from among the points in the acquired image. At this time, the plurality of points of the second group may be at least partially identical to the plurality of points of the first group. Alternatively, the plurality of points of the second group may be different from the plurality of points of the first group.

[0142] Then, the processor (12) can obtain estimated transformed GPS coordinates corresponding to the image coordinates of a plurality of points of the second group by using the obtained homography matrix.

[0143] - Perform image coordinate distortion correction using image coordinate and transformed GPS coordinate pairs.

[0144] The processor (12) can perform distortion correction of image coordinates using the image coordinates of the points of the first group and the points of the second group, and the corresponding actual transformed GPS coordinates or estimated transformed GPS coordinates.

[0145] - Update homography matrix

[0146] The processor (12) can update the homography matrix based on the corrected image coordinates of a plurality of points of the first group or the second group obtained according to distortion correction and the actual transformed GPS coordinates or the estimated transformed GPS coordinates.

[0147] The distortion correction process can be repeated, and for this purpose, a distance error (difference) can be calculated. The distance error refers to the difference between the estimated transformed GPS coordinates or estimated GPS coordinates and the actual transformed GPS coordinates or actual GPS coordinates. More specifically, the processor (12) can obtain the estimated transformed GPS coordinates of a plurality of points of the first group obtained using the updated homography matrix, and calculate the difference between the estimated transformed GPS coordinates and the actual transformed GPS coordinates.

[0148] The distortion correction process may be repeated until the distance error for the points in the first group is greater than the distance error for the points in the first group obtained using the homography matrix before the homography matrix is ​​updated. The processor (12) may select the image coordinate distortion correction-related information and the updated homography matrix of the distortion correction process with the smallest distance error.

[0149] Alternatively, the distortion correction process may be repeated a preset number (N), where N is an integer greater than or equal to 1. The processor (12) may obtain N difference values ​​between the estimated transformed GPS coordinates and the actual transformed GPS coordinates of the points of the first group. The processor (12) may select the image coordinate distortion correction-related information and the updated homography matrix of the distortion correction process corresponding to the minimum value among the obtained N difference values.

[0150] Distortion correction may include calculating or estimating the camera matrix or camera parameters of the camera that captured the acquired image.

[0151] The processor (12) may be configured to extract feature values ​​for a plurality of points of the first group, match the extracted feature values ​​with pairs of image coordinates and actual transformed GPS coordinates and store them, and detect changes in image coordinates of the plurality of points of the first group based on the extracted feature values.

[0152] The processor (12) may be configured to update a homography matrix based on the changed image coordinates and the actual transformed GPS coordinates as changes in the image coordinates of a plurality of points of the first group are detected.

[0153] The processor (12) can obtain estimated transformed GPS coordinates corresponding to image coordinates within the acquired image using the updated homography matrix, and convert the obtained estimated transformed GPS coordinates into estimated GPS coordinates. In addition, the processor (12) can be configured to transmit a traffic safety-related message to a user or user terminal located within a preset distance from a location corresponding to the converted estimated GPS coordinates.

[0154] The processor (12) can obtain estimated transformed GPS coordinates corresponding to image coordinates in the acquired image using the updated homography matrix, and convert the obtained estimated transformed GPS coordinates into estimated GPS coordinates. The processor (12) can be configured to transmit information related to a location corresponding to the converted estimated GPS coordinates to a server.

[0155] According to the present invention, since the accuracy of location estimation is improved, objects determined not to be in a dangerous location before the distortion correction process according to the present invention is applied may be determined to be in a dangerous location after the distortion correction process. Therefore, the present invention has the effect of eliminating blind spots in traffic safety caused by such location estimation errors. Furthermore, accurate location estimation is essential not only for traffic safety but also for providing services based on location information. The present invention improves the accuracy of location estimation, enabling information for providing location-based services to be transmitted to a server or other device providing the service.

[0156] Meanwhile, operations, processes, etc. described with reference to FIGS. 2 to 8, but not described with reference to FIG. 9, may be performed by the processor (12) or the camera distortion correction device (10).

[0157]

[0158] In addition, as another aspect of the present invention, the operation of the proposal or invention described above may be implemented, performed or executed by a “computer” (a comprehensive concept including a system on chip (SoC) or a (micro) processor, etc.), or may be provided as a code or a computer-readable storage medium storing or including the code or a computer program product, and the scope of the present invention may be extended to the code or the computer-readable storage medium storing or including the code or the computer program product.

[0159]

[0160] The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations of the present invention, as defined by the following claims, are possible. Accordingly, the present invention is not intended to be limited to the embodiments disclosed herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. As a device for camera distortion correction, A coordinate information collector that acquires image coordinates of a plurality of points of a first group in an acquired image and corresponding actual GPS coordinates; and Using a homography matrix obtained based on the image coordinates of a plurality of points of the first group and the actual transformed GPS coordinates based on the actual GPS coordinates, the estimated transformed GPS coordinates corresponding to the image coordinates of a plurality of points of the second group in the acquired image are obtained, Distortion correction of image coordinates is performed using image coordinates of a plurality of points of the first group and a plurality of points of the second group and the corresponding actual transformed GPS coordinates or the estimated transformed GPS coordinates, A camera distortion correction device, comprising a processor configured to update a homography matrix based on the corrected image coordinates of a plurality of points of the first group or the second group obtained according to the distortion correction and the actual transformed GPS coordinates or the estimated transformed GPS coordinates.

2. In the first paragraph, the processor, A camera distortion correction device configured to obtain estimated transformed GPS coordinates of a plurality of points of the first group obtained using the updated homography matrix, and to calculate the difference between the estimated transformed GPS coordinates and the actual transformed GPS coordinates.

3. In the first paragraph, the distortion correction process consisting of obtaining the estimated transformed GPS coordinates, correcting the distortion of the image coordinates, and updating the homography matrix is ​​set to be repeated. A camera distortion correction device, wherein the process is repeated until the difference between the estimated transformed GPS coordinates of the plurality of points of the first group and the actual transformed GPS coordinates is greater than the difference between the estimated transformed GPS coordinates of the plurality of points of the first group obtained using the homography matrix before the update of the homography matrix and the actual transformed GPS coordinates.

4. A camera distortion correction device according to claim 1, wherein the distortion correction process, which comprises obtaining the estimated transformed GPS coordinates, correcting the distortion of the image coordinates, and updating the homography matrix, is set to be repeated N times, where N is an integer greater than or equal to 1.

5. In the fourth paragraph, the processor, Obtain N difference values ​​between the estimated transformed GPS coordinates and the actual transformed GPS coordinates of the points of the first group, A camera distortion correction device that selects image coordinate distortion correction-related information and an updated homography matrix of a distortion correction process corresponding to the minimum value among the N difference values ​​obtained above.

6. A camera distortion correction device in the first paragraph, wherein at least some of the plurality of points of the first group and the plurality of points of the second group are identical.

7. A camera distortion correction device in the first paragraph, wherein the plurality of points of the first group and the plurality of points of the second group are different from each other.

8. A camera distortion correction device according to claim 1, wherein the distortion correction includes calculating or estimating a camera matrix or camera parameters of a camera that captured the acquired image.

9. In the first paragraph, the processor, Extracting feature values ​​for a plurality of points of the first group, and storing the extracted feature values ​​by matching them with pairs of the image coordinates and the actual transformed GPS coordinates, A camera distortion correction device configured to detect a change in image coordinates of a plurality of points of the first group based on the extracted feature values.

10. In the 9th paragraph, the processor, A camera distortion correction device configured to update a homography matrix based on the changed image coordinates and the actual transformed GPS coordinates when a change in the image coordinates of a plurality of points of the first group is detected.

11. In the first paragraph, the processor, Using the updated homography matrix, the estimated transformed GPS coordinates corresponding to the image coordinates in the acquired image are obtained, and the acquired estimated transformed GPS coordinates are converted into the estimated GPS coordinates. A camera distortion correction device configured to transmit the traffic safety-related message to a user or user terminal located within a preset distance from a location corresponding to the converted estimated GPS coordinates.

12. In the first paragraph, the processor, Using the updated homography matrix, the estimated transformed GPS coordinates corresponding to the image coordinates in the acquired image are obtained, and the acquired estimated transformed GPS coordinates are converted into the estimated GPS coordinates. A camera distortion correction device configured to transmit information related to a location corresponding to the above-mentioned converted estimated GPS coordinates to a server.

13. In the first paragraph, the actual converted GPS coordinates are: A camera distortion correction device comprising three-dimensional coordinates generated based on latitude and longitude of GPS coordinates measured at a plurality of points of the first group.

14. In the 13th, the actual converted GPS coordinates are, A camera distortion correction device comprising relative three-dimensional coordinates of the remaining points among the plurality of points with respect to a pre-designated point among the plurality of points.

15. As a method for camera distortion correction, A step of obtaining image coordinates of a plurality of points of a first group in an acquired image and corresponding actual GPS coordinates; A step of obtaining estimated transformed GPS coordinates corresponding to the image coordinates of a plurality of points of the second group in the obtained image by using a homography matrix obtained based on the image coordinates of the plurality of points of the first group and the actual transformed GPS coordinates based on the actual GPS coordinates; A step of performing distortion correction of image coordinates using image coordinates of a plurality of points of the first group and a plurality of points of the second group and the corresponding actual transformed GPS coordinates or the estimated transformed GPS coordinates; and A method comprising the step of updating a homography matrix based on the corrected image coordinates of a plurality of points of the first group or the second group obtained according to the distortion correction and the actual transformed GPS coordinates or the estimated transformed GPS coordinates.

Images