Camera calibration method based on single vanishing point recognition

Abstract

Provided is a camera calibration method based on single vanishing point recognition. The camera calibration method according to an embodiment of the present invention comprises: applying a vanishing line detection filter to detect vanishing line candidates for a single vanishing point in an acquired image; finding the single vanishing point from the vanishing line candidates; and extracting a plurality of points to calculate a parameter of a homography matrix of a camera having acquired the image. Accordingly, it is possible to accurately link a positional relationship between the real world and a virtual world by accurately modeling a relationship between a real space and the image without changing a position of a CCTV camera or using a calibration board.

Description

Camera calibration method based on single vanishing point recognition

[0001] The present invention relates to camera calibration, and more particularly, to a method and system for more easily and automatically calculating calibration parameters for numerous installed cameras, such as CCTV (Closed-Circuit Television).

[0002] Camera calibration is a technique that accurately models the relationship between real space and images by understanding the optical characteristics of the camera and the characteristics of the image sensor, thereby accurately linking the positional relationship between the real world and the virtual world.

[0003] In traditional camera calibration methods, a calibration plate with a chessboard pattern and a feature pattern is used to capture images from multiple angles with the camera, extract feature points from the images, and calculate calibration parameters.

[0004] Another way is to take multiple images, extract feature points from the images, and use the relationship between the specific points between the images to calculate the calibration parameters.

[0005] Although the above methods can effectively calculate the calibration parameters of a camera, it is difficult to apply them in reality because calculating the calibration parameters of countless installed cameras such as CCTV requires changing the position of the camera and taking pictures, or creating a calibration plate in a chessboard pattern and manually extracting the data for the calculations one by one. This is because fixed cameras such as CCTV cannot move, so it is impossible to take multiple pictures by changing the position, and carrying a large calibration plate and performing calibration requires an enormous amount of effort.

[0006] Accordingly, a method is needed to enable easier and automatic calibration of cameras.

[0007] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method for automatically finding a single vanishing point and calculating calibration parameters without a correction plate for CCTV images of a composition such as a corridor in a building.

[0008] According to one embodiment of the present invention for achieving the above object, a camera calibration method includes the steps of: detecting edges in an acquired image; detecting straight lines from the detected edges; applying a vanishing line detection filter to the detected straight lines to detect vanishing line candidates for a single vanishing point; drawing a straight line at the bottom of the image; finding a single vanishing point from the vanishing line candidates; generating final vanishing lines for the single vanishing point; extracting a plurality of points using the bottom straight line and the final vanishing lines of the image; and calculating parameters of a homography matrix of a camera that acquired the image using the extracted points.

[0009] A camera calibration method according to an embodiment of the present invention further includes a step of finding two intersection points between vanishing line candidates and the lowest straight line; and the generation step can generate two final vanishing lines by connecting the found single vanishing point and the two intersection points, respectively.

[0010] The extraction step extracts two additional points from the two identified intersections and the final vanishing lines, which can be used to calculate the parameters of the homography matrix. These two additional points have known location information, and the actual location information can be given in relative coordinates.

[0011] A camera calibration method according to an embodiment of the present invention further includes a step of dividing detected vanishing line candidates into left vanishing lines and right vanishing lines and adjusting them to the same number; and the step of finding a single vanishing point can find a single vanishing point using the adjusted vanishing line candidates. The adjustment step can determine whether the detected vanishing line candidates correspond to left vanishing lines or right vanishing lines based on the angles of the vanishing line candidates.

[0012] A camera calibration method according to an embodiment of the present invention may further include the steps of: acquiring a plurality of images; overlapping the acquired plurality of images; detecting edges in the overlapping images; detecting straight lines from the detected edges; and extracting some of the detected straight lines to generate a vanishing line detection filter.

[0013] The vanishing line detection filter generation step may further include a step of searching for a vanishing line condition among the detected straight lines and setting it as a filter condition; a step of extracting straight lines corresponding to the set filter condition from the detected straight lines and generating a filter for detecting the extracted straight lines; The filter condition may be conditioned on a range of distance and angle based on a starting point of the image.

[0014] A camera calibration method according to another embodiment of the present invention includes a processor that detects edges in an acquired image, detects straight lines from the detected edges, applies a vanishing line detection filter to the detected straight lines to detect vanishing line candidates for a single vanishing point, draws a straight line at the bottom of the image, finds a single vanishing point from the vanishing line candidates, generates final vanishing lines for the single vanishing point, extracts a plurality of points using the bottom straight line and the final vanishing lines of the image, and calculates parameters of a homography matrix of a camera that acquired the image using the extracted points; and a storage unit that provides storage space necessary for the operation of the processor.

[0015] In another embodiment of the present invention, a camera calibration method includes the steps of: detecting vanishing line candidates for a single vanishing point in an acquired image; drawing a straight line at the bottom of the image; finding a single vanishing point from the vanishing line candidates; generating final vanishing lines for the single vanishing point; extracting a plurality of points using the straight line at the bottom of the image and the final vanishing lines; and calculating parameters of a homography matrix of a camera that acquired the image using the extracted points.

[0016] In another embodiment of the present invention, a camera calibration system includes a processor that detects vanishing line candidates for a single vanishing point in an acquired image, draws a straight line at the bottom of the image, finds a single vanishing point from the vanishing line candidates, generates final vanishing lines for the single vanishing point, extracts a plurality of points using the bottom straight line and the final vanishing lines of the image, and calculates parameters of a homography matrix of a camera that acquired the image using the extracted points; and a storage unit that provides storage space required for the operation of the processor.

[0017] As described above, according to embodiments of the present invention, for CCTV images of a composition such as a corridor in a building, a single vanishing point can be automatically found without a correction plate and a calibration parameter can be calculated, so that the relationship between the actual space and the image can be accurately modeled and the positional relationship between the real world and the virtual world can be accurately linked without changing the position of the CCTV camera or utilizing a correction plate, etc.

[0018] In addition, according to embodiments of the present invention, through accurate calibration of CCTV cameras for corridors within a building, it is possible to accurately identify the location, movement, and entry / exit of objects in the corridors and each room, which can be utilized to identify the occupants of each room, and to lay the foundation for virtual environment services such as implementation of a digital twin for the building situation.

[0019] Figure 1 is a video composition of CCTV cameras installed inside a building.

[0020] Figure 2 is a CCTV image with a single vanishing point;

[0021] Figure 3 is a camera calibration method according to one embodiment of the present invention;

[0022] Figure 4 is a detailed flowchart of the vanishing line detection filter generation step illustrated in Figure 3.

[0023] Figure 5 is an example of the result of overlapping multiple CCTV images.

[0024] Figure 6 is an example of the results of detecting edges using the Canny edge detection algorithm.

[0025] Figure 7 is an example of the result of detecting a straight line through Hough transform.

[0026] Figure 8 is a method for expressing the distance (ρ) and angle (θ) of straight lines in an image.

[0027] Figure 9 is an example of a generated vanishing line detection filter.

[0028] Figure 10 is a detailed flowchart of the camera calibration parameter calculation step illustrated in Figure 3.

[0029] Figure 11 is an example of a single-image CCTV video.

[0030] Figure 12 is an example of edge detection results for CCTV footage.

[0031] Figure 13 is an example of the results of detecting straight lines in CCTV footage using Hough transform.

[0032] Figure 14 is an example of the results of detecting vanishing line candidates using a vanishing line detection filter.

[0033] Figure 15 shows examples of vanishing points, intersections, and additional points.

[0034] Figure 16 is a homography matrix,

[0035] Figure 17 shows the matching relationship between the image coordinates and the actual coordinates of the intersections and additional points, and

[0036] Fig. 18 is a camera calibration system according to another embodiment of the present invention.

[0037] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0038]

[0039] 1. Overview

[0040] Since most CCTV cameras installed in buildings are installed in the center space at the end of a corridor in order to monitor the entire corridor, due to the composition as shown in Fig. 1, the vanishing point in the image is often created in the upper horizontal center as shown in Fig. 2.

[0041] An embodiment of the present invention presents a method for calculating camera calibration parameters from images captured in this manner. This technique calculates calibration parameters by finding a single vanishing point in a fixed CCTV camera installed in a building's corridor, without the need for a calibration plate.

[0042] FIG. 3 is a diagram illustrating the process of a camera calibration method according to one embodiment of the present invention. As illustrated, the camera calibration method according to the embodiment of the present invention comprises a vanishing line detection filter generation step (S100) and a camera calibration parameter calculation step (S200). Each step is described in detail below.

[0043]

[0044] 2. Create a vanishing line detection filter

[0045] A vanishing line detection filter is a filter used to detect vanishing lines to extract points required for calculating camera calibration parameters for a camera image with a single vanishing point.

[0046] Specifically, as shown in Figure 2, this filter identifies vanishing points and vanishing lines, which are the corridor floor edge segments passing through the vanishing points, in an image where a single vanishing point is created. Four points are extracted from the identified vanishing lines to be used in calculating camera calibration parameters, which will be described in detail later.

[0047] Figure 4 is a detailed flowchart of the vanishing line detection filter generation step (S100) illustrated in Figure 3. The vanishing line detection filter is generated so that it can be commonly applied to images generated from various CCTV cameras installed in a building.

[0048] To this end, as illustrated, CCTV images generated by multiple CCTV cameras actually installed in different locations are first acquired (S110), and the acquired CCTV images are then superimposed (S120). Figure 5 illustrates the results of superimposing multiple CCTV images.

[0049] In step S120, edges are detected in the overlapping CCTV images (S130). Edge detection can be achieved using the Canny edge detection algorithm, but other methods are not excluded. Figure 6 shows the results of edge detection using the Canny edge detection algorithm.

[0050] Next, straight lines are detected from the edges detected in step S130 through the Hough transform (S140). Figure 7 shows the results of straight lines detected through the Hough transform.

[0051] Afterwards, in step S140, the conditions corresponding to vanishing lines are searched among the straight lines detected and set as filter conditions (S150). Specifically, as illustrated in FIG. 8, the range of distance (ρ) and angle (θ) based on the starting point of the image is set as conditions. In the embodiment of the present invention, the vanishing lines are straight lines at the edges of the corridor floor in the CCTV image.

[0052] And, among the straight lines detected in step S140, straight lines that meet the conditions in step S150 are extracted, and a filter for detecting the extracted straight lines is created (S160). Since the straight lines extracted in step S160 correspond to vanishing lines, the filter for detecting them becomes the vanishing line detection filter to be created in the embodiment of the present invention.

[0053] Figure 9 shows a vanishing line detection filter generated when applying the filter conditions “0.70 < θ < 0.79, 500 < ρ < 680” and “2.35 < θ < 2.50, -560 < ρ < -310.”

[0054]

[0055] 3. Calculating camera calibration parameters

[0056] The process of generating a vanishing line detection filter has been described in detail so far. Below, a method of performing camera calibration by detecting vanishing lines in CCTV footage using a vanishing line detection filter and calculating / determining a homography matrix, which is a camera calibration parameter, using points extracted from the detected vanishing lines will be described in detail with reference to FIG. 10. FIG. 10 is a detailed flowchart of the camera calibration parameter calculation step (S200) illustrated in FIG. 3.

[0057] As illustrated, edges are first detected using the Canny edge detection algorithm for the acquired CCTV footage (S210). The CCTV footage acquired in step S210 is a single image, not a composite of multiple overlapping images. Figure 11 illustrates a single CCTV footage, and Figure 12 illustrates the edge detection results for the CCTV footage.

[0058] Next, using the Hough transform, straight lines are detected from the edges detected in step S210 (S220). Figure 13 shows the results of detecting straight lines in CCTV footage using the Hough transform.

[0059] In addition, a vanishing line detection filter is applied to the CCTV video in which straight lines are detected in step S220 to detect vanishing line candidates for a single vanishing point (S230). Figure 14 shows the results of detecting vanishing line candidates using the vanishing line detection filter.

[0060] In the next step S230, the detected vanishing line candidates are divided into left and right vanishing lines and adjusted to the same number (S240). Whether the detected vanishing line candidates correspond to left or right vanishing lines can be determined based on the angles of the vanishing line candidates.

[0061] The number of left-vanishing lines and right-vanishing lines can be fixed, or can be set to the number of vanishing lines detected less frequently. For example, if two left-vanishing line candidates are detected and four right-vanishing line candidates are detected, two of the four right-vanishing line candidates can be randomly selected, so that there are two left-vanishing line candidates and two right-vanishing line candidates.

[0062] A straight line is drawn at the bottom of the following CCTV image (S250), and then, using the RANSEC algorithm, one vanishing point is found from the vanishing line candidates (S260), and similarly, using the RANSEC algorithm, two intersection points between the vanishing line candidates and the bottom straight line are found (S270).

[0063] And, in the two final vanishing lines that connect the vanishing point found in step S260 and the two intersection points found in step S270, two additional points whose actual location information is known are set (S280).

[0064] The results of performing steps S260 to S280 are shown in Fig. 15. In Fig. 15, A is the vanishing point found in step S260, B and C are two intersection points found in step S270, and D and E are two additional points set from the two final vanishing lines.

[0065] D and E must be points whose actual locations are known, but the actual location coordinates can be relative coordinates rather than absolute coordinates. For example, if the actual coordinates of B are (0%, 0%) and the actual coordinates of C are (100%, 0%), and if D and E are located 80% of the total corridor distance from B and C, then the actual coordinates of D can be (0%, 80%) and the actual coordinates of E can be (100%, 80%).

[0066] Meanwhile, the actual coordinates calculated as relative coordinates can be easily converted to absolute coordinates by obtaining information on the width and length of the actual corridor.

[0067] Afterwards, using the two intersection points (B and C in Fig. 15) found in step S270 and the two additional points (D and E in Fig. 15) set in step S280, the parameters of the homography matrix that converts the coordinates in the CCTV image into coordinates in real space are calculated (S290).

[0068] Since the homography matrix is ​​composed of camera internal parameters and camera external parameters as illustrated in Fig. 16, step S290 can be said to be a process of calculating and determining camera internal / external parameters.

[0069] Figure 17 shows the matching relationship between the image coordinates and the actual coordinates of two intersection points (B and C in Figure 15) found in step S270 and two additional points (D and E in Figure 15) set in step S280.

[0070] Once the homography matrix is ​​determined by Step S290, each coordinate in the corridor from the CCTV footage can be perspective transformed into real-world coordinates, allowing the location information of people appearing or moving in the corridor to be identified. This allows for the precise identification of people's locations, movements, and entry / exit times within the corridor, which can be utilized to determine the occupancy status of each room and enable the creation of a digital twin of the building's status.

[0071]

[0072] 4. Camera Calibration System

[0073] FIG. 18 is a diagram illustrating the configuration of a camera calibration system according to another embodiment of the present invention. The camera calibration system according to the embodiment of the present invention can be implemented as a computing system comprising a communication unit (310), an output unit (320), a processor (330), an input unit (340), and a storage unit (350), as illustrated.

[0074] The communication unit (310) is a communication interface for connection to an external network or external device, and receives video from a CCTV camera within the building. The output unit (320) is an output means for displaying the results of computations performed by the processor (330), and the input unit (340) is a user interface for receiving user commands and transmitting them to the processor (330).

[0075] The processor (330) performs camera calibration by generating a vanishing line detection filter and calculating camera calibration parameters according to the procedure illustrated in FIG. 3 described above. The storage unit (350) provides the storage space necessary for the processor (330) to function and operate.

[0076]

[0077] 5. Variations

[0078] So far, a preferred embodiment of a camera calibration method based on single vanishing point recognition has been described in detail.

[0079] For images with a single vanishing point composition within a building, if the camera calibration parameters can be calculated based on the vanishing point, the relationship between the image coordinates and the real-world coordinates can be resolved, allowing for more accurate digital twin construction of events in the real environment through AI recognition. The internal / external parameters of the camera can be inferred without the need for a correction plate or multiple photos from different angles required for camera calibration.

[0080] Accordingly, in an embodiment of the present invention, a method for calculating calibration parameters by finding a single vanishing point without a correction plate for fixed CCTV cameras installed in a hallway in a building is proposed, and a method for generating a vanishing line detection filter for this purpose is also proposed.

[0081] It is possible to convert a person recognized by the camera directly into coordinates on a digital twin, so that situational information recognized in the actual physical world can be directly mapped to the virtual environment world, enabling the provision of various real / virtual services.

[0082] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0083] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. A step of detecting edges in the acquired image; A step of detecting straight lines from the detected edges; A step of detecting vanishing line candidates for a single vanishing point by applying a vanishing line detection filter to the detected straight lines; Step 1: Draw a straight line at the bottom of the image; A step of finding a single vanishing point from the vanishing line candidates; Step of generating final vanishing lines for a single vanishing point; A step of extracting multiple points using the lowest straight line and final vanishing lines of the image; A camera calibration method, characterized by including a step of calculating parameters of a homography matrix of a camera that acquired an image using extracted points.

2. In claim 1, Further comprising the step of finding two intersection points between the vanishing line candidates and the lowest straight line; The creation phase is, A camera calibration method characterized in that two final vanishing lines are generated by connecting each of the found single vanishing point and two intersection points.

3. In claim 2, The extraction step is. A camera calibration method characterized by extracting two additional points on the two found intersection points and the final vanishing lines as points to be used for calculating parameters of a homography matrix.

4. In claim 3, Two additional points are, These are points that know the actual location information, Actual location information is, A camera calibration method characterized in that it is given in relative coordinates.

5. In claim 1, A step of dividing the detected vanishing line candidates into left-vanishing lines and right-vanishing lines and adjusting them to the same number is further included; The steps to find a single vanishing point are: A camera calibration method characterized by finding a single vanishing point using adjusted vanishing line candidates.

6. In claim 5, The adjustment phase is, A camera calibration method characterized in that it is determined whether the detected vanishing line candidates correspond to left-vanishing lines or right-vanishing lines based on the angles of the vanishing line candidates.

7. In claim 1, A step of acquiring multiple images; A step of overlapping multiple acquired images; A step of detecting edges in an overlapping image; A step of detecting straight lines from the detected edges; A camera calibration method, further comprising a step of extracting some of the detected straight lines and generating a vanishing line detection filter.

8. In claim 7, The vanishing line detection filter generation step is: A step of searching for a condition corresponding to a vanishing line among the detected straight lines and setting it as a filter condition; A camera calibration method, further comprising: a step of extracting straight lines corresponding to set filter conditions from detected straight lines and generating a filter for detecting the extracted straight lines.

9. In claim 8, The filter conditions are, A camera calibration method characterized by setting a range of distances and angles based on the starting point of an image.

10. A processor that detects edges in an acquired image, detects straight lines from the detected edges, applies a vanishing line detection filter to the detected straight lines to detect vanishing line candidates for a single vanishing point, draws a straight line at the bottom of the image, finds a single vanishing point from the vanishing line candidates, generates final vanishing lines for the single vanishing point, extracts a plurality of points using the bottom straight line and the final vanishing lines of the image, and calculates parameters of a homography matrix of a camera that acquired the image using the extracted points; and A camera calibration system characterized by including a storage unit that provides storage space required for the processor's operation.

11. A step of detecting vanishing line candidates for a single vanishing point in the acquired image; Step 1: Draw a straight line at the bottom of the image; A step of finding a single vanishing point from the vanishing line candidates; Step of generating final vanishing lines for a single vanishing point; A step of extracting multiple points using the lowest straight line and final vanishing lines of the image; A camera calibration method, characterized by including a step of calculating parameters of a homography matrix of a camera that acquired an image using extracted points.

12. A processor that detects vanishing line candidates for a single vanishing point in an acquired image, draws a straight line at the bottom of the image, finds a single vanishing point from the vanishing line candidates, generates final vanishing lines for the single vanishing point, extracts a plurality of points using the bottom straight line and the final vanishing lines of the image, and calculates parameters of a homography matrix of a camera that acquired the image using the extracted points; and A camera calibration system characterized by including a storage unit that provides storage space required for the processor's operation.

Images