Method and apparatus for calibration

Abstract

Methods and apparatuses for calibration are disclosed. The method includes detecting a preset pattern included in a surface of a road from a driving image of a vehicle; transforming image coordinates in an image domain of the pattern to world coordinates in a world domain; determining whether to calibrate a camera that captures the driving image by comparing a size predicted based on the world coordinates of the pattern to a reference size of the pattern; in response to determining to calibrate the camera, determining relative world coordinates of movement of the pattern relative to the camera using images captured by the camera at different points in time; transforming the relative world coordinates of the pattern to absolute world coordinates of the pattern; and calibrating the camera using a correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2021-0061237, filed on May 12, 2021, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a method and apparatus for calibration. Background Technology

[0003] Cameras can be used in a variety of applications, including, for example, autonomous driving (AD) and advanced driver assistance systems (ADAS). When a camera is installed in a vehicle, camera calibration can be performed. Through camera calibration, coordinate system transformation information is obtained to transform the coordinate system between the vehicle and the camera. This coordinate system transformation information can be used to implement various functions of AD and ADAS (e.g., functions for estimating the vehicle's attitude and functions for estimating the distance to vehicles ahead).

[0004] However, when deformation occurs in the initial state of calibration due to factors such as changes in tire pressure, vehicle posture, or the number of passengers, the position or angle of the camera may change. Therefore, there may be a difference between the information of the actual environment and the information determined based on the image of the environment captured by the camera. Summary of the Invention

[0005] The present invention is provided in a simplified form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a processor-implemented method for calibration includes: detecting a preset pattern included in a road surface from a driving image of a vehicle; transforming image coordinates in an image domain of the pattern to world coordinates in a world domain; determining whether to calibrate a camera capturing the driving image by comparing a predicted size based on the world coordinates of the pattern with a reference size of the pattern; in response to determining to calibrate the camera, determining relative world coordinates of the movement of the pattern relative to the camera using images captured by the camera at different time points; transforming the relative world coordinates of the pattern to absolute world coordinates of the pattern; and calibrating the camera using a correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern.

[0007] The step of determining the relative world coordinates of the pattern may include: determining camera movement information based on the correspondence between patterns in images captured by the camera at different time points; and using the correspondence between patterns and the movement information to determine the relative world coordinates of the movement of the pattern relative to the camera.

[0008] The step of transforming the relative world coordinates of the pattern into absolute world coordinates may include: in response to the pattern having a reference size, using the reference size to transform the relative world coordinates into absolute world coordinates.

[0009] The step of transforming the relative world coordinates of the pattern into absolute world coordinates may include: in response to the pattern having a reference range, transforming the relative world coordinates into absolute world coordinates based on the movement of the vehicle.

[0010] The steps of calibrating a camera may include: using the correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern to calibrate any one or any combination of the camera's pitch, roll, and altitude.

[0011] Preset patterns may include standardized road markings on roads on which vehicles are traveling.

[0012] The step of transforming the image coordinates of the pattern into world coordinates may include: transforming the image coordinates of the pattern into world coordinates based on the homography matrix.

[0013] The step of transforming the image coordinates of the pattern into world coordinates may include: transforming the image coordinates of the pattern into world coordinates based on camera parameters and the constraints of the pattern on the road surface.

[0014] The step of determining whether to calibrate the camera may include: determining whether to calibrate the camera based on whether the difference between the predicted size of the pattern and the reference size exceeds a preset threshold.

[0015] The method may include determining whether a preset number of feature points are extracted from a preset pattern.

[0016] The method may include estimating either or both of the vehicle's pose and distance to another vehicle based on images taken using a calibrated camera.

[0017] In another general aspect, one or more embodiments include: a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform any, any combination, or all of the operations and methods described herein.

[0018] In another general aspect, a calibration device includes: one or more processors configured to: detect a preset pattern included in a road surface from a driving image of a vehicle; transform image coordinates in an image domain of the pattern to world coordinates in a world domain; determine whether to calibrate a camera capturing the driving image by comparing a predicted size based on the world coordinates of the pattern with a reference size of the pattern; in response to determining to calibrate the camera, determine the relative world coordinates of the movement of the pattern relative to the camera using images captured by the camera at different time points; transform the relative world coordinates of the pattern to absolute world coordinates of the pattern; and calibrate the camera using the correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern.

[0019] To determine the relative world coordinates of the pattern, the one or more processors may be configured to: determine camera movement information based on the correspondence between patterns in images captured by the camera at different time points; and use the correspondence between patterns and the movement information to determine the relative world coordinates of the pattern's movement relative to the camera.

[0020] In order to transform the relative world coordinates of the pattern into absolute world coordinates, the one or more processors may be configured to: in response to the pattern having a reference size, use the reference size to transform the relative world coordinates into absolute world coordinates.

[0021] In order to transform the relative world coordinates of the pattern into absolute world coordinates, the one or more processors may be configured to transform the relative world coordinates into absolute world coordinates based on the movement of the vehicle, in response to the pattern having a reference range.

[0022] To calibrate the camera, the one or more processors may be configured to calibrate any one or any combination of the camera's pitch, roll, and altitude using the correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern.

[0023] Preset patterns may include standardized road markings on roads on which vehicles are traveling.

[0024] In order to transform the image coordinates of the pattern into world coordinates, the one or more processors can be configured to transform the image coordinates of the pattern into world coordinates based on the homography matrix.

[0025] In order to transform the image coordinates of the pattern into world coordinates, the one or more processors may be configured to transform the image coordinates of the pattern into world coordinates based on camera parameters and constraints of the pattern on the road surface.

[0026] To determine whether to calibrate the camera, the one or more processors may be configured to determine whether to calibrate the camera based on whether the difference between the predicted size of the pattern and the reference size exceeds a preset threshold.

[0027] The device may be a vehicle, and the device may also include a camera.

[0028] In another general aspect, a processor-implemented method for calibration includes: detecting a preset pattern in an image captured by a camera; determining whether to calibrate the camera by comparing a predicted size of the detected pattern with a preset reference threshold for the pattern; determining the absolute world coordinates of the pattern using images captured by the camera at different time points in response to determining that the camera should be calibrated; and calibrating the camera based on the absolute world coordinates.

[0029] The preset reference threshold can be a preset range, and the step of determining whether to calibrate may include: determining to calibrate the camera in response to the predicted size of the pattern being within the preset range.

[0030] Images captured by the camera at different points in time may include images of a pre-defined pattern.

[0031] The steps for determining absolute world coordinates may include: using images captured by a camera at different points in time to determine the relative world coordinates of the pattern's movement relative to the camera; and transforming the relative world coordinates of the pattern into absolute world coordinates by removing the scale ambiguity of the relative world coordinates by matching the relative length of the relative world coordinates with the reference length of preset pattern information.

[0032] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0033] Figure 1 An example of a calibration device is shown.

[0034] Figure 2 This shows an example of detecting a preset pattern from a driving image.

[0035] Figure 3 An example of a preset pattern is shown.

[0036] Figure 4 An example is shown of obtaining the absolute world coordinates of a pattern based on a structure from motion (SFM).

[0037] Figure 5 An example of performing calibration is shown.

[0038] Figure 6 An example of a calibration method is shown.

[0039] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0040] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known upon understanding this disclosure may be omitted.

[0041] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0042] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any one and any combination of any two or more of the associated listed items. It will also be understood that, as used herein, the terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof. The use of the term “may” herein with respect to examples or embodiments (e.g., regarding what an example or embodiment may include or implement) indicates the presence of at least one example or embodiment that includes or implements such features, but all examples are not limited thereto.

[0043] Throughout this specification, when a component is described as "connected to" or "attached to" another component, that component may be directly "connected to" or directly "attached to" said other component, or there may be one or more other components in between. Conversely, when an element is described as "directly connected to" or "directly attached to" another element, there may be no other elements in between. Similarly, similar expressions (e.g., "between" and "immediately between," and "adjacent to" and "closely adjacent to") should be interpreted in the same manner. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0044] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0045] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, based on an understanding of the disclosure of this application. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in the disclosure of this application, and shall not be interpreted in an idealized or overly formalized sense.

[0046] Furthermore, in the description of the exemplary embodiments, descriptions will be omitted where a detailed description of a structure or function known therefrom after understanding the disclosure of this application would lead to a vague interpretation of the exemplary embodiments. Hereinafter, the examples will be described in detail with reference to the accompanying drawings, and like reference numerals throughout the drawings denote like elements.

[0047] Figure 1 An example of a calibration device is shown.

[0048] Reference Figure 1Vehicle 100 can be or may include all types of transport vehicles that travel on roads or tracks. Vehicle 100 can be or may include any of the following: automobiles, motorcycles, etc., and automobiles can be or may include various types (such as buses, freight vehicles, and two-wheeled vehicles). Vehicle 100 can also be or may include any of the following: autonomous vehicles, intelligent vehicles, and vehicles equipped with driver assistance systems. The vehicle 100 described herein can be a vehicle equipped with calibration device 110. Nearby vehicles described herein can be vehicles adjacent to vehicle 100 in the forward, rearward, lateral, or diagonal direction. Nearby vehicles can also include vehicles surrounding vehicle 100 with another vehicle between them, or vehicles surrounding vehicle 100 with an unoccupied lane between them.

[0049] The calibration device 110 may include a memory 111 (e.g., one or more memories), a processor 113 (e.g., one or more processors), and a camera 115 (e.g., one or more cameras). The vehicle 100 and / or the calibration device 110 may perform any one or more of the operations and methods described herein.

[0050] Calibration device 110 can perform camera calibration by detecting a preset pattern indicated on the surface of the road over which vehicle 100 is traveling. The preset pattern described herein can be a standardized road marking on the driving road, and information regarding any one or any combination of the shape, size, length, and area of ​​the marking can be stored in calibration device 110 and used in the calibration operation described below. As a non-limiting example, the preset pattern may include any one of pedestrian crossings, direction-of-traffic markings, left-turn-allowed markings, directional signs, etc. However, examples of preset patterns are not limited to the foregoing examples, and preset patterns may also include various standardized road markings that may differ due to country or region, and the description of this disclosure can also be applied to these standardized road markings. Calibration device 110 of one or more embodiments can perform camera calibration during actual driving (e.g., driving of vehicle 100), thereby preventing errors that may occur due to the difference between calibration time and driving time, thus improving the technology (e.g., calibration, autonomous driving (AD), and / or advanced driver assistance system (ADAS) technology) that can implement calibration device 110 of one or more embodiments. Furthermore, calibration device 110 can efficiently obtain camera parameters suitable and / or accurate for actual driving environments using patterns on the road surface without the need for separate calibration tools. Online calibration can be achieved through calibration device 110 because camera calibration can be performed during actual driving. The position and pose angle of camera 115 attached to vehicle 100 can be obtained (e.g., determined) through calibration device 110. The calibration parameters can be used in the autonomous driving system and / or driver assistance system of vehicle 100.

[0051] Calibration can refer to the process of obtaining camera parameters that indicate the correspondence between points in the real world and each pixel in an image. Calibration can be an operation for recovering world coordinates from an image (e.g., estimating the actual distance between a vehicle and a vehicle ahead in the image) and / or an operation for obtaining the position of a point in the real world projected into the image. When the current environment of vehicle 100 changes from the initial calibration environment during actual driving due to factors such as changes in the number of passengers in the vehicle, changes in tire pressure, and changes in vehicle posture, the camera parameters may also change, and therefore, calibration may be performed again. However, unlike typical calibration operations, the calibration operations of one or more embodiments described below can be performed during driving, so that the calibration environment and the actual driving environment can be matched, and camera parameters suitable and / or accurate for the driving environment can be obtained, thereby improving the technology (e.g., calibration, AD, and / or ADAS technology) of the calibration device 110 that can implement one or more embodiments.

[0052] Images captured by camera 115 can be obtained as points projected from three-dimensional (3D) space onto a two-dimensional (2D) planar image. For example, the correspondence between coordinates (x, y) on the 2D image and coordinates in 3D space (i.e., 3D coordinates (X, Y, Z) in the world domain) can be represented by Equation 1 below.

[0053] Equation 1:

[0054]

[0055] In equation 1 above, by The "A" indicates intrinsic parameters (such as the focal length, aspect ratio, and principal point of camera 115 itself). x f y This indicates the focal length of the camera (115). x c y This represents the principal point of camera 115, and skew_cf x Indicator skew coefficient.

[0056] In addition, by The [R|t] represents the extrinsic parameter associated with the geometric relationship between camera 115 and the external space (e.g., the mounting height and orientation of camera 115 (e.g., translation and tilt)). 11 to r 33 The element representing the rotation of camera 115 is t1, which can be broken down into pitch, roll, and yaw. t1 to t3 are elements representing the translation of camera 115.

[0057] Image coordinates can be represented by x-axis and y-axis values ​​with respect to a reference point (e.g., upper left) of the image captured by camera 115, and world coordinates can be represented by x-axis, y-axis, and z-axis values ​​with respect to a feature point (e.g., center point) of vehicle 100.

[0058] Although described in detail below, the calibration device 110 can adjust any one or any combination of two or more of the pitch, roll and altitude values ​​of the camera 115 through the calibration operations described below.

[0059] Memory 111 may include computer-readable instructions. Processor 113 may be configured to perform any one or more of the operations described below when the instructions stored in memory 111 are executed by processor 113. Memory 111 may be volatile memory or non-volatile memory.

[0060] Processor 113 may be a device that executes instructions or programs or controls calibration device 110, and processor 113 may be or may include, for example, a central processing unit (CPU) and / or a graphics processing unit (GPU).

[0061] Processor 113 can transform a standardized road surface pattern detected from an image captured in a real-world driving environment into world coordinates to estimate the size of the pattern, and then determine whether to perform calibration based on a determined error between the estimated pattern size and the actual standard. For example, processor 113 can use structure from motion (SFM) and actual standard information to transform the image coordinates of the road surface pattern into world coordinates, and perform calibration on camera 115 based on the correspondence between world coordinates and image coordinates.

[0062] Camera 115 can output images by capturing scenes in front of or along the direction of travel of vehicle 100. Images captured by camera 115 can be sent to processor 113.

[0063] Despite Figure 1 In the examples, camera 115 is shown as included in or as a component of calibration device 110, but the examples are not limited thereto. In another example, calibration device 110 may receive images from an external camera and perform calibration based on the received images. Even in such examples, the description of this disclosure can be applied. Furthermore, although in Figure 1 In the example, calibration device 110 is shown as being disposed within or as a component of vehicle 100, but the example is not limited thereto. In another example, an image captured by camera 115 disposed within vehicle 100 can be sent to calibration device 110 disposed outside vehicle 100, and calibration can then be performed. In this example, calibration device 110 can be implemented in or as any of various computing devices (e.g., mobile phones, smartphones, personal computers (PCs), tablet PCs, laptop computers, remote servers, etc.).

[0064] Figure 2 This shows an example of detecting a preset pattern from a driving image.

[0065] Reference Figure 2 Images captured by a camera installed in the vehicle may include a preset pattern 210.

[0066] The calibration device can detect pattern 210 from an image. For example, the calibration device can detect pattern 210 from an image using segmentation methods and / or edge detection methods. When the calibration device is configured to use a high-definition (HD) map, the calibration device can use the vehicle position identified through positioning and information stored in the HD map to identify the presence of pattern 210 ahead and effectively detect pattern 210. The HD map can be a map that includes information about detailed roads and surrounding geographic features. In addition to lane unit information, the HD map can include 3D information of any of, such as traffic lights, signs, curbs, road markings, and various structures.

[0067] The calibration equipment can extract corner points from the detected pattern 210. Figure 2 In the example, the calibration device can extract corner points of blocks included in a crosswalk. For instance, the calibration device can use the Harris corner detection method to extract one or more corner points of pattern 210. Corner points may correspond to features of the pattern and may also be referred to as feature points. Corner points can be extracted from an image and can be represented by image coordinates in the image domain.

[0068] The calibration device can transform the image coordinates of the corner points of pattern 210 into world coordinates in the world domain. For example, the calibration device can use a homography matrix to transform the image coordinates of pattern 210 into world coordinates. A homography matrix represents the transformation or correspondence established between corresponding points of the projection when one plane is projected onto another plane. The homography matrix can be determined by camera parameters. As another example, the calibration device can use camera parameters and preset constraints to transform the image coordinates of pattern 210 into world coordinates. 3D world coordinates can be relatively easily transformed into 2D image coordinates by projecting them onto the image plane. However, additional information can be used to transform 2D image coordinates into 3D world coordinates. When constraints indicating the detected pattern 210 on the road surface are used as such additional information, the image coordinates of pattern 210 can be transformed into world coordinates.

[0069] The calibration device can predict the size of pattern 210 based on its world coordinates. The calibration device can calculate the difference between the predicted size of pattern 210 and a reference size of pattern 210, and determine whether to perform camera calibration based on this difference. The preset pattern 210 can be a standardized pavement marking, and the predicted size can be compared with a reference size according to relevant standards. The reference size can be stored in the calibration device's internal or external memory, and can be verified against an HD map when the HD map is available or determined to be used. For example, when the difference is greater than a threshold, the calibration device can determine to perform camera calibration. Conversely, when the difference is less than or equal to the threshold, the calibration device can determine not to perform camera calibration and / or continue using the current calibration value without performing camera calibration. In another example, when the predicted size of pattern 210 is within the reference range of pattern 210, the calibration device can determine to perform camera calibration, and when the predicted size is not within the reference range, the calibration device can determine not to perform camera calibration and / or continue using the current calibration value without performing camera calibration.

[0070] Figure 3 An example of a preset pattern is shown.

[0071] Reference Figure 3 This displays various standardized patterns on the road surface. Markings of various sizes, as determined by traffic regulations and rules (e.g., pedestrian crossing markings 310, direction of travel markings 320, permitted turning markings 330, and directional signs 340), may be present on the road surface. Figure 2 In the example, a pedestrian crossing is shown as an example for ease of description. However, various road surface patterns (such as any of the following, such as travel direction marking 320, permitted turning marking 330, directional sign 340, lane, stop line, guide line, etc.) may also be used.

[0072] The thickness I1 of the blocks of the pedestrian crossing mark 310 can have a reference range of 45 to 50 centimeters (cm). That is, the pattern can have a reference range that includes an upper and lower limit. In one example, the reference size used to determine whether to perform calibration, as described above, can be determined as the median or average of the reference range.

[0073] like Figure 3 As shown, the travel direction mark 320, the turn permission mark 330, and the indicator mark 340 may have their own reference sizes.

[0074] Figure 4 An example of obtaining absolute world coordinates of a pattern based on SFM is shown.

[0075] Reference Figure 4The absolute world coordinates of the pattern used to perform calibration can be obtained by performing an operation described below, and SFM can be used to perform this operation. SFM can represent a method that obtains motion information between images (e.g., rotation and / or translation matrices) using the correspondence between feature points in the images and obtains the actual 3D points of the feature points through triangulation.

[0076] In operation 410, the calibration device can determine the correspondence between patterns in images captured by a camera at different time points. When the same camera captures images at different time points while the vehicle is being driven or moving, the images captured by the camera can be obtained from different locations within the same scene. For example, the calibration device can detect a pattern in a first image captured at time t and extract corner points from the detected pattern as first feature points. The calibration device can also detect a pattern in a second image captured at time t+a (a>0) and extract corner points from the detected pattern as second feature points. The calibration device can match identical (or corresponding) points between the first and second feature points and determine the correspondence between the feature points. In one example, identical points can be points corresponding to the same portion of the same pattern, where the same pattern is included in both the first and second images.

[0077] In operation 420, the calibration device can estimate camera movement based on this feature point correspondence between the first and second images. For example, the calibration device can use the feature point correspondence to estimate camera translation and rotation information.

[0078] In operation 430, the calibration device can determine the relative world coordinates of the pattern's movement relative to the camera based on the correspondence of feature points and the camera's movement. In this case, when the actual distance of movement between time t and time t+a is unknown or undetermined, the relative world coordinates without applied scale can be determined. For example, when the camera movement is represented as k, the relative world coordinates of the pattern can be determined to be proportional to k (as in (kx, ky, kz)). The relative world coordinates can have values ​​without applied scale and, therefore, can be referred to as 3D points with scale ambiguity.

[0079] In operation 450, the calibration device can obtain absolute world coordinates by using pattern information 440 to remove scale ambiguity from the relative world coordinates. In one example, in response to the pattern having a reference size (e.g., reference length) as described above, the calibration device can use the reference size to transform the relative world coordinates into absolute world coordinates. For example, the calibration device can transform the relative world coordinates into absolute world coordinates by matching the relative length determined based on the relative world coordinates with the reference length of the pattern stored in the pattern information 440. For example, the pattern information 440 can be normalized and pre-stored pattern information or pattern information stored in an HD map.

[0080] For example, when a pattern (e.g., a pedestrian crossing) has the reference range described above, the scale ambiguity of relative world coordinates can be removed based on the vehicle's movement or motion. The vehicle's movement, determined based on its driving distance and direction from time t to time t+a, can be the same as the movement of a camera positioned within the vehicle. The calibration device can use the vehicle's movement to determine the camera movement k, and then determine the absolute world coordinates by applying the camera movement k to the relative world coordinates.

[0081] Figure 5 An example of performing calibration is shown.

[0082] Reference Figure 5 The calibration equipment can use the correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern to calibrate the camera. Figure 5 In the example, the image coordinates on the left (510) and the world coordinates on the right (520) correspond to each other.

[0083] Calibration equipment can be used to update any one, any two, or any combination of more of the pitch, roll, and altitude values ​​in the camera parameters. Typically, the vehicle's motion along the x-axis or y-axis may not change much, and pitch, roll, and altitude can vary due to changes in tire pressure or the weight of passengers. Therefore, these camera parameters can be updated through calibration. However, updating parameters associated with x-axis and y-axis motion can also be performed.

[0084] Figure 6 An example of a calibration method is shown.

[0085] The following will refer to Figure 6 Describe the calibration method performed by a processor included in the calibration device.

[0086] Reference Figure 6In operation 610, the calibration device determines whether a preset pattern is detected in the driving image. The calibration device may determine whether the preset pattern is detected from the driving image captured by the camera based on a segmentation method and / or an edge detection method. When no pattern is detected from the driving image, operation 610 can be performed again. Conversely, when a pattern is detected from the driving image, operation 620 can then be performed.

[0087] In operation 620, the calibration device can extract feature points from the pattern as image coordinates.

[0088] For example, the calibration device can determine whether six or more feature points have been extracted as corner points from a pattern extracted from a self-driving image. This is because six or more feature points can be used to determine translation and rotation in the x, y, and z axes through calibration. Therefore, the calibration device can determine whether six or more feature points have been extracted from the detected pattern. For example, if six or more feature points are not extracted because the pattern is occluded or hidden by nearby vehicles or other objects, or because the pattern is worn or blurred, operation 610 can be performed again. Conversely, when six or more feature points are extracted from the pattern, operation 630 can then be performed. However, the example is not limited to this, and according to other non-limiting examples, operation 630 can then be performed when fewer or more feature points are extracted from the pattern. For example, when fewer than six feature points are extracted from the pattern, additional feature points can be predicted for the pattern based on the extracted feature points, such that the total number of extracted and predicted feature points is six or more, and operation 630 can then be performed based on the extracted and predicted feature points.

[0089] In operation 630, the calibration device can transform the image coordinates to world coordinates using the current calibration values. In this case, the constraint of the pattern on the road surface can be used as additional information. In another example, the calibration device can use a homography matrix to transform the image coordinates of the pattern to world coordinates.

[0090] In operation 640, the calibration device calculates the difference between the predicted size of the pattern and a reference size of the pattern. The reference size can be a normalized size of the pattern pre-stored in memory or identified from an HD map.

[0091] In operation 650, the calibration device determines whether the calculated difference exceeds a threshold. When the calculated difference is less than or equal to the threshold, calibration is determined to be unnecessary, and operation 610 can then be performed. Conversely, when the calculated difference is greater than the threshold, calibration is determined to be necessary, and operation 660 can then be performed. For example, the threshold may be set to 10% of a reference size. However, various other standards may be set depending on the circumstances and region.

[0092] In operation 660, the calibration device can transform the relative world coordinates of a pattern determined based on SFM into absolute world coordinates. The calibration device can determine the relative world coordinates of the pattern's movement relative to the camera using images captured by the camera at different time points. The calibration device can transform the relative world coordinates of the pattern into absolute world coordinates based on the pattern's reference size and / or the vehicle's movement.

[0093] In operation 670, the calibration device can perform calibration using the correspondence between the image coordinates of the pattern and the world coordinates (e.g., absolute world coordinates) of the pattern.

[0094] In operation 680, the calibration device may update any one or any combination of two or more of the camera's pitch, roll, and altitude. In a non-limiting example, in operation 680, the calibration device may also estimate any one or both of the vehicle's attitude and distance to another vehicle based on images taken using a camera with updated pitch, updated roll, and updated altitude, or any combination of two or more of the camera's altitude.

[0095] To describe the above reference in more detail Figure 6 The described operations can be referenced above. Figures 1 to 5 The content described is for reference only; therefore, more detailed and repetitive descriptions will be omitted here.

[0096] Regarding Figures 1 to 6The described vehicle, calibration device, memory, processor, camera, vehicle 100, calibration device 110, memory 111, processor 113, camera 115, and other devices, apparatuses, units, modules, and components are implemented or represent hardware components by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memory, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For brevity, the singular terms "processor" or "computer" are used in the description of the examples described herein; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0097] Figures 1 to 6The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0098] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0099] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card-type storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other means configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide said instructions or software and any associated data, data files, and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0100] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A processor-implemented method for calibration, comprising: Detecting preset patterns from driving images of vehicles, including those on the road surface; Transform the image coordinates in the image domain of the pattern into world coordinates in the world domain; The camera used to capture driving images is calibrated by comparing the size predicted based on the world coordinates of the pattern with a reference size of the pattern. In response to determining the calibrated camera, the relative world coordinates of the pattern's movement relative to the camera are determined using images captured by the camera at different points in time; Transform the relative world coordinates of the pattern into the absolute world coordinates of the pattern; and The camera is calibrated using the correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern. The step of determining whether to calibrate the camera includes: determining whether to calibrate the camera based on whether the difference between the predicted size and the reference size of the pattern exceeds a preset threshold. The movement of the camera is determined based on the correspondence between patterns in images taken by the camera at different points in time.

2. The method according to claim 1, wherein, The steps for determining the relative world coordinates of the pattern include: The relative world coordinates of the pattern with respect to the camera's movement are determined using the correspondence between the patterns and the camera's movement.

3. The method according to claim 1, wherein, The steps for transforming the relative world coordinates of the pattern into absolute world coordinates include: In response to the pattern having a reference size, the relative world coordinates are transformed into absolute world coordinates using the reference size.

4. The method according to claim 1, wherein, The steps for transforming the relative world coordinates of the pattern into absolute world coordinates include: In response to the pattern having a reference range, relative world coordinates are transformed into absolute world coordinates based on the vehicle's movement.

5. The method according to claim 1, wherein, The steps to calibrate a camera include: The correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern is used to calibrate any one or any combination of two or more of the camera's pitch, roll, and altitude.

6. The method according to claim 1, wherein, The preset patterns include: standardized road markings on the road over which vehicles are traveling.

7. The method according to claim 1, wherein, The steps for transforming the image coordinates of the pattern into world coordinates include: The image coordinates of the pattern are transformed into world coordinates based on the homography matrix.

8. The method according to claim 1, wherein, The steps for transforming the image coordinates of the pattern into world coordinates include: Based on the camera parameters and the constraints of the pattern on the road surface, the image coordinates of the pattern are transformed into world coordinates.

9. The method according to any one of claims 1 to 8, further comprising: Determine whether a preset number of feature points have been extracted from the preset pattern.

10. The method according to any one of claims 1 to 8, further comprising: Based on images taken using a calibrated camera, estimate either or both of the vehicle's pose and distance to another vehicle.

11. A non-transitory computer-readable storage medium for storing instructions, wherein when the instructions are executed by one or more processors, the one or more processors are configured to perform the method according to any one of claims 1 to 10.

12. A device for calibration, comprising: One or more processors are configured as follows: Detecting preset patterns from driving images of vehicles, including those on the road surface; Transform the image coordinates in the image domain of the pattern into world coordinates in the world domain; The camera used to capture driving images is calibrated by comparing the size predicted based on the world coordinates of the pattern with a reference size of the pattern. In response to determining the calibrated camera, the relative world coordinates of the pattern's movement relative to the camera are determined using images captured by the camera at different points in time; Transform the relative world coordinates of the pattern into the absolute world coordinates of the pattern; and The camera is calibrated using the correspondence between the absolute world coordinates of the pattern and the image coordinates of the pattern. In order to determine whether to calibrate the camera, the one or more processors are configured to determine whether to calibrate the camera based on whether the difference between the predicted size of the pattern and the reference size exceeds a preset threshold. The movement of the camera is determined based on the correspondence between patterns in images taken by the camera at different points in time.

13. The device according to claim 12, wherein, To determine the relative world coordinates of the pattern, the one or more processors are configured to: The relative world coordinates of the pattern with respect to the camera's movement are determined using the correspondence between the patterns and the camera's movement.

14. The device according to claim 12 or 13, wherein, In order to transform the relative world coordinates of the pattern into absolute world coordinates, the one or more processors are configured to: In response to the pattern having a reference size, the relative world coordinates are transformed into absolute world coordinates using the reference size.

15. The device according to claim 12 or 13, wherein, The device is a vehicle, and the device also includes a camera.

16. A processor-implemented method for calibration, comprising: Detect preset patterns in images captured by a camera; The camera is calibrated by comparing the predicted size of the detected pattern with a preset reference threshold for the pattern. In response to determining the calibration camera, the absolute world coordinates of the pattern are determined using images captured by the camera at different points in time; and The camera is calibrated using the correspondence between the absolute world coordinates of the pattern and the image coordinates in the image domain of the pattern. The step of determining absolute world coordinates includes: using images captured by a camera at different time points to determine the relative world coordinates of the pattern's movement relative to the camera, and transforming the relative world coordinates of the pattern into its absolute world coordinates. The step of determining whether to calibrate the camera includes: determining whether to calibrate the camera based on whether the difference between the predicted size and the reference size of the pattern exceeds a preset threshold. The movement of the camera is determined based on the correspondence between patterns in images taken by the camera at different points in time.

17. The method according to claim 16, wherein, The preset reference threshold is a preset range, and the step of determining whether to calibrate includes: determining to calibrate the camera in response to the predicted size of the pattern being within the preset range.

18. The method according to claim 16, wherein, Images captured by the camera at different times include images of a pre-defined pattern.

19. The method according to any one of claims 16 to 18, wherein, The steps to determine absolute world coordinates include: The relative world coordinates of the pattern are transformed into absolute world coordinates by matching the relative length of the relative world coordinates with the reference length of the preset pattern information to remove the scale ambiguity of the relative world coordinates.

Images