Calibration method, calibration device, and non-transitory computer recording medium

Abstract

The present invention provides a calibration method, a calibration device and a non-transitory computer recording medium, wherein lines on a road are detected from an image and the coordinates of the lines in the image domain are transformed into coordinates in the world domain to update a homography matrix representing the transformation relationship between the image domain and the world domain, and a vehicle traveling on the road is controlled based on the lines.

Description

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS]

[0002] This application claims the priority of Korean Patent Application No. 10-2018-0126540 filed in the Korean Intellectual Property Office on October 23, 2018, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure relate to a calibration method and a calibration device. Background Art

[0004] The autonomous vehicle recognizes objects such as lanes and nearby vehicles through camera sensors, thereby maintaining the lane, maintaining the distance relative to other vehicles, and generating a path by estimating the position and trajectory of the vehicle. At this time, assuming that all objects detected in the image domain are at the same height as the self, the autonomous vehicle can implement a technology for reconstructing coordinates in the image domain into coordinates in the world domain through a homography matrix.

[0005] However, in an actual road environment, due to road curvature, inclined surfaces, and vehicle height, not all objects are at the same height, let alone at the same height as an autonomous driving vehicle. Therefore, in the case of using a fixed homography operation to transform the domain of coordinates, the result of the transformation may be inaccurate, and thus errors may occur in line detection, object detection, and position and trajectory estimation. These errors may lead to driving instability, and in particular, the accuracy of detection and / or estimation may be significantly degraded in an environment where the height changes (e.g., a ramp with different heights or a road change). Summary of the invention

[0006] According to an aspect of the present disclosure, a calibration method for steering assistance or autonomous driving of a vehicle is provided, the calibration method comprising: detecting a line in an image of the road captured by the vehicle traveling on the road; transforming the coordinates of the line in the image of the road from the image domain into the coordinates of the line in the world domain based on a homography matrix representing a transformation relationship between an image domain and a world domain in which the vehicle travels; correcting the coordinates of the line in the world domain based on characteristics of at least one of the lines; updating the transformation relationship of the homography matrix based on the corrected coordinates of the line in the world domain and the coordinates of the line in the image domain; and providing the updated output result for controlling the vehicle traveling on the road based on the line.

[0007] According to an aspect of the present disclosure, there is provided a non-transitory computer recording medium having stored thereon a computer program for executing the calibration method for steering assistance or automatic driving of a vehicle.

[0008] According to an aspect of the present disclosure, a calibration method for steering assistance or autonomous driving of a vehicle is provided, the calibration method comprising: detecting a pattern of lines in an image of a road captured by the vehicle traveling on the road; transforming the coordinates of the pattern of lines from the image domain into the coordinates of the pattern of lines in the world domain based on a homography matrix representing a transformation relationship between the image domain and the world domain; correcting the coordinates of the pattern of lines in the world domain based on characteristics of the pattern of lines; updating the transformation relationship of the homography matrix based on the corrected coordinates of the pattern of lines in the world domain and the coordinates of the pattern of lines in the image domain; and providing the updated output result for controlling the vehicle traveling on the road based on the pattern of lines.

[0009] According to an aspect of the present disclosure, a calibration device for steering assistance or autonomous driving of a vehicle is provided, the calibration device comprising: a sensor configured to obtain an image of a road on which the vehicle is traveling; and a processor configured to detect lines in the image of the road, transform coordinates of the lines in the image of the road from the image domain into coordinates of the lines in the world domain based on a homography matrix representing a transformation relationship between an image domain and a world domain on which the vehicle is traveling, correct the coordinates of the lines in the world domain based on characteristics of at least one of the lines, update the homography matrix based on the corrected coordinates of the lines in the world domain and the coordinates of the lines in the image domain, and provide the updated output result for controlling the vehicle traveling on the road based on the lines.

[0010] According to an aspect of the present disclosure, a calibration device for steering assistance or autonomous driving of a vehicle is provided, the calibration device comprising: a sensor configured to detect an image of a road on which the vehicle is traveling; and a processor configured to detect a pattern of lines in the image of the road, transform coordinates of the pattern of lines from the image domain into coordinates of the pattern of lines in the world domain based on a homography matrix representing a transformation relationship between the image domain and the world domain, correct the coordinates of the pattern of lines in the world domain based on characteristics of the pattern of lines, update the transformation relationship of the homography matrix based on the corrected coordinates of the pattern of lines in the world domain and the coordinates of the pattern of lines in the image domain, and provide the updated output result for controlling the vehicle traveling on the road based on the pattern of lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The embodiments of the present disclosure will be more clearly understood by reading the following detailed description in conjunction with the accompanying drawings, in which:

[0012] Figure 1A and Figure 1B is a diagram for explaining a calibration method according to an embodiment.

[0013] Figure 2 is a flow chart of a calibration method according to an embodiment.

[0014] Figure 3 is a diagram for explaining a method of transforming image-domain coordinates of a lane detected from a driving image into world-domain coordinates according to an embodiment.

[0015] Figure 4 and Figure 5 is a diagram for explaining a method of correcting world-domain coordinates of a lane based on pattern characteristics of the lane according to an embodiment.

[0016] Figure 6 , Figure 7 and Figure 8 is a flow chart of a calibration method according to an embodiment.

[0017] Fig. 9 is a flow chart of a calibration method according to an embodiment.

[0018] Fig.10 is a diagram illustrating a pattern detected from an image according to an embodiment.

[0019] Fig.11 is a block diagram of a calibration apparatus according to an embodiment. DETAILED DESCRIPTION

[0020] The embodiments of the present invention described below may be used to display lines in augmented reality navigation systems (such as smart vehicles), or to generate visual information to assist in steering of autonomous vehicles. In addition, embodiments of the present invention may assist in safe and comfortable driving by interpreting visual information in devices including smart systems (such as heads up displays (HUDs) installed in vehicles for driving assistance or fully autonomous driving). Embodiments of the present invention may be applicable, for example, to autonomous vehicles, smart vehicles, smart phones, and mobile devices.

[0021] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, in which the same reference numerals represent the same elements.

[0022] Expressions such as “at least one of…”, when preceding a series of elements, modify the entire series of elements rather than the individual elements in the series. For example, expressions in the form of “at least one of a, b, and c,” “at least one of a, b, or c,” and “at least one of a, b, and / or c,” or similar forms of “at least one of a, b, and c,” “at least one of a, b, or c,” and “at least one of a, b, and / or c” should be understood to include combinations of only a, only b, only c, both a and b, both a and c, both b and c, and all of a, b, and c.

[0023] Hereinafter, 'road' may include expressways, national highways, local roads, high-speed national highways, etc. on which the user's vehicle travels. A road may include one or more lanes. 'Driving lane' may correspond to any one of the multiple lanes occupied by the driving vehicle (the lane on which the driving vehicle travels). 'Lane' may be distinguished from each other by lane markings marked on the road surface. One lane may be identified by lane markings on the left and right sides of the lane. 'Road markings' are markings on the surface of the road on which the vehicle travels, and may include not only lane markings (such as center lines and guide lines or edge lines), but also non-lane markings, including signs such as lane change lines, left turn prohibition lines, reflectors, physical obstacles and walls, electrical signals emitted by transmitters for sensor detection, and driving direction guide lines and letters (such as child protection zones and deceleration zones).

[0024] Figure 1A and Figure 1B is a diagram for explaining a calibration method according to an embodiment. Figure 1A , Figure 1A Schematic diagram 110 , schematic diagram 130 , and schematic diagram 150 are shown. Schematic diagram 110 shows a lane, schematic diagram 130 shows a crosswalk, and schematic diagram 150 shows other road markings.

[0025] Each of graphs 110 , 130 , and 150 may include specific patterns that may be used to determine whether the position or pose of the camera has changed or to identify specific patterns that may be used as landmarks, such as lanes, crosswalks, and other road markings provided on the road surface.

[0026] The calibration device according to an embodiment may perform calibration by updating the homography matrix using pattern characteristics contained in the image when the position or posture of the camera changes, even without map information, reference points or other lane specification information. The pattern characteristics may include various characteristics with specific rules and / or repetitions, such as the size of the pattern, the symmetry of the pattern, the shape of the pattern, the parallelism between the shapes that constitute the pattern, the length of the shape, the spacing between the shapes, and the width of the shape. For example, when a single image includes all patterns (such as lanes, crosswalks, and road markings), the calibration device may use any one of the patterns or all of them together to improve the accuracy of the calibration. In the following, reference will be made to Figure 1B The pattern characteristics of a line as an example of a pattern are explained.

[0027] Reference Figure 1B , a diagram 170 shows pattern characteristics of lines detected in a driving image. The pattern characteristics of lines according to an embodiment may include, for example, parallel characteristics 171 of the lines, lengths 173 of the lines, widths 175 of lanes (e.g., latitudinal distances between lines), and intervals 177 between line segments.

[0028] The parallel characteristic 171 between lines indicates that the lines on the same road or the lines constituting (defining) one lane ('lane divider') are parallel to each other. For example, when the lines are intermittent dashed lines, the length 173 of the line may correspond to the length of a partial line included in one lane on the road. The width 175 of the lane may correspond to the latitudinal width (or distance) between the first line and the second line on the same road. The interval 177 between the dashed lines may correspond to, for example, the interval between the partial dashed lines included in the line, that is, the interval between the first partial line and the second partial line included in the second lane. The length 173 of the line, the width 175 of the lane, and the interval 177 between the lines may be determined according to the lane specification in the jurisdiction. The lane specification in the jurisdiction may vary according to the road type (or road class) according to the country or other jurisdiction.

[0029] Figure 2 is a flow chart of a calibration method according to an embodiment.

[0030] Reference Figure 2, the calibration device according to the embodiment detects lines included in the driving road from the driving image of the vehicle (operation 210). The vehicle can perform collision avoidance by detecting nearby vehicles, driving lanes, and road markings including lines from the captured driving image, and perform path finding and direction control by recognizing and / or detecting signs and traffic lights. In the following, 'vehicle' may be understood as a car with an autonomous driving function and / or an advanced driver assistance (ADA) function.

[0031] When driving a vehicle, a driving image may be obtained as a two-dimensional (2D) image using an image capture device mounted on the vehicle and configured to capture images of a front view, a rear view, and / or a side view of the vehicle. Alternatively, the driving image may be acquired via a communication interface (e.g., Fig.11 The communication interface 1170 shown in the figure may receive an external image of at least one of a front view, a rear view and / or a side view of the vehicle from outside the calibration device. The driving image may include, for example, a road image and / or a road surface image, such as Figure 3 A two-dimensional driving image 310 is shown. The driving image may include various images other than color images, such as infrared images. The driving image may include a single frame or may include multiple consecutive frames.

[0032] In operation 210, the calibration device may detect lines from the driving image. The lines may be detected by identifying the pixels of the lines from the driving image using, for example, a pixel-wise scene segmentation technique or an edge detection technique. Alternatively, the calibration device may detect the lines from the driving image using a neural network trained to recognize lines. For example, the neural network may be trained using a bounding box of a lane marking to detect the type of lane marking to be detected.

[0033] The calibration device transforms the coordinates of the lines in the image domain into coordinates in the world domain based on a preset homography matrix (operation 220). That is, the coordinates of the lines in the image can be transformed into entity dimensions in the real world environment surrounding the vehicle. The homography matrix can represent the constrained transformation relationship established between the corresponding projected points when one plane is projected onto another plane. The homography matrix can, for example, represent the transformation relationship (correspondence relationship) in the case of transforming the coordinates in the two-dimensional image domain into the corresponding three-dimensional world domain coordinates. The preset homography matrix can be, for example, an initial homography matrix determined based on the intrinsic and extrinsic parameters used during the calibration of the image capture device (camera). The following will refer to Figure 3 The method by which the calibration device transforms the coordinates of a line in the image domain into the coordinates in the world domain is described in detail.

[0034] The calibration device corrects the coordinates in the world domain transformed in operation 220 based on the pattern characteristics of the lines (operation 230). The pattern characteristics of the lines may include, for example, parallel characteristics of the lines, lengths of the lines, widths of lanes, and intervals between the lines.

[0035] In operation 230, the calibration device may select at least one reference line from the lines based on, for example, the position of the vehicle. The calibration device may measure reference line characteristics of the reference line. The calibration device may correct the world coordinates of at least one of the lines other than the reference line based on the reference line characteristics. The reference line may correspond to, for example, a line closest to the position of the vehicle or a line close to the center of the driving image. In general, calculations performed by the homography matrix may be more reliable when the distance relative to the line is close and toward the center of the image.

[0036] Therefore, according to an embodiment, a line close to the center of the driving image or a line closest to the position of the vehicle (e.g., any one of the lane dividing lines of the driving lane) may be selected as a reference line, and a reference line characteristic of the reference line may be measured, thereby improving the accuracy of the correction. In this case, the reference line characteristic may include, for example, a characteristic that the reference line is parallel to the remaining lines, the length of the reference line, the distance between the reference line and the remaining lines adjacent to each other in the latitudinal direction, the interval between some lines in the reference line, etc. The reference line characteristic may be referred to as a pattern or pattern characteristic of the reference line. The reference line characteristic may be for only the reference line or may be for one or more lines, such as the distance between lines spanning the width of the road.

[0037] According to an embodiment, after determining the accuracy of the selected reference line, the calibration device may use the reference line for calibration. The calibration device may, for example, select a line closest to the position of the vehicle as the reference line. The calibration device determines the length of the reference line, the distance between the reference line and a line close to the reference line in the longitudinal direction, and the distance between any one of the partial lines of the reference line and a partial line adjacent to the any one of the partial lines in the latitudinal direction. At this time, when there is information including the length of the line, the distance between the lines in the longitudinal direction, and the distance between the lines in the latitudinal direction on the high-precision map, the calibration device may perform correction without performing measurement.

[0038] For example, on the current road surface, when the differences between the length of each line, the distance between the lines in the longitudinal direction, and the distance between the lines in the latitudinal direction and Cambria Math are less than a preset threshold, the calibration device can correct the x-coordinate and y-coordinate in the world domain.

[0039] The calibration device may correct the x-coordinate (of the line) in the world domain, for example, based on at least one of the parallel characteristics between the lines and the distance between the lanes in the latitudinal direction. Figure 4The method of calibrating the x-coordinates of the lines in the world domain is described in detail. The calibration device may also calibrate the y-coordinates of the lines in the world domain, for example, based on at least one of the length of the lines and the spacing between the lines. Figure 5 A method for the calibration device to correct the y coordinate of a line in the world domain is described in detail.

[0040] The calibration device may update the reference line characteristic based on at least one of a line characteristic of the driving road according to the high-resolution map and a lane characteristic of the driving road according to a jurisdiction.

[0041] The calibration apparatus updates the homography matrix based on the pairs of the coordinates in the three-dimensional world domain and the coordinates in the image domain corrected in operation 230 (operation 240 ).

[0042] The calibration device may output the result of the calibration explicitly or implicitly. The output of the calibration result may explicitly include, for example, displaying the position of the line corrected by the calibration or the world coordinates of the corrected line on a screen (or map) and / or outputting the world coordinates of the corrected line through audio. On the other hand, the output of the calibration result may implicitly include, for example, controlling the vehicle based on the corrected position of the line, determining the position of the vehicle, and setting or changing the driving route.

[0043] Figure 3 is a diagram for explaining a method of transforming image-domain coordinates of a lane detected from a driving image into world-domain coordinates according to an embodiment.

[0044] Figure 3 A driving image 310 and a schematic diagram 330 are shown, which shows the result of transforming the coordinates of the lane detected from the driving image 310 in the image domain into coordinates in the world domain using a preset homography matrix.

[0045] The calibration device according to the embodiment can divide the portion corresponding to the object (e.g., line) included in the driving image 310 into semantic units by using, for example, semantic segmentation technology and detecting the object at the pixel level, thereby generating a segmented image. By using the semantic segmentation technology, the object can be densely predicted pixel by pixel, the object included in the driving image 310 can be determined, and the unit area including the object category can be output pixel by pixel. The calibration device can generate a segmented image by a classification network including, for example, a plurality of convolutional layers and a fully connected layer. The calibration device can use a well-known classification network (such as AlexNet, Visual Geometry Group (VGG) network and GoogleNet) to divide the object from the driving image 310 into semantic units, determine the meaning of the divided area pixel by pixel, and mark each category, thereby detecting the pixels of the line from the segmented image.

[0046] The calibration device may transform the coordinates of the pixels corresponding to the lines in the segmented image in the two-dimensional image domain into coordinates in the three-dimensional world domain using a preset homography matrix (or an initial homography matrix). As shown in diagram 330, the pixels corresponding to the lines whose coordinates are transformed into coordinates in the three-dimensional world domain may be detected.

[0047] The coordinates of the pixels corresponding to the lines in the three-dimensional world domain transformed using the initial homography matrix may be different from the actual coordinates of the lines due to, for example, changes in the pose and / or position of the camera used to capture the driving image, scale differences caused by differences between the heights of the road surface, and / or imbalances caused by road surface banks. According to an embodiment, using the pattern characteristics of the lines, such differences may be corrected more quickly and accurately.

[0048] Figure 4 is a diagram for explaining a method of correcting world-domain coordinates of a line based on pattern characteristics of the line according to an embodiment.

[0049] Figure 4 Schematic diagram 410 and schematic diagram 430 are shown. Schematic diagram 410 shows the result of transforming the coordinates of a line into coordinates in the world domain, and schematic diagram 430 shows the result of correcting the x coordinate of the line in the world domain.

[0050] The calibration device according to an embodiment may select a reference line (e.g., the second line from the left) from any of the lines shown in the schematic diagram 410 based on the position of the vehicle. The calibration device may, for example, select the pixel 415 of the second line closest to the vehicle from the pixels of the lines shown in the schematic diagram 410. Since the pixel closest to the vehicle is different from the vehicle in height, the calibration device may select such a pixel and correct the x-coordinates of other lines, thereby improving the accuracy of the correction.

[0051] The calibration device may measure the x-coordinates (latitude positions) of the pixels 415 of the second line. The calibration device may cluster the coordinates of the lines on the left and right sides of the second line in the world domain line by line based on the positions of the pixels 415 of the second line. At this time, the calibration device may cluster the pixels included in each line, for example, by a connected component operation. The connected component operation is an operation used in graph theory and is used to connect a dependent graph including vertices connected by at least one path in the dependent graph. The calibration device may cluster the pixels corresponding to the same line into the same group by a connected component operation.

[0052] The calibration device may correct the x-coordinates (latitude positions) of the lines in the world domain so that the clustered lines are parallel to each other around the x-coordinates of the pixel 415 of the second line and the distances between the clustered lines are the same. When the calibration device corrects the x-coordinates of the lines in the world domain, the lines may be arranged in parallel at regular intervals as shown in the diagram 430.

[0053] Figure 5 is a diagram for explaining a method of correcting world-domain coordinates of a line based on pattern characteristics of the line according to an embodiment.

[0054] Figure 5 Schematic diagram 510 and schematic diagram 530 are shown. Schematic diagram 510 shows the result of correcting the x-coordinate of the line in the world domain, and schematic diagram 530 shows the result of correcting the y-coordinate of the line in the world domain.

[0055] The calibration apparatus according to the embodiment may correct the y coordinate (longitudinal position) of the line in the world domain after correcting the x coordinate (latitude position) of the line in the world domain.

[0056] For example, in the case of correcting the x-coordinates of the lines in the world domain, the calibration device may select, for example, pixels 515 of the second line closest to the vehicle from the pixels of the lines shown in the diagram 510. The calibration device may cluster the lines whose x-coordinates (latitude positions) are corrected in the diagram 510 based on the partial lines 531, 532, 533, 534, and 535 shown in the diagram 530.

[0057] The calibration device may set the closest partial line 531 among the fully visible partial lines 531, 532, 533, 534, and 535 around the pixel 515 of the second line as a reference partial line to determine the lengths of the partial lines 532, 533, 534, and 535. The calibration device may correct the y coordinates of the remaining partial lines 532, 533, 534, and 535 so that the length of each of the remaining partial lines 532, 533, 534, and 535 is equal to the length of the reference partial line 531. In addition, the calibration device may set the partial line 533 included in the same line as the pixel 515 of the second line as a partial reference line to determine the interval between the partial lines. The calibration device may correct the y coordinates of the remaining partial lines 531, 532, 534, and 535 so that the interval between the pixel 515 of the second line and the reference partial line 533 is equal to the interval between the remaining partial lines 531, 532, 534, and 535.

[0058] Figure 6 is a flow chart of a calibration method according to another embodiment.

[0059] Reference Figure 6When acquiring a driving image of a vehicle (operation 610), the calibration device according to an embodiment may determine whether line pixels have been detected from the driving image (operation 620). When it is determined in operation 620 that line pixels have not been detected, the calibration device may terminate the operation.

[0060] In addition, when it is determined in operation 620 that the line pixels are detected, the calibration apparatus may transform the coordinates of the detected line pixels from the coordinates in the image domain to the coordinates in the world domain using the homography matrix preset in operation 630 (operation 640).

[0061] The calibration device may correct the coordinates of the line pixels transformed into the coordinates relative to the world domain in operation 640 in the latitudinal direction based on the latitudinal distance between the lines and / or the line parallel characteristics (operation 650).

[0062] The calibration device may correct the coordinates of the line pixels corrected in the latitudinal direction in operation 650 in the longitudinal direction based on the lengths of the lines and / or intervals between the lines (operation 660).

[0063] The calibration device may update the homography matrix preset in operation 630 based on the coordinates in the world domain and the coordinates in the image domain corrected in operation 660 (operation 670). The calibration device may perform calibration based on the updated homography matrix.

[0064] Figure 7 is a flow chart of a calibration method according to another embodiment.

[0065] Figure 7 The calibration device according to the embodiment is shown by not only taking into account the above reference Figure 6 An embodiment of updating the homography matrix by also taking into account pattern characteristics of road markings other than lines included in the driving road is described.

[0066] When a driving image of a vehicle is obtained (operation 710), the calibration device may detect line pixels from the driving image (operation 720). The calibration device may transform the coordinates of the detected line pixels from the coordinates in the image domain to the coordinates in the world domain using the homography matrix preset in operation 730 (operation 740).

[0067] The calibration device may correct the coordinates of the line pixels transformed into the coordinates relative to the world domain in operation 740 in the latitudinal direction based on the latitudinal distance between the lines and / or the line parallel characteristics (operation 750).

[0068] The calibration device may correct the coordinates of the line pixels corrected in the latitudinal direction in operation 750 in the longitudinal direction based on the lengths of the lines and / or intervals between the lines (operation 760).

[0069] Meanwhile, when obtaining a driving image of the vehicle (operation 710), the calibration device may detect pixels corresponding to road markings other than lines (e.g., crosswalk markings) included in the driving road (operation 770). The calibration device may transform the coordinates of the road markings in the image domain into the coordinates of the road markings in the world domain based on the homography matrix preset in operation 730 (operation 780).

[0070] The calibration device may correct the coordinates of the road marking in the world domain in the longitudinal direction and the latitudinal direction based on the pattern characteristics of the road marking (e.g., the length and / or parallel characteristics of the road marking, etc.) (operation 790). The pattern characteristics of the road marking may include, for example, the width of the road marking, the length of the road marking, the size of the road marking, the symmetry of the road marking, the parallel characteristics, and the shape of the road marking. For example, in operation 790, the calibration device may correct the x-coordinate of the road marking in the world domain based on the length of the road marking and correct the y-coordinate of the road marking in the world domain based on the width of the road marking.

[0071] The calibration apparatus may update the homography matrix based on the coordinates of the line pixels corrected through operations 720 to 760 and / or the coordinates of the road marking in the world domain corrected through operations 770 to 790 (operation 795 ).

[0072] According to an embodiment, the homography matrix may be updated by taking into account pattern characteristics of road markings other than lines included in the driving road in addition to the pattern characteristics of the lines, so that the accuracy of calibration may be improved.

[0073] Figure 8 is a flow chart of a calibration method according to an embodiment.

[0074] See also Figure 8 When a two-dimensional driving image of the vehicle is obtained (operation 805), the calibration device according to an embodiment may transform the two-dimensional driving image into a three-dimensional top view image (operation 815) using the initial calibration matrix preset in operation 810. At this time, the calibration device may transform the coordinates of the line pixels detected from the two-dimensional driving image into coordinates in the three-dimensional world domain.

[0075] The calibration apparatus may perform first clustering of the coordinates of the line pixels transformed into the coordinates in the world domain in operation 815 (operation 820). The calibration apparatus may perform first clustering of pixels corresponding to the same lane to cluster into the same group.

[0076] The calibration device may perform latitude correction (x-coordinate correction) of each line so that the lines of the first cluster are parallel to each other and the latitude distances between the clustered lanes are equal to each other (operation 825).

[0077] In operation 830, the calibration device may perform a second clustering of each of the line pixels corrected in the latitudinal direction. The calibration device may perform a second clustering of each of the line pixels corrected in the latitudinal direction according to the partial lines. The calibration device may longitudinally correct (y coordinate correction) the partial lines of the second cluster based on, for example, the length of the partial lines and / or the intervals between the lines (operation 835).

[0078] The calibration device may scale the line (part of the line) finally calibrated through the above operation (operation 840). The calibration device may adjust the height ratio (or size ratio) between the image domain and the world domain through scaling.

[0079] The calibration device may match the coordinates of the lines in the world domain scaled in operation 840 with the coordinates in the image domain corresponding to the coordinates in the world domain, respectively, into pairs (operation 845). The calibration device may update the calibration parameters based on the coordinate pairs matched in operation 845 (operation 850). The calibration device may also update the initial calibration matrix by applying the updated calibration parameters to the initial calibration matrix preset in operation 810.

[0080] Fig. 9 is a flow chart of a calibration method according to another embodiment. Fig.10 is a diagram illustrating a pattern detected from an image according to an embodiment.

[0081] Reference Fig. 9 and Fig.10 , the calibration apparatus according to the embodiment detects a pattern from an image (operation 910). For example, when Fig.10 As shown, when multiple patterns 1010 and 1030 (having one or more repeating features or shapes) are included in an image, the calibration device can detect any one or both of the patterns.

[0082] The calibration apparatus transforms the coordinates of the pattern in the image domain into the coordinates of the pattern in the world domain based on a preset homography matrix (operation 920).

[0083] The calibration device corrects the coordinates of the pattern in the world domain based on the characteristics of the pattern (operation 930). At this time, the pattern characteristics may include various characteristics, such as the size of the pattern, the symmetry of the pattern, the shape of the pattern, the parallelism between the shapes constituting the pattern, the length of the shape, the interval between the shapes, and the width of the shape.

[0084] In operation 930, the calibration device may correct the x-coordinate of the pattern in the world domain based on at least one of, for example, a parallel characteristic between shapes constituting the pattern and a distance between the shapes constituting the pattern. The calibration device may cluster the coordinates of the pattern in the world domain according to the shapes constituting the pattern. The calibration device may correct the x-coordinate of the pattern in the world domain so that the clustered shapes are parallel to each other and the distances between the clustered shapes are equal to each other.

[0085] In addition, the calibration device may correct the y coordinate of the pattern in the world domain based on at least one of, for example, the length of the shapes constituting the pattern and the intervals between the shapes constituting the pattern in operation 930. The calibration device may correct the y coordinate of the pattern in the world domain so that the clustered shapes are identical to each other and the intervals between the clustered shapes are equal to each other.

[0086] The calibration apparatus updates the homography matrix based on the pairs of the coordinates of the pattern in the world domain and the coordinates of the pattern in the image domain corrected in operation 930 (operation 940 ).

[0087] Fig.10 Patterns 1010 and 1030 detected from an image are shown. Pattern 1010 corresponds to a pattern detected at the ceiling of a room in an upper portion of the image, and pattern 1030 corresponds to a pattern detected at the floor of a room in a lower portion of the image. The calibration device according to an embodiment may perform calibration using the size of the pattern detected from the image, the symmetry of the pattern, and the shape of the pattern. In addition to the characteristics of the pattern, the calibration device may perform calibration by considering the parallel characteristics between the shapes constituting the pattern, the length of the shapes, the intervals between the shapes, and the latitudinal distances between the shapes.

[0088] According to an embodiment, the calibration device may perform calibration using the pattern 1010 of the upper portion of the image and may perform calibration using the pattern 1030 of the lower portion of the image.

[0089] The calibration device according to an embodiment can facilitate calibration for line detection and be used in various fields including mobile cleaning robots or autonomous factories by updating calibration parameters using patterns simply recognized from images (e.g., line patterns, crosswalk patterns, ground patterns, or ceiling patterns) without even requiring information such as map information or line specifications based on road specifications as landmarks.

[0090] Fig.11 is a block diagram of a calibration device according to an embodiment. Fig.11According to an embodiment, the calibration device 1100 includes a sensor 1110, a processor 1130, and a communication interface 1170. The calibration device 1100 may further include a memory 1150 and a display 1190. The sensor 1110, the processor 1130, the memory 1150, the communication interface 1170, and the display 1190 may communicate with each other via at least one communication bus 1105.

[0091] The sensor 1110 may include, for example, a camera sensor, an image sensor, and a visual sensor. The camera, the image sensor, and / or the visual sensor may be installed on the vehicle to capture a driving image of the vehicle.

[0092] The processor 1130 may execute the above-mentioned steps with reference to FIG. 1 to Fig.10 The algorithm of at least one of the methods described herein corresponds to computer readable instructions of the memory. The processor 1130 may execute the program from the memory and control the calibration device 1100. The program code executed by the processor 1130 may be stored in the memory 1150.

[0093] The processor 1130 may include, for example, a central processing unit (CPU) or a graphics processing unit (GPU).

[0094] The memory 1150 may store the driving image and the pairs of the corrected coordinates in the world domain and the coordinates in the image domain. The memory 1150 may also store the homography matrix updated by the processor 1130 and / or the result of calibration based on the updated homography matrix. The memory 1150 may include a volatile memory or a non-volatile memory.

[0095] The communication interface 1170 may receive a driving image captured outside the calibration device 1100 or information from various sensors outside the calibration device 1100, map information, etc. According to an embodiment, the communication interface 1170 may send the homography matrix updated by the processor 1130 and / or the result of the calibration performed based on the updated homography matrix to the outside of the calibration device 1100 or to the display 1190.

[0096] The display 1190 may display the position of the object modified by the calibration performed based on the updated homography matrix together with the driving image or separately from the driving image. The display 1190 may display the coordinates of the object in the world domain updated by the calibration, for example, in the form of the position of the object in the map information or navigation image or in the form of the updated coordinates of the object in the world domain. For example, when the calibration device 1100 is embedded in a vehicle, the display 1190 may be configured as a head-up display (HUD) installed in the vehicle.

[0097] The above embodiments may be implemented with hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented in a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (e.g., an array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device may be described as being used alone, but a person of ordinary skill in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. Other processing configurations (e.g., parallel processors) are also possible.

[0098] Software may include computer programs, codes, instructions, or a combination of one or more of the above to configure a processing device to run as needed or to command a processing device independently or generally. Software and / or data may be permanently or temporarily implemented on any type of machine, component, physical device, virtual device, computer storage medium or device or on a transmitted signal wave to be interpreted by a processing device or may be permanently or temporarily implemented to provide commands or data to a processing device. Software may be distributed through networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0099] The method according to the embodiment can be implemented in the form of program commands that can be executed by various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium can include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium can be specially designed and configured for the embodiment or can be published and used by a person of ordinary skill in the field of computer software. Examples of computer-readable recording media include hardware devices that are specially configured to store and execute program instructions, such as: magnetic media, such as hard disks, floppy disks and tapes; optical recording media, such as compact disc read-only memory (CD-ROM), digital versatile disk (DVD), etc.; magneto-optical media, such as optical magnetic floppy disks (floptical discs), read-only memory (ROM), random access memory (RAM), flash memory, etc. Examples of program commands include machine language codes (e.g., codes generated by a compiler) and high-level language codes that can be executed by a computer using an interpreter, etc. The above-described hardware devices may be configured to operate as one or more software modules to perform the operations of the described embodiments, and vice versa.

[0100] Although the present disclosure has been specifically shown and described with reference to the embodiments of the present disclosure, it should be understood that the disclosure is not limited to the embodiments. For example, it should be understood that even when the techniques can be implemented in a different order than the method, and / or the components of the system, structure, device, circuit, etc. can be replaced or substituted by other components or equivalent components, appropriate results can still be obtained. Therefore, other implementations, other embodiments, and equivalent forms of the claims are also within the scope of the above claims.

Claims

1. A calibration method for steering assistance or autonomous driving of a vehicle, the calibration method comprising: detecting lines in an image of the road captured by the vehicle traveling on the road; transforming the coordinates of the line in the image of the road from the image domain into the coordinates of the line in the world domain based on a homography matrix representing a transformation relationship between the image domain and the world domain in which the vehicle travels; Correcting the coordinates of the lines in the world domain based on characteristics of at least one of the lines, wherein the characteristics of the at least one of the lines include at least one of parallel characteristics between the lines, lengths of the lines, latitudinal distances between the lines, and intervals between line segments, wherein the line is a dashed line consisting of a plurality of the line segments; updating the transformation relationship of the homography matrix based on the corrected coordinates of the line in the world domain and the coordinates of the line in the image domain; and providing the updated output for controlling the vehicle traveling on the road based on the line; Wherein the correcting the coordinates in the world domain comprises: correcting the x-coordinates of the lines in the world domain based on at least one of the parallel characteristics between the lines and the latitudinal distances between the lines; and The y-coordinate of the line in the world domain is corrected based on at least one of the length of the line and the spacing between the line segments.

2. The calibration method according to claim 1, wherein the correcting the x-coordinate in the world domain comprises: clustering the coordinates in the world domain according to the line; as well as The x-coordinates in the world domain are corrected so that the lines on which the coordinates are clustered are parallel to each other and the latitude distances between the lines are equal to each other.

3. The calibration method according to claim 2, wherein the correcting the y coordinate in the world domain comprises: clustering the clustered lines according to partial line segments; as well as The y coordinates in the world domain are corrected so that lengths of the clustered partial line segments are equal to each other and intervals between the clustered partial line segments are equal to each other.

4. The calibration method according to claim 1, wherein the correcting the coordinates in the world domain comprises: selecting at least one reference line from the lines based on the position of the vehicle; measuring a reference pattern characteristic of the selected at least one reference line; as well as The coordinates of the lines in the world domain other than the selected at least one reference line are corrected based on the reference pattern characteristics of the selected at least one reference line.

5. The calibration method according to claim 4, wherein the reference pattern characteristics include at least one of a property that the at least one reference line and the lines in the world domain other than the at least one selected reference line are parallel to each other, a length of the at least one reference line, a distance between the at least one reference line and the lines in the world domain other than the at least one selected reference line adjacent to the at least one reference line in the latitudinal direction, and an interval between some line segments in the at least one reference line.

6. The calibration method of claim 4, further comprising updating the reference pattern characteristics based on at least one of line characteristics of a driving road according to a high-resolution map and line characteristics of the road according to road regulations of a jurisdiction of the road.

7. The calibration method according to claim 1, further comprising: detecting road markings other than the line in the image of the road; transforming the coordinates of the road marking in the image domain into the coordinates of the road marking in the world domain based on the homography matrix; correcting the coordinates of the road marker in the world domain based on characteristics of the road marker; updating the transformation relationship of the homography matrix based on the corrected coordinates of the road sign in the world domain and the coordinates of the road sign in the image domain; as well as The updated output is provided for controlling the vehicle traveling on the road based on the road markings. 8 . The calibration method of claim 7 , wherein the pattern characteristics of the road marking include at least one of a width of the road marking, a length of the road marking, a size of the road marking, a symmetry of the road marking, and a shape of the road marking.

9. The calibration method according to claim 8, wherein the correcting the coordinates of the road markings in the world domain comprises: correcting an x-coordinate of the road marker in the world domain based on the length of the road marker; as well as A y-coordinate of the road marking in the world domain is corrected based on the width of the road marking.

10. The calibration method of claim 1, wherein the detecting comprises detecting pixels of the line in the image of the road using at least one of a pixel-based scene segmentation technique and an edge detection technique. The calibration method of claim 1 , wherein the image comprises at least one of a single frame and continuous frames.

12. A non-transitory computer recording medium having stored thereon a computer program for executing the calibration method according to claim 1.

13. A calibration method for steering assistance or autonomous driving of a vehicle, the calibration method comprising: detecting patterns in an image of the road captured by the vehicle traveling on the road; transforming the coordinates of the pattern from the image domain into the coordinates of the pattern in the world domain based on a homography matrix representing a transformation relationship between the image domain and the world domain; correcting the coordinates of the pattern in the world domain based on characteristics of the pattern, wherein the characteristics of the pattern include at least one of a size of the pattern, a symmetry of the pattern, a shape of the pattern, parallel characteristics between shapes constituting the pattern, and a length of the shape; updating the transformation relationship of the homography matrix based on the corrected coordinates of the pattern in the world domain and the coordinates of the pattern in the image domain; as well as providing the updated output for controlling the vehicle traveling on the road based on the pattern; Wherein the correcting the coordinates of the pattern in the world domain comprises: correcting an x-coordinate of the pattern in the world domain based on the parallelism between the shapes constituting the pattern; as well as A y-coordinate of the pattern in the world domain is corrected based on the length of the shape.

14. A calibration device for steering assistance or automatic driving of a vehicle, comprising: a sensor configured to obtain an image of a road on which the vehicle is traveling; as well as a processor configured to detect lines in the image of the road, transform coordinates of the lines in the image of the road from the image domain into coordinates of the lines in the world domain based on a homography matrix representing a transformation relationship between the image domain and a world domain in which the vehicle travels, correct the coordinates of the lines in the world domain based on a characteristic of at least one of the lines, update the homography matrix based on the corrected coordinates of the lines in the world domain and the coordinates of the lines in the image domain, and provide the updated output for controlling the vehicle traveling on the road based on the lines; wherein the characteristic of the at least one of the lines comprises at least one of a parallel characteristic between the lines, a length of the lines, a latitudinal distance between the lines, and an interval between line segments, wherein the line is a dashed line composed of a plurality of the line segments; wherein the processor corrects the x-coordinates of the lines in the domain of the world based on at least one of the parallel characteristics between the lines and the latitudinal distances between the lines and corrects the y-coordinates of the lines in the domain of the world based on at least one of the lengths of the lines and the spacings between the line segments.

15. The calibration device according to claim 14, wherein the processor clusters the coordinates in the world domain according to the lines and corrects the x coordinates in the world domain so that the clustered lines are parallel to each other and the latitude distances between the clustered lines are equal to each other.

16. The calibration device according to claim 15, wherein the processor clusters the clustered lines according to partial line segments and corrects the y coordinates in the world domain so that the lengths of the clustered partial line segments are equal to each other and the intervals between the clustered partial line segments are equal to each other.

17. A calibration device according to claim 14, wherein the processor selects at least one reference line from the lines based on the position of the vehicle, measures reference pattern characteristics of the selected at least one reference line, and corrects the coordinates of the lines in the world domain other than the selected at least one reference line based on the reference pattern characteristics of the selected at least one reference line.

18. The calibration device according to claim 17, wherein the reference pattern characteristics include at least one of a property that the at least one reference line is parallel to the lines in the world domain except the at least one selected reference line, a length of the at least one reference line, a distance between the at least one reference line and the lines in the world domain except the at least one selected reference line adjacent to the at least one reference line in the latitudinal direction, and a spacing between some line segments in the at least one reference line.

19. The calibration device of claim 17, wherein the processor updates the reference pattern characteristics based on at least one of line characteristics of a driving road according to a high-resolution map and line characteristics of the road according to road regulations of a jurisdiction of the road.

20. The calibration device of claim 14, wherein the processor detects road markings other than the line in the image of the road, transforming the coordinates of the road sign in the image domain into the coordinates of the road sign in the world domain based on the homography matrix, correcting the coordinates of the road marker in the world domain based on characteristics of the road marker, updating the transformation relationship of the homography matrix based on the corrected coordinates of the road sign in the world domain and the coordinates of the road sign in the image domain; and The updated output is provided for controlling the vehicle traveling on the road based on the road markings.

Images