A method and device for calibrating the camera spacing of a vehicle, and a method and device for its continuous learning of a vanishing point estimation model

Abstract

Disclosed is a method for calibrating the camera spacing of a vehicle. A calibration device: (a) inputs multiple driving images from a camera into an object detection network to detect multiple objects and generate multiple object detection information, and inputs the same into a lane detection network to detect multiple lanes and generate lane detection information; (b) generates multiple object profiling information by analyzing the multiple object detection information, and generates multiple lane profiling information by analyzing the multiple lane detection information, inputs the multiple object profiling information into an object-based spacing estimation module to select a first target object and a second target object and generate a first spacing and a second spacing, and generates a third spacing and a fourth spacing by inputting vanishing point detection information and the multiple lane profiling information into a lane-based spacing estimation module; (c) inputs the first spacing to the fourth spacing into a spacing determination module to generate a determined spacing.

Description

[0001] This invention claims the priority and benefits of U.S. Patent Application No. 63 / 014,884, filed with the U.S. Patent Office on April 24, 2020, and U.S. Patent Application No. 17 / 125,087, filed with the U.S. Patent Office on December 17, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to a method and apparatus for calibrating the pitch of a vehicle's camera using vanishing point, object, and lane information, and a method for continuously learning a vanishing point estimation model. Background Art

[0003] Today's vehicles incorporate IT technology to provide various functions and are developing in the form of installing various Advanced Driver Assist Systems (ADAS) to improve the driving stability of the vehicle and ensure user convenience.

[0004] At this time, the Advanced Driver Assist System (ADAS) performs functions such as adaptive headlights, forward collision avoidance, lane departure prevention, blind spot monitoring, and improved rear monitoring using advanced sensing devices and intelligent imaging devices.

[0005] The driver assistance system analyzes the driving image transmitted from video image detection devices such as cameras, lidar sensors, and radar sensors based on a perception model to detect the surrounding environment information of the vehicle.

[0006] Moreover, when analyzing the driving image, it is necessary to measure the distance between an object in the driving image and the vehicle. For this purpose, after calculating the pitch of the camera, the distance between the object and the vehicle in the actual driving environment is calculated using the calculated pitch.

[0007] On the other hand, during vehicle driving, the vehicle rolls in grooves on the road surface of the lane, and accordingly, the pitch of the camera fixedly installed in the vehicle also changes, so it is necessary to accurately estimate it.

[0008] In addition, to estimate the pitch of the camera, usually, the vanishing point in the driving image is used to estimate the pitch, or ego-motion, that is, the motion of the camera, is used to estimate the pitch.

[0009] For a method using the vanishing point of a driving image, the vanishing point in the driving image is detected and estimated by calculating the angle between the origin of the camera and the vanishing point and the center point of the driving image. The advantage is that no separate point for distance estimation is required and it can be fully automated. However, the disadvantage is that depending on the driving environment, it may be difficult to detect the vanishing point in the driving image, resulting in a decrease in the accuracy of distance estimation.

[0010] In addition, for a method using self-motion, the moving state of an object in each frame is calculated by using the front and rear frames of the driving image to estimate the distance. The advantage is that it has higher accuracy compared to the method using the vanishing point. However, the disadvantage is that since the front and rear frames of the driving image need to be analyzed, more computational effort is required compared to the method using the vanishing point, thus requiring more computational resources. Moreover, due to unstable features or irregular illuminance changes occurring according to the driving environment, the motion cannot be accurately detected. Summary of the Invention

[0011] Technical Problem

[0012] An object of the present invention is to solve all the above problems.

[0013] Another object of the present invention is to be able to accurately estimate the camera distance of a vehicle compared to the prior art.

[0014] Still another object of the present invention is to be able to estimate the camera distance of a vehicle with less computational effort compared to the prior art.

[0015] Yet another object of the present invention is to be able to continuously learn a vanishing point estimation model for detecting the vanishing point.

[0016] Technical Solution

[0017] According to an embodiment of the present invention, a method for calibrating the camera spacing of a vehicle, comprising: (a) when obtaining a driving image from a camera during vehicle driving, a calibration device inputs the driving image into an object detection network and a lane detection network respectively, so that the object detection network detects a plurality of objects on the driving image to output a plurality of object detection information, and the lane detection network detects a plurality of lanes on the driving image to output a plurality of lane detection information; (b) the calibration device performs the following processing: generating a plurality of object analysis information corresponding to each object by analyzing the plurality of object detection information, and generating a plurality of lane analysis information corresponding to each lane by analyzing the plurality of lane detection information, inputting the plurality of object analysis information into an object-based spacing estimation module so that the object-based spacing estimation module (i) selects a first target object from the plurality of objects with reference to the plurality of object analysis information, and generates a first spacing by using a first spacing estimation of a first height of the first target object, (ii) selects a second target object from the plurality of objects with reference to the plurality of object analysis information, and generates a second spacing by using a second spacing estimation of a width of the second target object; inputting the vanishing point detection information of a vanishing point estimation network for detecting a vanishing point by analyzing the driving image and the plurality of lane analysis information into a lane-based spacing estimation module so that the lane-based spacing estimation module (i) generates a third spacing by using a third spacing estimation of the plurality of lane analysis information, (ii) generates a fourth spacing by using a fourth spacing estimation of the vanishing point detection information; and (c) the calibration device inputs the first spacing to the fourth spacing into a spacing determination module to enable the spacing determination module to synthesize the first spacing to the fourth spacing to output a determined spacing corresponding to the driving image.

[0018] According to an embodiment, in the process of (c), the calibration device enables the distance determination module to select a third target object from multiple objects with reference to multiple object analysis information, calculate a second height of the third target object using the third distance, and then verify the third distance by confirming whether the second height is within a height threshold. (i) When the third distance is valid, output the third distance as the determined distance. (ii) When the third distance is invalid, compare a first target object corresponding to the first distance with a second target object corresponding to the second distance. When the first target object is the same as the second target object, output either the first distance or the second distance as the determined distance. When the first target object is different from the second target object, output a specific distance corresponding to a specific target object with a smaller lateral distance from the vehicle among the first target object and the second target object as the determined distance. (iii) When no object and lane are detected in the driving image, output the fourth distance as the determined distance.

[0019] According to an embodiment, the calibration device enables the distance determination module to select a first specific object with the smallest lateral distance from the vehicle among multiple first candidate objects as the third target object. The lateral distances of the multiple first candidate objects are less than or equal to a first distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, and they do not overlap.

[0020] According to an embodiment, in the process of (c), the calibration device enables the distance determination module to apply a previous distance value and smoothing in the previous frame to the determined distance for distance smoothing, and perform tolerance processing. The tolerance processing uses a distance change threshold to limit the maximum distance change between frames.

[0021] According to an embodiment, in (d), the calibration device obtains a distance loss with reference to the determined distance and the fourth distance, and uses the distance loss to perform in-vehicle continuous learning on the vanishing point estimation network.

[0022] According to an embodiment, the calibration device (i) performs instance-wise incremental learning on the vanishing point estimation network using the distance loss during the vehicle operation to achieve fast adaptation, and (ii) after the vehicle has finished driving, performs balanced continuous learning on the vanishing point estimation network using the data sampled during the driving process to recover the catastrophic forgetting phenomenon that may occur due to the fast adaptation.

[0023] According to an embodiment, during the operation of the vehicle, the calibration device (i) uses the pitch loss to perform instance-level incremental learning on the vanishing point estimation network to achieve fast adaptation, (ii) transmits the sampling data for the instance-level incremental learning to a learning server so that the learning server uses the sampling data to perform server-side continuous learning on a reference vanishing point estimation network corresponding to the vanishing point estimation network, and transmits a plurality of parameters updated through the server-side continuous learning to the calibration device, and (iii) after the vehicle has finished traveling, updates the vanishing point estimation network using the plurality of parameters received from the learning server.

[0024] According to an embodiment, (e) the calibration device (i) transmits the trained vanishing point estimation network model learned on the device to a learning server so that the learning server evaluates at least one other trained vanishing point estimation network model transmitted from at least one other vehicle and the trained vanishing point estimation network model to select an optimal vanishing point estimation network model, and transmits the optimal vanishing point estimation network model to the calibration device, and (ii) updates the vanishing point estimation network using the optimal vanishing point estimation network model transmitted from the learning server.

[0025] According to an embodiment, in step (b), the calibration device causes the object-based pitch estimation module to select, with reference to a plurality of object profiling information, a second specific object having the smallest lateral distance from the vehicle among a plurality of second candidate objects as the first target object. The lateral distances of the plurality of second candidate objects are less than or equal to a second distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, and they do not overlap. The average height of the first target object is obtained with reference to the detection history of the first target object. When the average height is greater than or equal to a minimum height threshold and less than or equal to a maximum height threshold, the average height is determined as the first height. When the average height is less than the minimum height threshold or greater than the maximum height threshold, the average of the minimum height threshold and the maximum height threshold is determined as the first height.

[0026] According to an embodiment, in step (b), the calibration device causes the object-based pitch estimation module to select, with reference to a plurality of object profiling information, a third specific object having the smallest lateral distance from the vehicle among a plurality of third candidate objects as the second target object. The lateral distances of the plurality of third candidate objects are less than or equal to a third distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, they do not overlap, the difference between the 2D bounding box and the 3D bounding box is less than or equal to a box difference threshold, and the aspect ratio of the bounding box is greater than or equal to an aspect ratio threshold.

[0027] According to one embodiment, in the step (b), the calibration device enables the lane-based spacing estimation module to select a first lane and a second lane with reference to a plurality of the lane profiling information, detect a target vanishing point using the first lane and the second lane, and perform the third spacing estimation using the target vanishing point, wherein the first lane and the second lane are straight lines, are greater than or equal to a length threshold, and are parallel to each other in the coordinate system of the vehicle.

[0028] According to another embodiment of the present invention, a calibration device for calibrating the camera spacing of a vehicle, comprising: a memory that stores instructions for calibrating the camera spacing of the vehicle; and a processor that performs operations for calibrating the camera spacing of the vehicle according to the instructions stored in the memory, and the processor performs the following processing: (I) When obtaining a driving image from a camera during the driving of the vehicle, inputting the driving image into an object detection network and a lane detection network respectively, enabling the object detection network to detect a plurality of objects on the driving image to output a plurality of object detection information, and enabling the lane detection network to detect a plurality of lanes on the driving image to output a plurality of lane detection information; (II) Generating a plurality of object profiling information corresponding to each object by analyzing a plurality of the object detection information, and generating a plurality of lane profiling information corresponding to each lane by analyzing a plurality of the lane detection information, inputting the plurality of object profiling information into an object-based spacing estimation module to enable the object-based spacing estimation module (i) to select a first target object from a plurality of the objects with reference to a plurality of the object profiling information, and generate a first spacing by performing a first spacing estimation using a first height of the first target object, (ii) to select a second target object from a plurality of the objects with reference to a plurality of the object profiling information, and generate a second spacing by performing a second spacing estimation using a width of the second target object; inputting the vanishing point detection information of a vanishing point estimation network for detecting a vanishing point by analyzing the driving image and a plurality of the lane profiling information into a lane-based spacing estimation module to enable the lane-based spacing estimation module (i) to generate a third spacing by performing a third spacing estimation using a plurality of the lane profiling information, (ii) to generate a fourth spacing by performing a fourth spacing estimation using the vanishing point detection information; and (III) Enabling a spacing determination module to synthesize the first spacing to the fourth spacing by inputting the first spacing to the fourth spacing into the spacing determination module to output a determined spacing corresponding to the driving image.

[0029] According to an embodiment, in the process of (III), the processor enables the spacing determination module to select a third target object from multiple objects with reference to multiple object analysis information, calculate a second height of the third target object using the third spacing, and then verify the third spacing by confirming whether the second height is within a height threshold. (i) When the third spacing is valid, output the third spacing as the determined spacing. (ii) When the third spacing is invalid, compare a first target object corresponding to the first spacing with a second target object corresponding to the second spacing. When the first target object and the second target object are the same, output either the first spacing or the second spacing as the determined spacing. When the first target object and the second target object are different, output a specific spacing corresponding to a specific target object with a smaller lateral distance from the vehicle among the first target object and the second target object as the determined spacing. (iii) When no object and lane are detected in the driving image, output the fourth spacing as the determined spacing.

[0030] According to an embodiment, the processor enables the spacing determination module to select a first specific object with the smallest lateral distance from the vehicle among multiple first candidate objects as the third target object. The lateral distances of the multiple first candidate objects are less than or equal to a first distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, and they do not overlap.

[0031] According to an embodiment, in the process of (III), the processor enables the spacing determination module to apply a previous spacing value and smoothing in the previous frame to the determined spacing for spacing smoothing, and perform tolerance processing. The tolerance processing uses a spacing change threshold to limit the maximum spacing change between frames.

[0032] According to an embodiment, the processor further performs the following process: (IV) Obtain a spacing loss with reference to the determined spacing and the fourth spacing, and use the spacing loss to perform in-vehicle continuous learning on the vanishing point estimation network.

[0033] According to an embodiment, the processor (i) performs instance-wise incremental learning on the vanishing point estimation network using the spacing loss during the running of the vehicle to achieve fast adaptation, and (ii) after the vehicle has finished driving, performs balanced continuous learning on the vanishing point estimation network using the data sampled during the driving process to recover the catastrophic forgetting phenomenon that may occur due to the fast adaptation.

[0034] According to one embodiment, the processor (i) during the operation of the vehicle, uses the spacing loss to perform instance-level incremental learning on the vanishing point estimation network to achieve fast adaptation; (ii) transmits the sampling data for the instance-level incremental learning to a learning server so that the learning server uses the sampling data to perform server-side continuous learning on a reference vanishing point estimation network corresponding to the vanishing point estimation network, and transmits a plurality of parameters updated through the server-side continuous learning to the calibration device; (iii) after the vehicle has finished driving, updates the vanishing point estimation network using the plurality of parameters received from the learning server.

[0035] According to one embodiment, the processor further performs the following processing: (V) (i) transmits the trained vanishing point estimation network model learned on the device to a learning server so that the learning server evaluates at least one other trained vanishing point estimation network model transmitted from at least one other vehicle and the trained vanishing point estimation network model to select the best vanishing point estimation network model, and transmits the best vanishing point estimation network model to the calibration device; (ii) updates the vanishing point estimation network using the best vanishing point estimation network model transmitted from the learning server.

[0036] According to one embodiment, in the process of (II), the processor causes the object-based spacing estimation module to refer to a plurality of object profiling information to select a second specific object with the smallest lateral distance from the vehicle among a plurality of second candidate objects as the first target object. The lateral distances of the plurality of second candidate objects are less than or equal to a second distance threshold, there is no bounding box cutting, the object category is a vehicle category, and they do not overlap. When obtaining the average height of the first target object by referring to the detection history of the first target object, and the average height is greater than or equal to a minimum height threshold and less than or equal to a maximum height threshold, the average height is determined as the first height; when the average height is less than the minimum height threshold or greater than the maximum height threshold, the average of the minimum height threshold and the maximum height threshold is determined as the first height.

[0037] According to one embodiment, in the process of (II), the processor causes the object-based spacing estimation module to refer to a plurality of object profiling information to select a third specific object with the smallest lateral distance from the vehicle among a plurality of third candidate objects as the second target object. The lateral distances of the plurality of third candidate objects are less than or equal to a third distance threshold, there is no bounding box cutting, the object category is a vehicle category, they do not overlap, the difference between the 2D bounding box and the 3D bounding box is less than or equal to a box difference threshold, and the aspect ratio of the bounding box is greater than or equal to an aspect ratio threshold.

[0038] According to an embodiment, in the process of (II), the processor causes the lane-based spacing estimation module to select a first lane and a second lane with reference to a plurality of the lane profile information, detect a target vanishing point using the first lane and the second lane, and perform the third spacing estimation using the target vanishing point, wherein the first lane and the second lane are straight lines, are greater than or equal to a length threshold, and are parallel to each other in the vehicle coordinate system.

[0039] In addition, the present invention also provides a computer-readable recording medium for recording a computer program for executing the method of the present invention.

[0040] Advantageous Effects

[0041] By using the output of the perception module that constitutes the driving assistance device of the vehicle, the present invention can estimate the camera spacing of the vehicle more accurately compared with the prior art.

[0042] By using the output of the perception module that constitutes the driving assistance device of the vehicle, the present invention can estimate the camera spacing of the vehicle with less computational amount compared with the prior art.

[0043] By continuously learning the vanishing point estimation model for detecting the vanishing point, the present invention can accurately detect the vanishing point, and thus can more accurately estimate the camera spacing of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The following drawings for describing the embodiments of the present invention are only a part of the embodiments of the present invention, and those of ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary technicians") can obtain other drawings based on these drawings without any creative work.

[0045] Figure 1 A schematic diagram of a calibration device for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0046] Figure 2 A schematic diagram of a method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0047] Figure 3a and Figure 3b A schematic diagram of a state of estimating the spacing using the object height in the method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0048] Figure 4a and Figure 4b A schematic diagram of a state of estimating the spacing using the object width in the method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0049] Figure 5 It is a process diagram of selecting the ego lane on the driving image in the method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0050] Figure 6 It is a schematic diagram of the vanishing point estimation network in the method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0051] Figure 7 It is a process diagram of determining the spacing of the driving image among the calculated spacings in the method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention.

[0052] Figures 8 to 10 They are respectively state diagrams of the continuous learning vanishing point estimation network in the method for calibrating the camera spacing of a vehicle according to an embodiment of the present invention. Detailed Embodiments

[0053] The following detailed description of the present invention refers to the accompanying drawings, which illustrate specific embodiments in which the present invention can be implemented in an illustrative manner to clarify the purpose, technical solution, and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to implement the present invention.

[0054] In addition, in the content and claims of the present invention, the term "comprising" and its variations are not intended to exclude other technical features, additives, components, or steps. For those of ordinary skill in the art of the present invention, some of the other purposes, advantages, and characteristics of the present invention can be learned from this specification, and some can be learned from the implementation of the present invention. The following examples and drawings are provided as examples and are not intended to limit the present invention.

[0055] Furthermore, the present invention includes all possible combinations of the embodiments shown in this specification. It should be understood that although the various embodiments of the present invention are different, they do not necessarily exclude each other. For example, the specific shapes, structures, and characteristics described herein can be implemented by other embodiments without departing from the spirit and scope of the present invention in one embodiment. And, it should be understood that the positions or configurations of the components in each disclosed embodiment can be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description does not limit the present invention, and the scope of the present invention should be defined according to all scopes equivalent to the scope of its claims and the scope of the appended claims as long as appropriate explanations can be made. Similar reference numerals in the drawings indicate the same or similar functions in multiple aspects.

[0056] In order to enable those of ordinary skill in the art to easily implement the present invention, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] Figure 1Schematic diagram of a calibration device for calibrating the camera spacing of a vehicle according to an embodiment of the present invention. Refer to Figure 1 The calibration device 1000 may include a memory 1001 that stores instructions for estimating the camera spacing using object detection information and lane detection information for driving images obtained from the cameras of the vehicle, and a processor 1002 that performs operations for estimating the camera spacing of the vehicle according to the instructions stored in the memory 1001.

[0058] Specifically, the calibration device 1000 can generally use a combination of a computing device (e.g., a computer processor, memory, storage device, input and output devices, and a device that can include other components of a conventional computing device; an electronic communication device such as a router, switch, etc.; an electronic information storage system such as a network attached storage (NAS) and a storage area network (SAN)) and computer software (i.e., instructions that cause the computing device to operate in a specific manner) to achieve the required system performance.

[0059] In addition, the processor of the computing device can include hardware configurations such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus. In addition, the computing device can further include an operating system and a software configuration for executing application programs for specific purposes.

[0060] However, it is not excluded that the computing device includes an integrated processor in the form of integrating a medium, a processor, and a memory for implementing the present invention.

[0061] Next, reference will be made to Figure 2 A method for calibrating the camera spacing of a vehicle using the calibration device 1000 according to an embodiment of the present invention configured as described above will be described.

[0062] First, when obtaining a driving image from a camera installed in a vehicle, for example, an autonomous vehicle, the calibration device 1000 can input the driving image into an object detection network 1100 and a lane detection network 1200 respectively, so that the object detection network 1100 detects objects on the driving image and outputs object detection information, and the lane detection network 1200 detects lanes on the driving image and outputs lane detection information.

[0063] At this time, the object detection network 1100 can be implemented as a CNN (convolutional neural network)-based object detection network using deep learning, or can be implemented as a classifier based on visual features and shallow learning, but is not limited thereto, and various algorithms capable of detecting multiple objects from a driving image can be used. In addition, the output of the object detection network 1100 can include multiple object category information, which is obtained by analyzing object region information such as multiple 2D bounding boxes or multiple 3D bounding boxes and the categories of multiple objects corresponding to each bounding box, but the present invention is not limited thereto, and can include various information related to the object, such as information related to vehicle types such as sedans, trucks, or SUVs, etc.

[0064] In addition, the lane detection network 1200 can be implemented as a CNN-based lane detection network using deep learning, or can be implemented as an image processing and computer vision algorithm, but is not limited thereto, and various algorithms capable of detecting multiple lanes from a driving image can be used, and the output of the lane detection network 1200 can be a lane model represented by an nth-order polynomial equation.

[0065] Next, the calibration device 1000 can generate multiple lane profiling information corresponding to each lane by profiling multiple object detection information and profiling multiple object profiling information corresponding to each object and multiple lane detection information.

[0066] At this time, the multiple object profiling information can include the width and height for each object, but the present invention is not limited thereto, and can include various information about each object.

[0067] In addition, the multiple lane profiling information can include multiple straightness, multiple length, and multiple inclination on the vehicle coordinates for each lane, but the present invention is not limited thereto, and can include various information for each lane.

[0068] Next, the calibration device 1000 can input the driving image or lane detection information into the vanishing point estimation network 1210 so that the vanishing point estimation network 1210 detects the vanishing point on the driving image through learning operations and tracks the vanishing point.

[0069] Next, the calibration device 1000 can perform the following processes: input multiple object analysis information into the object-based spacing estimation module 1110 so that the object-based spacing estimation module 1100 selects a first target object from multiple objects with reference to the multiple object analysis information, generate a first spacing by using a first spacing estimation of the first height of the first target object, select a second target object from multiple objects with reference to the multiple object analysis information, and generate a second spacing by using a second spacing estimation of the width of the second target object.

[0070] In addition, the calibration device 1000 can perform the following processes: the vanishing point detection information of the vanishing point estimation network 1210 that detects a vanishing point by analyzing a driving image or lane detection information, and input multiple lane analysis information into the lane-based spacing estimation module 1220 so that the lane-based spacing estimation module 1220 generates a third spacing by using a third spacing estimation of the multiple lane analysis information, and generate a fourth spacing by using a fourth spacing estimation of the vanishing point detection information.

[0071] The object-based spacing estimation module 1000 generates the first spacing and the second spacing. The process of the lane-based spacing estimation module 1220 generating the third spacing and the fourth spacing will be described in more detail below.

[0072] First, the object-based spacing estimation module 1110 selects multiple second candidate objects from multiple objects detected in the driving image with reference to the multiple object analysis information. The lateral distance between the multiple second candidate objects and the vehicle is less than or equal to a second distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, and they do not overlap. The second specific object with the smallest lateral distance among the multiple selected second candidate objects can be selected as the first target object.

[0073] At this time, the lateral distance can be the distance between the vehicle and the object in a direction perpendicular to the vehicle width direction. The second distance threshold can be, for example, 10 m, but the present invention is not limited thereto, and any lateral distance that can be clearly recognized can be set as the second distance threshold. In addition, the object without a cut bounding box can be an object whose bounding box does not cross the boundary of the driving image. Furthermore, an object as a vehicle category can be an object having a category for vehicle types such as sedans, SUVs, trucks, buses, etc. The non-overlapping objects can be objects whose bounding boxes do not overlap with the bounding boxes of other objects, or even if they overlap, the bottom coordinates of the bounding boxes are closer to the lower end of the driving image.

[0074] Then, based on the detection history of the first target object, the object-based spacing estimation module 1100 can obtain the average height of the first target object. When the average height is greater than or equal to the minimum height threshold and less than or equal to the maximum height threshold, the average height is determined as the first height. When the average height is less than the minimum height threshold or greater than the maximum height threshold, the average of the minimum height threshold and the maximum height threshold is determined as the first height.

[0075] At this time, the minimum height threshold and the maximum height threshold can be set for each vehicle category. As an example, according to the vehicle category, for cars, SUVs or vehicles of similar size, the minimum height threshold can be set to 1.4m and the maximum height threshold can be set to 1.8m. For buses, trucks and other vehicles, the minimum height threshold can be set to 2.5m and the maximum height threshold can be set to 4m.

[0076] In addition, the object-based spacing estimation module 1100 can generate the first spacing through binary search using the first height of the first target object, or can generate the first spacing through direct calculation.

[0077] That is, the object-based spacing estimation module 1100 receives the current spacing P, the spacing search angle α, the bounding box B of the first target object, and the first height H of the first target object as inputs through binary search to output a newly estimated spacing angle P'.

[0078] At this time, min_angle can be expressed as P - α, max_angle can be expressed as P + α. When the function for calculating the height of the bounding box B of the first target object in the vehicle coordinate system is VCS_H(B), in the state where VCS_H(B)!= H, it can be estimated through P' = binary search(B, H, min_angle, max_angle), and min_angle and max_angle can be updated. On the other hand, binary search(B, H, min_angle, max_angle) can be a function for searching for the spacing to make the height of the bounding box B of the first target object become the first height H.

[0079] In addition, referring to Figure 3a and Figure 3b , the object-based spacing estimation module 1100 uses a triangle-like proportional expression similar to the pinhole camera model in Figure 3a . According to the focal length: bounding box height = Z: target object height (the first height), the distance Z is calculated through the target object distance Z = (focal length) * (target object height) / (bounding box height), as in Figure 3bAs shown, the first spacing θ can be estimated by the following mathematical formula using the calculated distance Z.

[0080]

[0081]

[0082]

[0083] Second, the object-based spacing estimation module 1100 can select the third specific object with the smallest lateral distance from the vehicle among multiple third candidate objects with reference to multiple object profiling information as the second target object. The lateral distances of the multiple third candidate objects are less than or equal to the third distance threshold, there is no cutting of the bounding box, the object category is the vehicle category, they do not overlap, the difference between the 2D bounding box and the 3D bounding box is less than or equal to the box difference threshold, and the aspect ratio of the bounding box is greater than or equal to the aspect ratio threshold.

[0084] At this time, the lateral distance can be the distance between the vehicle and the object in the direction perpendicular to the vehicle's vehicle width direction. The third distance threshold can be, for example, 3m, but the present invention is not limited thereto, and any lateral distance that can be clearly recognized can be set as the third distance threshold. And the object without a cut bounding box can be an object whose bounding box does not cross the boundary of the driving image. In addition, the object as the vehicle category can be an object with a category for vehicle types such as sedans, SUVs, trucks, buses, etc. The non-overlapping objects can be objects whose bounding boxes do not overlap with the bounding boxes of other objects, or even if they overlap, the bottom coordinates of the bounding boxes are closer to the lower end of the driving image. Furthermore, the object with the difference between the 2D bounding box and the 3D bounding box less than or equal to the box difference threshold can be an object almost in front of the vehicle, and the object with the aspect ratio of the bounding box greater than or equal to the aspect ratio threshold can be an object whose side cannot be seen from the front. That is, at intersections, etc., when the side of an object can be seen from the front of the vehicle, the bounding box of this object has a smaller aspect ratio. To prevent this, objects with an aspect ratio of the bounding box less than the aspect ratio threshold are not selected.

[0085] Thereafter, the object-based spacing estimation module 1100 can determine the width of the second target object with reference to the vehicle category of the second target object.

[0086] At this time, the width of the second target object can be determined by the width set according to each vehicle category of the second target object. As an example, according to the vehicle category, for sedans, SUVs or vehicles of similar size, it can be set to 1.8m, while for vehicles such as buses and trucks, it can be set to 1.8m, but the present invention is not limited thereto, and it can be set with a specific constant according to each vehicle category. This can be set to have the same width for the same vehicle category because the width difference according to the vehicle type in the same vehicle category is not large.

[0087] Also, see Figure 4a and Figure 4b The object-based distance estimation module 1100 uses Figure 4a The triangle-like proportional expression of the pinhole camera model in , according to the bounding box width: f = actual width: Z, through Calculate the distance Z of the second target object, such as Figure 4b As shown, the second distance θ can be estimated by the following mathematical formula using the calculated distance Z of the second target object.

[0088]

[0089]

[0090]

[0091] Third, the lane-based spacing estimation module 1200 can select the first lane and the second lane with reference to multiple lane analysis information, use the first lane and the second lane to detect the target vanishing point, and use the target vanishing point to estimate the third spacing, wherein the first lane and the second lane are straight lines, greater than or equal to the distance threshold, and parallel to each other in the vehicle's coordinate system.

[0092] As an example, the lane-based spacing estimation module 1220 may refer to a plurality of lane analysis information, the lane analysis information including straightness indicating the straightness of the lane detected on the driving image, the length of the lane detected on the driving image, and the inclination of the lane on the vehicle coordinate system, and select the first lane and the second lane as lanes for detecting the vanishing point, the first lane and the second lane satisfying the conditions of being straight, the lane length being above the distance threshold, and being parallel on the vehicle coordinate system. At this time, when converting the lane from the driving image coordinate system to the vehicle coordinate system, the camera parameters of the previous frame may be used.

[0093] Also, the lane-based distance estimation module 1220 may calculate an intersection point using lane equations of the selected first and second lanes to detect a vanishing point, and estimate a third distance using the detected vanishing point and the center of the camera.

[0094] At this time, when n lanes are detected, the following Figure 5 The search tree shown is used to select two lanes.

[0095] That is, the ego lane corresponding to the vehicle is detected from multiple lanes. When both the left and right ego lanes are detected, it is confirmed whether the availability conditions are met, that is, it is straight and greater than or equal to a specific distance threshold. If both the left and right ego lanes meet the availability conditions, it is confirmed whether the inclination validity verification condition is met, that is, the two lanes are parallel. If only one lane meets the availability conditions, an alternative lane that meets the availability conditions is detected to confirm whether the validity verification condition is met. Moreover, when only the left ego lane or the right ego lane is detected, alternative lanes that respectively meet the availability conditions are detected to confirm whether the inclination verification condition is met. In this way, two lanes that meet the availability conditions and the inclination verification condition can be selected.

[0096] Fourth, the lane-based distance estimation module 1220 tracks the vanishing point detected in the vanishing point estimation network 1210 by using the vanishing point detection information of the vanishing point estimation network 1210, and can estimate the fourth distance by using the vanishing point tracked with the EOL (end of line) distance set as the default when preparing the camera.

[0097] At this time, referring to Figure 6 , in order to directly detect the vanishing point from the driving image, the vanishing point estimation network 1210 can use a CNN (convolutional neural network), or can use a perception-based network that simultaneously detects lanes and vanishing points with a multi-task CNN structure such as the vanishing point GNet (Guided Network for Lane and Road Marking Detection and Recognition), but the present invention is not limited thereto and can be implemented as an independent vanishing point detection network.

[0098] Next, the calibration device 1000 can input the first to fourth distances into the distance determination module 1300 to enable the distance determination module 1300 to integrate the first to fourth distances (ensemble) to output the determined distance corresponding to the driving image.

[0099] On the other hand, each of the first distance estimation to the fourth distance estimation has the following advantages and disadvantages.

[0100] The advantage of the first distance estimation is that since the distance is adjusted to maintain the height, stable distance estimation can be performed, and the object selection range is relatively wide, so it can operate in a wider range than the second distance estimation. However, its disadvantage is that it is difficult to apply the distance estimation to objects with too wide a height range, such as pedestrians or two-wheeled vehicles.

[0101] Moreover, the advantage of the second spacing estimation is that, since the spacing is adjusted to maintain the width, stable spacing estimation can be performed, and while the height has a minimum / maximum area, the width varies very little between vehicles and can be fixed as a constant, so its accuracy is higher than that of the first spacing estimation. However, its disadvantage is that it cannot operate on objects such as pedestrians or two-wheel vehicles, and due to the limitation of object selection, it must operate within a narrower motion range than the first spacing estimation.

[0102] In addition, the advantage of the third spacing estimation is that it is very accurate in a straight-line area and can also operate well on a flat road. However, its disadvantage is that it is inaccurate on a curve and can only operate when there are lanes.

[0103] Furthermore, the advantage of the fourth spacing estimation is that it outputs in all cases regardless of the object and the lane. However, its disadvantage is that it is difficult to keep the spacing variance at a low level due to accuracy issues.

[0104] Therefore, the calibration device 1000 enables the spacing determination module 1300 to integrate the first to fourth spacings to utilize the advantages of each of the first to fourth spacing estimations and compensate for the disadvantages, so that the spacing in each frame of the driving image can be determined.

[0105] As an example, referring to Figure 7 , the spacing determination module 1300 can select a third target object from multiple objects with reference to multiple object profiling information, calculate the second height of the third target object using the third spacing, and then verify the third spacing by confirming whether the second height is within the height threshold. At this time, the spacing determination module 1300 can select the first specific object with the smallest lateral distance from the vehicle among multiple first candidate objects as the third target object. The lateral distances of the multiple first candidate objects are less than or equal to the first distance threshold, there is no cutting of the bounding box, the object category is the vehicle category, and they do not overlap.

[0106] Thereafter, as a result of verifying the third spacing, when the third spacing is valid, the spacing determination module 1300 can output the third spacing as the determined spacing.

[0107] However, when the third spacing is invalid, the spacing determination module 1300 can compare the first target object corresponding to the first spacing with the second target object corresponding to the second spacing. When the first target object and the second target object are the same, either the first spacing or the second spacing is output as the determined spacing. When the first target object and the second target object are different, the specific spacing corresponding to the specific target object with the smaller lateral distance from the vehicle among the first target object and the second target object is output as the determined spacing.

[0108] Moreover, when no object and lane are detected from the driving image, the spacing determination module 1300 may output a fourth spacing as the determined spacing.

[0109] In addition, the spacing determination module 1300 applies a previous spacing value and smoothing in the previous frame to the determined spacing to smooth the spacing of the determined spacing, and performs tolerance processing, where the tolerance processing limits the maximum spacing change between frames using a spacing change threshold.

[0110] That is, the spacing determination module 1300 can perform spacing smoothing for the determined spacing by applying the previous spacing value and smoothing in the previous frame to prevent sudden changes in the spacing, and limit the maximum spacing change between frames using a threshold through tolerance processing. However, the spacing determination module 1300 can lift the restriction on the spacing change when the object first appears after it no longer exists. In addition, tolerance processing can be performed when the object is on a road plane different from the vehicle, i.e., the ego-vehicle.

[0111] On the other hand, the vanishing point estimation network 1210 can be trained to detect the vanishing point from the driving image through continuous learning.

[0112] At this time, for the continuous learning of the vanishing point estimation network 1210, in-device continuous learning, cooperative continuous learning, server-side continuous learning, etc. can be used.

[0113] First, referring to Figure 8 , in in-device continuous learning, the calibration device 1000 can obtain a spacing loss by referring to the determined spacing and the fourth spacing, and perform in-device continuous learning on the vanishing point estimation network 1210 using the spacing loss. At this time, although the calibration device 1000 performs in-device continuous learning, differently, in-device continuous learning can also be performed by a separate learning device.

[0114] At this time, during the operation of the vehicle, the calibration device 1000 can perform instance-wise incremental learning on the vanishing point estimation network 1210 using the spacing loss to achieve fast adaptation, and after the vehicle has finished driving, perform balanced continuous learning on the vanishing point estimation network 1210 using the data sampled during driving to recover the catastrophic forgetting phenomenon that may occur due to fast adaptation.

[0115] That is, generally speaking, since a continuous image sequence is used as input when detecting the driving environment of a vehicle, if the vanishing point is incorrectly estimated in a specific frame, it is very likely that the incorrect vanishing point will be maintained in subsequent frames. Thus, during driving, a spacing loss is obtained by comparing the determined spacing output from the spacing determination module 1300 with the fourth spacing. When the spacing loss is greater than a threshold, instance-level incremental learning is performed to achieve rapid adaptation. After driving is completed, in order to recover from the catastrophic forgetting phenomenon that may occur due to rapid adaptation during driving, the data sampled during driving can be used for balanced continuous learning.

[0116] In addition, referring to Figure 9 , in cooperative continuous learning, the calibration device 100 can transmit the trained vanishing point estimation network model learned on the device to the learning server so that the learning server evaluates at least one other trained vanishing point estimation network model transmitted from at least one other vehicle and the trained vanishing point estimation network model to select the best vanishing point estimation network model, transmit the best vanishing point estimation network model to the calibration device 1000, and update the vanishing point estimation network 12010 using the best vanishing point estimation network model transmitted from the learning server.

[0117] That is, referring to Figure 8 , each vehicle performs continuous learning through the above-mentioned on-device continuous learning. When transmitting the learned model and data to the learning server, the learning server evaluates the models of each vehicle and selects the best model. Then, by transmitting the selected best model to each vehicle, each vehicle can update the vanishing point estimation network through cooperative continuous learning.

[0118] In addition, referring to Figure 10 , in server-side continuous learning, during vehicle operation, the calibration device 1000 can perform instance-level incremental learning on the vanishing point estimation network 1210 using the spacing loss to achieve rapid adaptation, transmit the sampled data for instance-level incremental learning to the learning server so that the learning server performs server-side continuous learning on the reference vanishing point estimation network corresponding to the vanishing point estimation network 1210 using the sampled data, transmit the multiple parameters updated through server-side continuous learning to the calibration device 1000, and update the vanishing point estimation network 1210 using the multiple parameters received from the learning server after the vehicle has finished driving.

[0119] That is, in each vehicle, only rapid adaptation is performed, continuous learning is performed in the learning server, and then the continuously learned vanishing point estimation network or multiple corresponding parameters are transmitted and updated to each vehicle.

[0120] In addition, the embodiments according to the present invention described above can be implemented in the form of program instructions executable by various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. individually or in combination. The program instructions recorded in the computer-readable recording medium may be specifically designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, flash memories, etc. Examples of program instructions include not only machine language codes such as those generated by compilers, but also high-level language codes that can be executed by computers using interpreters, etc. The hardware devices may be configured to operate as at least one software module to execute the processing according to the present invention, and vice versa.

[0121] In the foregoing, the present invention has been described with reference to specific matters such as specific components, as well as limited embodiments and drawings, but this is only helpful for a more comprehensive understanding of the present invention, and the present invention is not limited to the above embodiments. Those of ordinary skill in the art to which the present invention pertains can design various modifications and variations based on these descriptions.

[0122] Therefore, the spirit of the present invention should not be limited to the above embodiments, and any modifications equivalent or identical to these claims, except for the appended claims, should be included within the spirit of the present invention.

Claims

1. A method for calibrating the camera spacing of a vehicle, wherein, Including: (a)When obtaining a driving image from a camera during vehicle driving, the calibration device inputs the driving image into an object detection network and a lane detection network respectively, enabling the object detection network to detect multiple objects on the driving image to output multiple object detection information, and enabling the lane detection network to detect multiple lanes on the driving image to output multiple lane detection information; (b)The calibration device performs the following processing: generating multiple object analysis information corresponding to each object by analyzing the multiple object detection information, and generating multiple lane analysis information corresponding to each lane by analyzing the multiple lane detection information, and inputting the multiple object analysis information into an object-based spacing estimation module to enable the object-based spacing estimation module (i) to select a first target object from the multiple objects with reference to the multiple object analysis information, and generate a first spacing by using a first spacing estimation of the first height of the first target object, (ii) to select a second target object from the multiple objects with reference to the multiple object analysis information, and generate a second spacing by using a second spacing estimation of the width of the second target object; Inputting the vanishing point detection information of a vanishing point estimation network for detecting a vanishing point by analyzing the driving image and the multiple lane analysis information into a lane-based spacing estimation module to enable the lane-based spacing estimation module (i) to generate a third spacing by using a third spacing estimation of the multiple lane analysis information, and (ii) to generate a fourth spacing by using a fourth spacing estimation of the vanishing point detection information; And (c)The calibration device enables the spacing determination module to comprehensively consider the first spacing to the fourth spacing by inputting the first spacing to the fourth spacing into the spacing determination module to output a determined spacing corresponding to the driving image; Wherein, in step (c): The calibration device enables the spacing determination module to select a third target object from the multiple objects with reference to the multiple object analysis information, calculate a second height of the third target object by using the third spacing, and then verify the third spacing by confirming whether the second height is within a height threshold. (i)When the third spacing is valid, output the third spacing as the determined spacing. (ii)When the third spacing is invalid, compare the first target object corresponding to the first spacing with the second target object corresponding to the second spacing. When the first target object is the same as the second target object, output either the first spacing or the second spacing as the determined spacing. When the first target object is different from the second target object, output a specific spacing corresponding to a specific target object with a smaller lateral distance from the vehicle among the first target object and the second target object as the determined spacing. (iii)When no object and lane are detected in the driving image, output the fourth spacing as the determined spacing.

2. The method according to claim 1, wherein: The calibration device causes the spacing determination module to select a first specific object with the smallest lateral distance from the vehicle among multiple first candidate objects as the third target object. The lateral distances of the multiple first candidate objects are less than or equal to a first distance threshold, there is no cutting of the bounding box, the object category is the vehicle category, and they do not overlap.

3. The method according to claim 1, wherein In the step (c): The calibration device causes the spacing determination module to apply a previous spacing value and smoothing in the previous frame to the determined spacing for spacing smoothing of the determined spacing, and perform tolerance processing, where the tolerance processing uses a spacing change threshold to limit the maximum spacing change between frames.

4. The method according to claim 1, wherein, It further includes: (d) The calibration device obtains a spacing loss with reference to the determined spacing and the fourth spacing, and uses the spacing loss to perform on-vehicle continuous learning on the vanishing point estimation network.

5. The method according to claim 4, wherein: The calibration device (i) during the operation of the vehicle, performs instance-level incremental learning on the vanishing point estimation network using the spacing loss to achieve rapid adaptation, and (ii) after the vehicle has finished traveling, uses the data sampled during the traveling process to perform balanced continuous learning on the vanishing point estimation network to recover the catastrophic forgetting phenomenon that may occur due to the rapid adaptation.

6. The method according to claim 4, wherein: The calibration device (i) during the operation of the vehicle, performs instance-level incremental learning on the vanishing point estimation network using the spacing loss to achieve rapid adaptation, (ii) transmits the sampled data used for the instance-level incremental learning to a learning server so that the learning server uses the sampled data to perform server-side continuous learning on a reference vanishing point estimation network corresponding to the vanishing point estimation network, and transmits multiple parameters updated through the server-side continuous learning to the calibration device, and (iii) after the vehicle has finished traveling, updates the vanishing point estimation network using the multiple parameters received from the learning server.

7. The method according to claim 4, wherein It further includes: (e) The calibration device (i) transmits the trained vanishing point estimation network model of the on-vehicle continuous learning to a learning server so that the learning server evaluates at least one other trained vanishing point estimation network model transmitted from at least one other vehicle and the trained vanishing point estimation network model to select the best vanishing point estimation network model, and transmits the best vanishing point estimation network model to the calibration device, and (ii) updates the vanishing point estimation network using the best vanishing point estimation network model transmitted from the learning server.

8. The method according to claim 1, wherein In the step (b): The calibration device causes the object-based spacing estimation module to select, with reference to the plurality of object profiling information, a second specific object with the smallest lateral distance from the vehicle among a plurality of second candidate objects as the first target object. The lateral distances of the plurality of second candidate objects are less than or equal to a second distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, and they do not overlap. The average height of the first target object is obtained with reference to the detection history of the first target object. When the average height is greater than or equal to a minimum height threshold and less than or equal to a maximum height threshold, the average height is determined as the first height. When the average height is less than the minimum height threshold or greater than the maximum height threshold, the average of the minimum height threshold and the maximum height threshold is determined as the first height.

9. The method according to claim 1, wherein In the step (b): The calibration device causes the object-based spacing estimation module to select, with reference to the plurality of object profiling information, a third specific object with the smallest lateral distance from the vehicle among a plurality of third candidate objects as the second target object. The lateral distances of the plurality of third candidate objects are less than or equal to a third distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, they do not overlap, the difference between the 2D bounding box and the 3D bounding box is less than or equal to a box difference threshold, and the aspect ratio of the bounding box is greater than or equal to an aspect ratio threshold.

10. The method according to claim 1, wherein, In the step (b): The calibration device causes the lane-based spacing estimation module to select a first lane and a second lane with reference to the plurality of lane profiling information, detect a target vanishing point using the first lane and the second lane, and estimate the third spacing using the target vanishing point. Among them, the first lane and the second lane are straight lines, greater than or equal to a length threshold, and are parallel to each other in the coordinate system of the vehicle.

11. A calibration device for calibrating the camera spacing of a vehicle, wherein, Comprising: A memory that stores instructions for calibrating the camera spacing of a vehicle; And A processor that performs operations for calibrating the camera spacing of a vehicle according to the instructions stored in the memory, The processor performs the following processing: (I) When obtaining a driving image from a camera during the driving of the vehicle, the driving image is respectively input into an object detection network and a lane detection network, causing the object detection network to detect a plurality of objects on the driving image to output a plurality of object detection information, and causing the lane detection network to detect a plurality of lanes on the driving image to output a plurality of lane detection information; (II) Generate a plurality of object analysis information corresponding to each object by analyzing the plurality of object detection information, and generate a plurality of lane analysis information corresponding to each lane by analyzing the plurality of lane detection information. Input the plurality of object analysis information into the object-based spacing estimation module so that the object-based spacing estimation module (i) selects a first target object from the plurality of objects with reference to the plurality of object analysis information, and generates a first spacing by using a first spacing estimation of the first height of the first target object; (ii) selects a second target object from the plurality of objects with reference to the plurality of object analysis information, and generates a second spacing by using a second spacing estimation of the width of the second target object; Input the vanishing point detection information of the vanishing point estimation network for detecting the vanishing point by analyzing the driving image and the plurality of lane analysis information into the lane-based spacing estimation module so that the lane-based spacing estimation module (i) generates a third spacing by using a third spacing estimation of the plurality of lane analysis information; (ii) generates a fourth spacing by using a fourth spacing estimation of the vanishing point detection information; And (III) input the first spacing to the fourth spacing into the spacing determination module so that the spacing determination module synthesizes the first spacing to the fourth spacing to output a determined spacing corresponding to the driving image; Wherein, in the process of (III), the processor enables the spacing determination module to select a third target object from the plurality of objects with reference to the plurality of object analysis information, calculate a second height of the third target object by using the third spacing, and then verify the third spacing by confirming whether the second height is within a height threshold. (i) When the third spacing is valid, output the third spacing as the determined spacing; (ii) when the third spacing is invalid, compare the first target object corresponding to the first spacing with the second target object corresponding to the second spacing. When the first target object and the second target object are the same, output either the first spacing or the second spacing as the determined spacing. When the first target object and the second target object are different, output a specific spacing corresponding to a specific target object with a smaller lateral distance from the vehicle among the first target object and the second target object as the determined spacing; (iii) when no object and lane are detected in the driving image, output the fourth spacing as the determined spacing.

12. The calibration device according to claim 11, wherein: The processor enables the spacing determination module to select a first specific object with the smallest lateral distance from the vehicle among a plurality of first candidate objects. The lateral distances of the plurality of first candidate objects are less than or equal to a first distance threshold, there is no cutting of the bounding box, the object category is a vehicle category, and they do not overlap.

13. The calibration device according to claim 11, wherein, In the processing of (III), the processor causes the spacing determination module to apply a previous spacing value and smoothing in a previous frame to the determined spacing for spacing smoothing of the determined spacing, and performs tolerance processing, where the tolerance processing limits the maximum spacing change between frames using a spacing change threshold.

14. The calibration device according to claim 11, wherein: The processor further performs the following processing: (IV) obtaining a spacing loss with reference to the determined spacing and the fourth spacing, and using the spacing loss to perform in-vehicle continuous learning on the vanishing point estimation network.

15. The calibration device according to claim 14, wherein: The processor (i) performs instance-level incremental learning on the vanishing point estimation network using the spacing loss during the running of the vehicle to achieve fast adaptation, and (ii) performs balanced continuous learning on the vanishing point estimation network using the data sampled during the driving of the vehicle after the vehicle has finished driving to recover the catastrophic forgetting phenomenon that may occur due to the fast adaptation.

16. The calibration device according to claim 14, wherein The processor (i) performs instance-level incremental learning on the vanishing point estimation network using the spacing loss during the running of the vehicle to achieve fast adaptation, (ii) transmits the sampled data for the instance-level incremental learning to a learning server so that the learning server performs server-side continuous learning on a reference vanishing point estimation network corresponding to the vanishing point estimation network using the sampled data, and transmits a plurality of parameters updated through the server-side continuous learning to the calibration device, and (iii) updates the vanishing point estimation network using the plurality of parameters received from the learning server after the vehicle has finished driving.

17. The calibration device according to claim 14, wherein The processor further performs the following processing: (V) (i) transmitting the trained vanishing point estimation network model of the in-vehicle continuous learning to a learning server so that the learning server evaluates at least one other trained vanishing point estimation network model transmitted from at least one other vehicle and the trained vanishing point estimation network model to select an optimal vanishing point estimation network model, and transmits the optimal vanishing point estimation network model to the calibration device, and (ii) updating the vanishing point estimation network using the optimal vanishing point estimation network model transmitted from the learning server.

18. The calibration device according to claim 11, wherein: In the process of (II), the processor causes the object-based spacing estimation module to select, with reference to a plurality of object analysis information, a second specific object with the smallest lateral distance from the vehicle among a plurality of second candidate objects as the first target object. The lateral distances of the plurality of second candidate objects are less than or equal to a second distance threshold, there is no cutting of the bounding box, the object category is the vehicle category, and they do not overlap. The average height of the first target object is obtained with reference to the detection history of the first target object. When the average height is greater than or equal to a minimum height threshold and less than or equal to a maximum height threshold, the average height is determined as the first height; when the average height is less than the minimum height threshold or greater than the maximum height threshold, the average of the minimum height threshold and the maximum height threshold is determined as the first height.

19. The calibration device according to claim 11, wherein In the process of (II), the processor causes the object-based spacing estimation module to select, with reference to a plurality of object analysis information, a third specific object with the smallest lateral distance from the vehicle among a plurality of third candidate objects as the second target object. The lateral distances of the plurality of third candidate objects are less than or equal to a third distance threshold, there is no cutting of the bounding box, the object category is the vehicle category, they do not overlap, the difference between the 2D bounding box and the 3D bounding box is less than or equal to a box difference threshold, and the aspect ratio of the bounding box is greater than or equal to an aspect ratio threshold.

20. The calibration device according to claim 11, wherein In the process of (II), the processor causes the lane-based spacing estimation module to select a first lane and a second lane with reference to a plurality of lane analysis information, detect a target vanishing point using the first lane and the second lane, and estimate the third spacing using the target vanishing point. Among them, the first lane and the second lane are straight lines, greater than or equal to a length threshold, and are parallel to each other in the coordinate system of the vehicle.

Images