Calibration method and apparatus for camera

Abstract

The present application relates to a kind of calibration method and device for camera.The method includes: according to the movement of camera, the first optical flow data between time series image data is generated;From the second optical flow data mapped to three-dimensional space based on the first optical flow data, ground optical flow data located on estimated ground is extracted;The attitude information of camera is estimated, and the view error of the view aligned to multiple ground optical flow data based on image data, with predetermined viewpoint.

Description

[0001] Cross-reference to related applications This application claims priority to Korean Patent Application No. 10-2025-0048965, filed with the Korean Intellectual Property Office on April 15, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to a camera calibration method and apparatus, and to such a camera calibration method and apparatus that calibrates the camera based on the ground without a reference target and estimates all pose elements of the camera without involving a large amount of data. Background Technology

[0003] The descriptions in this background section are for the purpose of enhancing understanding of the background of this invention and should not be construed as an admission that they correspond to prior art known to those skilled in the art.

[0004] Autonomous driving capabilities are under development, enabling vehicles to control their own movement while reducing or eliminating driver intervention entirely. Technologies supporting this are gradually being applied to vehicles.

[0005] To achieve autonomous driving, vehicles collect information about their surroundings through various sensors. Based on the data acquired from these sensors, the vehicle can perceive its environment and use this perceived information to identify the situation around it. A control plan for autonomous driving is then established based on the identified situation, and this plan allows for the execution of autonomous driving by controlling the vehicle's drive system.

[0006] Autonomous driving utilizes a vehicle's camera sensors to accurately identify its surroundings. Camera calibration can be used to correct lens distortion and perform accurate pose estimation. Calibration can also be used to estimate pose and map it between spaces with different dimensions.

[0007] During calibration, the vanishing point is detected to estimate pitch and yaw, and roll is assessed using the horizon slope. In pure longitudinal motion, the focus of expansion (FoE) is the vanishing point, but it may not be orthogonal because roll cannot be used for estimation. This can increase uncertainty. In real-world road environments, horizon detection is only effective if the Manhattan world assumption is satisfied. Calibration primarily utilizes features from the Manhattan world environment to detect the horizon and accumulates vanishing point coordinates to identify the vanishing line, but uncertainty may still exist. The vanishing point moves during vehicle cornering and does not fully account for the correlation between pitch, yaw, and roll, thus potentially reducing the accuracy of attitude estimation. Summary of the Invention

[0008] Various examples of the present invention provide a method and apparatus for calibrating a camera based on the ground without a reference target and for estimating all attitude elements of the camera without involving a large amount of data.

[0009] The various examples to be implemented in this invention are not limited to the technical problems described above, and those skilled in the art will clearly understand other technical problems not mentioned through the following description.

[0010] According to the present invention, a method performed by a vehicle may include: generating time-series image data related to the vehicle's surrounding environment via the vehicle's sensors; obtaining first optical flow data between adjacent image frames of the time-series image data via the sensors based on sensor movement; mapping the first optical flow data to a three-dimensional space to generate second optical flow data; extracting ground optical flow data related to key points located on the ground from the second optical flow data; aligning the ground optical flow data to a view based on the time-series image data and having a predetermined viewpoint; determining the view error of the aligned ground optical flow data; determining attitude parameters related to the sensors, which, when applied, reduce the view error; outputting a signal indicating the attitude parameters; and adjusting the sensors based on the signal.

[0011] This method can be further clarified to include: the first optical flow data, the second optical flow data, and the ground optical flow data all having an optical flow vector format. The method can also be further clarified to include: obtaining the first optical flow data can include: extracting a first feature from time-series image data; generating a feature descriptor describing the internal relationships between keypoints related to the first feature and the external relationships between keypoints matched between adjacent image frames in the time-series image data; generating global coarse matching information based on the feature descriptor; extracting a second feature with finer granularity than the first feature; performing precise matching between adjacent frames based on the global coarse matching information and the second feature; and generating the first optical flow data based on the precise matching. The feature descriptor can be generated by: applying a transformer-based self-attention process to keypoints related to the first feature to generate internal relationships; and applying a transformer-based cross-attention process to keypoints matched between adjacent image frames to generate external relationships.

[0012] The first optical flow data may include location information of keypoints in each image frame and matching pair information indicating the correspondence between keypoints in adjacent image frames. The first optical flow data may further include location information of keypoints in each image frame and matching pair information indicating the correspondence between keypoints in adjacent image frames. Second optical flow data can be generated based on the matching pair information and the locations obtained by mapping keypoints in the time-series image data into three-dimensional space.

[0013] According to the present invention, a vehicle may include: a sensor configured to generate time-series image data relating to the vehicle's surrounding environment; and a processor circuit configured to cause the vehicle to: acquire first optical flow data between adjacent image frames of the time-series image data based on sensor movement; map key points associated with the first optical flow data into three-dimensional space to generate a plurality of second optical flow data; form a cluster of at least two second optical flow data, including the normal vectors of each of the plurality of second optical flow data satisfying a clustering condition based on the angle difference between the normal vectors; the cluster corresponding to a candidate plane; identify the candidate plane as ground based on vehicle movement; acquire a plurality of ground optical flow vectors associated with key points located on the ground; convert the plurality of ground optical flow vectors into a view at a predetermined viewpoint; determine a view error of the converted plurality of ground optical flow vectors based on at least one of amplitude difference and angle difference; determine an attitude parameter that reduces the view error and indicates at least one of yaw, pitch, and roll of the sensor; output a signal indicating the attitude parameter; and adjust the sensor based on the signal. The processor circuit may further determine whether at least two second optical flow data satisfy the clustering condition based on the angle difference between their respective normal vectors. The processor circuit can further select second optical flow data related to key points on the ground from the multiple second optical flow data to generate multiple ground optical flow vectors. The processor circuit can further determine attitude parameters by reducing view errors, ensuring that the converted multiple ground optical flow vectors have uniform amplitude and parallel direction.

[0014] According to the present invention, a vehicle apparatus may include a sensor, a processor, and a memory; the sensor is configured to generate time-series image data relating to the vehicle's surrounding environment; the memory stores at least one instruction, which, when executed by the processor, is configured to cause the apparatus to: obtain first optical flow data between adjacent image frames of the time-series image data based on sensor movement; map the first optical flow data to a three-dimensional space to generate multiple second optical flow data; extract ground optical flow data relating to key points located on the ground from the multiple second optical flow data; align the ground optical flow data to a view based on the time-series image data and having a predetermined viewpoint; determine a view error of the aligned ground optical flow data; determine attitude parameters associated with the sensor, the attitude parameters reducing the view error; output a signal indicating the attitude parameters; and adjust the sensor based on the signal. The first optical flow data, the second optical flow data, and the ground optical flow data may have an optical flow vector format. The instructions can further enable the device to: extract a first feature from time-series image data; generate a feature descriptor describing the internal relationships between keypoints associated with the first feature and the external relationships between keypoints matched between adjacent image frames; generate global coarse matching information based on the feature descriptor; extract a second feature with finer granularity than the first feature; perform precise matching between adjacent image frames based on the global coarse matching information and the second feature; and generate first optical flow data based on the precise matching. The feature descriptor can be generated by applying a transformer-based self-attention process to keypoints associated with the first feature; and applying a transformer-based cross-attention process to keypoints matched between adjacent image frames. The first optical flow data may include location information of keypoints in each image frame and matching pair information indicating the correspondence between keypoints in adjacent image frames.

[0015] The first optical flow data may further include the location information of key points in each image frame and matching pair information indicating the correspondence between key points in adjacent image frames. The multiple second optical flow data can be generated based on the matching pair information and the positions obtained by mapping key points to three-dimensional space.

[0016] The instructions may further enable the device to: determine the similarity between multiple sets of second optical flow data; cluster at least two sets of second optical flow data that satisfy the similarity condition to determine candidate planes; identify the candidate planes as the ground based on the sensor's position and the vehicle's motion information; and extract second optical flow data related to key points located on the ground as ground optical flow data. The instructions may further enable the device to determine whether the at least two sets of second optical flow data satisfy the similarity condition based on the angle between their respective normal vectors.

[0017] View error can be determined based on the difference in amplitude and direction of the ground optical flow vectors associated with the aligned ground optical flow data; the instructions can further enable the device to reduce view error by ensuring that the ground optical flow vectors have uniform amplitude and are parallel to each other. Attitude parameters can represent the spatial translation and rotation of the sensor used to convert the ground optical flow data into a view with a predetermined viewing angle, which may include a bird's-eye view.

[0018] The above is a brief summary of the features of this invention, which are merely examples of the detailed description of the invention below and are not intended to limit the scope of the invention. Attached Figure Description

[0019] The above and other objects, features, and advantages of the invention will become more apparent to those skilled in the art from the detailed description of one or more examples of the invention with reference to the accompanying drawings, wherein: Figure 1 An example of a module that constitutes a camera calibration device is shown.

[0020] Figure 2 An example of a camera calibration method is shown.

[0021] Figure 3 An example of the process of generating the first optical flow data is shown.

[0022] Figure 4 An example of generating a feature descriptor from a single image dataset is shown.

[0023] Figure 5 An example of generating feature descriptors from multiple time-series image data is shown.

[0024] Figure 6 An example of generating ground optical flow data is shown.

[0025] Figure 7 An example of estimating the same plane is shown.

[0026] Figure 8 An example of ground optical flow data from an aerial image is shown.

[0027] Figure 9 An example is shown where a vehicle communicates with another device to send and receive data.

[0028] Figure 10 An example of the modules that make up a vehicle is shown.

[0029] Figure 11 An exemplary computing system is shown. Detailed Implementation

[0030] Hereinafter, examples of the invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can readily implement the invention. However, the invention can be implemented in various ways and is not limited to the examples described herein.

[0031] In describing examples of the invention, well-known functions or configurations will not be described in detail, as they may unnecessarily obscure the understanding of the invention. In the accompanying drawings, the same constituent elements are indicated by the same reference numerals, and repeated descriptions of the same elements will be omitted.

[0032] In this invention, when an element is simply referred to as being "connected to," "linked to," or "attached to" another element, this may mean that an element is "directly connected to," "directly linked to," or "directly linked to" another element, or is connected to, linked to, or linked to another element, with other elements interposed therebetween. Furthermore, when an element "comprises" or "has" another element, this means that, unless otherwise specifically stated, an element may further include another element without excluding another component.

[0033] In this invention, the terms first, second, etc., are used only to distinguish one element from another and do not limit the order or importance of the elements unless specifically mentioned. Therefore, without departing from the scope of this invention, a first element in one example may be referred to as a second element in another example, and similarly, a second element in one example may be referred to as a first element in another example.

[0034] In this invention, the distinguishing elements are used to clearly describe each feature and do not necessarily mean that these elements are separate. That is, multiple elements may be integrated into a single hardware or software unit, or a single element may be distributed and formed in multiple hardware or software units. Therefore, such examples of integration or distribution are included within the scope of this invention, even if not otherwise mentioned.

[0035] In this invention, the elements described in the various examples do not necessarily mean basic elements, and some may be optional elements. Therefore, examples consisting of a subset of the elements described in the examples are also included within the scope of this invention. Furthermore, examples including elements other than those described in the examples are also included within the scope of this invention.

[0036] The advantages and features of the invention, as well as the ways in which they are implemented, will become apparent from the examples described in detail below with reference to the accompanying drawings. However, the examples may be embodied in many different forms and should not be construed as limiting to the examples set forth herein. Rather, these examples are provided to complete the invention and to fully convey its scope to those skilled in the art.

[0037] For the purposes of this application and claims, the exemplary phrases “at least one of A; B; or C” or “at least one of A, B, or C” are used, which means “at least one A, or at least one B, or at least one C, or any combination of at least one A, at least one B, and at least one C.” Furthermore, exemplary phrases such as “at least one of A, B, or C,” “at least one of A, B, and C,” “at least one of A, B, or C,” etc., as used herein, may refer to each of the enumerated items or all combinations of the enumerated items. For example, “at least one of A or B” may mean: (1) at least one A; (2) at least one B; or (3) at least one A and at least one B.

[0038] The terms "module" or "unit" used in this specification refer to software and / or hardware components, and a "module" or "unit" performs a specific operation / function / action. However, a "module" or "unit" is not to be construed as limited to software or hardware. A "module" or "unit" may be configured to reside in addressable storage media or to be executed by one or more processors. Thus, by way of example, a "module" or "unit" may include at least one of the following: components (such as software components, object-oriented software components, class components, and task components), processes, functions, attributes, programs, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The functionality provided in a component, "module," or "unit" may be combined into a smaller number of components, "units," or "modules," or may be further divided into additional components, "modules," or "units."

[0039] In this invention, a "module" or "unit" can be implemented as a processor and a memory. "Processor" should be broadly interpreted to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), microcontrollers, state machines, etc. In some contexts, "processor" can refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or field-programmable gate arrays (FPGAs). For example, "processor" can refer to a combination of processing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such combination. Furthermore, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" can refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage devices, and registers. Memory can be in a state of electronic communication with the processor when the processor can read information from the memory and / or record information in the memory. Memory integrated into the processor is in a state of electronic communication with the processor.

[0040] One or more features described herein can be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium may continuously store the computer-executable program or temporarily store the program for execution or download. Furthermore, the medium can be a variety of recording or storage devices in the form of a single hardware device or a combination of multiple hardware devices, and is not limited to media directly connected to certain computer systems, but may also be distributed across a network. Examples of such media include magnetic media (such as hard disks, floppy disks, or magnetic tapes), optical recording media (such as CD-ROMs or DVDs), magneto-optical media (such as floppy disks), and ROM, RAM, or flash memory configured to store program instructions. Other examples of such media include media or storage media managed by application stores that distribute applications or various other websites or servers that provide or distribute software.

[0041] In a hardware implementation, the processing unit for performing the technology may be implemented within one or more ASICs, DSPs, digital signal processing devices, programmable logic devices, field-programmable gate arrays, processors, controllers, microcontrollers, microprocessors, electronic devices, or computers or combinations thereof, designed to perform the functions described in this invention.

[0042] According to the American Society of Automotive Engineers (SAE), the automation levels of autonomous vehicles can be classified as follows: At Level 0, the SAE classification corresponds to "No Automation," where the autonomous driving system temporarily intervenes in emergency situations (e.g., automatic emergency braking) and / or only provides warnings (e.g., blind spot warning, lane departure warning, etc.) and expects the driver to operate the vehicle. At Level 1, the SAE classification corresponds to "Driver Assistance," where the system performs some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control, etc.) while the driver is operating the vehicle normally on the road. The driver needs to determine the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. At Level 2, the SAE classification corresponds to "Partial Automation," where the system performs steering, acceleration, and / or braking under the driver's supervision. The driver needs to determine the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. At Level 3 of autonomous driving, the SAE classification standard can correspond to "conditional automation," where the system drives the vehicle under limited conditions (e.g., performing driving functions such as steering, acceleration, and / or braking), but transfers driving control to the driver when the required conditions are not met. The driver needs to determine the system's operating status and / or timing, and take over control in emergency situations but not operate the vehicle otherwise (e.g., steering, acceleration, and / or braking). At Level 4 of autonomous driving, the SAE classification standard can correspond to "high automation," where the system performs all driving functions, and the driver only needs to control the vehicle in emergency situations. At Level 5 of autonomous driving, the SAE classification standard can correspond to "full automation," where the system performs all driving functions without any driver assistance (including in emergency situations), and the driver does not need to perform any driving functions except to determine the system's operating status. Although this invention can apply the SAE classification standard to autonomous driving classification, other classification methods and / or algorithms can be used in one or more configurations described herein.

[0043] One or more features related to autonomous driving control can be controlled based on features from ground-driven sensor self-calibration. For example, when the calibration device updates attitude parameters that reduce view errors in ground optical flow data, the vehicle can adjust its autonomous driving behavior (e.g., lane keeping control, adaptive cruise following distance, or collision warning timing). Using the updated calibration, the processor can optimize perception outputs and adjust downstream controls (e.g., braking control, acceleration control, steering correction, or sensor fusion weighting) to maintain accurate vehicle operation.

[0044] One or more auxiliary devices can also be controlled based on the self-calibration characteristics of ground-driven sensors. For example, when updated attitude parameters improve ground-based optical flow alignment, the vehicle can adjust auxiliary braking functions (e.g., engine brakes, exhaust brakes, hydraulic reducers, electric reducers, or regenerative brakes) to maintain smooth deceleration, stabilize vehicle motion on ramps, or correct braking force distribution during autonomous maneuvering.

[0045] One or more communication devices (e.g., modems, network adapters, radio transceivers, or antennas supporting Ethernet, Wi-Fi, NFC, Bluetooth, LTE, 5G NR, or V2X) can also be controlled based on the characteristics of ground-driven sensor self-calibration. For example, when updated attitude parameters improve calibration quality, the vehicle can trigger a priority V2X uplink for the calibration state, request an updated perception map from a server, or adjust bandwidth allocation to transmit optimized ground-based optical flow data for collaborative perception.

[0046] Minimum risk maneuver (MRM) operation can also be controlled based on characteristics of ground-driven sensor self-calibration. For example, when the ground-derived pose update indicates a calibration deterioration (e.g., inconsistent ground optical flow alignment), the vehicle can trigger MRM by reducing speed, tightening lane centering, or safely pulling over until calibration stabilizes. During MRM, one or more processors can continuously apply updated ground-based attitude estimates to maintain a minimum risk state while performing controlled braking or lateral stabilization maneuvers.

[0047] Biased driving can also be controlled based on features of ground-driven sensor self-calibration. For example, when ground-derived attitude updates indicate a slight camera pitch or roll bias, the driving control unit can adjust the target lateral distance of the bias to compensate (e.g., slightly shifting the vehicle to the left or right within the lane) to maintain a safer clearance from road edges or adjacent vehicles. During maneuvers (e.g., lane changes or merging), the unit can utilize the updated calibration to select the lateral offset of the bias to maintain stability on uneven surfaces, avoid roadside obstacles, or increase buffer space with nearby traffic. One or more sensors (e.g., IMU sensors, cameras, lidar, radar, blind spot sensors, or parking sensors, etc.) can also be controlled based on features of ground-driven sensor self-calibration. For example, when ground-derived attitude updates indicate a camera pitch or roll bias, the system can adjust camera exposure or region of interest settings to maintain stable perception, recalibrate IMU drift using ground-aligned motion cues, or update lidar / radar fusion weights to reflect improved attitude accuracy. These calibration-driven adjustments may help maintain accurate detection and support downstream autonomous driving controls (e.g., lane keeping, collision avoidance, or adaptive cruise control). Autonomous driving levels and / or autonomous driving activation or deactivation can also be controlled based on characteristics of ground-driven sensor self-calibration. For example, when the attitude update derived from the ground detects a decline in calibration quality (e.g., increased view error or unstable ground optical flow alignment), the system may automatically reduce the autonomous driving level, increase driver attention requirements (e.g., more frequent hands-on steering wheel or gaze confirmation) or deactivate autonomous driving until calibration stabilizes. Conversely, when calibration confidence increases, the system may restore a higher level of autonomous driving or re-enable hands-free functions (e.g., display unlocking for certain driver-assistance visualizations). In this invention, for ease of explanation, terms such as "up," "down," "left," and "right" are used in this specification, and the positional relationships described in the specification can be understood in reverse if the accompanying drawings are reversed.

[0048] According to the present invention, sensors mounted on a vehicle (e.g., cameras) can be automatically calibrated using only images captured while the vehicle is in motion, without requiring any special calibration targets or large datasets. As the vehicle moves, the sensors acquire a series of images, and the vehicle observes the offset of small features on the road surface between the images. By analyzing these offsets, the vehicle identifies motion patterns belonging to the ground and maps them into three-dimensional space. The sensor's orientation (e.g., its yaw, pitch, and roll angles) can then be adjusted so that, when the images are converted to a bird's-eye view, the features of the ground appear consistent in size and orientation. When the orientation is correct, the ground motion vectors become uniform and parallel; if the orientation is incorrect, the vehicle detects deformation and makes further adjustments. Through this process, the vehicle can determine the correct alignment required for accurate perception, enabling the vehicle to calibrate the sensors during normal driving.

[0049] In the following description, examples of the invention will be described with reference to the accompanying drawings.

[0050] In the following text, reference will be made to Figure 1 A calibration apparatus for performing a camera calibration method according to an example of the present invention is described. Figure 1 An example of a module constituting a calibration device for a camera according to an example of the present invention is shown.

[0051] Reference Figure 1The calibration device 100 can provide camera parameters (e.g., geometric parameters) by utilizing continuous image data, wherein the camera detects the surrounding environment in a time-series manner as the device equipped with the camera moves in the actual environment. The geometric parameters can be camera pose information. The pose information can include parameters related to spatial translation and rotation applied when converting image data acquired from the camera into a view with a predetermined viewpoint (e.g., bird's-eye-view transformation, top-down map layout, occupancy-grid viewpoint, or depth-augmented viewpoint). These parameters can be external geometric parameters. For example, the view with the predetermined viewpoint can be a view from a bird's-eye-view perspective. The view can have various viewpoints, such as, but not limited to, three-dimensional views, views with depth information, semantic-view projections, etc. (e.g., side-view projection, oblique-view projection, or fused multi-camera views). The calibration device 100 can correct distortions in cameras with wide-angle, stereo, or panoramic lenses, or provide internal geometric parameters based on focal length, principal point, etc., to provide depth information (e.g., for depth-from-motion, stereo disparity estimation, or 3D reconstruction).

[0052] For example, calibration device 100 can receive time-series image data from an external device equipped with a camera while moving in a real environment, and generate camera parameters. Calibration device 100 can send the camera parameters to the camera or the vehicle 200 equipped with the camera. As another example, calibration device 100 can be a sensor calibration device with a camera installed while moving in a real environment. Calibration device 100 can generate camera parameters based on time-series image data acquired in the surrounding environment while moving. Calibration device 100 can send these parameters to the camera or the vehicle 200 equipped with the camera. As another example, calibration device 100 can be configured in a vehicle 200 that uses image data acquired from a camera for driving control. When the vehicle 200 updates the camera parameters while driving, calibration device 100 can output parameters based on the image data acquired while driving and update the camera parameters (e.g., pose drift correction, rolling-shutter compensation, or temporal recalibration under vibration, etc.). In this invention, for ease of description, Figure 1 The calibration device 100 is described as an example of receiving time-series image data from a camera transmitted from an external device, but it can also be applied to another example of a sensor calibration device. Another example is a vehicle 200 equipped with the calibration device 100. Figure 10 As shown in the figure, it will be described later.

[0053] On the other hand, the vehicle 200, which receives camera parameters such as attitude information from the calibration device 100, can refer to a mobile device. The vehicle 200 is a ground vehicle that travels on the ground and can be a conventional passenger vehicle or commercial vehicle, a purpose-built vehicle (PBV), etc. The vehicle 200 can be a robot in a broad sense (e.g., a mobile device), and the robot can move using wheels, tracks, or other mobility modules (e.g., articulated legs, hybrid wheel-leg mechanisms, or omnidirectional drive units, etc.). The vehicle 200 can move autonomously or passively.

[0054] The autonomous vehicle 200 can achieve semi-autonomous or fully autonomous driving. Fully autonomous driving can provide automatic movement where the vehicle 200 can be fully controlled without user intervention, even when driving conditions are uncertain (e.g., in low visibility conditions, heavy traffic, or complex intersections). Semi-autonomous driving can provide automatic movement where driver intervention is required depending on specific driving conditions (e.g., construction zones, inclement weather, or unstructured roads).

[0055] The attitude information of the camera according to the present invention can be applied not only to vehicles 200 used for ground driving, but also to various transportation devices such as flying mobile devices (e.g., drones, urban air traffic aircraft or unmanned aerial vehicles).

[0056] The calibration device 100 may be a device provided separately from the vehicle 200, operated by the vehicle manufacturer or by a regulatory authority providing autonomous driving services (e.g., a server). (See server) Figure 9 Server 100 can not only provide information related to camera calibration, but also send applications or learning models (e.g., object detection models, lane segmentation models, sensor fusion models, or trajectory prediction models) to the vehicle for autonomous driving. When server 100 is operated by the vehicle manufacturer or an autonomous driving management agency, it can receive connection data from vehicle 200, or send data required for autonomous driving (e.g., HD map updates, cloud-based calibration data, or remote software modules). In response to requests and data from vehicle 200 and user devices, server 100 can send various information and software modules for controlling vehicle 200 to support autonomous driving and various services (e.g., over-the-air updates, feature activation packages, or remote diagnostics). In this invention, the camera calibration function among the various functions of the server is described.

[0057] The calibration device 100 may include a communication unit 102, an input unit 104, an output unit 106, a memory 108, and a processor 110.

[0058] The communication unit 102 can support communication with external devices, vehicles 200 and 400, etc. In this invention, the communication unit 102 can be a communication interface for receiving time-series image data from cameras received from external devices or vehicles 200 and 400. The communication interface can send camera parameters (e.g., attitude information) estimated using the image data to the cameras or vehicles 200 and 400 (e.g., when attitude drift occurs, when recalibration is required after vibration, or when the camera is replaced, etc.).

[0059] The input unit 104 may include an interface for receiving user requests, and the output unit 106 may include an interface for visually providing information generated according to the present invention. This information may include, for example, camera parameters and intermediate data output during parameter generation (e.g., intermediate feature maps, optical-flow visualizations, or confidence indices, etc.).

[0060] The memory 108 can store programs and various data for controlling the calibration device 100, and can load programs or read and write data according to the request of the processor 110.

[0061] In this invention, memory 108 can store multiple consecutive image data in a time series, a model for generating optical flow data based on the multiple image data, and a software module for estimating camera pose information. The model for generating the optical flow data can be, for example, a learned model (e.g., a CNN-based model, a transformer-based model, or an encoder-decoder optical flow network, etc.), which will be described in detail later. The software module can be configured to estimate the camera pose information that minimizes the view error of multiple ground optical flow data aligned to a view based on the image data with a predetermined viewpoint (e.g., BEV alignment, ground-plane rectification, or multi-view geometric correction, etc.).

[0062] Processor 110 can perform overall control of calibration device 100. Processor 110 can be configured to execute application programs and instructions stored in memory 108. Specifically, processor 110 can perform a process of generating first optical flow data between time-series image data based on camera movement. Processor 110 can perform a process of extracting ground optical flow data located on the estimated ground from second optical flow data mapped to three-dimensional space based on the first optical flow data (e.g., using keypoint classification, plane fitting logic, or orientation-based filtering, etc.). Furthermore, processor 110 can perform a process of estimating camera pose information that minimizes view error when aligning multiple ground optical flow data to a view based on image data and a predetermined viewpoint.

[0063] Then, processor 110 can send the camera's attitude information to the camera or vehicle 200, 400. Vehicle 200, 400 can process the camera's image data based on the attitude information, and control the movement of vehicle 200 based on the processed image data and other sensor data (e.g., lidar data, radar data, or ultrasonic measurement results). The above processing by processor 110 will be described in detail later.

[0064] In this invention, processor 110 can be implemented as, for example, a single processing module. As another example, the processing according to the above can be distributed processing among multiple processing modules. In this invention, processor 110 can be collectively referred to as multiple processing modules (e.g., a vision processor, a neural network accelerator, or a motion estimation module, etc.).

[0065] In the following text, reference will be made to Figures 2 to 8 A camera calibration method according to another example of the present invention is described. Figure 2 An example of a camera calibration method is shown. Hereinafter, the method can be described as being performed by a calibration device 100, which generates camera parameters based on time-series image data received from an external device equipped with and moving a camera capturing the surrounding environment. However, unless technically contradictory, the following description can be applied to a sensor calibration device or vehicle 200. The above method is performed by a processor 110 of the calibration device 100, but for ease of description, processor 110 may be abbreviated as calibration device 100, or these terms may be used interchangeably.

[0066] Reference Figure 2 In step S105, the processor 110 of the calibration device 100 can acquire continuous image data in a time series based on the movement of an external device equipped with a camera. In this invention, from a time series perspective, the image data can be referred to as image frames (e.g., video frames, captured still images, or compressed keyframes, etc.).

[0067] Reference Figure 2 In S110, the processor 110 can generate first optical flow data between consecutive image data in a time series. In this invention, consecutive image data are represented as time-series adjacent image frames, but even if they are not adjacent, they can be image frames from different times (e.g., frames skipped due to low frame rate, sensor latency, or computational decimation, etc.).

[0068] Reference Figure 3 Describe the generation of the first optical flow data. Figure 3 An example of the process of generating the first optical flow data is shown.

[0069] The processor 110 can load the model that generates the first optical flow data from the memory 108 to generate the data. The model can be, for example, a deep learning model (e.g., a PWC-Net model, a RAFT model, or a FlowNet model, etc.) that serves as a learning model.

[0070] Processor 110 can extract a first feature and a second feature from multiple time-series image data. Feature extraction can be, for example, by an encoder of a Convolutional Neural Network (CNN). The encoder can include multiple layers, and features can be extracted by applying filters to each layer. The encoder can output feature maps from the individual image data through feature extraction. The second feature can have a more precise level than the first feature, for example, having higher resolution or higher-level data (e.g., finer texture details, sharper edges, or denser feature points, etc.) compared to the first feature. For example, the second feature can be generated by decoding the first feature. As another example, the second feature can be output from an intermediate layer in the layer used to extract the first feature.

[0071] Processor 110 can generate feature descriptors that describe the internal relationships between keypoints associated with a first feature of each image data and the external relationships between keypoints matched across multiple image data. Keypoints can be, for example, but not limited to, portions of edges or vertices (e.g., corners, intersections, or blobular points) included in the feature maps of the respective image data. Processor 110 can generate feature descriptors, for example, by utilizing a local feature matching model with a transformer (e.g., LoFTR, SuperGlue, or SparseFormer). This model can convert each first feature into a one-dimensional vector and add positional encoding to each one-dimensional vector to flatten the first feature (or first feature map) of each image data. The model can then input each positionally encoded one-dimensional vector into self-attention and cross-attention to generate feature descriptors with internal and external relationships. The feature descriptors can be output, for example, as vector maps (e.g., feature descriptor grids or dense embedding maps).

[0072] Internal relationships can be generated by a transformer that utilizes self-attention between keypoints 122 and 124 of the first feature map 120 in each image data, such as... Figure 4 As shown. Figure 4 An example of generating feature descriptors in a single image dataset is shown. Specifically, the internal relationship can be a feature descriptor describing the relationship between any keypoint 122 and surrounding keypoints 124 (e.g., spatial proximity, edge continuity, or local pattern similarity).

[0073] External relations can be generated by a transformer that utilizes cross-attention between keypoints 122 and 132 matched in first feature maps 120 and 130 across multiple image data, such as... Figure 5 As shown. Figure 5 An example of generating feature descriptors across multiple time-series image data is illustrated. Processor 110 can perform matching between keypoints 122 and 132 based on the vectors of keypoints 123 and 132 (e.g., using cosine similarity, relevance scores, or learned attention weights). The external relationship can be a feature descriptor describing the relationship between matched keypoints 122 and 132 across the time-series image data in terms of their matching states.

[0074] In this invention, even in low-texture environments without special features (e.g., on the ground or road surface of a road environment (e.g., asphalt, concrete, soil, or sprayed road markings, etc.)), first optical flow data based on feature descriptors can be generated well. Furthermore, since self-attention defines the position of the keypoint itself in the corresponding image, repeating patterns between multiple image data can also be smoothly matched (e.g., patchwork road surfaces, repeating lane markings, or patterned floors, etc.).

[0075] Processor 110 can generate global coarse matching information based on feature descriptors. Specifically, processor 110 can input a vector map of feature descriptors into a differentiable matching layer to generate global coarse matching information. The global coarse matching information can be, for example, a confidence matrix, and the matching of features in the confidence matrix can be selected according to a mutual nearest neighbor criterion to generate coarse-level matching predictions (e.g., region-level correspondence, approximate optical migration, or coarse trajectory alignment). The coarse-level matching predictions can be provided for local regions with windows of a specific size in the feature map.

[0076] Processor 110 can generate first optical flow data between consecutive image data by performing precise matching between multiple image data based on global coarse matching information and second features of multiple image data (e.g., fine patch-level feature descriptors, high-resolution feature embeddings, or detail-enhancing feature maps). For example, processor 110 can crop each second feature of the image data according to the window size of the coarse-level matching prediction provided in the global coarse matching information, and generate first optical flow data between consecutive image data for each cropped second feature with reference to the global coarse matching information. The first optical flow data can be generated as having location information of keypoints of each image data and corresponding keypoint matching pairs between multiple image data. Here, the first optical flow data can be generated in a format with optical flow vectors (e.g., 2D pixel displacement, subpixel motion vectors, or direction-magnitude vector pairs, etc.). Thus, the global coarse matching information is refined, and first optical flow data with subpixel-level precise matching can be output.

[0077] Refer again Figure 2In step S115, the processor 110 of the calibration device 100 can extract ground optical flow data located on the estimated ground from second optical flow data mapped from the first optical flow data to three-dimensional space. Here, the ground can be a road surface (e.g., asphalt, concrete, gravel, or soil surface).

[0078] Reference Figure 6 Description of S115. Figure 6 An example of generating ground optical flow data is shown.

[0079] Reference Figure 6 In S205, processor 110 can map key points of time-series image data into three-dimensional space. This mapping can be a reconstruction (e.g., triangulation-based reconstruction, depth estimation from motion, or multi-view geometric reconstruction, etc.).

[0080] The reconstruction of 3D space is exemplified by using epipolar geometry to map matched keypoints in continuous image data into 3D space. The reconstruction shown can be, but is not limited to, Structure-from-Motion (SFM). According to the example, matched keypoints in the first optical flow data can be moved to a normalized plane by correcting distortion and removing the internal matrix using internal geometric parameters (e.g., focal length, principal point, or lens distortion coefficients). A fundamental matrix can be estimated based on the matched keypoints in the normalized plane. The fundamental matrix can reveal the epipolar geometry corresponding to the geometric relationship between the coordinates of two keypoints in the normalized plane. The 3D space can be derived using the geometric relationship between the two coordinates and triangulation.

[0081] Reference Figure 6 In S210, processor 110 can generate second optical flow data in three-dimensional space based on key point locations mapped to key points in three-dimensional space and key point matching pair information. The second optical flow data can be generated in a format with optical flow vectors (e.g., 3D displacement vectors, direction-amplitude representations, or 3D point motion fields, etc.).

[0082] Reference Figure 6 In S215, processor 110 can determine the similarity between multiple second optical flow data in three-dimensional space. Specifically, as... Figure 7 As shown, the processor 110 can determine the similarity between multiple second optical flow data 150, 152, 154 based on the angle between the normal vectors 160, 162 of each of the planes 156, 158 formed by adjacent second optical flow data 150, 152, 154. Figure 7An example of evaluating the same plane is shown. For example, the closer the cosine similarity between adjacent normal vectors 160 and 162 is to 1, the more similar the second optical flow data 150, 152, and 154 can be determined (e.g., similarity thresholds of 0.95, 0.98, or 0.99, etc.).

[0083] Reference Figure 6 In S220, processor 110 can estimate the same plane by clustering similar second optical flow data (e.g., using angle thresholds, cosine distance metrics, or neighborhood-based grouping). Figure 7 As shown, the processor 110 can estimate two planes 156 and 158 as the same plane based on the determination that the two planes 156 and 158 are similar.

[0084] Reference Figure 6 The S225 processor 110 can identify the same plane as a ground feature based on the position of the camera and the motion information of the device with the camera installed.

[0085] Motion information may include, for example, motion vectors generated from multiple image data based on the movement of the device, relative positions between features on the same plane and motion vectors, road surface and continuity information from previous image frames, etc. (e.g., self-motion vectors, IMU-based motion estimates, or visual odometry outputs, etc.). The relative positions between the same plane and motion vectors (e.g., features on the same plane) may include at least one of the same plane, second optical flow data constituting the same plane, or normal vectors (e.g., vertical plane normals, ground-aligned normals, or forward-facing plane normals, etc.). The relative positions may relate, for example, to whether the features on the same plane are located below the motion vector, or to whether the features on the same plane are substantially perpendicular to the direction of motion (e.g., the motion vector points forward, while the ground normal points upward, etc.). When features on the same plane are located below the motion vector or perpendicular to the direction of motion, the same plane can be identified as a ground feature.

[0086] Reference Figure 6 In steps S230 and S235, processor 110 can estimate the ground based on identified ground features and extract second optical flow data located on the estimated ground as ground optical flow data. The ground optical flow data can have an optical flow vector format (e.g., 2D pixel motion vector, 3D displacement vector, or direction-amplitude optical flow pairs, etc.). Figure 8 As shown, the estimated second optical flow data on the ground or road surface can be extracted as ground optical flow data 176 and 178. Figure 8 An example of ground optical flow data from an aerial image is shown.

[0087] Refer again Figure 2In S120, the processor 110 can estimate the camera's attitude information, which minimizes the view error of multiple ground optical flow data aligned to a view based on image data and a predetermined viewpoint (e.g., BEV alignment, top-down correction, or viewpoint normalization transformation).

[0088] A view from a particular perspective can be, for example, but not limited to, a bird's-eye view. A bird's-eye view can have a viewpoint in the vertical direction above the vehicle 200, pointing towards the ground or road surface, such as... Figure 8 As shown (e.g., top view projection, lowest point view projection, or vertical orthogonal view, etc.).

[0089] When the camera's orientation is correct in the bird's-eye view, multiple ground optical flow data 176 and 178 in 3D space can have the same size or length, and data 176 and 178 can be parallel (e.g., their flow vectors point in the same direction and have the same amplitude). When the camera's orientation is incorrect in the bird's-eye view, view errors may occur in the bird's-eye view, and the above conditions may not be met.

[0090] View error can be defined by the amplitude difference between multiple ground optical flow data 176, 178 and the angular difference between multiple ground optical flow data 176, 178 (e.g., mismatched length, angular deviation, or inconsistent flow direction, etc.). In this invention, view error can be referred to as view deviation. Specifically, view error can be generated by the sum of the absolute value of the amplitude difference of the ground optical flow vectors and α (1 - cosine similarity between ground optical flow vectors). The unit of the amplitude difference of the ground optical flow vectors is pixels, and α (1 - cosine similarity between ground optical flow vectors) can be represented unitlessly in the range of 0 to 1. The weight α can be determined experimentally (e.g., α = 0.3, α = 0.5, or α = 0.7, etc.). Figure 8 As shown, even if the approximately square vehicle 172 and the actually straight parking line 174 are distorted in the bird's-eye view 170, when there is no weight α, if the lengths of the ground optical flow vectors 176 and 178 that constitute the trapezoidal sides are basically the same, the view error can be considered to be close to zero.

[0091] View error can be minimized to make it close to zero. Specifically, view error can be minimized so that multiple ground optical flow data 176, 178 have the same size and are nearly parallel to each other (e.g., aligned flow directions, uniform amplitude or symmetrical displacement patterns, etc.).

[0092] Processor 110 can search for camera attitude information that minimizes view error. The camera attitude information can be generated as parameters relating to spatial movement and rotation applied to transformations from image data to a view with a specific viewpoint (e.g., a bird's-eye view), such as translation along an axis, rotation around yaw-pitch-tilt, or a combination of SE(3) transformations. Furthermore, the camera attitude information is generated as attitude elements of the camera, and these attitude elements may include yaw, pitch, and tilt.

[0093] According to the present invention, camera calibration can be performed based on the ground without a reference target. Regardless of the vehicle's direction of travel and the camera's shooting direction, and without involving a large amount of data, the camera's attitude elements (especially yaw, pitch, and roll) can be estimated based on the vehicle's longitudinal motion and the ground it travels on (e.g., forward motion, smooth curved motion, or low-speed crawling motion). Furthermore, the vehicle's behavior can be estimated for each frame of image data from the time-series camera (e.g., frame-by-frame self-motion, frame-by-frame calibration, or continuous tracking updates). This allows for... Figure 10 The vehicle shown may be important when performing camera calibration.

[0094] The generation will be given Figure 2 Description of the vehicle 200 that generates camera parameters (specifically, attitude information) or receives camera attitude information from server 100 (calibration device) and other devices that communicate with server 100.

[0095] Figure 9 This is a schematic diagram illustrating a vehicle communicating with another device to send and receive data.

[0096] As described above, vehicle 200 can refer to a device capable of moving to a specific location. Although vehicle 200 is described in this invention as an example of a vehicle traveling on land, the invention can also be applied to mobile devices used for air or water transportation (e.g., drones, unmanned aerial vehicles, or autonomous boats, etc.). Referring to the above... Figure 1 The vehicle 200 can be autonomously controlled and driven, and the autonomous driving can be implemented as semi-autonomous driving or fully autonomous driving (e.g., highway navigation mode, parking assistance mode or fully autonomous driving mode, etc.).

[0097] Vehicle 200 can be powered by either electricity or fossil fuels. In the case of electricity, vehicle 200 can be, for example, a pure battery vehicle powered solely by a high-voltage battery, or a gas fuel cell (e.g., a hydrogen fuel cell, a methane fuel cell, or an ammonia fuel cell). Furthermore, the fuel cell can utilize various types of gases capable of generating electricity; for example, the gas could be hydrogen. However, various gases are applicable, and not limited to these. In the case of fossil fuels, vehicle 200 can be powered by fuel (e.g., gasoline, diesel, or liquefied petroleum gas) and can be equipped with an engine that drives the wheels of actuator 214 through the combustion of fuel. The engine can be included in power source unit 212 to provide the driving rotational force to the wheels of actuator 214. Alternatively, vehicle 200 can be powered by a hybrid of electricity and fossil fuels (e.g., a plug-in hybrid, a mild hybrid, or a series hybrid).

[0098] On the other hand, vehicle 200 can communicate with other devices 100, 300, or another vehicle 400. Other devices may include, for example, a calibration device 100 supporting various controls, status management, and driving functions of vehicle 200; an ITS device 300 for receiving information from the ITS; and various types of user devices (e.g., smartphones, wearable devices, or in-vehicle infotainment units). (Refer to the above text.) Figure 1 The calibration device 100 may be an external device operated by the vehicle manufacturer or by an administrative agency providing autonomous driving services.

[0099] The ITS device 300 is, for example, a roadside base station (RSU), and can exchange vehicle identification data, driving control and status data, environmental data around the vehicle, map data, etc. with the vehicle 200 via V2I to assist the user in driving the vehicle or support the autonomous driving of the vehicle 200 (e.g., traffic signal timing, congestion alerts, or work area notifications). The vehicle 200 can exchange the data listed above with other vehicles 400 via V2V to support manual or autonomous driving (e.g., collision warnings, cooperative lane changing, or platooning coordination).

[0100] Vehicle 200 can communicate with other vehicles or other devices based on cellular communication, Wireless Access in Vehicular Environment (WAVE) communication, Dedicated Short Range Communication (DSRC) or Near Field Communication or other communication solutions (e.g., BLE, UWB or WiFi Direct).

[0101] For example, in order to communicate with server 100, ITS device 300, or other vehicles 400, vehicle 200 can use a cellular communication network such as LTE or 5G, WiFi, or WAVE. As another example, DSRC or similar technologies used in vehicle 200 can be used for inter-vehicle communication. The communication methods between vehicle 200, server 100, ITS device 300, another vehicle 400, and user equipment are not limited to the examples above.

[0102] Figure 10 An example of the modules that make up a vehicle is shown.

[0103] The vehicle 200 may include a sensor unit 202, an operation unit 204, a display 206, a load device 208, and a communication unit 210.

[0104] The sensor unit 202 may include various types of sensors (e.g., temperature sensors, cabin monitoring sensors, or vibration sensors, etc.) for sensing various states and conditions occurring in the external surrounding environment, internal systems, user operations, and passenger space of the vehicle 200.

[0105] Specifically, the sensor unit 202 may include an externally facing camera 202a, radar sensor 202b, etc., to identify dynamic and static objects (e.g., vehicles, pedestrians, cyclists, or road signs) around the vehicle 200.

[0106] Camera 202a can identify external objects as images during vehicle 200 use, generate image data, and send the image data to processor 218. Camera 202a can be installed at specific locations on vehicle 200, such as the front, rear, and sides of vehicle 200 (e.g., bumper, roof, exterior rearview mirror housing, or windshield area). In this invention, when multiple cameras 202a are installed at various locations, it can be referred to as a multi-camera system. Image data acquired by multiple cameras 202a can be referred to as multi-view image data or multi-view (e.g., front view, rear view, side view, or surround view images, etc.).

[0107] Radar sensor 202b can emit radio waves of a specific frequency around vehicle 200 and generate radar data by using radio waves reflected from external objects, thereby detecting the presence and relative distance, speed, direction, etc., of external objects (e.g., approaching vehicles, roadside obstacles, or pedestrians). While the present invention is shown without a lidar sensor (not shown), in another example, a lidar sensor (e.g., a roof-mounted lidar, a bumper-mounted lidar, or a corner-mounted lidar, etc.) can be mounted on vehicle 200.

[0108] Sensor unit 202 may include attitude sensor 202c. Attitude sensor 202c can detect, for example, the three-axis states of vehicle 200 (e.g., yaw, pitch, and roll) and output various attitude states of the vehicle based on the aforementioned factors (e.g., tilt angle, angular velocity, or rotational estimation based on acceleration). The attitude sensor may be configured as, for example, an IMU sensor, a gyroscope sensor, etc. Processor 218 can generate attitude data based on the attitude states.

[0109] In addition, sensor unit 202 may include a positioning sensor 202d for confirming the position of vehicle 200. Positioning sensor 202d is, for example, but not limited to, a GPS sensor or a GNSS sensor (e.g., a satellite-based RTK module, a differential GPS receiver, or a multi-band GNSS module, etc.).

[0110] The present invention describes the sensor of the sensor unit 202 mentioned in the specification, and may further include sensors for sensing various situations not listed herein (e.g., ultrasonic sensors, cockpit cameras, or rain / light sensors, etc.).

[0111] The operating unit 204 can be configured as a module for user steering to drive. For example, the operating unit 204 can be a steering wheel for manual driving, an automatic or manual transmission actuator, an accelerator pedal, a brake pedal, a transmission, etc. (e.g., paddle shifters, a driving mode selector, or an electronic shift-by-wire lever, etc.). The operating unit 204 may further include an interface for detailed functions for using, unlocking, and selecting user-requested autonomous driving modes so that the user can use autonomous driving functions (e.g., enabling lane keeping assist, activating highway navigation, or selecting automatic parking, etc.).

[0112] Display 206 can be used as a user interface. Display 206 can be displayed via processor 218 to output the vehicle 200's operating status, control status, route / traffic information, remaining energy information, and content requested by the driver (e.g., navigation maps, ADAS alerts, or media playback). Furthermore, display 206 can be configured as a touchscreen, on which driver input can be detected to receive driver requests directed to processor 218.

[0113] The load device 208 is mounted on the vehicle 200 and can be a non-driving electric device other than a drive power system such as a wheel drive unit (not shown). The load device 208 is an auxiliary device that receives power from the power supply unit 212 and can be, for example, an air conditioning system, an indicator light system, a lighting system, a seat system, and various devices installed in the vehicle 200 (e.g., an infotainment unit, window actuators, or interior comfort modules, etc.).

[0114] The communication unit 210 can support communication with the server 100, the ITS device 300, surrounding vehicles 400, etc. The communication unit 210 may include, for example, modules for processing cellular communication, WAVE, DSRC communication, etc. (e.g., a 5G-NR module, a V2X modem, or a WiFi communication processor). In this invention, the communication unit 210 can send data generated or stored during driving to the server 100 and receive data and software modules sent from the server 100. The communication unit 210 can support communication with... Figure 10 Communication between components that send or receive signals related to data, control, instructions, and requests. These components can be... Figure 10 The instruments or modules referred to by reference numerals 202-218 in the accompanying drawings are shown. Communication unit 210 can transmit signals (i.e., signals related to data, control, commands, and requests) between components to a communication bus and to specific components via, for example, Controller Area Network (CAN) communication. The data, control, commands, and requests transmitted in CAN communication can be referred to as CAN data. Communication unit 210 can support communication with electronic devices carried by passengers within vehicle 200 (e.g., smartphones, tablets, or wearable devices).

[0115] In addition, vehicle 200 may include power supply unit 212 and actuator 214.

[0116] The power supply unit 212 can generate and supply power and electricity for the drive power system and non-drive electrical systems (such as actuator 214). The non-drive electrical systems may be, for example, but not limited to, sensor unit 202, operation unit 204, display 206, load device 208, communication unit 210, etc., and may include various components (e.g., HVAC system, in-cabin electronics, or door / seat control modules, etc.) that realize sensing, interface, communication, and convenience functions, in addition to the components that directly participate in the drive operation.

[0117] Actuator 214 includes at least one module for implementing driving operations and can perform driving operations based on signals from user requests from operating unit 204 or according to requests from processor 218. Signals according to such requests can be associated with at least one of longitudinal control (e.g., acceleration or deceleration), lateral control (e.g., steering), or gear shifting (e.g., downshifting, upshifting, or switching driving modes, etc.). Here, gear shifting can be processed according to requests from manual driving users utilizing a gear shifter or from processor 218 in autonomous driving.

[0118] Actuator 214 may include a wheel drive unit (not shown) and mechanical components and electronic modules for implementing drive operations in the wheel drive unit to perform driving operations according to instructions from processor 218, either manually or automatically by a user. When vehicle 200 operates on electric power, vehicle 200 may include components for transmitting the requested drive motion to the wheel drive unit (e.g., an inverter, motor controller, or reduction gear assembly, etc.). When vehicle 200 operates on fossil fuel power, actuator 214 may include a transmission and gear module (e.g., an automatic transmission, dual-clutch transmission, or continuously variable transmission, etc.) for transmitting power from an internal combustion engine.

[0119] The wheel drive unit may include multiple wheels, a drive force generating module for generating and applying or transmitting driving force to the wheels, a braking module for decelerating the wheels, and a steering module for achieving lateral control of the wheels (e.g., a steer-by-wire module, electric power steering, or a four-wheel steering unit). When the vehicle 200 is electrically driven, the drive force generating module may be configured with a motor assembly that generates driving force based on the power output from the battery (e.g., a front motor, a rear motor, or a dual-motor AWD unit). The braking module of the electric vehicle 200 may further have regenerative braking functionality (e.g., mild regenerative braking, strong regenerative braking, or a driver-selectable level of regenerative braking).

[0120] In addition, vehicle 200 may include memory 216 and processor 218.

[0121] The memory 216 can store applications and various data used to control the vehicle 200, allowing the application to be loaded or data to be read and written according to requests from the processor 218. In this invention, the memory 216 can receive and manage parameters from the camera 202a of the server 100. Furthermore, the memory 216 can receive and manage information required for driving, such as map information, traffic information, weather information, accident information, etc. (e.g., HD maps, road closure warnings, or real-time congestion data).

[0122] Processor 218 can perform overall control of vehicle 200. Processor 218 can be configured to execute application programs and instructions stored in memory 216. Processor 218 can execute... Figure 2The calibration method generates parameters (e.g., attitude information) for camera 202a. In another example, processor 218 may receive attitude information of camera 202a generated by server 100. Specifically, processor 218 may convert image data acquired from camera 202a into a view from a predetermined viewpoint (e.g., BEV conversion, corrected front view, or oblique view projection, etc.) based on the attitude information and internal geometric parameters of camera 202a. Processor 218 may identify objects (e.g., vehicles, pedestrians, or road boundaries, etc.) based on the view at a predetermined time point. Processor 218 may control driving based on the identified objects. Furthermore, processor 218 may perform autonomous driving control (e.g., lane keeping control, adaptive cruise control, or collision avoidance control, etc.) based on information estimated from the view at a predetermined time point and various data identified from radar sensor 202b, attitude sensor 202c, and positioning sensor 202d.

[0123] In this invention, as an example, processor 218 can be implemented as a single processing module. As another example, the processing according to the above can be distributed processing among multiple processing modules, and in this invention, processor 218 can be collectively referred to as multiple processing modules (e.g., a perception processor, a planning processor, or a control processor, etc.).

[0124] Figure 11 An exemplary computing system (e.g., a computing device in a vehicle or any other device) is illustrated. Figure 11 As shown, one or more controllers, processors, etc. (e.g., one or more components of vehicle 200, one or more components of server 100, one or more components of another vehicle 400, and any other components and devices disclosed herein) can be provided by Figure 11 The computing system shown is implemented or implemented in the computing system.

[0125] The computing system 1000 may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1060, and a network interface 1700 interconnected via a bus 1200. The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and / or the storage device 1600. Each of the memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include read-only memory (ROM) and random access memory (RAM).

[0126] A communication interface, such as network interface 1700 (also referred to as a communication device, communicator, communication module, communication unit, etc.), enables the transmission of software and / or data between the device and one or more external devices and / or between one or more components of the device. The communication interface may include a receiver, transmitter, transceiver, modem, network interface and / or adapter (e.g., Ethernet adapter), radio transceiver, antenna, communication port, Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. The software and data transmitted through the communication interface may be in the form of signals, which may be electronic signals, electromagnetic signals, optical signals, infrared signals, or other signals that can be received by the communication interface. These signals can be provided to the communication interface through a communication path of the device, which may utilize, for example, wires or cables, optical fibers, cellular links, radio frequency (RF) links, and / or other communication channels. The communication interface can communicate using one or more communication protocols (e.g., Ethernet, Wi-Fi, near-field communication (NFC), Infrared Data Association (IrDA), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Long-Term Evolution (LTE), 5G New Radio (NR), vehicle-to-everything (V2X), controller area network (CAN), or local interconnect network (LIN)).

[0127] Therefore, the steps of the methods or algorithms described in conjunction with the examples disclosed in this specification can be implemented by hardware modules, software modules, or a combination of hardware and software modules executed by processor 1100. Software modules can reside on storage media (e.g., memory 1300 and / or storage device 1600), such as RAM, flash memory, ROM, erasable programmable ROM (EPROM), electrically EPROM (EEPROM), registers, hard disk drives, removable disks, or optical disc ROMs (CD-ROM).

[0128] The storage medium can be connected to the processor 1100. The processor 1100 can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor 1100. The processor and storage medium can be implemented using an application-specific integrated circuit (ASIC). The ASIC can be located in the user terminal. Alternatively, the processor and storage medium can be implemented as separate components in the user terminal.

[0129] For example, a camera calibration method is provided, comprising: generating first optical flow data between time-series image data based on camera movement; extracting ground optical flow data located on an estimated ground surface from second optical flow data mapped to three-dimensional space based on the first optical flow data; and estimating camera attitude information that minimizes view error among a plurality of ground optical flow data aligned with a view based on image data and having a predetermined viewpoint. In some embodiments, the first optical flow data, the second optical flow data, and the ground optical flow data may be in the format of optical flow vectors.

[0130] In some implementations, generating the first optical flow data may include: extracting a first feature from multiple time-series image data; generating a feature descriptor describing the internal relationships between key points associated with the first feature of each image data and the external relationships between key points matched across the multiple image data; generating global coarse matching information based on the feature descriptor; and generating the first optical flow data by performing precise matching across the multiple image data based on the global coarse matching information and second features of the multiple image data that are more precise than the first feature.

[0131] In some implementations, internal relationships can be generated by a transformer that utilizes self-attention between keypoints in the various image data, while external relationships can be generated by a transformer that utilizes cross-attention between matched keypoints.

[0132] In some implementations, the first optical flow data can be generated as including location information of key points for each image data and key point matching pair information regarding the correspondence between multiple image data.

[0133] In some implementations, the first optical flow data can be generated as key point location information for each image data and key point matching pair information about the correspondence between multiple image data, and the second optical flow data can be generated in three-dimensional space based on the key point matching pair information and the key point mapping of the time series image data to the key point location in three-dimensional space.

[0134] In some implementations, extracting ground optical flow data may include: determining the similarity between multiple second optical flow data; estimating the same plane by clustering similar second optical flow data; identifying the same plane as a ground feature based on the location of the camera and the motion information of the device with the camera installed; and extracting the second optical flow data located on the ground estimated based on the ground feature as ground optical flow data.

[0135] In some implementations, determining similarity may include determining the similarity between multiple second optical flow data based on the angle between the normal vectors of the respective planes formed by adjacent second optical flow data.

[0136] In some implementations, the view error can be defined by the amplitude difference between multiple ground optical flow data and the angular difference between multiple ground optical flow data, and the view error can be minimized such that the multiple ground optical flow data have the same amplitude and are approximately parallel to each other.

[0137] In some implementations, the camera's pose information can be generated as parameters relating to spatial translation and rotation applied to convert image data into a view with a predetermined perspective, and the view with the predetermined perspective is provided as a bird's-eye view.

[0138] For example, a camera calibration apparatus is provided, comprising: a memory storing at least one instruction; and at least one processor executing the at least one instruction stored in the memory. The at least one processor is configured to: generate first optical flow data between temporally adjacent image data based on camera movement; extract ground optical flow data located on an estimated ground surface from second optical flow data mapped to three-dimensional space based on the first optical flow data; and estimate camera attitude information that minimizes view error among a plurality of ground optical flow data aligned with a view based on image data and having a predetermined viewing angle.

[0139] For example, a method and apparatus for calibrating a camera can be provided, which calibrates the camera against the ground without a reference target and estimates all attitude elements of the camera without involving a large amount of data.

[0140] For example, regardless of the vehicle's direction of travel and the camera's shooting direction, the camera's attitude elements (especially yaw, pitch, and roll) can be estimated based on the vehicle's longitudinal movement and the ground it is traveling on. Furthermore, the vehicle's behavior can be estimated for each frame of image data from the time-series camera.

[0141] The effects that can be obtained in this invention are not limited to those described above, and those skilled in the art will clearly understand other effects not mentioned through the following description.

[0142] Although the exemplary method of the present invention described above is represented as a series of steps for clarity, it is not intended to limit the order in which the steps are performed, and these steps may be performed simultaneously or in different orders as needed. To implement the above method, the described steps may further include other steps, may include the remaining steps in addition to some steps, or may include other additional steps in addition to some steps.

[0143] The various examples of the present invention are not a list of all combinations, and are intended to describe representative examples of the invention. The contents described in the various examples can be applied independently or in combination of two or more.

[0144] Furthermore, various examples of the present invention can be implemented in hardware, firmware, software, or a combination thereof. When the present invention is implemented in hardware, it can be implemented using application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, etc.

[0145] The scope of this invention includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) for enabling the steps of methods according to various examples to be executed on a device or computer, as well as non-volatile computer-readable media including such software or instructions stored thereon and executable on a device or computer.

Claims

1. A method performed by a vehicle, the method comprising: Time-series image data relating to the vehicle's surrounding environment is generated using the vehicle's sensors. Based on the movement of the sensor, the first optical flow data between adjacent image frames of time-series image data is obtained through the sensor; The first optical flow data is mapped onto three-dimensional space to generate the second optical flow data; Extract ground optical flow data related to key points located on the ground from the second optical flow data; Align ground optical flow data to a view based on time-series image data with a predetermined perspective; Determine the view error of the aligned ground optical flow data; Determine sensor-related attitude parameters that, when applied, reduce the view error; Output a signal indicating the attitude parameters; Adjust the sensor based on the signal.

2. The method according to claim 1, wherein, The first optical flow data, the second optical flow data, and the ground optical flow data have an optical flow vector format.

3. The method according to claim 1, wherein, The first optical flow data obtained includes: Extracting the first feature from time-series image data; Generate feature descriptors that describe the following: The internal relationships between key points related to the first feature; External relationships of key points matched between adjacent image frames in time-series image data; Generate global coarse matching information based on feature descriptors; Extract a second feature from time-series image data, wherein the second feature has a finer granularity than the first feature; Based on global coarse matching information and second features, precise matching is achieved between adjacent image frames in time series image data; The first optical flow data is generated based on precise matching.

4. The method according to claim 3, wherein, The generated feature descriptors include: A transformer-based self-attention process is applied to keypoints associated with the first feature to generate internal relationships; A transformer-based cross-attention process is applied to key points in time-series image data to match adjacent image frames in order to generate external relationships.

5. The method according to claim 3, wherein, The first optical flow data includes: Location information of key points in each image frame of time-series image data; Matching pair information indicating the correspondence between key points in adjacent image frames.

6. The method according to claim 1, wherein, The first optical flow data includes: Location information of key points in each image frame of time-series image data; Matching pair information indicating the correspondence between key points in adjacent image frames; The second optical flow data is generated based on the matching pair information and the position obtained by mapping key points of time-series image data to three-dimensional space.

7. A vehicle comprising: Sensors configured to generate time-series image data relating to the vehicle’s surrounding environment; as well as The processor circuitry is configured to enable the vehicle to: Based on the movement of the sensor, the first optical flow data between adjacent image frames of the time-series image data is obtained; Key points related to the first optical flow data are mapped to three-dimensional space to generate multiple second optical flow data. Form a cluster comprising at least two of the plurality of second optical flow data; Each of the at least two second optical flow data has its own normal vector; Wherein, each of the normal vectors satisfies the clustering condition based on the angle difference between the normal vectors; Wherein, the cluster corresponds to the candidate plane; Based on the motion information of vehicles equipped with sensors, candidate planes are identified as ground. Obtain multiple ground optical flow vectors associated with key points located on the ground; Convert the plurality of ground optical flow vectors into a view from a predetermined viewpoint; The view error of the transformed multiple ground optical flow vectors is determined based on at least one of the following: The magnitude difference between the converted ground optical flow vectors; The angular difference between the converted ground optical flow vectors; Determine attitude parameters that, when applied, reduce the view error, wherein the attitude parameters indicate at least one of the sensor's yaw, pitch, and roll. Output a signal indicating the attitude parameters; Adjust the sensor based on the signal.

8. The vehicle according to claim 7, wherein, The processor circuit is further configured to: enable the vehicle to determine whether the at least two second optical flow data satisfy the clustering condition based on the angle difference between the normal vectors of at least two of the multiple second optical flow data.

9. The vehicle according to claim 7, wherein, The processor circuit is further configured to enable the vehicle to generate the plurality of ground optical flow vectors by selecting data related to key points located on the ground from the plurality of second optical flow data.

10. The vehicle according to claim 7, wherein, The processor circuit is further configured to enable the vehicle to determine the attitude parameters by reducing the view error, such that the converted multiple ground optical flow vectors have a uniform amplitude and parallel direction.

11. A vehicle apparatus, the apparatus comprising: Sensors configured to generate time-series image data relating to the vehicle’s surrounding environment; processor; as well as A memory storing at least one instruction, which, when executed by a processor, configures the device to: Based on the movement of the sensor, the first optical flow data between adjacent image frames of time-series image data is obtained through the sensor; The first optical flow data is mapped onto a three-dimensional space to generate multiple second optical flow data. Extract ground optical flow data related to key points located on the ground from the multiple second optical flow data; Align ground optical flow data to a view based on time-series image data with a predetermined perspective; Determine the view error of the aligned ground optical flow data; Determine sensor-related attitude parameters that, when applied, reduce the view error.

12. The device for a vehicle according to claim 11, wherein, The first optical flow data, the plurality of second optical flow data, and the ground optical flow data have an optical flow vector format.

13. The device for a vehicle according to claim 11, wherein, When executed by a processor, the at least one instruction is configured to cause the device to: Extracting the first feature from time-series image data; Generate feature descriptors that describe the following: The internal relationships between key points related to the first feature; External relationships of key points matched between adjacent image frames in time-series image data; Generate global coarse matching information based on feature descriptors; Extract a second feature from time-series image data, wherein the second feature has a finer granularity than the first feature; Based on global coarse matching information and second features, precise matching is performed between adjacent image frames in time series image data; The first optical flow data is generated based on precise matching.

14. The device for a vehicle according to claim 13, wherein, The feature descriptor is generated in the following manner: A transformer-based self-attention process is applied to keypoints associated with the first feature to generate internal relationships; A transformer-based cross-attention process is applied to key points in time-series image data to match adjacent image frames in order to generate external relationships.

15. The device for a vehicle according to claim 13, wherein, The first optical flow data includes: Location information of key points in each image frame of time-series image data; Matching pair information indicating the correspondence between key points in adjacent image frames.

16. The device for a vehicle according to claim 11, wherein, The first optical flow data includes: Location information of key points in each image frame of time-series image data; Matching pair information indicating the correspondence between key points in adjacent image frames; The multiple second optical flow data are generated based on matching pair information and the positions obtained by mapping key points of time-series image data to three-dimensional space.

17. The device for a vehicle according to claim 11, wherein, When executed by a processor, the at least one instruction is configured to cause the device to: Determine the similarity between the multiple second optical flow data; Based on the fact that the similarity condition between at least two second optical flow data is satisfied, the at least two second optical flow data are clustered to determine candidate planes; Based on the sensor's location and the motion information of the vehicle equipped with the sensor, the candidate plane is identified as the ground. Extract the second optical flow data associated with key points located on the ground as ground optical flow data.

18. The device for a vehicle according to claim 17, wherein, When executed by a processor, the at least one instruction is configured to cause the device to: Based on the angle between the normal vectors of the at least two second optical flow data, determine whether the at least two second optical flow data satisfy the similarity condition.

19. The device for a vehicle according to claim 11, wherein, The view error is determined based on the following: The magnitude difference of the ground optical flow vector associated with the aligned ground optical flow data; The difference in the direction of the ground optical flow vector; When executed by a processor, the at least one instruction is configured to reduce the view error of the device such that the ground optical flow vectors have a uniform amplitude and are parallel to each other.

20. The device for a vehicle according to claim 11, wherein, The attitude parameters represent the spatial translation and rotation of the sensor used to convert ground optical flow data into a view with a predetermined perspective. The predetermined perspective includes a bird's-eye view.

Images