System and method for processing avm image based on vehicle driving state, and computer-readable recording medium having program for implementing the recorded thereon

Abstract

A system and method for processing AVM images based on vehicle driving conditions, and a computer-readable recording medium storing a program for implementing the method are disclosed. An AVM image processing system based on vehicle driving conditions according to an embodiment of the present invention includes multiple cameras for capturing images in the shape of a curved surface, an image correction device for correcting the captured images, and an autonomous driving device for controlling the autonomous driving of a vehicle using the corrected images. The image correction device includes an image conversion unit for converting the images in the shape of a curved surface into planar projection images, an information collection unit for collecting driving information of a vehicle from the autonomous driving device, a field of view correction unit for correcting changes in the field of view in real time using the converted planar projection images and the collected driving information, a FoV / ROI adjustment unit for adjusting the field of view angle and region of interest according to the vehicle's driving mode based on the corrected field of view, a reliability determination unit for determining reliability information for the adjusted images, an image matching execution unit for performing relational image matching between multiple cameras by fusing the determined reliability information and image clues obtained from the images, and a top-view image generation unit for generating a top-view image using the image matching results.

Description

Technology Field

[0001] The present invention relates to an AVM image processing system and method based on vehicle driving conditions and a computer-readable recording medium storing a program for implementing the method. More specifically, the invention relates to an AVM image processing system and method based on vehicle driving conditions capable of providing highly reliable AVM top-view images in various driving situations, and a computer-readable recording medium storing a program for implementing the method. Background Technology

[0003] Generally, vehicles are equipped with an Around View Monitoring (AVM) system using front, rear, left, and right cameras to intuitively provide the driver with surrounding conditions while driving. This AVM system combines images from multiple cameras to generate a top-view image, which resembles a view from above the vehicle, and provides it to the driver.

[0004] The AVM system has made it possible to provide users with convenience and safety in situations such as parking, low-speed driving, and passing through narrow sections. Nevertheless, the AVM system still exhibits the following technical limitations.

[0005] Firstly, since it is based on video captured by cameras, field of view distortion occurs due to constraints such as the lenses and installation positions of each camera, and it fails to adequately reflect dynamically changing camera coordinates during vehicle rotation or acceleration / deceleration.

[0006] Secondly, since most AVM systems perform matching based on limited cues such as image feature points, colors, and edges, they fail to reflect factors in actual driving environments, such as lighting changes, shadows, road surface patterns, and vehicle vibrations.

[0007] Thirdly, while some AVM systems attempt to correct visibility by utilizing gyroscope sensors or vehicle attitude information, accurate visibility correction is difficult in most cases due to missing sensor errors and complex dynamic situations.

[0008] Consequently, at a time when the accuracy of image-based cognitive information is becoming increasingly important, there is a need to develop technology that can resolve the problems of conventional AVM systems and improve matching quality by efficiently integrating multiple cues. Prior art literature

[0010] Published Patent Application No. 10-2021-0023174 Japanese Registered Patent Application JP6890855 The problem to be solved

[0011] To solve the aforementioned problems, the technical objective of the present invention is to provide a vehicle driving state-based AVM image processing system and method that provide stable and highly reliable AVM top-view images in various driving environments by dynamically correcting the field of view and aligning multiple cues to generate a top-view image, and to present a computer-readable recording medium storing a program for realizing this method.

[0012] The problems solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0014] As a means to solve the aforementioned technical problem, an AVM image processing system based on a vehicle driving state according to an embodiment of the present invention comprises a multi-camera system for capturing a curved image, an image correction device for correcting the captured image, and an autonomous driving device for controlling the autonomous driving of a vehicle using the corrected image. The image correction device comprises an image conversion unit for converting the curved image into a planar projection image, an information collection unit for collecting driving information of a vehicle from the autonomous driving device, a field of view correction unit for correcting changes in the field of view in real time using the converted planar projection image and the collected driving information, a FoV / ROI adjustment unit for adjusting the field of view angle and region of interest according to the vehicle's driving mode based on the corrected field of view, a reliability determination unit for determining reliability information for the adjusted image, an image matching unit for performing relational image matching between the multi-cameras by fusing the determined reliability information and image clues obtained from the image, and a top-view image generation unit for generating a top-view image using the image matching result.

[0015] The above image conversion unit may include an image normalization unit that converts a curved image into a planar projection image and normalizes it, a distortion removal unit that removes distortion from the normalized image, a coordinate system alignment unit that aligns the distortion-removed image to a reference coordinate system, and an image standardization unit that standardizes the quality of the image aligned to the reference coordinate system.

[0016] The driving information of the above vehicle may include yaw, pitch, roll, speed, and steering information.

[0017] The above-mentioned field of view correction unit may include a projection matrix update unit that updates the projection matrix of a multi-camera image using driving information, and a ground coordinate system matching unit that performs matching with a ground coordinate system using the updated projection matrix.

[0018] The projection matrix update unit above can determine the projection matrix by the following formula:

[0019]

[0020] Here, H i (t) is the projection matrix of the i-th camera, H i0 is the initial fixed projection matrix, s(t) is the change in vehicle state at time t, is a rotation correction matrix determined by s(t), is a fine position shift correction matrix caused by changes in vehicle state.

[0021] The above reliability determination unit may include a resolution adjustment unit that improves the resolution of an image using a Super-Resolution (SR) network, an uncertainty calculation unit that calculates the uncertainty of areas with potential quality degradation in the image with improved resolution, and a reliability calculation unit that calculates the reliability of the image.

[0022] The above uncertainty calculation unit can calculate the uncertainty using the following formula:

[0023]

[0024] Here, U(u,v) is the uncertainty value at coordinates (u,v), N is the number of times the resolution enhancement operation is performed, is the (u,v) pixel value of the n-th output image after running the SR network N times, is the average pixel value for the results of N SR network operations.

[0025] The above reliability calculation unit can calculate reliability by the following formula:

[0026]

[0027] Here, w(u,v) is the confidence value at coordinates (u,v), is a sigmoid function, edge(u,v) is the edge strength of pixel (u,v), c(u,v) is a preset confidence prediction value for the pixel, are the weights set for c(u,v) and edge(u,v), respectively.

[0028] The above uncertainty calculation unit can calculate the uncertainty using the following formula:

[0029]

[0030] Here, U(u,v) is the uncertainty value of pixel (u,v), and w(u,v) is the confidence value at pixel (u,v).

[0031] The above image matching unit may include a lane continuity calculation unit that calculates a lane continuity score, an object alignment calculation unit that calculates an object alignment score, a map alignment calculation unit that calculates a map alignment score, and a transformation matrix selection unit that selects a final transformation matrix by combining the lane continuity score, the object alignment score, and the map alignment score.

[0032] The above lane continuity calculation unit can calculate a lane continuity score by the following formula:

[0033]

[0034] Here, S lane (H k ) is the candidate transformation matrix H k Lane continuity score for, |L| is the number of reference lane point sets, is H k This is the predicted position of the lane projected onto the top-view image using .

[0035] The above object consistency calculation unit can calculate an object consistency score by the following formula:

[0036]

[0037] Here, S obj (H k ) is the candidate transformation matrix H k Object consistency score for, N o is the total number of objects simultaneously observed by multiple cameras, and H is the position of object o seen by cameras i and j kThese are the top-view coordinates when converted.

[0038] The above map match calculation unit can determine a map match score using IoU and distance-based indicators between structures in the HD map and structures in the image.

[0039] The above transformation matrix selection unit can determine the final transformation matrix by the following formula:

[0040]

[0041] Here, H * is the final transformation matrix, H k is the transformation matrix of the k-th candidate, S total (H k ) is H k This is the overall matching score for.

[0042] Meanwhile, an AVM image processing method based on a vehicle driving state according to another embodiment of the present invention comprises the steps of: converting a curved image captured by multiple cameras into a planar projection image; collecting driving information of the vehicle; correcting changes in the field of view in real time using the converted planar projection image and the collected driving information; adjusting the field of view angle and region of interest according to the vehicle's driving mode based on the corrected field of view; determining reliability information for the adjusted image; fusing the determined reliability information and image clues obtained from the image to perform relational image matching between multiple cameras; and generating a top-view image using the image matching result.

[0043] The above-mentioned transforming step may include the step of transforming a curved image into a planar projected image and normalizing it, the step of removing distortion from the normalized image, the step of aligning the distortion-removed image to a reference coordinate system, and the step of standardizing the quality of the image aligned to the reference coordinate system.

[0044] The driving information of the above vehicle may include yaw, pitch, roll, speed, and steering information.

[0045] The above correction step may include a step of updating the projection matrix of a multi-camera image using driving information and a step of performing alignment with the ground coordinate system using the updated projection matrix.

[0046] The above updating step can determine the projection matrix by the following formula:

[0047]

[0048] Here, H i (t) is the projection matrix of the i-th camera, H i0 is the initial fixed projection matrix, s(t) is the change in vehicle state at time t, is a rotation correction matrix determined by s(t), is a fine position shift correction matrix caused by changes in vehicle state.

[0049] The above-mentioned determining step may include a step of improving the resolution of an image using a Super-Resolution (SR) network, a step of calculating uncertainty regarding areas with potential quality degradation in the image with improved resolution, and a step of calculating confidence in the image.

[0050] The step of calculating the above uncertainty can calculate the uncertainty using the following formula:

[0051]

[0052] Here, U(u,v) is the uncertainty value at coordinates (u,v), N is the number of times the resolution enhancement operation is performed, is the (u,v) pixel value of the n-th output image after running the SR network N times, is the average pixel value for the results of N SR network operations.

[0053] The step of calculating the above reliability can calculate the reliability using the following formula:

[0054]

[0055] Here, w(u,v) is the confidence value at coordinates (u,v), is a sigmoid function, edge(u,v) is the edge strength of pixel (u,v), c(u,v) is a preset confidence prediction value for the pixel, are the weights set for c(u,v) and edge(u,v), respectively.

[0056] The step of calculating the above uncertainty can calculate the uncertainty (U(u,v)) by the following formula:

[0057]

[0058] The steps performed above may include a step of calculating a lane continuity score, a step of calculating an object consistency score, a step of calculating a map consistency score, and a step of selecting a final transformation matrix by combining the lane continuity score, the object consistency score, and the map consistency score.

[0059] The step of calculating the lane continuity score above can calculate the lane continuity score by the following formula:

[0060]

[0061] Here, S lane (H k ) is the candidate transformation matrix H k Lane continuity score for, |L| is the number of reference lane point sets, is H k This is the predicted position of the lane projected onto the top-view image using .

[0062] The step of calculating the object consistency score above can calculate the object consistency score using the following formula:

[0063]

[0064] Here, S obj (H k ) is the candidate transformation matrix H k Object consistency score for, N o is the total number of objects simultaneously observed by multiple cameras, and H is the position of object o seen by cameras i and j k These are the top-view coordinates when converted.

[0065] The step of calculating the map match score above can determine the map match score using IoU and distance-based indicators between structures in the HD map and structures in the image.

[0066] The step of selecting the final transformation matrix above can determine the final transformation matrix by the following formula:

[0067]

[0068] Here, H * is the above final transformation matrix, H k is the transformation matrix of the k-th candidate, S total (H k ) is H k This is the overall matching score for.

[0069] Additionally, according to another embodiment of the present invention, a computer-readable recording medium is disclosed that records a program for realizing an AVM image processing method based on a vehicle driving state according to any one of claims 16 to 28. Effects of the invention

[0071] According to the present invention, by updating the projection matrix of multiple cameras in real time based on the driving state of the vehicle, the effect is to provide a vehicle driving state-based AVM image processing system and method capable of maintaining a stable field of view aligned with a designated coordinate system despite changes occurring during driving, and a computer-readable recording medium recording a program for realizing this method.

[0072] In addition, by dynamically adjusting the field of view and region of interest according to the vehicle's driving speed and driving mode, optimized image information can be provided for various driving situations. This limits unnecessary image processing and improves the recognition accuracy of areas critical for driving judgment.

[0073] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0075] FIG. 1 is a block diagram of an AVM image processing system based on vehicle driving state according to an embodiment of the present invention. FIG. 2 is a block diagram of the autonomous driving device illustrated in FIG. 1, FIG. 3 is a block diagram of the image correction device illustrated in FIG. 1, FIG. 4 is a detailed block diagram of the image conversion unit illustrated in FIG. 3, FIG. 5 is a detailed block diagram of the field of view correction unit illustrated in FIG. 3, FIG. 6 is a conceptual diagram for explaining the operation of the Fov / ROI control unit illustrated in FIG. 3. FIG. 7 is a detailed block diagram of the reliability determination unit illustrated in FIG. 3, FIG. 8 is a detailed block diagram of the image matching unit illustrated in FIG. 3, FIG. 9 is a conceptual diagram for explaining the operation of the image matching unit illustrated in FIG. 8, and, FIG. 10 is a flowchart illustrating an AVM image processing method based on vehicle driving conditions according to an embodiment of the present invention. Specific details for implementing the invention

[0076] The above objects, other objects, features, and advantages of the present invention will be easily understood through the following preferred embodiments associated with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided to ensure that the disclosed content is thorough and complete and to ensure that the spirit of the invention is sufficiently conveyed to those skilled in the art.

[0077] In this specification, when a component is described as being on another component, it means that it may be formed directly on the other component or that a third component may be interposed between them.

[0078] Where terms such as "first," "second," etc. are used in this specification to describe components, these components shall not be limited by such terms. These terms are used merely to distinguish one component from another. The embodiments described and illustrated herein also include complementary embodiments.

[0079] Furthermore, when it is stated that the first element (or component) operates or is executed on (ON) the second element (or component), it should be understood that the first element (or component) operates or is executed in the environment where the second element (or component) operates or is executed, or operates or is executed through direct or indirect interaction with the second element (or component).

[0080] Where any element, component, device, or system is described as including a component consisting of a program or software, it should be understood that, even without explicit mention, that element, component, device, or system includes hardware (e.g., memory, CPU, etc.) or other programs or software (e.g., an operating system or drivers required to run the hardware) necessary for the execution or operation of that program or software.

[0081] Furthermore, unless otherwise specified regarding the implementation of any element (or component), it should be understood that the element (or component) may be implemented in software, hardware, or in any form that combines both software and hardware.

[0082] Furthermore, the terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components to the mentioned components.

[0084] FIG. 1 is a block diagram of an AVM image processing system based on vehicle driving conditions according to one embodiment of the present invention.

[0085] Referring to FIG. 1, an AVM image processing system (1000) based on vehicle driving conditions according to one embodiment of the present invention includes multiple cameras (100), an image correction device (200), and an autonomous driving device (300).

[0086] Multiple cameras (100) are installed on the exterior of a vehicle to capture the surrounding environment of the vehicle, and are generally positioned to capture the front, rear, left, and right sides. The multiple cameras (100) may be configured to capture images in a curved shape, and accordingly, each camera may include a wide-angle or fisheye lens. In this case, while wide-angle or fisheye lenses have the advantage of securing a wide field of view around the vehicle, there is a problem that image distortion occurs due to the characteristics of the lens.

[0087] A multi-camera system (100) is mounted on a vehicle with the relative positions and orientations between each camera fixed, and the projection relationship of each camera can be established through initial calibration. Images captured by the multi-camera system (100) can be transmitted to an image correction device (200) and an autonomous driving device (300).

[0088] The image correction device (200) processes a curved image captured by multiple cameras (100) to correct field of view distortion according to the vehicle driving state and performs image matching, thereby generating a top-view image to be provided to the autonomous driving device (300). The image correction device (200) will be described in more detail in FIGS. 3 to 9, which will be described later.

[0089] The autonomous driving device (300) includes one or more processors and memory for recognizing the driving environment, determining the driving situation, and controlling the vehicle, and controls the driving of the vehicle using a top-view image or an image matching result provided from an image correction device (200).

[0090] More specifically, the autonomous driving device (300) controls at least one of steering, acceleration, deceleration, and braking of the vehicle based on the results of recognizing the surrounding environment of the vehicle, thereby enabling the vehicle to move autonomously according to the driving situation. In addition, the autonomous driving device (300) uses an image corrected and aligned by the image correction device (200) that reflects the vehicle driving state, thereby enabling accurate recognition of the surrounding environment despite changes in the vehicle's tilt, rotation, or road surface condition.

[0092] Figure 2 is a block diagram of the autonomous driving device illustrated in Figure 1.

[0093] Referring to FIG. 2, an autonomous driving device (300) according to one embodiment of the present invention includes a perception module (310), a judgment module (320), a vehicle control module (330), and a map-based path module (340).

[0094] The perception module (310) performs object detection, lane recognition, and traffic light recognition by utilizing sensor data from various sensors, such as radar and lidar, including multiple cameras (100). When the perception module (310) operates using a top-view image corrected by an image correction device (200), the perception accuracy can be improved.

[0095] The judgment module (320) analyzes the current driving situation and determines driving behavior using recognition information about the vehicle's surrounding environment provided by the perception module (310), such as object information, lane information, and driving environment information, and a top-view image or image matching result provided by the image correction device (200).

[0096] The vehicle control module (330) generates a control signal to perform an actual driving action of the vehicle based on the driving judgment result provided by the judgment module (320). That is, the vehicle control module (330) outputs a control command to at least one of the vehicle's steering device, acceleration device, and braking device, thereby causing the vehicle to move according to the determined driving action.

[0097] More specifically, the vehicle control module (330) controls the steering angle, acceleration, or braking force in response to driving behaviors such as straight driving, deceleration, stopping, evasive driving, and lane change. Additionally, the vehicle control module (330) can enable control that takes into account the driving stability of the vehicle by referring to the top-view image provided by the image correction device (200) and the recognition result of the recognition module (310).

[0098] The map-based path module (340) sets a target path that the vehicle must drive and generates driving path information by considering the vehicle's current location and road environment. More specifically, the map-based path module (340) can generate path data including lane structure, road boundary, intersection, stop line, and speed limit information using HD map or navigation map information, and can modify and reset by checking whether the map information matches the actual road environment by referring to the top-view image or image matching result provided by the image correction device (200).

[0100] Figure 3 is a block diagram of the image correction device illustrated in Figure 1.

[0101] Referring to FIG. 3, an image correction device (200) according to one embodiment of the present invention includes an image conversion unit (210), an information collection unit (220), a field of view correction unit (230), a Fov / ROI adjustment unit (240), a reliability determination unit (250), an image matching execution unit (260), a top-view image generation unit (270), a storage unit (280), and a control unit (290).

[0102] The image conversion unit (210) converts a curved image captured by multiple cameras (100) into a planar projection image. More specifically, the image conversion unit (210) converts the curved image into a planar image corresponding to a reference coordinate system by reflecting the lens characteristics and shooting geometric information of each camera, and aligns the coordinate system of the image into a normalized form. Accordingly, the image conversion unit (210) minimizes geometric differences between images input from multiple cameras (100) and provides an input image that allows subsequent image processing to be performed stably. The image conversion unit (210) will be described in more detail in FIG. 3, which will be described later.

[0103] The information collection unit (220) collects driving information of the vehicle from the autonomous driving device (300). The vehicle is equipped with various sensors, and the information collection unit (220) can collect driving information from the autonomous driving device or various sensors. The driving information collected by the information collection unit (220) may include yaw, pitch, roll, speed, and steering information.

[0104] The field of view correction unit (230) corrects changes in the field of view that occur according to the vehicle driving state in real time using a planar projection image provided by the image conversion unit (210) and vehicle driving information provided by the information collection unit (220). The field of view correction unit (230) will be described in more detail in FIG. 5, which will be described later.

[0105] The Fov / ROI adjustment unit (240) adjusts the field of view and region of interest according to the vehicle's driving mode based on the field of view corrected by the field of view correction unit (230). The Fov / ROI adjustment unit (240) will be described in more detail in FIG. 6, which will be described later.

[0106] The reliability determination unit (250) determines reliability information for the image adjusted by the Fov / ROI adjustment unit (240). The reliability determination unit (250) will be described in more detail in FIG. 7, which will be described later.

[0107] The image matching unit (260) fuses the reliability information determined by the reliability determination unit (250) with image clues obtained from the image to perform relationship image matching between each camera of the multiple cameras (100). The image matching unit (260) will be described in more detail in FIGS. 8 and 9, which will be described later.

[0108] The top-view image generation unit (270) generates a top-view image using the image matching result from the image matching unit (260). The top-view image refers to an image reconstructed from a viewpoint looking down at the vehicle from above by matching the vehicle surrounding images captured by multiple cameras (100) on a common coordinate system. The top-view image is generated as a result of reflecting the field of view correction according to the vehicle's driving state and the matching between each camera of the multiple cameras (100) through the image conversion unit (210), the field of view correction unit (230), and the image matching unit (260), and provides the vehicle surrounding environment as a single continuous planar image. In particular, the top-view image according to this embodiment includes real-time field of view correction results reflecting the vehicle's yaw, pitch, roll, speed, and steering information, so even if the vehicle tilts or rotates while driving, the image distortion or matching error can be maintained in a minimized state.

[0109] The top-view image generation unit (270) can additionally perform image blending or weight-based synthesis processing to minimize visual seams that may occur in the boundary area between cameras.

[0110] The storage unit (280) stores all information necessary for the operation of the image correction device (200). For example, the storage unit (280) can store initial calibration information for multiple cameras (100), initial projection matrices of each camera, various parameters, weights, and setting information used in the image conversion unit (210), field of view correction unit, image matching unit (260), etc.

[0111] The control unit (290) controls the overall operation of the image correction device (200). That is, the control unit (290) controls the signal input and output between the image conversion unit (210), the information collection unit (220), the field of view correction unit (230), the Fov / ROI adjustment unit (240), the reliability determination unit (250), the image matching execution unit (260), the top-view image generation unit (270), and the storage unit (280).

[0113] Figure 4 is a detailed block diagram of the image conversion unit illustrated in Figure 3.

[0114] Referring to FIG. 4, the image conversion unit (210) constituting the image correction device (200) includes an image normalization unit (211), a distortion removal unit (213), a coordinate system alignment unit (215), and an image standardization unit (217).

[0115] The image normalization unit (211) converts the curved image into a planar projection image and normalizes it. More specifically, the image normalization unit (211) receives a curved image captured by multiple cameras (100) and performs an operation to standardize the format and criteria of the image so that the subsequent image processing process can be performed stably.

[0116] The image normalization unit (211) minimizes the difference between images input from multiple cameras (100) by adjusting the resolution, scale, pixel density, or image size of the input image according to a preset standard. Additionally, the image normalization unit (211) normalizes the brightness, contrast, and color distribution of the image to mitigate image quality deviations caused by changes in illumination or differences in sensitivity between cameras.

[0117] The distortion removal unit (213) receives an image normalized by the image normalization unit (211) and removes geometric distortion caused by camera lens characteristics. More specifically, the distortion removal unit (213) corrects radial distortion and tangential distortion caused by a wide-angle lens or a fisheye lens using preset camera parameters or calibration information. Through this, the distortion removal unit (213) transforms the linear structure of the image, which appeared curved, to correspond to the actual spatial structure.

[0118] The coordinate system alignment unit (215) aligns the image, from which distortion has been removed by the distortion removal unit (213), to a reference coordinate system. That is, the coordinate system alignment unit (215) corrects the coordinate differences according to the installation position and direction of each camera so that the images input from the multiple cameras (100) are aligned to be expressed in a common reference coordinate system.

[0119] More specifically, the coordinate system alignment unit (215) adjusts the coordinate axes, origin, and direction of each image to match the reference coordinate system using external parameters and initial calibration information for each camera. By doing so, the coordinate system alignment unit (215) can prevent alignment errors caused by coordinate mismatch between cameras from occurring during subsequent field of view correction, image alignment, and top-view image generation processes.

[0120] The image standardization unit (217) receives an image aligned to a reference coordinate system by the coordinate system alignment unit (215) and standardizes the quality of the image so that subsequent image processing and image matching can be performed stably.

[0121] More specifically, the image standardization unit (217) adjusts the resolution, pixel arrangement, color space, and data format of the image to match a preset standard specification, thereby maintaining the same processing conditions between images from multiple cameras (100). In addition, the image standardization unit (217) corrects the brightness range, contrast, and color distribution of the image to mitigate image quality deviations caused by differences in lighting conditions or camera sensitivity.

[0123] Figure 5 is a detailed block diagram of the field of view correction unit illustrated in Figure 3.

[0124] Referring to FIG. 5, the field of view correction unit (230) constituting the image correction device (200) includes a projection matrix update unit (231) and a ground coordinate system matching unit (233).

[0125] The projection matrix update unit (231) updates the projection matrix (Homography) corresponding to each of the multiple cameras (100) in real time using the vehicle driving information collected by the information collection unit (220).

[0126] More specifically, the projection matrix update unit (231) updates the projection matrix of each camera over time by reflecting the yaw, pitch, and roll changes of the vehicle and the fine positional movement according to the vehicle state change, based on the basic projection matrix set through initial calibration. Accordingly, the projection matrix update unit (231) ensures that the field of view of the multi-camera (100) image corresponds consistently with the actual ground despite tilt, rotation, and attitude changes occurring while driving the vehicle.

[0127] A radius (r) of one pixel in a curved image acquired from multiple cameras (100) d The relationship between ) and angle (θ) can be expressed as a generalized model. This is shown in Equation 1.

[0128] [Mathematical Formula 1]

[0129]

[0130] The projection matrix update unit (231) uses the coefficients (f, k1, k2, …) estimated during the calibration process and the lookup table to update each pixel (u d , v d ) distortion-removed coordinates (u u' , v u It is converted to ). The conversion relationship at this time is expressed by mathematical formula 2.

[0131] [Mathematical Formula 2]

[0132]

[0133] In mathematical equation 2, K u is a new intrinsic parameter matrix of the multi-camera (100) updated by the projection matrix update unit (231).

[0134] For each camera i, a point on the ground plane and image coordinates Between them is the projection matrix H i (t) exists and can be defined by mathematical equation 3.

[0135] [Mathematical Formula 3]

[0136]

[0137] In conventional autonomous driving methods, a fixed H i (t) was applied. However, the projection matrix update unit (231) H i (t) is not a fixed value, but the vehicle state vector s i It is defined as a function of (t). Through this, the projection matrix update unit (231) determines the projection matrix by mathematical formula 4.

[0138] [Mathematical Formula 4]

[0139]

[0140] In mathematical equation 4, H i (t) is the projection matrix of the i-th camera, H i0 is the initial fixed projection matrix, s(t) is the change in vehicle state at time t, is a rotation correction matrix determined by s(t), is a fine position shift correction matrix caused by changes in vehicle state.

[0141] The projection matrix update unit (231) can reflect changes in vehicle attitude measured by the IMU. It is represented in mathematical formula 5.

[0142] [Mathematical Formula 5]

[0143]

[0144] As defined in mathematical formula 5, it corrects for forward / backward / left / right rotation according to yaw (Δψ), pitch (θ), and roll (φ), and corrects so that the ground appears flat even when the vehicle is tilted or rotated.

[0145] The ground coordinate system matching unit (233) performs matching between each camera image and the ground coordinate system using the projection matrix updated by the projection matrix update unit (231). More specifically, the ground coordinate system matching unit (233) converts the pixel coordinates of each camera image into ground reference coordinates so that the images of the multiple cameras (100) are aligned to a common ground coordinate system. Through this, the ground coordinate system matching unit (233) can minimize geometric discrepancies that may occur during the image matching process between cameras.

[0147] FIG. 6 is a conceptual diagram for explaining the operation of the Fov / ROI control unit illustrated in FIG. 3.

[0148] The Fov / ROI adjustment unit (240) dynamically adjusts the field of view and the region of interest based on the image corrected by the field of view correction unit (230) according to the vehicle's driving condition, driving speed, or driving mode. As illustrated, the Fov / ROI adjustment unit (240) can control the size and position of the field of view and the region of interest to selectively emphasize at least one of the near, medium, and far regions relative to the front of the vehicle.

[0149] (a) is an example corresponding to a case where the vehicle is parked or driving at a low speed. With the near-field area in front of the vehicle set as the region of interest (near-field ROI), the Fov / ROI control unit (240) emphasizes the near-field area within a relatively wide field of view (FoV) so as to accurately recognize the nearby environment, such as obstacles, pedestrians, or parking spaces around the vehicle.

[0150] (b) is an example corresponding to the case where the vehicle is traveling at a normal driving speed. With the mid-range area in front of the vehicle set as the region of interest (mid-range ROI), the Fov / ROI control unit (240) improves driving stability by setting the region of interest centered on the mid-range area, which is important for determining the driving path, such as lane keeping and the main vehicle ahead.

[0151] (c) is an example corresponding to a case where the vehicle is traveling at high speed or there is no need to predict the situation ahead in advance. With the distant area in front of the vehicle set as the region of interest (distant ROI), the Fov / ROI control unit (240) sets the ROI to intensively include the distant area within a relatively narrow field of view, thereby enabling preemptive judgment regarding changes in the road conditions ahead.

[0152] The Fov / ROI control unit (240) can define the top-view projection area of ​​each camera according to the vehicle speed (v) and driving mode (parking / low speed / high speed) using Equations 6 and 7.

[0153] Mathematical formula 6 is used to adjust the forward FoV.

[0154] [Mathematical Formula 6]

[0155]

[0156] As the vehicle speed increases, the forward FoV (FoV) front Expand (t)) and secure a long-distance ROI in preparation for high-speed driving.

[0157] Mathematical formula 7 is used to adjust the rear FoV.

[0158] [Mathematical Formula 7]

[0159]

[0160] As the vehicle speed increases, the rear FoV (FoV) rear (t)) is reduced so that unnecessary operations are reduced.

[0161] For example, in parking mode, a wide FoV is secured for a range of [-5m, +5m] around the vehicle, and in high-speed driving mode, the ROI is configured by focusing on a range of [0m, +60m] ahead.

[0162] Since the FoV and region of interest are controlled by the Fov / ROI control unit (240), subsequent SR (Super-Resolution) network and multi-cue fusion calculations can be performed only on pixels within the region of interest, thereby significantly reducing the amount of computation.

[0164] Figure 7 is a detailed block diagram of the reliability determination unit illustrated in Figure 3.

[0165] Referring to FIG. 7, the reliability determination unit (250) constituting the image correction device (200) includes a resolution adjustment unit (251), an uncertainty calculation unit (253), and a reliability calculation unit (255).

[0166] The resolution adjustment unit (251) improves the resolution of an image using a Super-Resolution (SR) network. The object whose resolution is adjusted by the resolution adjustment unit (251) is an image within the region of interest.

[0167] An SR network is a deep learning-based image restoration model for restoring high-resolution images from low-resolution input images. An SR network may include a Convolutional Neural Network (CNN), a Residual Network, or a neural network structure capable of probabilistic inference, and can be trained to restore detailed structures important for image registration, such as lanes, object contours, and road boundaries.

[0168] The resolution adjustment unit (251) restores spatial information contained in the low-resolution image and converts it into a high-resolution image, thereby allowing detailed features important for image matching, such as lanes, object outlines, and road boundaries, to be expressed more clearly.

[0169] In addition, the resolution adjustment unit (251) can effectively improve the image quality of the region important for alignment accuracy by reducing the amount of computation by performing SR (Super-Resolution) processing only on the region of interest rather than the entire image.

[0170] The resolution adjustment unit (251) is an input image patch I for the region of interest LR From SR network F SR High-resolution package I through SR It can generate. This is shown in mathematical formula 8.

[0171] [Mathematical Formula 8]

[0172]

[0173] The uncertainty calculation unit (253) calculates the uncertainty regarding areas where quality degradation is possible in an image whose resolution has been enhanced by the resolution adjustment unit (251). More specifically, the uncertainty calculation unit (253) receives multiple high-resolution output images obtained by repeatedly running an SR network on the same input image multiple times, and analyzes the output variability at the pixel level. That is, the uncertainty calculation unit (253) quantitatively evaluates how stable the restoration result is at a given pixel by calculating the variance between the output images of the repeatedly run SR network. At this time, pixels with large fluctuations in output values ​​are judged to have high uncertainty, and pixels with small fluctuations in output values ​​are judged to have low uncertainty.

[0174] The uncertainty calculation unit (253) may convert the uncertainty value into a supplementary indicator of the reliability value or define it as the reciprocal or supplementary value of the reliability value according to the relationship with the reliability value calculated by the reliability calculation unit (255).

[0175] The uncertainty calculation unit (253) can calculate the uncertainty using mathematical formula 9.

[0176] [Mathematical Formula 9]

[0177]

[0178] In Equation 9, U(u,v) is the uncertainty value at coordinates (u,v), N is the number of times the resolution enhancement operation is performed, is the (u,v) pixel value of the n-th output image after running the SR network N times, is the average pixel value for the N above SR network operation results.

[0179] The uncertainty calculation unit (253) plays a role in improving the stability of image alignment and the reliability of the final top-view image by identifying in advance areas where image quality may be degraded.

[0180] The reliability calculation unit (255) calculates a reliability at the pixel level by combining the uncertainty information calculated by the uncertainty calculation unit (253) and a plurality of image clues obtained from the image. That is, the reliability calculation unit (255) receives image clues, such as edge strength, texture strength, or feature point distribution, and a reliability value that is predicted or set in advance for each pixel in the image as input, and quantitatively calculates the reliability of the corresponding pixel.

[0181] The reliability calculation unit (255) can calculate the reliability w(u,v) at pixel (u,v) using the sigmoid function by Equation 10.

[0182] [Mathematical Formula 10]

[0183]

[0184] In mathematical formula 10, is a sigmoid function, edge(u,v) is the edge strength of pixel (u,v), c(u,v) is a preset confidence prediction value for the pixel, is the weight set for c(u,v) and edge(u,v), respectively.

[0185] The reliability value calculated by the reliability calculation unit (255) may be used to supplement the uncertainty value in the uncertainty calculation unit (253). That is, the uncertainty calculation unit (253) can calculate the uncertainty using mathematical formula 11.

[0186] [Mathematical Formula 11]

[0187]

[0188] The reliability value calculated by the reliability calculation unit (255) can be used as a weight in the multi-queue fusion process of the image matching unit (260) so that pixels or regions with high reliability are preferentially reflected in the calculation of lane continuity, object matching, and map matching.

[0190] Figure 8 is a detailed block diagram of the image matching unit illustrated in Figure 3.

[0191] Referring to FIG. 8, the image matching unit (260) constituting the image correction device (200) includes a lane continuity calculation unit (261), an object matching calculation unit (263), a map matching calculation unit (265), and a transformation matrix selection unit (267).

[0192] The lane continuity calculation unit (261) calculates a lane continuity score to evaluate whether the lanes are spatially continuous in the image transformed by the candidate transformation matrix. The lane continuity score can be calculated by Equation 12.

[0193] [Mathematical Formula 12]

[0194]

[0195] In mathematical equation 12, S lane (H k ) is the candidate transformation matrix H kLane continuity score for, |L| is the number of reference lane point sets, is H k This is the predicted position of the lane projected onto the top-view image using .

[0196] The object alignment calculation unit (263) calculates an object alignment score to evaluate whether the same object observed simultaneously by multiple cameras (100) is aligned to a matching position when transformed into a top-view coordinate system by a candidate transformation matrix.

[0197] The object matching calculation unit (263) calculates the object position in each camera for the same object detected in the front, rear left, and right cameras using a candidate transformation matrix H k Projected into the top-view coordinate system using . At this time, the object alignment calculation unit (263) calculates the top-view coordinates of the same object o observed by cameras i and j. and Calculate the distance error between the two coordinates by comparing them.

[0198] The object consistency score can be calculated by averaging the distance errors of simultaneously observed objects and normalizing them into an exponential function. This is shown in Equation 13.

[0199] [Mathematical Formula 13]

[0200]

[0201] In mathematical equation 13, S obj (H k ) is the candidate transformation matrix H k Object consistency score for, N o is the total number of objects simultaneously observed by the above multiple cameras, and H is the position of object o seen by cameras i and j k These are the top-view coordinates when converted.

[0202] The map matching score (265) calculates a map matching score to evaluate how well the image transformed by the candidate transformation matrix matches the structural information of the pre-constructed HD map. To this end, the map matching score (265) compares the structural information on the map, such as road boundaries, lanes, crosswalks, stop lines, and intersection structures included in the HD map, with the structural information on the top-view image generated by applying the candidate transformation matrix.

[0203] More specifically, the map matching calculation unit (265) evaluates the matching quality of the candidate transformation matrix by calculating the intersection over union (IoU) or distance-based error index between the road structure detected in the top-view image and the corresponding structure on the HD map.

[0204] For example, it is determined that the higher the IoU value between the lane area detected in the top-view image and the lane area on the HD map, the higher the map match, and the smaller the distance error between structures, the higher the map matching accuracy.

[0205] The transformation matrix selection unit (267) selects a final transformation matrix to be used for image matching of multiple cameras (100) by combining lane continuity scores, object matching scores and map matching scores.

[0206] The transformation matrix selection unit (267) calculates the total matching score by applying each weight to each score according to mathematical formula 14.

[0207] [Mathematical Formula 14]

[0208]

[0209] In mathematical formula 14, is the overall matching score, is a weight representing the relative importance of each lane, object, and map clue.

[0210] Afterwards, the transformation matrix selection unit (267) determines the final transformation matrix by mathematical formula 15.

[0211] [Mathematical Formula 15]

[0212]

[0213] In mathematical equation 15, H * is the above final transformation matrix, H k is the transformation matrix of the k-th candidate, S total (H k ) is H k This is the overall matching score for.

[0214] The final transformation matrix H that is finally determined in the transformation matrix selection unit (267). * The image is then provided to the top-view image generation unit (270) and used to ensure that the images of the multiple cameras (100) are accurately aligned in a common coordinate system.

[0215] As a result, by operating the transformation matrix selection unit (267), the stability and reliability of image matching can be improved by comprehensively considering multiple matching clues without relying on a single image clue.

[0217] FIG. 9 is a conceptual diagram for explaining the operation of the image matching unit illustrated in FIG. 8.

[0218] In this embodiment, a multi-cue-based image matching structure is illustrated, which calculates lane, object, and map-based matching metrics for a plurality of candidate transformation matrices and selects a final transformation matrix by weighted combination thereof.

[0219] A set of multiple candidate transformation matrices generated during the field of view correction and image transformation process {H k} is provided as input. Each candidate transformation matrix H k represents different projection relationships or coherence hypotheses.

[0220] Each candidate transformation matrix H k Regarding this, the image matching unit (260) calculates a plurality of matching metrics. The plurality of matching metrics are lane continuity metrics (S lane ), object consistency metric (Sobj ), and map match metric(S map )am.

[0221] The lane continuity metric is an indicator that evaluates the degree to which a lane projected by a candidate transformation matrix is ​​spatially continuous with a reference lane, reflecting the geometric consistency of the lane structure.

[0222] The object alignment metric is an indicator that evaluates the degree to which the same object observed in multiple cameras (100) is aligned to a position that matches each other in a top view, and reflects the consistency of the object position between cameras.

[0223] The map match metric is an indicator that evaluates how well the top-view image generated by the candidate transformation matrix matches the structural information of the HD map, reflecting map-based spatial fitting accuracy.

[0224] The lane continuity metric, object alignment metric, and map match metric calculated in this way are subjected to respective weights, and then the weighted sum-based comprehensive alignment score S total It is combined as.

[0225] Finally, H * The selection block is transmitted to the top-view image generation unit (270) and used to ensure that the images of the multiple cameras (100) are accurately aligned in a common coordinate system.

[0227] FIG. 10 is a flowchart illustrating an AVM image processing method based on vehicle driving conditions according to an embodiment of the present invention.

[0228] Hereinafter, with reference to FIGS. 1 to 10, an AVM image processing method based on a vehicle driving state according to an embodiment of the present invention is described.

[0229] A curved image of the surrounding environment of the vehicle is captured using multiple cameras (100) positioned at the front, rear, left, and right of the vehicle (S510). Each camera of the multiple cameras (100) includes a wide-angle or fisheye lens to secure a wide field of view around the vehicle. At this time, the image captured by the multiple cameras (100) includes lens distortion and coordinate distortion.

[0230] The image conversion unit (210) converts a curved image captured by multiple cameras (100) into a planar projection image (S520). Step S520 includes image normalization, distortion removal, coordinate system alignment, and image standardization.

[0231] The information collection unit (220) collects driving information of the vehicle from a sensor mounted on the autonomous driving device (300) or the vehicle (S530). Here, the driving information of the vehicle may include yaw, pitch, roll, speed, and steering information.

[0232] The field of view correction unit (230) corrects changes in the field of view in real time due to tilt, rotation, and position changes occurring while the vehicle is driving, using driving information and a planar projection image (S540). To do this, the field of view correction unit (230) updates the projection matrix of each camera and performs alignment with the ground coordinate system.

[0233] The Fov / ROI adjustment unit (240) dynamically adjusts the viewing angle and region of interest according to the vehicle's driving mode or speed based on the viewing angle corrected by the viewing correction unit (230) (S550). Through this, image processing is performed centered on the area important to the driving situation.

[0234] When the field of view and region of interest are adjusted by the Fov / ROI adjustment unit (240), the reliability determination unit (250) improves the resolution of the image within the region of interest using an SR network, calculates the uncertainty in pixel units based on the iterative inference results, and determines reliability information by combining the uncertainty and image clues (S560).

[0235] When reliability information is determined by the reliability determination unit (250), the image matching unit (260) evaluates lane continuity, object alignment, and map alignment using the reliability information and image clues, and performs image matching between multiple cameras (100) by selecting a final transformation matrix through weighted combination (S570).

[0236] The top-view image generation unit (270) generates a top-view image in the form of a vehicle viewed from above using the image matching result of the image matching unit (260) (S580). The top-view image generated by the top-view image generation unit (270) is provided to the autonomous driving device (300) or the driver.

[0237] As described above, the AVM image processing method based on vehicle driving status according to the present invention resolves all problems of field of view distortion, alignment error, and computational inefficiency associated with existing AVM systems by reflecting the vehicle's driving status in real time during the image processing process.

[0239] The AVM image processing method based on vehicle driving conditions according to the present invention can be implemented as a program to realize the same. In addition, such a program may include programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, assembly language, machine language, etc.

[0240] In addition, a program for realizing an AVM image processing method based on vehicle driving conditions may be stored on a computer-readable recording medium. Here, any device that is in a form that can be read by a computer and can store data, and whose whole or part contains instructions that can be executed by a computer, may be applied. For example, a computer-readable recording medium may include CD-ROM, RAM, ROM, flash memory, magnetic disk, optical disk, hard disk drive, and solid state disk.

[0242] Those skilled in the art to which the present invention pertains will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0244] 1000: AVM image processing system based on vehicle driving status 100: Multiple cameras 200: Image correction device 300: Autonomous driving device 210: Image converter 220: Information Gathering Unit 230: View Correction Unit 240: FoV / ROI Adjustment Unit 250: Reliability Determination Unit 260: Image registration unit 270: Top-view image generation unit 280: Storage unit 290: Control unit

Claims

Claim 1 A multi-camera system for capturing a curved image; an image correction device for correcting the captured image; and an autonomous driving device for controlling the autonomous driving of a vehicle using the corrected image; wherein the image correction device comprises: an image conversion unit for converting the curved image into a planar projection image; an information collection unit for collecting driving information of the vehicle from the autonomous driving device; a field of view correction unit for correcting changes in the field of view in real time using the converted planar projection image and the collected driving information; a FoV / ROI adjustment unit for adjusting the field of view angle and region of interest according to the driving mode of the vehicle based on the corrected field of view; a reliability determination unit for determining reliability information for the adjusted image; and an image matching execution unit for performing relational image matching between the multi-cameras by fusing the determined reliability information and image clues obtained from the image. AVM image processing system based on vehicle driving state, comprising: a top-view image generation unit that generates a top-view image using the image matching result; wherein the reliability determination unit comprises: a resolution adjustment unit that improves the resolution of the image using a Super-Resolution (SR) network; an uncertainty calculation unit that calculates the uncertainty regarding areas where quality degradation is possible in the image with improved resolution; and a reliability calculation unit that calculates the reliability of the image. Claim 2 The AVM image processing system based on vehicle driving state according to claim 1, wherein the image conversion unit comprises: an image normalization unit that converts the curved image into a planar projection image and normalizes it; a distortion removal unit that removes distortion from the normalized image; a coordinate system alignment unit that aligns the distortion-removed image to a reference coordinate system; and an image standardization unit that standardizes the quality of the image aligned to the reference coordinate system. Claim 3 An AVM image processing system based on vehicle driving state according to claim 1, characterized in that the driving information of the vehicle includes yaw, pitch, roll, speed, and steering information. Claim 4 An AVM image processing system based on vehicle driving state according to claim 1, wherein the field of view correction unit comprises: a projection matrix update unit that updates the projection matrix of the multi-camera image using the driving information; and a ground coordinate system matching unit that performs matching with the ground coordinate system using the updated projection matrix. Claim 5 In claim 4, the projection matrix update unit is characterized by determining the projection matrix by the following formula, in an AVM image processing system based on vehicle driving state: Here, H i (t) is the projection matrix of the i-th camera, H i0 is the initial fixed projection matrix, s(t) is the change in vehicle state at time t, is a rotation correction matrix determined by s(t), is a fine position shift correction matrix caused by changes in vehicle state. Claim 6 delete Claim 7 AVM image processing system based on vehicle driving state according to claim 1, wherein the uncertainty calculation unit calculates uncertainty by the following formula: Here, U(u,v) is the uncertainty value at coordinates (u,v), N is the number of times the resolution enhancement operation is performed, is the (u,v) pixel value of the n-th output image after running the above SR network N times, is the average pixel value for the N above SR network operation results. Claim 8 AVM image processing system based on vehicle driving state according to claim 1, wherein the reliability calculation unit calculates the reliability by the following formula: Here, w(u,v) is the confidence value at coordinates (u,v), is a sigmoid function, edge(u,v) is the edge strength of pixel (u,v), c(u,v) is a preset confidence prediction value for the pixel, is the weight set for c(u,v) and edge(u,v), respectively. Claim 9 In claim 8, the above uncertainty calculation unit is characterized by calculating the uncertainty using the following formula: AVM image processing system based on vehicle driving state Here, U(u,v) is the uncertainty value of pixel (u,v), and w(u,v) is the confidence value at pixel (u,v). Claim 10 An AVM image processing system based on vehicle driving state according to claim 1, wherein the image matching unit comprises: a lane continuity calculation unit for calculating a lane continuity score; an object matching calculation unit for calculating an object matching score; a map matching calculation unit for calculating a map matching score; and a transformation matrix selection unit for selecting a final transformation matrix by combining the lane continuity score, the object matching score, and the map matching score. Claim 11 In claim 10, the lane continuity calculation unit is characterized by calculating the lane continuity score using the following formula in an AVM image processing system based on vehicle driving conditions: Here, S lane (H k ) is the candidate transformation matrix H k Lane continuity score for, |L| is the number of reference lane point sets, is H k This is the predicted position of the lane projected onto the top-view image using . Claim 12 In claim 10, the object matching calculation unit is characterized by calculating the object matching score using the following formula in an AVM image processing system based on vehicle driving state: Here, S obj (H k ) is the candidate transformation matrix H k Object consistency score for, N o is the total number of objects simultaneously observed by the above multiple cameras, and H is the position of object o seen by cameras i and j k These are the top-view coordinates when converted. Claim 13 AVM image processing system based on vehicle driving state according to claim 10, wherein the map matching score calculation unit determines the map matching score using IoU and distance-based indicators between a structure in an HD map and a structure in an image. Claim 14 AVM image processing system based on vehicle driving state according to claim 10, wherein the transformation matrix selection unit determines the final transformation matrix by the following formula: Here, H * is the above final transformation matrix, H k is the transformation matrix of the k-th candidate, S total (H k ) is H k This is the overall matching score for. Claim 15 A method for processing AVM images based on vehicle driving state applied to an image correction device that receives images from multiple cameras capturing images of a curved shape, comprising: a step of converting the curved shape images captured by the multiple cameras into planar projection images; a step of collecting driving information of the vehicle; a step of correcting changes in the field of view in real time using the converted planar projection images and the collected driving information; a step of adjusting the field of view angle and region of interest according to the driving mode of the vehicle based on the corrected field of view; a step of determining reliability information for the adjusted images; a step of performing relational image matching between the multiple cameras by fusing the determined reliability information and image clues obtained from the images; and a step of generating a top-view image using the image matching results; wherein the determining step comprises: a step of improving the resolution of the images using an SR (Super-Resolution) network; a step of calculating uncertainty regarding areas with potential quality degradation in the resolution-improved images; and a step of calculating reliability for the images. Claim 16 AVM image processing method based on vehicle driving state according to claim 15, wherein the transforming step comprises: a step of normalizing the curved image by converting it into a planar projected image; a step of removing distortion from the normalized image; a step of aligning the distortion-removed image to a reference coordinate system; and a step of standardizing the quality of the image aligned to the reference coordinate system. Claim 17 AVM image processing method based on vehicle driving state according to claim 15, characterized in that the driving information of the vehicle includes yaw, pitch, roll, speed, and steering information. Claim 18 AVM image processing method based on vehicle driving state according to claim 15, wherein the correction step comprises: a step of updating the projection matrix of the multi-camera image using the driving information; and a step of performing alignment with the ground coordinate system using the updated projection matrix. Claim 19 AVM image processing method based on vehicle driving state according to claim 18, wherein the updating step is characterized by determining the projection matrix by the following formula: Here, H i (t) is the projection matrix of the i-th camera, H i0 is the initial fixed projection matrix, s(t) is the change in vehicle state at time t, is a rotation correction matrix determined by s(t), is a fine position shift correction matrix caused by changes in vehicle state. Claim 20 delete Claim 21 AVM image processing method based on vehicle driving state according to claim 15, wherein the step of calculating the uncertainty above is characterized by calculating the uncertainty by the following formula: Here, U(u,v) is the uncertainty value at coordinates (u,v), N is the number of times the resolution enhancement operation is performed, is the (u,v) pixel value of the n-th output image after running the above SR network N times, is the average pixel value for the N above SR network operation results. Claim 22 AVM image processing method based on vehicle driving state according to claim 15, wherein the step of calculating the reliability is characterized by calculating the reliability by the following formula: Here, w(u,v) is the confidence value at coordinates (u,v), is a sigmoid function, edge(u,v) is the edge strength of pixel (u,v), c(u,v) is a preset confidence prediction value for the pixel, is the weight set for c(u,v) and edge(u,v), respectively. Claim 23 AVM image processing method based on vehicle driving state according to claim 22, wherein the step of calculating the uncertainty is characterized by calculating the uncertainty by the following formula: Here, U(u,v) is the uncertainty value of pixel (u,v), and w(u,v) is the confidence value at pixel (u,v). Claim 24 AVM image processing method based on vehicle driving state according to claim 15, wherein the steps performed include: a step of calculating a lane continuity score; a step of calculating an object alignment score; a step of calculating a map alignment score; and a step of selecting a final transformation matrix by combining the lane continuity score, the object alignment score, and the map alignment score. Claim 25 AVM image processing method based on vehicle driving state according to claim 24, wherein the step of calculating the lane continuity score is characterized by calculating the lane continuity score by the following formula: Here, S lane (H k ) is the candidate transformation matrix H k Lane continuity score for, |L| is the number of reference lane point sets, is H k This is the predicted position of the lane projected onto the top-view image using . Claim 26 In claim 24, the step of calculating the object alignment score is characterized by calculating the object alignment score using the following formula: AVM image processing method based on vehicle driving state Here, S obj (H k ) is the candidate transformation matrix H k Object consistency score for, N o is the total number of objects simultaneously observed by the above multiple cameras, and H is the position of object o seen by cameras i and j k These are the top-view coordinates when converted. Claim 27 AVM image processing method based on vehicle driving state according to claim 24, wherein the step of calculating the map match score is characterized by determining the map match score using IoU and distance-based indicators between a structure in an HD map and a structure in an image. Claim 28 AVM image processing method based on vehicle driving state according to claim 24, wherein the step of selecting the final transformation matrix is ​​characterized by determining the final transformation matrix by the following formula: Here, H * is the above final transformation matrix, H k is the transformation matrix of the k-th candidate, S total (H k ) is H k This is the overall matching score for. Claim 29 A computer-readable recording medium having a program for realizing an AVM image processing method based on a vehicle driving state according to any one of paragraphs 15 through 19 and 21 through 28.

Images