Autonomous Driving Control Apparatus and Method Using Sensor Reliability Commonly for Contribution Adjustment in a Reasoning Layer and Safety Constraint Variation in a Safety Layer

Abstract

The present invention relates to an autonomous driving control device and method, comprising: a sensor interface unit that acquires sensing data from a plurality of sensors; a reliability determination unit that calculates a performance degradation index of each sensor from the sensing data and determines reliability information for each sensor; a context generation unit that converts the reliability information into context data that can be processed by a language-based inference model; an inference unit that receives the sensing data and the context data as inputs and outputs driving behavior candidates while adjusting the contribution of information derived from each sensor differently; and a safety arbitration unit that receives the reliability information in parallel with the inference unit, varies safety constraints, compares the driving behavior candidates with the varied safety constraints, and transmits them to a vehicle control unit only when they satisfy the conditions. Since a single reliability information simultaneously determines the contribution of information in the inference layer and the strictness of judgment criteria in the safety layer, the degradation of the quality of perception information is consistently reflected in the content of the judgment and its verification criteria, thereby reducing the risk of misjudgment based on degraded information passing through the safety layer.

Description

Technology Field

[0001] The present invention relates to autonomous driving control technology, and more specifically, to an autonomous driving control device and method that provides reliability information, which quantifies the degree of performance degradation of a plurality of sensors mounted on a vehicle, in parallel to a language-based inference layer and a safety verification layer, so that the reliability information determines the contribution of information derived from each sensor in the inference layer and determines the strictness of safety constraints in the safety verification layer. Background Technology

[0002] Autonomous vehicles collect information about the surrounding environment from multiple sensors, such as cameras, radar, and lidar, and fuse this data to recognize drivable areas and surrounding objects, after which they determine a driving path. Among these, cameras play a pivotal role in the perception system because they provide semantic information that is difficult for other sensors to acquire, such as color, text, traffic light status, and pedestrian posture and intent.

[0003] Automotive cameras are constantly exposed to a wide range of temperature variations, from -40°C to 105°C, as well as vibration, ultraviolet rays, and debris flying off the road surface. These environmental stresses cause out-of-focus due to creep deformation of the optical adhesive that secures the lens and image sensor, yellowing and micro-scratches on the lens surface, accumulation of defective pixels, and drift of external calibration parameters. As a result, the camera enters a state of so-called silent degradation, in which the spatial frequency response and contrast of the image gradually deteriorate while the camera continues to operate electrically normally.

[0004] In conventional autonomous driving systems, sensor status monitoring was generally performed binaryly. That is, when a clear failure event—such as failure to receive a frame, failure of data integrity checks, or disconnection of the communication link—was detected, the corresponding sensor was invalidated and switched to a predefined compressed mode. However, latent degradation does not satisfy these binary judgment criteria. Since a degraded camera still transmits valid frames at a normal cycle, the system treats it as a healthy sensor; consequently, driving continues without any warning even if the recognition distance of distant objects is shortened or traffic light colors are misjudged.

[0005] Furthermore, while there have been conventional attempts to incorporate sensor quality indicators into perception algorithms, these have generally been limited to numerical weighting adjustments during the fusion stage, such as the adjustment of observation noise covariance in Kalman filters or the weighted average of detection results. This approach merely adjusts the precision of location estimation based on the premise that a detected object exists, and it has a fundamental limitation in that it fails to address false negatives caused by degraded sensors failing to detect the object itself.

[0006] Recently, there has been active research aimed at applying large-scale language models and visual-language-behavior models to the situation interpretation and behavior decision layers of autonomous driving. These models demonstrate superior judgment capabilities in long-tail scenarios that are difficult to describe by rules, such as interpreting temporary signs in construction zones, inferring the intent of a signaling operator, and determining whether to avoid objects based on the type of debris on the road. However, conventional approaches are built upon the implicit premise that all input sensing data is equally reliable. Since inference models lack the means to determine whether the input image comes from an out-of-focus camera or was acquired from a locally contaminated field of view, they interpret the situation by adopting inaccurate information provided by degraded sensors with the same certainty as sound information.

[0007] Furthermore, language-based inference models produce probabilistic outputs that lack guaranteed deterministic reproducibility, and inference delays often exceed several hundred milliseconds. Consequently, configurations that directly link the outputs of such models to vehicle steering and braking struggle to meet the deterministic response times and verifiability requirements of functional safety standards. Therefore, a configuration in which the outputs of the inference models are verified by a separate safety layer before being incorporated into vehicle control may be considered.

[0008] However, simply arranging the inference layer and the safety layer in series as described above is not sufficient. Conventional safety layers possess fixed constraints, such as minimum safety distances and maximum acceleration, and determine only whether a received behavior candidate violates these constraints. In this case, the safety layer does not know the quality of the information from which the behavior candidate was derived, and it evaluates candidates derived from a healthy sensor and those derived from inaccurate information from a degraded sensor using the same criteria. Consequently, since the safety margin is not expanded even when the reliability of the perceived information is reduced, a risk remains that a misjudgment by the inference layer may pass through the safety layer.

[0009] Ultimately, in conventional technology, the information weighting of the inference layer and the judgment criteria of the safety layer were determined independently of each other, and a configuration in which a single indicator representing the actual state of the sensor governs the operation of both layers together was not presented. Prior art literature

[0010] Republic of Korea Published Patent Application No. 10-2020-0000000 United States Published Patent Application No. 2021 / 0000000 The problem to be solved

[0011] The present invention aims to solve the problems of the prior art as described above by providing an autonomous driving control device and method that quantifies not only clear sensor failure but also gradual and partial performance degradation into continuous reliability values.

[0012] In addition, the present invention aims to provide an autonomous driving control device and method in which a single reliability information determines the contribution of each sensor-derived information in a language-based inference layer and determines the strictness of safety constraints in a safety verification layer, so that the degradation of the quality of cognitive information is simultaneously reflected in the content of the judgment and the criteria for the judgment.

[0013] Furthermore, the present invention aims to provide an autonomous driving control device and method that does not cause all-around performance degradation due to localized degradation by assigning different reliability levels to different partial regions of the field of view even within a single sensor, and selectively varying safety constraint conditions only for the orientation corresponding to the partial region with reduced reliability.

[0014] Furthermore, the present invention aims to provide an autonomous driving control device and method that identify common cause degradation in which a plurality of sensors degrade simultaneously due to the same external factor, and in this case, vary safety constraint conditions by an expanded range compared to the degradation of individual sensors, thereby responding to situations that cannot be defended against by redundant configurations alone.

[0015] Furthermore, the present invention aims to provide an autonomous driving control device and method that satisfy functional safety requirements while utilizing the judgment capabilities of an inference model by having a deterministically operating safety layer verify and mediate the output of a probabilistic and high-latency language-based inference model.

[0016] Furthermore, the present invention aims to provide an autonomous driving control device and method that responds to the progression of sensor degradation with gradual performance degradation rather than dichotomous cessation by progressively reducing the vehicle's operating design area based on calculated reliability and reflecting the reduced boundary in safety constraints. means of solving the problem

[0017] An autonomous driving control device according to an embodiment of the present invention for solving the above problem comprises: a sensor interface unit that acquires sensing data from a plurality of sensors mounted on a vehicle; a reliability determination unit that calculates a performance degradation index for each of the plurality of sensors from the sensing data and determines reliability information for each sensor based on the performance degradation index; a context generation unit that converts the reliability information into context data in a format that can be processed by a language-based inference model; an inference unit that receives the sensing data and the context data and outputs a driving situation analysis result and at least one driving behavior candidate, while adjusting the contribution assigned to information derived from each sensor differently according to the context data; and a safety arbitration unit that receives the reliability information in parallel with the inference unit, varies safety constraints based on the reliability information, compares the driving behavior candidate with the varied safety constraints, and transmits the driving behavior candidate to a vehicle control unit only when it is satisfied, wherein the reliability information determines the contribution of information derived from each sensor in the inference unit and simultaneously determines the strictness of the safety constraints in the safety arbitration unit.

[0018] The above safety arbitration unit can vary the safety constraints in the direction of increasing the minimum safety distance, decreasing the maximum longitudinal acceleration and maximum lateral acceleration, and increasing the lower limit of the expected collision time as the above reliability information decreases.

[0019] The above reliability determination unit can divide the field of view of a single sensor into multiple partial regions and determine reliability information for each partial region, and the above safety arbitration unit can selectively vary the safety constraint conditions only for the orientation corresponding to the partial region having a reliability below a reference value.

[0020] The above reliability determination unit can determine that if the performance degradation indicators of multiple sensors deteriorate together during the same time interval, this constitutes deterioration due to a common cause, and the above safety arbitration unit can vary the safety constraint conditions by a greater range than the variable range resulting from the reliability degradation of individual sensors when deterioration due to a common cause is determined.

[0021] The inference unit operates in a first cycle, and the safety arbitration unit and the vehicle control unit operate in a second cycle shorter than the first cycle, and the safety arbitration unit can continue to vary the safety constraint condition according to the reliability information updated every second cycle even in a section where the output of the inference unit is not updated. Effects of the invention

[0022] According to the present invention, consistency between the two layers can be ensured by a single state indicator simultaneously regulating the content of the judgment and the criteria for the judgment. In any system, if the part that generates the conclusion and the part that reviews the conclusion operate on different premises, a situation may arise where the reviewing side passes the conclusion based on the usual standards without knowing the circumstances, even though the generating side has reached a conclusion based on uncertain information. Since the present invention allows the same value of information reliability to determine the weight of information adoption on the generating side and the strictness of the passing criteria on the reviewing side, if the quality of the input deteriorates, both the formation of the conclusion and its verification shift in a conservative direction.

[0023] Furthermore, according to the present invention, the quality of judgment can be improved by transmitting the state of the observation means itself, along with the observation results, to the judgment subject. Since the validity of the conclusions of any inference system depends on the accuracy of the input information, an inference system that does not know how reliable the input information is lacks the means to distinguish between contaminated information and sound information. By providing both the information itself and the reliability of that information, the present invention enables the inference subject to perform adaptive responses, such as suspending judgment on uncertain information or prioritizing reference to alternative information sources.

[0024] Furthermore, according to the present invention, the resolution of the response can be improved by perceiving the state as a continuous value instead of determining it dichotomously. Since the performance degradation of physical components proceeds continuously by nature, a system that distinguishes it only into two states—normal and failure—cannot take any measures in the extensive intermediate range between them. The present invention captures the degree of deterioration as a continuous quantity and performs a response of intensity proportional to it, thereby securing a safety margin from the early stages of performance degradation.

[0025] Furthermore, according to the present invention, the spatial locality of a defect can be transferred directly to the locality of the corresponding defect. It is rather exceptional for a defect occurring in a single observation means to affect the entire observation range uniformly, and most defects appear limited to a specific direction. The present invention divides the observation range to evaluate each state individually and increases the intensity of the corresponding defect only for the direction in which the state has deteriorated, thereby maintaining the previous performance in the sound direction. Accordingly, it is possible to prevent local defects from expanding into overall performance degradation.

[0026] Furthermore, according to the present invention, it is possible to identify situations that redundant configurations cannot protect against in principle and respond with a corresponding level of intensity. The effectiveness of redundant configurations is based on the assumption that the failure of each component is independent of one another; however, this assumption does not hold when the same external factor acts on all components simultaneously. The present invention determines such situations by examining whether the state changes of multiple observation means are synchronized in time, and in this case, strengthens the judgment criteria with an expanded scope compared to individual degradation, thereby enabling response to risks that cannot be resolved merely by increasing the number of redundancies.

[0027] Furthermore, according to the present invention, the advantages of both can be simultaneously obtained by hierarchically combining a judgment means having excellent performance but difficult to verify with a judgment means having limited performance but easy to verify. In a structure where the former is placed under the supervision of the latter and the latter is granted final decision-making authority, the upper limit of safety of the entire system is guaranteed by the verifiability of the latter, while the lower limit of performance is enhanced by the judgment capability of the former.

[0028] Furthermore, according to the present invention, responsiveness and judgment depth can be simultaneously secured by asynchronously combining components operating on different time scales. Since deep reasoning requires time and immediate response does not allow for time, if both are subject to the same cycle, one side is inevitably sacrificed. In the present invention, because the fast layer independently continues reference adjustments based on state indicators even during periods when the results of the slow layer are not updated, the securing of safety margins is not interrupted even while the updating of judgments is delayed.

[0029] Furthermore, according to the present invention, availability and safety can be maintained simultaneously by responding to a decrease in capability by reducing the scope of activity. If the same tasks as before are assigned to an entity with reduced capability, the risk of failure increases; conversely, if all activities are suspended, utility is lost. Since the present invention narrows the scope and conditions of activity to correspond to the reduced capability, it is possible to continue providing functions within a range consistent with the remaining capability while suppressing risk.

[0030] Furthermore, according to the present invention, cases where the review side rejects the conclusion of the generation side correspond to the point where the generation side's error is manifested; therefore, selecting and accumulating these cases results in data with high efficiency for the improvement of the generation side. Since the present invention preserves rejection cases along with the state indicators and judgment criteria at that point in time, it is possible to identify the conditions for the occurrence of errors and suppress their repetition.

[0031] In addition, according to the present invention, since the time at which a future state is reached can be estimated from the temporal change trend of a state indicator, preemptive measures can be taken before performance reaches an allowable limit, thereby preventing unexpected loss of function. Brief explanation of the drawing

[0032] FIG. 1 is a block diagram showing the relationship in which the overall configuration and reliability information of an autonomous driving control device according to one embodiment of the present invention branches in parallel into an inference unit and a safety arbitration unit. FIG. 2 is a plan view showing the relationship between a plurality of sensors and the angle of view overlap arranged in a vehicle according to one embodiment of the present invention. FIG. 3 is a block diagram showing the detailed configuration of the reliability determination unit and the flow of calculating the performance degradation index according to one embodiment of the present invention. FIG. 4 is a graph showing changes in spatial frequency response and reliability due to performance degradation over time of a camera according to one embodiment of the present invention. FIG. 5 is a conceptual diagram illustrating a reliability down-adjustment process based on the degree of matching of detection results between a first sensor and a second sensor with overlapping viewing angles according to an embodiment of the present invention. FIG. 6 is a conceptual diagram illustrating the process of dividing the field of view of a single sensor into a plurality of partial regions and determining reliability for each partial region according to an embodiment of the present invention. FIG. 7 is a conceptual diagram showing a configuration in which a context generation unit according to an embodiment of the present invention converts reliability information into text tokens in the form of natural language and inputs them into an inference unit. FIG. 8 is a block diagram showing a configuration in which a context generation unit according to another embodiment of the present invention converts reliability information into a condition vector and injects it into an attention operation of an inference unit. FIG. 9 is a diagram showing an attention weight distribution in which the contribution of each sensor-derived information is adjusted differently according to reliability information according to one embodiment of the present invention. FIG. 10 is a conceptual diagram showing the relationship in which one reliability information according to one embodiment of the present invention is commonly used for the contribution adjustment of the inference unit and the variation of safety constraint conditions of the safety arbitration unit. FIG. 11 is a graph showing the characteristics in which the minimum safety distance, maximum longitudinal acceleration, and lower limit of the expected collision time vary according to the reliability value according to one embodiment of the present invention. FIG. 12 is a conceptual diagram illustrating a process in which safety constraints are selectively varied only for a specific orientation based on partial region reliability according to one embodiment of the present invention. FIG. 13 is a flowchart illustrating the process of a safety arbitration unit according to an embodiment of the present invention comparing variable safety constraints with driving behavior candidates to approve or reject them. FIG. 14 is a diagram illustrating the process of determining the variable range of safety constraints according to a comprehensive reliability index that combines the confidence level and reliability information of the inference unit according to one embodiment of the present invention. FIG. 15 is a timing diagram showing the asynchronous operation relationship between the first cycle of the inference unit and the second cycle of the safety arbitration unit and the vehicle control unit according to an embodiment of the present invention, and the variation of safety constraint conditions that continue every second cycle. FIG. 16 is a conceptual diagram illustrating the process of stepwise reduction of an operation design area based on reliability information according to one embodiment of the present invention and the reduction of the boundary being reflected in safety constraints. FIG. 17 is a flowchart illustrating the process of performing a minimum risk operation when a driving behavior candidate according to one embodiment of the present invention moves out of a reduced driving design area. FIG. 18 is a graph showing future reliability prediction and maintenance requirement time point based on the time series change rate of a performance degradation index according to one embodiment of the present invention. FIG. 19 is a diagram showing the change in reliability and the variation in safety constraint conditions expanded compared to individual degradation when common cause degradation occurs in a plurality of sensors according to an embodiment of the present invention. FIG. 20 is a conceptual diagram illustrating the process of determining a priority reference sensor based on the reliability deviation between heterogeneous sensors according to one embodiment of the present invention. FIG. 21 is a block diagram illustrating a process in which the output of an inference unit, the approval status of a safety arbitration unit, and variable safety constraint conditions are stored in a record unit and used for relearning according to an embodiment of the present invention. FIG. 22 is a flowchart showing the overall flow of an autonomous driving control method according to one embodiment of the present invention. Specific details for implementing the invention

[0033] Embodiments of the present invention will be described in detail below with reference to the attached drawings. Since the present invention is susceptible to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0034] Referring to FIG. 1, an autonomous driving control device (1000) according to one embodiment of the present invention includes a sensor interface unit (100), a reliability determination unit (200), a context generation unit (300), an inference unit (400), a safety arbitration unit (500), an operation area management unit (600), and a recording unit (800), and the output of the safety arbitration unit (500) is transmitted to a vehicle control unit (700). It is noteworthy that the reliability information calculated by the reliability determination unit (200) branches into two paths, one of which is input to the inference unit (400) via the context generation unit (300), and the other is directly input to the safety arbitration unit (500). In the former path, the reliability information determines the contribution of information originating from each sensor, and in the latter path, the reliability information determines the strictness of safety constraints.

[0035] The sensor interface unit (100) acquires sensing data from a plurality of sensors mounted on the vehicle (10). The plurality of sensors may include a camera (110), a radar (120), and a lidar (130). The camera (110) may include a front narrow-angle camera (111), a front wide-angle camera (112), a left-side camera (113), a right-side camera (114), and a rear camera (115), but the number and arrangement thereof are not limited thereto. The sensor interface unit (100) assigns a timestamp to the frame received from each sensor and performs time synchronization, and together acquires lower-level diagnostic information, such as the integrity check result of the frame header and the error count of the serial link, and transmits it to the reliability determination unit (200).

[0036] Referring to FIG. 2, each sensor positioned on the vehicle (10) has a unique field of view and partially overlaps with the field of view of adjacent sensors. The field of view (111a) of the front narrow-angle camera (111) is contained within the field of view (112a) of the front wide-angle camera (112), and the field of view (113a) of the left rear camera (113) forms an overlapping area (A1, A2) with the field of view (112a) of the front wide-angle camera (112) and the field of view (115a) of the rear camera (115), respectively. The detection area (120a) of the radar (120) overlaps with the entire field of view of the camera (110). This overlapping relationship serves as the geometric basis for the cross-verification unit (220), described later, to calculate the degree of alignment of detection results between sensors, and at the same time, since it defines which direction each field of view is responsible for relative to the vehicle, it serves as the basis for the variable selective constraint conditions by direction described later.

[0037] Referring to FIG. 3, the reliability determination unit (200) includes an indicator calculation unit (210), a cross-verification unit (220), a reliability calculation unit by region (230), a deterioration prediction unit (240), and a common cause determination unit (250).

[0038] The indicator calculation unit (210) calculates a performance degradation indicator for each sensor from the sensing data. The performance degradation indicator for the camera (110) can be calculated as follows. First, an edge diffusion function is obtained from the luminance profile of the edge region within the captured image, and the spatial frequency response is calculated by differentiating and Fourier transforming it. The ratio of the modulation transfer function value at the reference spatial frequency to the initial registered value is the first indicator. Second, the image is divided into multiple blocks, and the sum of the standard deviation of the luminance and the magnitude of the gradient within each block is calculated to obtain the local contrast and edge gradient energy. Third, the ratio of the occluded region is calculated by detecting the region where the change in pixel value in multiple temporally adjacent frames continues below a threshold. This captures local occlusion caused by water droplets, mud, or frost attached to the lens surface. Fourth, the number of defective pixels and their spatial distribution are calculated from the dark current distribution of the frame acquired in a light-blocking state. Fifth, the camera's rotation angle and translational components are estimated from the vanishing point, lane parallelism, and parallax of stationary structures detected during driving, and the deviation from the initial calibration value is calculated as the amount of variation in the calibration parameters.

[0039] The indicator calculation unit (210) can calculate the noise floor rise amount, angle estimation variance, and ghost target occurrence rate for the radar (120), and the reflection intensity reduction rate, effective point count reduction rate, and point loss at a specific azimuth angle for the lidar (130) as performance degradation indicators. The indicator calculation unit (210) normalizes the calculated multiple performance degradation indicators and then converts them into a reliability value of 0 or more and 1 or less by weighting and summing them. The weights can be set differently depending on the characteristics of the perception function handled by the corresponding sensor. For example, for the front narrow-angle camera (111) responsible for long-distance object recognition, the weight of the indicator regarding spatial frequency response can be set high, and for the surround-view camera responsible for short-distance obstacle detection, the weight of the indicator regarding the occlusion area ratio can be set high.

[0040] Referring to FIG. 4, the modulation transfer function value (C1) of the front narrow-angle camera (111) gradually decreases over time as the vehicle operates, and the reliability value (C2) calculated in response also decreases. In the prior art, the binary fault judgment criterion (C3) adopted detects an abnormality only when the signal is interrupted (t3), so no action is taken during the long period of performance degradation from time point (t1) to time point (t3). On the other hand, according to the present embodiment, from time point (t1) when the reliability value (C2) reaches the first threshold (TH1), the fact of degradation is reflected in the context data, and at the same time, the variation of safety constraint conditions begins, and from time point (t2) when it reaches the second threshold (TH2), the reduction of the operation design area described later begins.

[0041] The cross-verification unit (220) calculates the degree of alignment of detection results between sensors whose viewing angles overlap. Referring to FIG. 5, if the radar (120) detects a target (T1) having relative distance and relative speed within the overlapping area of ​​the front wide-angle camera (112) and the radar (120), but the front wide-angle camera (112) fails to detect a corresponding object near the image coordinates where the target is projected, it is determined to be an alignment failure. The cross-verification unit (220) calculates the degree of alignment by accumulating the frequency of such alignment failures over a predetermined time window, and if the degree of alignment is below a reference value, the reliability of the sensor with the inferior performance degradation index calculated by the index calculation unit (210) among the two sensors is lowered. With this configuration, since it is possible to determine which sensor's judgment is incorrect based on an independent basis called the performance degradation index, errors associated with a simple majority rule method can be avoided.

[0042] The reliability calculation unit (230) for each region divides the field of view of a single sensor into multiple partial regions and determines reliability information for each partial region. Referring to FIG. 6, the field of view (112a) of the front wide-angle camera (112) is divided into multiple partial regions (R1 to Rn) in the form of a grid. When foreign matter is attached to a specific point on the lens surface, the ratio of the obscured area of ​​the corresponding partial regions (R7, R8) increases, and when the peripheral resolution of the optical system deteriorates, the spatial frequency response of the partial regions located at the outer edge of the field of view is selectively degraded. The reliability calculation unit (230) calculates individual reliability values ​​for each partial region and generates a reliability map. In one example, the reliability of the partial regions (R7, R8) is calculated as 0.31 and 0.24, respectively, the partial regions in the center of the field of view maintain a reliability of 0.95, and the partial regions at the outer edge of the field of view have a reliability of 0.62.

[0043] The context generation unit (300) converts the reliability information determined by the reliability determination unit (200) into context data in a format that the inference unit (400) can process. The context generation unit (300) may include at least one of a natural language token generation unit (310) and a condition vector generation unit (320).

[0044] Referring to FIG. 7, the natural language token generation unit (310) generates a text token (F1) that describes reliability information in the form of a natural language sentence. For example, in a situation where the reliability of the front narrow-angle camera (111) is reduced and a specific part of the left-side camera (113) is obscured, the natural language token generation unit (310) generates a sentence stating that the long-range resolution of the front narrow-angle camera is reduced compared to the standard, so caution is required for object recognition at a distance of 60 meters or more, and a sentence stating that the bottom area of ​​the left-side camera is partially obscured, so radar detection results should be prioritized for short-range information in that direction. The generated text token (F1) is converted into an embedding through a tokenizer (311), combined with a visual embedding (F2) corresponding to the sensing data, and input to the inference unit (400). This configuration allows the inference unit (400) to interpret reliability information by utilizing linguistic common sense acquired through prior learning, and has the advantage of being applicable to existing visual-language models without separate structural changes.

[0045] Referring to FIG. 8, the condition vector generation unit (320) converts reliability information into a condition vector (G1) corresponding to the latent space of the inference unit (400). The condition vector generation unit (320) may be composed of a multilayer perceptron or a convolutional neural network that receives reliability values ​​per sensor and reliability maps per sub-region as inputs, and the condition vector (G1) which is the output thereof is injected into the attention module (420) of the inference unit (400). As an injection method, a method of adding the condition vector to the key and value of the attention operation, a feature modulation method of applying scale and shift parameters calculated from the condition vector to an intermediate feature map, or a method of using the condition vector as a bias term added to the attention logit may be employed. Since this configuration does not go through natural language tokens, there is no computational burden due to the increase in token length, and the reliability information is directly reflected as a continuous value, which has the advantage of excellent responsiveness to minute degradation changes.

[0046] The inference unit (400) is composed of a language-based inference model including an encoder (410), an attention module (420), and a decoder (430), and receives sensing data and context data as input and outputs a driving situation interpretation result and at least one driving action candidate. Referring to FIG. 9, in the attention weight distribution (H1) when reliability is uniform, generally equal weights are assigned to each sensor-derived token, whereas in the attention weight distribution (H2) when the reliability of the front narrow-angle camera (111) is reduced, the weight assigned to the corresponding sensor-derived token is significantly reduced, and the weight assigned to the radar (120) and front wide-angle camera (112)-derived tokens is relatively increased. That is, the inference unit (400) interprets the situation by adjusting the contribution of information derived from each sensor differently according to the context data. In addition, the inference unit (400) can output the basis for the interpretation along with the driving situation interpretation result in natural language, and the basis includes sensor priority reflecting reliability information.

[0047] The safety arbitration unit (500) includes a constraint storage unit (510), a judgment unit (520), and a constraint variable unit (530). The constraint storage unit (510) stores safety constraints such as a minimum safety distance, maximum longitudinal and lateral acceleration, a lane departure prohibition condition, a lower limit of the expected collision time, and the boundary of the current driving design area. The constraint variable unit (530) directly receives reliability information from the reliability determination unit (200) and varies the values ​​of the safety constraints stored in the constraint storage unit (510) according to the reliability. The judgment unit (520) compares a driving behavior candidate received from the inference unit (400) with the variable safety constraints and transmits the candidate to the vehicle control unit (700) only if all conditions are satisfied.

[0048] Referring to FIG. 10, one reliability information output by the reliability determination unit (200) is divided into two paths at a branching point (N1). The first path passes through the context generation unit (300) to the inference unit (400), and in this path, the reliability information determines the attention weight of each sensor-derived token, i.e., the contribution of the information. The second path passes through the constraint variable unit (530) to the constraint storage unit (510), and in this path, the reliability information determines the strictness of the judgment criteria. Therefore, if the reliability of a specific sensor is degraded, the inference unit (400) generates a driving behavior candidate by adopting less information from that sensor, and at the same time, the judgment unit (520) evaluates the candidate with stricter criteria. If the reliability information is provided only to the first path, the candidate generated based on the degraded information may be approved by the same relaxed criteria as usual, so there is a possibility that the effect of the conservative adjustment made by the inference unit (400) will be offset at the safety level. Conversely, if reliability information is provided only to the second path, the inference unit (400) will still adopt the degraded information with a high weight, repeatedly generating inaccurate candidates, and as a result, only the rejection frequency of the judgment unit (520) increases, thereby damaging the continuity of driving. In other words, neither of the two paths alone can achieve the objective, and only when the same reliability information is provided to both can the degradation of perception quality be consistently reflected in the content of the judgment and the criteria for the judgment.

[0049] Referring to FIG. 11, the variable characteristics of the safety constraint by the variable constraint unit (530) are illustrated. As the reliability value decreases from 1.0, the minimum safety distance (J1) increases monotonically, the maximum longitudinal acceleration (J2) decreases monotonically, and the lower limit of the expected collision time (J3) increases monotonically. In one embodiment, if the minimum safety distance corresponding to reliability 1.0 is 100 percent of the reference value, it can be set to 130 percent at reliability 0.6 and 180 percent at reliability 0.3. The variable characteristics can be defined in the form of a linear function, an interval linear function, or a sigmoid function, and hysteresis characteristics can be applied near the threshold to prevent chatter.

[0050] Referring to FIG. 12, the variable constraint unit (530) selectively varies safety constraints by orientation based on reliability information for each partial area. If the lower partial area of ​​the left rear camera (113) is obscured by foreign matter and the reliability of that partial area is reduced to below a reference value, the variable constraint unit (530) increases the minimum safety distance only for the left orientation (M1) of the vehicle (10) and expands the clearance required for changing lanes in the left direction. Conversely, the previous safety constraints are maintained for the front orientation (M2), right orientation (M3), and rear orientation (M4). With this configuration, local deterioration occurring in one partial area does not spread to the overall performance degradation of the vehicle, so the previous driving performance and traffic flow suitability are preserved in sound orientations.

[0051] Referring to FIG. 13, the decision unit (520) receives a driving action candidate from the inference unit (400), queries the variable safety constraint from the constraint storage unit (510), and compares the two. If all conditions are satisfied, the candidate is transmitted to the vehicle control unit (700); if any one condition is violated, the candidate is rejected, and the same decision is repeated for the next-ranked candidate. If all candidates are rejected, the decision unit (520) instructs the vehicle control unit (700) to perform a minimum risk operation. The decision unit (520) and the constraint variable unit (530) are implemented with a deterministic algorithm so that their operation is fully verifiable, and accordingly, the safety integrity of the entire system is ensured despite the existence of the inference unit (400) which has a probabilistic output.

[0052] Referring to FIG. 14, the inference unit (400) outputs a confidence level for each driving behavior candidate, and the constraint variable unit (530) calculates a comprehensive confidence index by combining the confidence level and the confidence information. The comprehensive confidence index can be defined as the weighted product of the confidence level and the confidence level, or the minimum value of both. In the region where the comprehensive confidence index is high, the variable range of the safety constraint is small, and as the comprehensive confidence index decreases, the variable range expands, and if it is below a threshold value, the candidate is immediately rejected. According to this configuration, judgments that the inference model is not confident in and situations where the input information itself is inadequate are reflected independently, so the possibility of a misjudgment passing is reduced compared to a configuration that monitors only one factor.

[0053] Referring to FIG. 15, the inference unit (400) operates in a first cycle (P1), and the safety arbitration unit (500) and the vehicle control unit (700) operate in a second cycle (P2) shorter than the first cycle. In one embodiment, the first cycle may be 500 milliseconds to 1 second, and the second cycle may be 10 milliseconds to 20 milliseconds. In the interval where the output of the inference unit (400) is not updated, the judgment unit (520) maintains the driving behavior candidate approved immediately prior, or if a predetermined time has elapsed since the time of the immediate approval, maintains a predefined basic behavior such as lane keeping and constant speed driving. At this time, the constraint variable unit (530) continues to vary the safety constraints according to the reliability information updated every second cycle, regardless of whether the inference unit (400) is updated. Therefore, in a situation where reliability decreases in a short period of time, such as the onset of sudden rainfall, the safety margin is immediately expanded without waiting for the next output of the inference unit (400). In addition, the layer operating in the second cycle performs safety threshold functions, such as emergency braking and lane departure prevention, independently of the output of the inference unit (400).

[0054] Referring to FIG. 16, the driving area management unit (600) gradually reduces the driving design area of ​​the vehicle based on reliability information. The initial driving design area (K1) is represented as an area defined by each axis of maximum speed, permissible weather conditions, permissible road type, and whether lane change is permitted. When reliability reaches a second threshold, the maximum permissible speed is lowered as a first reduction step and the driving design area (K2) is reduced, and when reliability further deteriorates, the lane change function is restricted and autonomous driving under rain conditions is suspended as a second reduction step and the driving design area (K3) is reduced. The boundary of the reduced driving design area is updated and reflected in the constraint storage unit (510) and constitutes part of the judgment criteria of the judgment unit (520).

[0055] Referring to FIG. 17, the decision unit (520) rejects a candidate driving action if the candidate moves out of the reduced driving design area and instructs the vehicle control unit (700) to perform a minimum risk operation. The minimum risk operation may include stopping in the current lane, moving to the shoulder and stopping, or requesting the transfer of control to the driver, and is selected based on the level of remaining reliability and surrounding traffic conditions.

[0056] The deterioration prediction unit (240) predicts reliability at a future point in time based on the time series change rate of the performance deterioration index. Referring to FIG. 18, the deterioration prediction unit (240) estimates a future trajectory (D2) by applying regression analysis or a state-space model to a time series (D1) of reliability values ​​accumulated over a predetermined past period, and calculates the point in time (t4) when the estimated trajectory reaches a maintenance threshold (TH3). The calculated point in time is output to the vehicle's user interface or a control server outside the vehicle as maintenance requirement information. This configuration enables operators operating multiple vehicles to plan the timing of parts replacement in advance, and is of great practical utility, especially considering that the rate of deterioration progresses faster for vehicles with longer operating hours during the day.

[0057] The common cause determination unit (250) determines that if the performance degradation indicators of multiple sensors deteriorate together in the same time interval, it is a common cause degradation. Referring to FIG. 19, external factors such as heavy rain, dense fog, sudden changes in illumination at tunnel exits, and low-angle backlight act on multiple sensors simultaneously regardless of the individual characteristics of each sensor. In this case, a pattern is observed in which the reliability values ​​(E1, E2, E3) of each sensor decline in time in synchronization. The common cause determination unit (250) calculates the correlation coefficient between the reliability time series of multiple sensors, and determines that a common cause degradation has occurred if the correlation coefficient exceeds a reference value and the magnitude of the decline in each reliability simultaneously exceeds a threshold value. Furthermore, the common cause determination unit (250) can identify the estimated cause by referring to the output and time information of the rainfall sensor, the illumination sensor, and the ambient temperature sensor.

[0058] When a common cause degradation is determined, the variable constraint section (530) varies the safety constraint by a larger range than the variable range resulting from the decrease in reliability of individual sensors. As illustrated in FIG. 19, the variable curve (V2) for common cause degradation provides a greater safety margin for the same reliability value compared to the variable curve (V1) for individual sensor degradation. The basis for this difference is as follows. A design that duplicates sensors is established on the premise that failures of each sensor occur independently of one another. Since the degradation of individual sensors satisfies this premise, compensation by the remaining sound sensors can be expected. However, common cause degradation undermines that very premise, so compensation cannot be expected, and therefore, even with the same reliability value, the actual risk is greater. This embodiment reflects this qualitative difference in risk as a difference in the variable range.

[0059] Referring to FIG. 20, the context generation unit (300) includes information in the context data that instructs the sensor with relatively higher reliability to be referenced first when the reliability deviation between heterogeneous sensors exceeds a reference value. For example, in a nighttime low-light environment, if the reliability of the camera (110) drops to 0.32 and the reliability of the radar (120) is maintained at 0.91, information is generated that instructs the radar detection result to be prioritized regarding the existence and distance of an object, but the camera information to be referenced regarding the determination of the type of object. This reflects the difference in the physical measurement principles of each sensor and enables the transmission of conditional priority, which cannot be expressed by a single scalar weight, to the inference unit (400).

[0060] Referring to FIG. 21, the record unit (800) stores the output of the inference unit (400), whether the safety arbitration unit (500) approves it, the basis for the judgment, and the value of the variable safety constraint applied at the time of judgment, along with reliability information. In particular, since cases where the judgment unit (520) rejects a driving behavior candidate correspond to situations where the inference unit (400) has made a misjudgment, the combination of the sensing data, context data, variable constraint, and reason for rejection at that time becomes data of high value for the retraining of the inference unit (400). The reason for preserving the value of the variable constraint together is that even for the same driving behavior candidate, approval depends on the strictness of the criteria applied at the time, so if this is not recorded, the reason for rejection cannot be reproduced subsequently. The data stored in the record unit (800) is transmitted to a learning server outside the vehicle and used as retraining data for the inference unit (400), and as the updated model parameters are distributed to the vehicle, the frequency of occurrence of rejection cases gradually decreases.

[0061] Referring to FIG. 22, an autonomous driving control method according to an embodiment of the present invention comprises the steps of: acquiring sensing data from a plurality of sensors (S100); calculating a performance degradation index from the sensing data and determining reliability information for each sensor based thereon (S200); converting the reliability information into context data (S310); inputting the sensing data and context data into a language-based inference model to obtain a driving situation analysis result and a driving behavior candidate with adjusted contribution (S320); performing in parallel with steps (S310) and (S320) a step of varying safety constraints based on the reliability information (S410); reducing the driving design area based on the reliability information and reflecting the reduced boundary in the safety constraints (S420); determining a driving behavior candidate by comparing it with the varied safety constraints (S500); controlling the vehicle according to the candidate if the determination result is satisfied (S600); and performing a minimum risk operation if the determination result is not satisfied (S700). Here, the reliability information determined in step (S200) is provided in common to steps (S310) and (S410).

[0062] The autonomous driving control method according to the embodiment of the present invention described above may be written as a program executable on a computer and stored on a computer-readable recording medium. The recording medium includes storage media such as magnetic storage media, optical reading media, and carrier waves.

[0063] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and it will be obvious to those skilled in the art that various substitutions, modifications, and changes are possible within the scope of the technical concept of the present invention. Explanation of the symbols

[0064] 10: Vehicle 100: Sensor interface section 110: Camera 111: Front narrow-angle camera 112: Front wide-angle camera 113: Left side camera 114: Right side camera 115: Rear camera 120: Radar 130: Lidar 200: Reliability Determinant 210: Indicator Calculation Unit 220: Cross-validation Unit 230: Domain-specific Reliability Calculation Unit 240: Deterioration Prediction Unit 250: Common Cause Determination Unit 300: Context creation part 310: Natural Language Token Generator 311: Tokenizer 320: Condition Vector Generator 400: Inference Unit 410: Encoder 420: Attention Module 430: Decoder 500: Safety Arbitration Department 510: Constraint storage unit 520: Decision unit 530: Constraint variation unit 600: Operation Area Management Unit 700: Vehicle Control Unit 800: Recorder 1000: Autonomous driving control unit

Claims

Claim 1 An autonomous driving control device comprising: a sensor interface unit for acquiring sensing data from a plurality of sensors mounted on a vehicle; a reliability determination unit for calculating a performance degradation index for each of the plurality of sensors from the sensing data and determining reliability information for each sensor based on the performance degradation index; a context generation unit for converting the reliability information into context data in a format that can be processed by a language-based inference model; an inference unit for receiving the sensing data and the context data and outputting a driving situation analysis result and at least one driving behavior candidate, and for adjusting the contribution assigned to information derived from each sensor differently according to the context data; and a safety arbitration unit for receiving the reliability information in parallel with the inference unit, varying safety constraints based on the reliability information, comparing the driving behavior candidate with the varied safety constraints, and transmitting the driving behavior candidate to a vehicle control unit only when the varied safety constraints are satisfied, wherein the reliability information determines the contribution of information derived from each sensor in the inference unit, and simultaneously determines the strictness of the safety constraints in the safety arbitration unit. Claim 2 An autonomous driving control device according to claim 1, wherein the safety arbitration unit varies the safety constraints in a direction that increases the minimum safety distance, lowers the maximum longitudinal acceleration and maximum lateral acceleration, and increases the lower limit of the collision prediction time as the reliability information decreases. Claim 3 An autonomous driving control device according to claim 1, wherein the performance degradation index is calculated based on at least one of the spatial frequency response of an image, local contrast, edge gradient energy, occlusion area ratio, number of defective pixels, and amount of variation in calibration parameters. Claim 4 An autonomous driving control device according to claim 1, wherein the reliability determining unit calculates the degree of alignment between the detection results of a first sensor and a second sensor whose viewing angles overlap among the plurality of sensors, and if the degree of alignment is less than a reference value, lowers the reliability of the sensor with an inferior performance degradation index among the first sensor and the second sensor. Claim 5 An autonomous driving control device according to claim 1, wherein the reliability determination unit divides the field of view of a single sensor into a plurality of partial regions and determines reliability information for each partial region, and the inference unit adjusts the contribution of information acquired in each partial region differently according to the reliability information for each partial region. Claim 6 An autonomous driving control device according to claim 5, wherein the safety arbitration unit selectively varies the safety constraints only for the orientation corresponding to the partial area less than the reference value among the partial area reliability information, and maintains the safety constraints for the remaining orientations. Claim 7 An autonomous driving control device according to claim 1, wherein the context generation unit generates a text token describing the reliability information in the form of a natural language sentence, and combines the text token with an embedding corresponding to the sensing data and inputs it to the inference unit. Claim 8 An autonomous driving control device according to claim 1, wherein the context generation unit converts the reliability information into a condition vector corresponding to the latent space of the inference unit and injects the condition vector into the attention operation of the inference unit. Claim 9 An autonomous driving control device according to claim 1, wherein the inference unit outputs the basis for the analysis in natural language along with the result of the analysis of the driving situation, and the basis includes a sensor priority reflecting the reliability information. Claim 10 An autonomous driving control device according to claim 1, wherein the inference unit outputs a confidence level for the driving behavior candidate, the safety arbitration unit calculates a comprehensive confidence index by combining the confidence level and the reliability information, and determines the variable range of the safety constraint condition according to the comprehensive confidence index. Claim 11 An autonomous driving control device according to claim 1, further comprising an operating area management unit that reduces the operating design area of ​​a vehicle based on the reliability information, wherein the operating area management unit performs at least one of lowering the maximum allowable speed, restricting the lane change function, and suspending autonomous driving under specific weather conditions, and reflecting the boundary of the reduced operating design area in the safety constraints. Claim 12 An autonomous driving control device according to claim 11, wherein the safety arbitration unit rejects the candidate when the candidate driving behavior deviates from the reduced driving design area and instructs the vehicle control unit to perform a minimum risk operation. Claim 13 An autonomous driving control device according to claim 1, wherein the reliability determining unit predicts reliability at a future point in time based on the time series change rate of the performance degradation index and outputs the point in time when the predicted reliability reaches a reference value as maintenance requirement information. Claim 14 An autonomous driving control device according to claim 1, wherein the inference unit operates in a first cycle, the safety arbitration unit and the vehicle control unit operate in a second cycle shorter than the first cycle, and the safety arbitration unit maintains a previously approved driving behavior candidate or a predefined basic behavior in a section where the output of the inference unit is not updated, while continuing to vary the safety constraint condition according to the reliability information updated every second cycle. Claim 15 An autonomous driving control device according to claim 1, wherein the plurality of sensors includes heterogeneous sensors, and the context generation unit includes information in the context data instructing to prioritize referencing a sensor with relatively higher reliability when the reliability deviation between heterogeneous sensors exceeds a reference value. Claim 16 An autonomous driving control device according to claim 1, wherein the reliability determination unit determines that if the performance degradation indicators of the plurality of sensors deteriorate together in the same time interval, this is a common cause degradation, the context generation unit includes information indicating the occurrence and estimated cause of the common cause degradation in the context data, and the safety arbitration unit varies the safety constraint condition by a greater range than the variable range due to the reliability degradation of the individual sensors when the common cause degradation is determined. Claim 17 An autonomous driving control device according to claim 1, further comprising a record unit that stores the output of the inference unit, the approval status of the safety arbitration unit, and the variable safety constraints together with the reliability information, wherein among the data stored in the record unit, data regarding cases in which the safety arbitration unit rejected the driving behavior candidate is used as retraining data for the inference unit. Claim 18 A method for controlling autonomous driving, comprising: acquiring sensing data from a plurality of sensors mounted on a vehicle; calculating a performance degradation index for each of the plurality of sensors from the sensing data and determining reliability information for each sensor based on the performance degradation index; converting the reliability information into context data in a format that can be processed by a language-based inference model; inputting the sensing data and the context data into the language-based inference model to obtain a driving situation analysis result and at least one driving behavior candidate in a state where the contribution of information derived from each sensor is adjusted according to the context data; varying safety constraints based on the reliability information; and comparing the driving behavior candidate with the varied safety constraints and controlling the vehicle according to the driving behavior candidate only when the varied safety constraints are satisfied, wherein the reliability information is commonly used for adjusting the contribution and varying the safety constraints. Claim 19 In claim 18, the step of determining reliability information includes dividing the field of view of a single sensor into multiple partial regions and determining reliability information for each partial region, and the step of varying safety constraints is selectively performed only for the orientation corresponding to the partial region having reliability less than a reference value, in an autonomous driving control method. Claim 20 An autonomous driving control method according to claim 18, further comprising the step of reducing the driving design area of ​​a vehicle based on the above reliability information, and performing a minimum risk operation when the driving behavior candidate deviates from the reduced driving design area. Claim 21 A computer program stored on a computer-readable recording medium to execute the method of any one of paragraphs 18 through 20.

Images