Artificial intelligence device, method of operation of artificial intelligence device, and computer-readable recording medium storing a program for performing the method of operation of artificial intelligence device

The AI device with an LLM improves AEB control by inferring failure causes and adapting ECU logic, addressing limitations of conventional rule-based systems and enhancing safety in complex driving conditions.

KR103045035B1Active Publication Date: 2026-11-11LG ELECTRONICS INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
KR1020257036252
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2024-06-20
Filing Date
2024-11-04
Publication Date
2026-11-11
Estimated Expiration
2044-11-04

AI Technical Summary

Technical Problem

Conventional rule-based Autonomous Emergency Braking (AEB) control systems struggle with limited response to complex driving situations, inaccurate object recognition in multi-object environments, and sensor limitations, leading to potential safety issues.

Method used

An artificial intelligence device equipped with a Large Language Model (LLM) that infers the cause of AEB control failure and provides adaptive control logic to the Electronic Control Unit (ECU) by learning from driving-related data, updating the LLM through a server, and transmitting control logic to the ECU.

Benefits of technology

Enhances AEB control safety by enabling flexible response to various driving scenarios and real-time analysis of failure causes, preventing collisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025120568531-PCT00001_ABST
    Figure 112025120568531-PCT00001_ABST
Patent Text Reader

Abstract

An artificial intelligence device according to one embodiment of the present disclosure may include: a communication interface for communicating with a server; a memory; and one or more processors that transmit driving-related state data to the server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor, receive a Large Language Model (LLM) learned based on the driving-related state data from the server and store it in the memory, acquire new driving-related state data, infer the cause of failure of the AEB control from the new driving-related state data through the LLM, acquire control logic of an Electronic Control Unit (ECU) corresponding to the inferred cause of failure of the AEB control, and transmit the acquired ECU control logic to the ECU.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present disclosure relates to an artificial intelligence device for enhancing vehicle control performance. Background Technology

[0002] A vehicle is a device that moves the user in the desired direction. A typical example is the automobile. Recent vehicles are equipped with Advanced Driver Assistance Systems (ADAS) to assist the driver.

[0003] ADAS is an active safety system that uses sensors or cameras to detect dangerous situations, warns the driver of the risk of an accident, and helps the driver make judgments and take action.

[0004] Among ADAS features, Autonomous Emergency Braking (AEB) control is a function that automatically applies the brakes to prevent vehicle accidents.

[0005] Conventional roll-based AEB control operates according to set rules and conditions, and these rules evaluate and determine collision risk based on data collected from the vehicle's sensors.

[0006] However, conventional rule-based AEB control relies on predefined rules, so its response to complex or unexpected driving situations is limited.

[0007] In addition, conventional rule-based AEB control has the problem of being difficult to accurately recognize and evaluate all objects in complex environments where multiple objects exist simultaneously.

[0008] In addition, conventional rule-based AEB control may have limited performance depending on the accuracy and range of the sensor. The problem to be solved

[0009] The purpose of the present disclosure may be to improve the safety factor of AEB control through an LLM embedded in an edge device.

[0010] The purpose of the present disclosure may be to relearn or update the LLM so as to respond to various driving situations.

[0011] The purpose of the present disclosure may be to infer the cause of failure of AEB control through LLM and to efficiently control the ECU according to the cause of failure. means of solving the problem

[0012] An artificial intelligence device according to one embodiment of the present disclosure may include: a communication interface for communicating with a server; a memory; and one or more processors that transmit driving-related state data to the server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor, receive a Large Language Model (LLM) learned based on the driving-related state data from the server and store it in the memory, acquire new driving-related state data, infer the cause of failure of the AEB control from the new driving-related state data through the LLM, acquire control logic of an Electronic Control Unit (ECU) corresponding to the inferred cause of failure of the AEB control, and transmit the acquired ECU control logic to the ECU.

[0013] A computer-readable recording medium storing a program for performing an operation method of an artificial intelligence device according to one embodiment of the present disclosure, wherein the operation method may include: a step of transmitting driving-related state data to a server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor; a step of receiving and storing a Large Language Model (LLM) learned based on the driving-related state data from the server; a step of acquiring new driving-related state data; a step of inferring the cause of failure of the AEB control from the new driving-related state data through the LLM; a step of acquiring control logic of an Electronic Control Unit (ECU) corresponding to the inferred cause of failure of the AEB control; and a step of transmitting the acquired ECU control logic to the ECU.

[0014] A method of operation of an artificial intelligence device according to an embodiment of the present disclosure may include: transmitting driving-related status data to a server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor; receiving and storing a Large Language Model (LLM) learned based on the driving-related status data from the server; acquiring new driving-related status data; inferring the cause of failure of the AEB control from the new driving-related status data through the LLM; acquiring control logic of an Electronic Control Unit (ECU) corresponding to the inferred cause of failure of the AEB control; and transmitting the acquired ECU control logic to the ECU. Effects of the invention

[0015] According to an embodiment of the present disclosure, it is possible to efficiently respond to a fail-safe of AEB control.

[0016] According to an embodiment of the present disclosure, various driving situations are analyzed through the LLM, and AEB control can be performed flexibly.

[0017] According to an embodiment of the present disclosure, the cause of failure of AEB control is analyzed in real time, and the Electronic Control Unit (ECU) is controlled in response to the cause of failure, thereby preventing the possibility of a vehicle collision. Brief explanation of the drawing

[0018] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle. Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles. FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle. FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle. Figure 4 is an example of an internal block diagram of the vehicle of Figure 1. FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure. FIG. 6 is a drawing for explaining the operation of a vehicle system according to an embodiment of the present disclosure. FIG. 7 is a sequence diagram for explaining an LLM-based fail-safe response method of a system according to one embodiment of the present disclosure. FIG. 8 is a diagram illustrating the learning process of an LLM according to one embodiment of the present disclosure. FIG. 9 is a diagram illustrating a table matching the cause of failure of AEB control and the ECU control logic according to one embodiment of the present disclosure. FIGS. 10a and FIGS. 10b are drawings illustrating a warning message including the cause of failure of AEB control according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating the frame structure of a Controller Area Network (CAN) message transmitted to a server by an artificial intelligence device according to one embodiment of the present disclosure. FIG. 12 is a diagram illustrating modes related to data processed by an ECU according to one embodiment of the present disclosure. FIG. 13 is a flowchart illustrating a method of operation of an artificial intelligence device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0019] The present disclosure will be described in more detail below with reference to the drawings.

[0020] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0021] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0022] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0023] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0024] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0025] In FIG. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). Other displays such as a HUD (Head Up Display) are also possible.

[0026] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named the Center Information Display.

[0027] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0028] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0029] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0030] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0031] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the vehicle communication gateway (GWDa).

[0032] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0033] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0034] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0035] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0036] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0037] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0038] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0039] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0040] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0041] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0042] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0043] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0044] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0045] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0046] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0047] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0048] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0049] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0050] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0051] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may be equipped with a lamp drive unit (751), a steering drive unit (752), a brake drive unit (753), a power source drive unit (754), a suspension drive unit (756), an air conditioning drive unit (757), a window drive unit (758), a seat drive unit (761), and a signal processing device (170).

[0052] Meanwhile, the vehicle (200) may further be equipped with an ECU (770), a plurality of sensor devices (SN), and a plurality of communication modules (EMa~EMd).

[0053] Meanwhile, the vehicle (200) according to the embodiment of the present disclosure may further be equipped with a vehicle display device (100).

[0054] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0055] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0056] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0057] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0058] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), a lidar (196), a radar (197), or a position sensor (198).

[0059] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0060] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0061] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0062] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0063] The communication device (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (800) or a server (900). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0064] Meanwhile, the communication device (120) can exchange data wirelessly with an adjacent vehicle.

[0065] For example, the communication device (120) can exchange vehicle messages wirelessly with an adjacent vehicle through V2X (Vehicle-to-everything) communication.

[0066] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0067] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0068] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0069] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0070] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0071] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and vehicle surrounding obstacle distance information from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0072] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0073] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0074] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0075] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0076] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100) or vehicle (200).

[0077] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0078] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0079] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0080] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0081] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0082] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0083] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0084] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0085] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508).

[0086] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0087] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0088] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0089] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0090] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0091] Referring to FIG. 5, the server (900) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the server (900) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network.

[0092] The server (900) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0093] The server (900) may include a communication interface (510), memory (530), a learning processor (540), and a processor (560).

[0094] The communication interface (510) can transmit and receive data with the vehicle (200) or an external device.

[0095] The memory (530) may include a model storage unit (931). The model storage unit (531) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (540).

[0096] The learning processor (540) can train the artificial neural network (531a) using training data. The training model may be used while mounted on the server (900) of the artificial neural network, or it may be used while mounted on an external device such as a vehicle (200).

[0097] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (530).

[0098] The processor (560) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0099] FIG. 6 is a drawing for explaining the operation of a system according to an embodiment of the present disclosure.

[0100] Referring to FIG. 6, the system (60) may include a server (900), an artificial intelligence device (600), and an ECU (770).

[0101] The server (900) may include a preprocessing unit (910), a learning unit (920), a model unit (930), and a model distribution unit (940).

[0102] The preprocessing unit (910) can preprocess driving-related status data received from the artificial intelligence device (600).

[0103] The learning unit (920) can learn a Large Language Model (LM) using a training data set that includes preprocessed driving-related state data and labeled data labeled on the driving-related state data. The LLM may be a model that infers one or more causes of failure of AEB control or AEB control logic from the driving-related state data.

[0104] As another example, the LLM may be a model that outputs the braking timing of the brake, the braking force of the brake, the state of the engine intake valve, and a warning message from the sensing data collected from the artificial intelligence device (600).

[0105] The model unit (930) can store the LLM after training or updating is complete. The model unit (930) can be included in the model storage unit (531) of FIG. 5.

[0106] The model distribution unit (940) can transmit multiple learning embedding vectors that match the failure causes of the LLM and multiple AEB controls, which have been learned or updated, to an edge device. The edge device may be an artificial intelligence device (600). The model distribution unit (940) may be included in the communication interface (510) of FIG. 5.

[0107] The preprocessing unit (910) and the learning unit (920) may be included in the learning processor (540) or processor (560) of FIG. 5.

[0108] The artificial intelligence device (600) may include a data expansion unit (610) and a data inference unit (620).

[0109] The data expansion unit (610) may include an LLM (Large Language Model) agent (611), a vector conversion unit (612), and a vector storage unit (613).

[0110] The LLM (Large Language Model) agent (611) can determine the suitability of the data inference unit (620). The data inference unit (620) may be referred to as LLM.

[0111] The LLM agent (611) can generate ECU control logic corresponding to the cause of AEB failure. The LLM agent (611) can generate a control command including the ECU control logic and transmit the generated control command to the ECU (770).

[0112] The vector conversion unit (612) can convert driving-related state data into an embedding vector and can place the converted embedding vector in an embedding space.

[0113] The vector storage unit (613) may store multiple learning embedding vectors corresponding to multiple causes of failure of multiple AEB control.

[0114] The data inference unit (620) may be an LLM that infers the cause of failure of AEB control from driving-related status data.

[0115] The data inference unit (620) can compare the embedding vector converted by the vector conversion unit (612) with a plurality of learning embedding vectors stored in the vector storage unit (613). As a result of the comparison, the data inference unit (620) can obtain the learning embedding vector that is most similar to the converted embedding vector among the plurality of learning embedding vectors. The data inference unit (620) can output the cause of failure of the AEB control matched to the obtained learning embedding vector.

[0116] The LLM agent (611) can obtain ECU control logic corresponding to the cause of failure of AEB control inferred by the data inference unit (620), and can transmit a control command corresponding to the obtained ECU control logic to the ECU (770). The LLM agent (611) can transmit the control command to the ECU (770) via Ethernet communication.

[0117] The artificial intelligence device (600) may be the signal processing device (170) of FIG. 1. In this case, the data expansion unit (610) and the data inference unit (620) may be included in the signal processing device (170). In particular, the data expansion unit (610) and the data inference unit (620) may be included in the processor (175) of the signal processing device (170). The artificial intelligence device (600) may further include a communication device (120) and a plurality of communication modules (EMa~EMd).

[0118] The artificial intelligence device (600) may be the vehicle display device (100) of FIG. 1.

[0119] The artificial intelligence device (600) may be the vehicle (200) of Fig. 1.

[0120] The artificial intelligence device (600) may be a cockpit domain controller (CDC). The CDC may be an electronic control unit for vehicles that manages various electronic systems within the vehicle. The CDC may manage an instrument panel, infotainment system, driver assistance system, etc. The CDC may be included in the vehicle (200) or may be included in the signal processing unit (170a).

[0121] FIG. 7 is a sequence diagram for explaining an LLM-based fail-safe response method of a system according to one embodiment of the present disclosure.

[0122] In the following, the artificial intelligence device (600) may include all components of the vehicle (200). In particular, the artificial intelligence device (600) may include the memory (140, 952), communication device (120), and processor (175) of the vehicle (200). The communication device (120) may be referred to as a communication interface (120). The processor (175) may be provided in at least one or more units.

[0123] The processor (175) of the artificial intelligence device (600) can obtain a safety factor according to the execution of Autonomous Emergency Braking (AEB) control (S701).

[0124] The safety factor can be the rate at which the AEB system successfully prevents a collision.

[0125] In one embodiment, the processor (175) can obtain a safety rate based on the execution of rule-based AEB control. The processor (175) can obtain a safety rate based on the success rate of AEB control performed according to a predefined rule or condition stored in memory (140).

[0126] In another embodiment, the processor (175) can obtain a safety rate based on the execution of an artificial intelligence model-based AEB control. The processor (175) can obtain the success rate of AEB control based on the AEB control logic output by the artificial intelligence model stored in memory (140) as the safety rate.

[0127] The processor (175) of the artificial intelligence device (600) can determine whether the obtained safety rate is less than the reference safety rate (S703).

[0128] In one embodiment, the standard safety factor may be 95%, but this is merely an example.

[0129] The processor (175) of the artificial intelligence device (600) can collect driving-related status data based on whether the safety factor is less than the reference safety factor (S705).

[0130] In one embodiment, driving-related status data may include at least one of ECU information, vehicle (200) aging information, or surrounding information.

[0131] ECU information may include at least one of the sensing information of the camera (195) of the vehicle (200), the sensing information of the lidar (196), the sensing information of the radar (197), the control information of the steering wheel (150), the control information of the accelerator, the control information of the brake pad, or the control information of the gear. ECU information may further include at least one of the failure status of vehicle parts, the condition of the engine, the condition of the coolant, the RPM, and the condition of the battery.

[0132] The control information of the steering wheel (150) may include at least one of the steering angle or steering torque.

[0133] The control information of the brake pad may include at least one of the time the brake pad is pressed or the pressure (pressure) applied to the brake pad.

[0134] The control information of the accelerator may include at least one of the engine rotation speed, the speed of the vehicle (200), the engine ignition time, or the engine fuel filler opening time.

[0135] The control information of the gear may include information regarding the shifting of the gear.

[0136] The aging information of the vehicle (200) may include the degree of aging of at least one of the engine, brake pads, accelerator, gear, or steering wheel (150).

[0137] Surrounding information may include at least one of weather information, temperature, humidity, or precipitation.

[0138] The processor (175) of the artificial intelligence device (600) can transmit driving-related status data collected through the communication device (120) to the server (900) (S707).

[0139] The processor (175) can transmit driving-related status data to the server (900) via CAN communication. The server (900) can collect driving-related status data from the artificial intelligence device (600) and other vehicles.

[0140] The processor (560) of the server (900) can learn LLM based on driving-related state data received from the artificial intelligence device (600) (S709).

[0141] In one embodiment, the LLM may be a model that infers one or more of the causes of failure of AEB control or AEB control logic from driving-related state data.

[0142] The learning processor (540) or processor (560) of the server (900) can learn LLM using driving-related state data. LLM can be learned through supervised learning. The learning processor (540) or processor (560) of the server (900) can also receive driving-related state data through devices other than the artificial intelligence device (600).

[0143] FIG. 8 is a diagram illustrating the learning process of an LLM according to one embodiment of the present disclosure.

[0144] Referring to FIG. 8, the training data set used for supervised learning of the LLM may include training driving-related state data and labeling data indicating the cause of failure of the AEB control corresponding to the driving-related state data. The training driving-related state data may be data received from an artificial intelligence device (600) and other vehicles.

[0145] The learning processor (540) or processor (560) of the server (900) can preprocess driving-related state data and convert the preprocessed information into an embedding vector.

[0146] The learning processor (540) or processor (560) of the server (900) can train an LLM model by assigning labeling data indicating the cause of AEB failure to the converted embedding vector.

[0147] The labeled cause of AEB failure may be any one of the following: failure to recognize a forward obstacle, calculation error of Time to Collision (TTC) due to weather or road conditions, sensor error, or brake aging.

[0148] The cause of failure of the labeled AEB may be either a TTC overestimation error or a TTC underestimation error.

[0149] The learning processor (540) or processor (560) of the server (900) can train the LLM so that the loss function representing the difference between the inference result inferred from the training driving-related state data and the correct cause of failure of the AEB control is minimized.

[0150] The learning processor (540) or processor (560) of the server (900) can convert training driving-related state data into an embedding vector and can place the converted embedding vector in an embedding space by matching it with the cause of failure of the AEB control.

[0151] The learning processor (540) or processor (560) of the server (900) can update the existing LLM using driving-related state data received from the artificial intelligence device (600). The learned LLM may refer to the LLM that has been updated by the new data.

[0152] Again, Figure 7 is explained.

[0153] The server (900) can transmit the learned LLM to the artificial intelligence device (600) through the communication interface (510) (S711).

[0154] The server (900) can transmit parameters of the learned LLM, weights, code or libraries for executing the LLM, etc., to the artificial intelligence device (600) through the communication interface (510). Transmitting the LLM may mean transmitting parameters of the LLM, weights, code or libraries for executing the LLM, etc.

[0155] The server (900) can additionally transmit learning embedding vectors matched to the cause of failure of AEB control placed in the embedding space to the artificial intelligence device (600) through the communication interface (510).

[0156] The processor (175) of the artificial intelligence device (600) can store LLM and learning embedding vectors received from the server (900) in memory (140, 952).

[0157] The processor (175) of the artificial intelligence device (600) can acquire new driving-related state data (S713), and can infer the cause of failure of AEB control from the new driving-related state data acquired through LLM (S715).

[0158] Driving-related status data may include at least one of the ECU information described in step S705, the aging information of the vehicle (200), or surrounding information.

[0159] The processor (175) can obtain the cause of failure of AEB control from new driving-related status data through the LLM stored in memory (140, 952).

[0160] The processor (175) can convert new driving-related data into embedding vectors. The processor (175) can measure the similarity between the converted embedding vectors and the learning embedding vectors stored in memory (140, 952). The processor (175) can obtain the cause of failure of the AEB control that matches the learning embedding vector with the greatest similarity to the converted embedding vector.

[0161] The processor (175) of the artificial intelligence device (600) can obtain ECU control logic corresponding to the inferred cause of failure of AEB control (S717), and can transmit the obtained ECU control logic to the ECU (S719).

[0162] The processor (175) can transmit ECU control logic to the ECU (770) via Ethernet communication.

[0163] One ECU control logic may be matched to the cause of failure of AEB control. The memory (140, 952) may store multiple ECU control logics corresponding to each of the multiple causes of failure.

[0164] The memory (140, 952) may store a table that matches multiple causes of failure and multiple ECU control logics corresponding to each of the multiple causes of failure.

[0165] The processor (175) can extract ECU control logic corresponding to the cause of failure of AEB control inferred from a table stored in memory (140, 952).

[0166] The processor (175) can transmit the extracted ECU control logic to the ECU (770), and the ECU (770) can control the brake drive unit (753) according to the received ECU control logic.

[0167] According to an embodiment of the present disclosure, the cause of failure of AEB control is analyzed in real time, and the ECU is controlled in response to the cause of failure, thereby preventing the possibility of a vehicle collision.

[0168] Meanwhile, even after the embedding of the LLM received from the server (900), there may be cases where the inference of the cause of failure due to AEB control is incorrect. The artificial intelligence device (600) can transmit the correct cause of failure of AEB control and the corresponding driving-related status data to the server (900), and the server (900) can re-learn the LLM using the correct cause of failure of AEB control and the corresponding driving-related status data. The server (900) can transmit the re-learned LLM to the artificial intelligence device (600), and the artificial intelligence device (600) can infer the cause of failure of AEB control using the re-learned LLM.

[0169] FIG. 9 is a diagram illustrating a table matching the cause of failure of AEB control and the ECU control logic according to one embodiment of the present disclosure.

[0170] Referring to FIG. 9, a table (901) including multiple causes of failure and multiple control logics corresponding to each of the multiple causes of failure is shown.

[0171] For example, the first cause of failure may be that the TTC is overcalculated. The first ECU control logic corresponding to the first cause of failure may be to advance the braking timing of the brake and increase the braking force (or braking force value) of the brake to the maximum.

[0172] Table (901) may further include detailed causes of failure of AEB control of the first cause of failure. Detailed causes of failure of AEB control may be sensing errors of the LiDAR (196).

[0173] For example, a second cause of failure may be that the TTC is undercalculated. A second ECU control logic corresponding to the second cause of failure may be to delay the braking timing of the brake and gradually increase the braking force of the brake. Table (901) may further include detailed causes of failure of AEB control for the second cause of failure. A detailed cause of failure of AEB control may be a sensing error of the camera (195).

[0174] Another example of a cause for AEB control failure is a weakening of the connection in the intake valve's Parker connector. If the connection weakens, the interval during which the connector's signal is interrupted may increase over time. The corresponding ECU control logic may be a logic that lowers the RPM relative to the accelerator pedal pressure (a logic that reduces the vehicle's speed).

[0175] As another example, the cause of AEB control failure may be brake malfunction due to the brake pads getting stuck. The corresponding ECU control logic may be a logic that gradually reduces the RPM.

[0176] Meanwhile, the processor (175) of the artificial intelligence device (600) may display the inferred cause of failure of AEB control on at least one of the first to third displays (180a, 180b, 180c) or transmit it to the mobile terminal (800).

[0177] The processor (175) may display a warning message containing the inferred cause of failure of AEB control and a warning phrase on at least one of the first to third displays (180a, 180b, 180c) or transmit it to a mobile terminal (800).

[0178] FIGS. 10a and FIGS. 10b are drawings illustrating a warning message including the cause of failure of AEB control according to an embodiment of the present disclosure.

[0179] Referring to FIG. 10a, the first warning message (1010) may be a message corresponding to the first cause of failure of FIG. 9. The first warning message (1010) may include the first cause of failure and the first ECU control logic corresponding to the first cause of failure.

[0180] Referring to FIG. 10b, the second warning message (1020) may be a message corresponding to the second cause of failure of FIG. 9. The second warning message (1020) may include the second cause of failure and a second ECU control logic corresponding to the second cause of failure.

[0181] The first warning message (1010) or the second warning message (1020) may be displayed on the first to third displays (180a, 180b, 180c) of the vehicle (200). Through this, the user may receive information about the cause of the failure of the AEB control and attempt to take appropriate action.

[0182] FIG. 11 is a diagram illustrating the frame structure of a Controller Area Network (CAN) message transmitted to a server by an artificial intelligence device according to one embodiment of the present disclosure, and FIG. 12 is a diagram illustrating modes related to data processed by an ECU according to one embodiment of the present disclosure.

[0183] The CAN message (1100) of Fig. 11 and the data byte of Fig. 12 may follow the OBD-II (On-Board Diagnostics-II) standard protocol, which is a standard protocol for obtaining information from the ECU (770) of the vehicle (200).

[0184] Referring to FIG. 11, the CAN message (1100) may include a Header 1 field, a Header 2 field, a Header 3 field, Data 1 to Data 7 fields, and a CRC (Cyclic Redundancy Check) field.

[0185] Each of the Header 1 field, Header 2 field, and Header 3 field may include metadata of the CAN message (1100). The metadata may include at least one of the type of the CAN message (1100), the identifier of the artificial intelligence device (600), the identifier of the CAN message (1100), or the status of the CAN message (1100).

[0186] Each of the Data 1 to Data 7 fields may contain data collected from sensors or an ECU (770) equipped in the vehicle (200). The Data 2 field may be a field representing a PID (Parameter ID) and may be a field identifying parameters (e.g., speed, RPM, etc.) so that the user can check the data received from the ECU (770).

[0187] Each of the Data 3 to Data 7 fields may be a field containing ECU information. The ECU information may include at least one of the following: whether a vehicle part is faulty, the condition of the engine, the condition of the coolant, the RPM, and the condition of the battery.

[0188] The CRC field may be a field for verifying the integrity of the data. The server (900) can determine whether the data is corrupted during the transmission of the CAN message (1100) through the CRC field.

[0189] Each mode illustrated in FIG. 12 may represent a function of requesting specific data collected by the ECU (770) when the artificial intelligence device (600) or server (900) communicates with the ECU (770).

[0190] FIG. 13 is a flowchart illustrating a method of operation of an artificial intelligence device according to one embodiment of the present disclosure.

[0191] In the following, the artificial intelligence device (600) may include all components of the vehicle (200). In particular, the artificial intelligence device (600) may include the memory (140, 952), communication device (120), and processor (175) of the vehicle (200).

[0192] The processor (175) may be provided in at least one or more units.

[0193] The detailed explanation below regarding content that overlaps with the steps of Fig. 5 is taken from the explanation of Fig. 5.

[0194] The processor (175) of the artificial intelligence device (600) can obtain a safety factor according to the execution of Autonomous Emergency Braking (AEB) control (S1301).

[0195] The processor (175) of the artificial intelligence device (600) can determine whether the obtained safety rate is less than the reference safety rate (S1303).

[0196] The processor (175) of the artificial intelligence device (600) can collect driving-related status data based on whether the safety factor is less than the reference safety factor (S1305).

[0197] The processor (175) of the artificial intelligence device (600) can learn LLM based on driving-related state data received from the artificial intelligence device (600) (S1307).

[0198] The processor (175) can learn LLM using driving-related state data. LLM can be learned through supervised learning. The processor (175) can receive driving-related state data from a device other than the artificial intelligence device (600).

[0199] The processor (175) can learn the LLM in the manner shown in FIG. 8 and can embed the learned LLM into the artificial intelligence device (600). The learned LLM or the updated LLM can be stored in memory (140, 952).

[0200] The processor (175) of the artificial intelligence device (600) can acquire new driving-related state data (S1309) and can infer the cause of failure of AEB control from the new driving-related state data acquired through LLM (S1311).

[0201] The processor (175) of the artificial intelligence device (600) can obtain ECU control logic corresponding to the inferred cause of failure of AEB control (S1313), and can transmit the obtained ECU control logic to the ECU (S1315).

[0202] In this way, according to the embodiment of FIG. 13, the artificial intelligence device (600) may directly perform the collection of training data sets for learning LLM and the learning of LLM.

[0203] An artificial intelligence device (600) according to an embodiment of the present disclosure may include a communication interface (120) for communicating with a server (900); a memory (140, 952); and one or more processors (175) that transmit driving-related state data to the server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor, receive a Large Language Model (LLM) learned based on the driving-related state data from the server and store it in the memory, obtain new driving-related state data, infer the cause of failure of the AEB control from the new driving-related state data through the LLM, obtain control logic of an Electronic Control Unit (ECU) corresponding to the inferred cause of failure of the AEB control, and transmit the obtained ECU control logic to the ECU.

[0204] If the cause of failure inferred by the LLM is incorrect, the above one or more processors (175) can transmit the cause of failure of the correct AEB control and the corresponding driving-related status data to the server through the communication interface, receive the relearned LLM from the server, and store the relearned LLM in the memory.

[0205] The cause of failure of the above AEB control is an error in calculating the Time to Collision (TTC), and the control logic of the above ECU may be the braking timing and braking force value.

[0206] The artificial intelligence device (600) may further include one or more displays (180a to 180c), and the one or more processors (175) may display a warning message including the cause of failure of the AEB control on the one or more displays.

[0207] The memory (140, 952) can store multiple learning embedding vectors corresponding to each of the multiple causes of failure of AEB control received from the server.

[0208] The above LLM may be a model that converts the new driving-related state data into an embedding vector, obtains the learning embedding vector most similar to the converted embedding vector among the plurality of learning embedding vectors, and infers the cause of failure of AEB control corresponding to the obtained learning embedding vector.

[0209] The memory (140, 952) can store a table including the causes of failure of the plurality of AEB control and the plurality of ECU control logics matched to the causes of failure of the plurality of AEB control.

[0210] One or more processors (175) can obtain the ECU control logic that matches the cause of failure of the inferred AEB control from the table.

[0211] The above driving-related status data includes at least one of ECU information, vehicle aging information, or surrounding information, and the ECU information may include at least one of sensing information, steering wheel control information, accelerator control information, brake pad control information, or gear control information.

[0212] The artificial intelligence device (600) above may be a CDC (Cockpit domain controller).

[0213] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (175).

[0214] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

Claim 1 An artificial intelligence device comprising: a communication interface for communicating with a server; a memory; and one or more processors that, based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor, transmit driving-related status data to the server, receive a Large Language Model (LLM) learned based on the driving-related status data from the server and store it in the memory, acquire new driving-related status data, infer one of a plurality of AEB control failure causes from the new driving-related status data through the LLM, acquire control logic of an Electronic Control Unit (ECU) corresponding to the inferred failure cause, and transmit the acquired ECU control logic to the ECU. Claim 2 An artificial intelligence device according to claim 1, wherein the one or more processors transmit the correct cause of failure of AEB control and corresponding driving-related status data to the server through the communication interface when the cause of failure inferred by the LLM is incorrect, receive the re-learned LLM from the server, and store the re-learned LLM in the memory. Claim 3 In claim 1, the cause of failure of the AEB control is an error in calculating the Time To Collision (TTC), and the control logic of the ECU is an artificial intelligence device that is the braking timing and braking force value of the brake. Claim 4 An artificial intelligence device according to claim 1, further comprising one or more displays, wherein the one or more processors display a warning message including the cause of failure of the AEB control on the one or more displays. Claim 5 In claim 1, the memory is an artificial intelligence device that stores a plurality of learning embedding vectors corresponding to each of the causes of failure of the plurality of AEB control received from the server. Claim 6 In claim 5, the LLM is an artificial intelligence device that is a model for converting the new driving-related state data into an embedding vector, obtaining the learning embedding vector most similar to the converted embedding vector among the plurality of learning embedding vectors, and inferring the cause of failure of AEB control corresponding to the obtained learning embedding vector. Claim 7 In claim 6, the memory is an artificial intelligence device that stores a table including the plurality of causes of failure of AEB control and the plurality of ECU control logics matched to the plurality of causes of failure of AEB control. Claim 8 In claim 7, the artificial intelligence device wherein one or more processors obtain the ECU control logic that matches the cause of failure of the inferred AEB control from the table. Claim 9 An artificial intelligence device according to claim 1, wherein the driving-related state data includes at least one of ECU information, vehicle aging information, or surrounding information, and the ECU information includes at least one of sensing information, steering wheel control information, accelerator control information, brake pad control information, or gear control information. Claim 10 In paragraph 1, the artificial intelligence device is an artificial intelligence device that is a CDC (Cockpit domain controller). Claim 11 A computer-readable recording medium having a program for performing a method of operation of an artificial intelligence device, wherein the method of operation comprises: a step of transmitting driving-related state data to a server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor; a step of receiving and storing a Large Language Model (LLM) learned based on the driving-related state data from the server; a step of acquiring new driving-related state data; a step of inferring one of a plurality of AEB control failure causes from the new driving-related state data through the LLM; a step of acquiring control logic of an Electronic Control Unit (ECU) corresponding to the inferred failure cause; and a step of transmitting the acquired ECU control logic to the ECU. Claim 12 In claim 11, the above-described operation method further comprises the steps of: transmitting to a server the cause of failure of the correct AEB control and corresponding driving-related status data when the cause of failure inferred by the LLM is incorrect; and receiving the re-learned LLM from the server and storing the re-learned LLM. Claim 13 In Clause 11, the cause of failure of the above AEB control is an error in the calculation of the Time To Collision (TTC), and the control logic of the above ECU is a recording medium that is the braking timing and braking force value of the brake. Claim 14 In paragraph 11, the above-described operation method further comprises a recording medium including the step of displaying a warning message containing the cause of failure of the above-described AEB control. Claim 15 In claim 11, the above-described operation method further comprises a recording medium including the step of storing a plurality of learning embedding vectors corresponding to each of the causes of failure of the plurality of AEB control received from the server. Claim 16 In claim 15, the above LLM converts the new driving-related state data into an embedding vector, obtains the learning embedding vector most similar to the converted embedding vector among the plurality of learning embedding vectors, and infers the cause of failure of AEB control corresponding to the obtained learning embedding vector, and is a recording medium that is a model. Claim 17 In claim 16, the above-described operation method further comprises a recording medium including the step of storing a table comprising the plurality of AEB control failure causes and the plurality of ECU control logics matched to the plurality of AEB control failure causes. Claim 18 In claim 17, the step of obtaining the ECU control logic includes the step of obtaining the ECU control logic that matches the cause of failure of the AEB control inferred from the table. Claim 19 In claim 11, the driving-related status data comprises at least one of ECU information, vehicle aging information, or surrounding information, and the ECU information comprises at least one of sensing information, steering wheel control information, accelerator control information, brake pad control information, or gear control information. Claim 20 A method of operation of an artificial intelligence device comprising: a step of transmitting driving-related state data to a server based on the fact that the safety factor resulting from the execution of rule-based Autonomous Emergency Braking (AEB) control is less than a reference safety factor; a step of receiving and storing a Large Language Model (LLM) learned based on the driving-related state data from the server; a step of acquiring new driving-related state data; a step of inferring one of a plurality of AEB control failure causes from the new driving-related state data through the LLM; a step of acquiring control logic of an Electronic Control Unit (ECU) corresponding to the inferred failure cause; and a step of transmitting the acquired ECU control logic to the ECU.

Citation Information

Patent Citations

  • Travel support device

    JP2017117343A

  • Deep Learning-based Vehicle Trajectory Prediction Method and Apparatus using Rasterized Lane Information

    KR102176483B1

  • Comprehensive user control system for vehicle

    US20240132079A1